A rapidly increasing effort to merge functional community ecology and phylogenetic biology has increased our understanding of community assembly. However, studies using both phylogenetic- and trait-based methods have been mainly conducted in old-growth forests, with fewer studies in human-disturbed communities, which play an increasingly important role in providing ecosystem services as primary forests are degraded. We used data from 18 1-ha plots in tropical old-growth forests and secondary forests with different disturbance histories (logging and shifting cultivation) and vegetation types (tropical lowland and montane forests) on Hainan Island, southern China. The distributions of 11 functional traits were compared among these six forest types. We used a null model approach to assess the effects of disturbance regimes on variation in response and effect traits and community phylogenetic structure across different stem sizes (saplings, treelets, and adult trees) and spatial scales (10–50 m). We found significant differences in the distribution of functional traits in highly disturbed lowland sites versus other forest types. Many individuals in highly disturbed lowland sites were deciduous, spiny, with non-fleshy fruits and seeds dispersed passively or by wind, and low SLA. The response traits of coexisting species were clustered in all sites except for highly disturbed lowland sites where evenness was evident. There were different distributions of effect traits for saplings and treelets among different forest types but adult trees showed stronger clustering of trait values with increasing spatial scale among all forest types. Phylogenetic clustering predominated across all size classes and spatial scales in the highly disturbed lowland sites, and evenness in other forest types. High disturbance can lead to abiotic filtering, generating a community dominated by closely related species with disturbance-adapted traits, where biotic interactions play a relatively minor role. In lightly disturbed and old growth forests, multiple processes simultaneously drive the community assembly, but biotic processes dominate at the fine scale.