Understanding what governs community assembly and the maintenance of biodiversity is a central issue in ecology, but has been a continuing debate. A key question is the relative importance of habitat specialization (niche assembly) and dispersal limitation (dispersal assembly). In the middle of the Loess Plateau, northwestern China, we examined how species turnover in Liaodong oak (Quercus wutaishanica) forests differed between observed and randomized assemblies, and how this difference was affected by habitat specialization and dispersal limitation using variation partitioning. Results showed that expected species turnover based on individual randomization was significantly lower than the observed value (P < 0.01). The turnover deviation significantly depended on the environmental and geographical distances (P < 0.05). Environmental and spatial variables significantly explained approximately 40% of the species composition variation at all the three layers (P < 0.05). However, their contributions varied among forest layers; the herb and shrub layers were dominated by environmental factors, whereas the canopy layer was dominated by spatial factors. Our results underscore the importance of synthetic models that integrate effects of both dispersal and niche assembly for understanding the community assembly. However, habitat specialization (niche assembly) may not always be the dominant process in community assembly, even under harsh environments. Community assembly may be in a trait-dependent manner (e.g., forest layers in this study). Thus, taking more species traits into account would strengthen our confidence in the inferred assembly mechanisms.