In natural forests, hundreds of fungal species colonize plant roots. The preference or specificity for partners in these symbiotic relationships is a key to understanding how the community structures of root-associated fungi and their host plants influence each other. In an oak-dominated forest in Japan, we investigated the root-associated fungal community based on a pyrosequencing analysis of the roots of 33 plant species. Of the 387 fungal taxa observed, 153 (39.5%) were identified on at least two plant species. Although many mycorrhizal and root-endophytic fungi are shared between the plant species, the five most common plant species in the community had specificity in their association with fungal taxa. Likewise, fungi displayed remarkable variation in their association specificity for plants even within the same phylogenetic or ecological groups. For example, some fungi in the ectomycorrhizal family Russulaceae were detected almost exclusively on specific oak (Quercus) species, whereas other Russulaceae fungi were found even on “non-ectomycorrhizal” plants (e.g., Lyonia and Ilex). Putatively endophytic ascomycetes in the orders Helotiales and Chaetothyriales also displayed variation in their association specificity and many of them were shared among plant species as major symbionts. These results suggest that the entire structure of belowground plant–fungal associations is described neither by the random sharing of hosts/symbionts nor by complete compartmentalization by mycorrhizal type. Rather, the colonization of multiple types of mycorrhizal fungi on the same plant species and the prevalence of diverse root-endophytic fungi may be important features of belowground linkage between plant and fungal communities.