Estimation of carbon sequestration in the forest sector should take into consideration changes in carbon stock in all carbon pools, including above-ground and below-ground biomass, litter, deadwood and soil. In this review, we discuss current knowledge of carbon stocks in litter, deadwood and soil in Japan’s forest sector. According to data from published reports and nationwide surveys, the carbon stock in forest litter is less than that indicated in the Intergovernmental Panel on Climate Change (IPCC) guidelines for temperate and cool temperate forests; for example, coniferous species showed 4.4 Mg C ha−1 for Cryptomeria japonica and 3.1 Mg C ha−1 for Chamaecyparis obtusa, and broad-leaved species ranged from 3.5 Mg C ha−1 for Castanopsis spp. to 7.3 Mg C ha−1 for Fagus spp. For deadwood carbon stock, coniferous plantations with a record of non-commercial thinning showed 17.1 Mg C ha−1 and semi-natural broad-leaved forests showed 5.3 Mg C ha−1 on average, although only limited data were available. The black soil group (comparable to Andosols and Andisols) showed large carbon stocks in soil layers 0–30 cm deep (130 Mg C ha−1). The brown forest soil group (Cambisols and Inceptisols), occupying the most dominant area, showed a carbon stock of 87.0 Mg C ha−1 on average, which was similar to the data shown in the IPCC guidelines. In a comparison of land use between the forest sector and the agricultural sector for the same soil group, the carbon stock in the agricultural soil was 21% lower and in the grassland soil it was 18% higher than the stock in the forest soil. In this review, we also discuss issues for improving the estimation method and inventory of carbon stock in litter, deadwood and soil in Japan.