The role of dietary fiber on the risk of colon and rectal cancer has been investigated in numerous studies, but findings have been inconsistent. The purpose of this study was to examine associations between intake of dietary fiber and risk of incident colon (including distal and proximal colon) and rectal cancer in the prospective Scandinavian HELGA cohort and to determine if fiber source (vegetables, fruits, potatoes, cereals) impacted the association. We included 1,168 incident cases (691 colon, 477 rectal cancer), diagnosed during a median of 11.3 years, among 108,081 cohort members. Sex-specific incidence rate ratios (IRRs) of colon and rectal cancer were related to intake of total or specific fiber source using Cox proportional hazards models. For men, an inverse association was observed between intake of total fiber and the risk of colon cancer per an incremental increase of 10 g day−1, IRR (95% CI): 0.74 (0.64–0.86). Intake of cereal fiber per 2 g day−1 was associated with an IRR of 0.94 (0.91–0.98), which was also seen for intake of cereal fiber from foods with high fiber content (≥5 g per 100 g product), where the IRR per 2 g day−1 was 0.94 (0.90–0.98). In women, intake of cereal fiber per 2 g day−1 was also associated with lower risk of colon cancer, 0.97 (0.93–1.00). No clear associations were seen for rectal cancer. Our data indicate a protective role of total and cereal fiber intake, particularly from cereal foods with high fiber content, in the prevention of colon cancer.