Estrogen receptor (ER)-β signaling has generally been implicated in protection against colorectal cancer. The ER-β gene cytosine-adenine (ESR2 CA) repeat polymorphism was reported to be associated with colorectal cancer, although showing contradicting results probably caused by ethnicity or age distribution of the subjects. We investigated the association between this polymorphism and the colorectal cancer risk in a community-based case-control study in Japan (685 cases/778 controls), including only subjects younger than 75. The effect modifications of the body mass index (BMI) and isoflavone intake were also examined. ESR2 CA repeat polymorphism was determined by polymerase chain reaction using fluorescein-labeled primers. CA repeat alleles were classified into short (S) allele (<22 repeats) and long (L) allele (≥22 repeats). Subjects were divided into three genotype groups (SS/SL/LL). The risk of colon cancer, but not of rectal cancer, was increased with an increasing number of L alleles among postmenopausal women; age-adjusted odds ratio (OR) for SL and LL genotypes compared with the SS genotype were 1.78 and 2.91, respectively (trend p = 0.002). Increased risks of colon cancer associated with the L allele were more evident among postmenopausal women with low BMI (<25 kg m−2) or with high isoflavone intake. Such associations were not observed among men or premenopausal women. Having longer ESR2 CA repeat increases colon cancer risk among postmenopausal women younger than 75, possibly with modification of BMI and isoflavone intake. Aging and estrogenic condition may be important in the colon cancer pathogenesis associated with ESR2 CA repeat polymorphism.