Current prebiotics, such as fructo-oligosaccharides (FOS), are limited in their persistence in the distal colon and are predominantly fermented in the proximal colon. In order to identify a potential alternative, the differences in the fermentation profile of arabinogalactan (AG) and FOS have been assessed in the Simulator of the Human Intestinal Microbial Ecosystem. The effect of each product on the composition and activity of the microbial community was analysed during a 3-week treatment period at a dose of 5 g day−1. While FOS indeed was mainly fermented in the simulated proximal colon, AG was still available for fermentation in the simulated distal colon as shown by pH profiles, size exclusion chromatography and analyses of specific enzymatic activities. As a consequence, the main effect of the products (increase in propionate and butyrate and decrease in ammonium production) occurred in different intestinal areas. DGGE and qPCR analyses confirmed that the main modulation of the microbiota by the two products occurred in different areas of the gut. AG was associated with a statistically significant increase in the concentration of total bacteria, Bacteroidetes, Faecalibacterium prausnitzii, a delayed bifidogenic effect and a decrease of the pathogenic Clostridium perfringens. FOS led to a strong lactobacillogenic effect.