Recent studies have shown that α Centauri B might be, from an observational point of view, an ideal candidate for the detection of an Earth-like planet in or near its habitable zone (0.5–0.9 au). We study here if such habitable planets can form, by numerically investigating the planet-formation stage which is probably the most sensitive to binarity effects: the mutual accretion of km-sized planetesimals. Using a state-of-the-art algorithm for computing the impact velocities within a test planetesimal population, we find that planetesimal growth is only possible, although marginally, in the innermost part of the habitable zone (HZ) around 0.5 au. Beyond this point, the combination of secular perturbations by the binary companion and gas drag drives the mutual velocities beyond the erosion limit. Impact velocities might later decrease during the gas removal phase, but this probably happens too late for preventing most km-sized objects to be removed by inward drift, thus preventing accretion from starting anew. A more promising hypothesis is that the binary formed in a crowded cluster, where it might have been wider in its initial stages, when planetary formation was ongoing. We explore this scenario and find that a starting separation roughly 15 au wider, or an eccentricity 2.5 times lower than the present ones, is required to have an accretion-friendly environment in the whole HZ.