The failure mechanisms induced by a wedge-shaped tool indenting normally against a rock surface are investigated using the discrete element method (DEM). The main focus of this study is to explore the conditions controlling the transition from a ductile to a brittle mode of failure. The development of a damage zone and the initiation and propagation of a brittle fracture is well captured by the DEM simulations. The numerical results support the conjecture that initiation of brittle fractures is governed by a scaled flaw length Λ, a ratio between the flaw size λ and the characteristic length (where KIc is the toughness and σc the uniaxial compressive strength). The size of the damage zone agrees well with analytical predictions based on the cavity expansion model. The effects of a far-field confining stress and the existence of a relief surface near the indenter are also examined.Copyright © 2012 John Wiley & Sons, Ltd.