Several specific lipids of the cell envelope are implicated in the pathogenesis of M. tuberculosis (Mtb), including phthiocerol dimycocerosates (DIM) that have clearly been identified as virulence factors. Others, such as trehalose-derived lipids, sulfolipids (SL), diacyltrehaloses (DAT) and polyacyltrehaloses (PAT), are believed to be essential for Mtb virulence, but the details of their role remain unclear. We therefore investigated the respective contribution of DIM, DAT/PAT and SL to tuberculosis by studying a collection of mutants, each with impaired production of one or several lipids. We confirmed that among those with a single lipid deficiency, only strains lacking DIM were affected in their replication in lungs and spleen of mice in comparison to the WT Mtb strain. We found also that the additional loss of DAT/PAT, and to a lesser extent of SL, increased the attenuated phenotype of the DIM-less mutant. Importantly, the loss of DAT/PAT and SL in a DIM-less background also affected Mtb growth in human monocyte-derived macrophages (hMDMs). Fluorescence microscopy revealed that mutants lacking DIM or DAT/PAT were localized in an acid compartment and that bafilomycin A1, an inhibitor of phagosome acidification, rescued the growth defect of these mutants. These findings provide evidence for DIM being dominant virulence factors that mask the functions of lipids of other families, notably DAT/PAT and to a lesser extent of SL, which we showed for the first time to contribute to Mtb virulence.