Mild fractionation/pretreatment processes are becoming the most preferred choices for biomass processing within the biorefinery framework. To further explore their advantages, new developments are needed, especially to increase the extent of the hydrolysis of poly- and oligosaccharides. A possible way forward is the use of solid acid catalysts that may overcome many current drawbacks of other common methods. In this Review, the advantages and limitations of the use of heterogeneous catalysis for the main groups of solid acid catalysts (zeolites, resins, carbon materials, clays, silicas, and other oxides) and their relation to the hydrolysis of model soluble disaccharides and soluble poly- and oligosaccharides are presented and discussed. Special attention is given to the hydrolysis of hemicelluloses and hemicellulose-derived saccharides into monosaccharides, the impact on process performance of potential catalyst poisons originating from biomass and biomass hydrolysates (e.g., proteins, mineral ions, etc.). The data clearly point out the need for studying hemicelluloses in natura rather than in model compound solutions that do not retain the relevant factors influencing process performance. Furthermore, the desirable traits that solid acid catalysts must possess for the efficient hemicellulose hydrolysis are also presented and discussed with regard to the design of new catalysts.