Infectious particles can be deposited on surfaces. Susceptible persons who contacted these contaminated surfaces may transfer the pathogens to their mucous membranes via hands, leading to a risk of respiratory infection. The exposure and infection risk contributed by this transmission route depend on indoor surface material, ventilation, and human behavior. In this study, quantitative infection risk assessments were used to compare the significances of these factors. The risks of three pathogens, influenza A virus, respiratory syncytial virus (RSV), and rhinovirus, in an aircraft cabin and in a hospital ward were assessed. Results showed that reducing the contact rate is relatively more effective than increasing the ventilation rate to lower the infection risk. Nonfabric surface materials were found to be much more favorable in the indirect contact transmission for RSV and rhinovirus than fabric surface materials. In the cases considered in this study, halving the ventilation rate and doubling the hand contact rate to surfaces and the hand contact rate to mucous membranes would increase the risk by 3.7–16.2%, 34.4–94.2%, and 24.1–117.7%, respectively. Contacting contaminated nonfabric surfaces may pose an indirect contact risk up to three orders of magnitude higher than that of contacting contaminated fabric surfaces. These findings provide more consideration for infection control and building environmental design.