It is widely recognized that adsorption, distribution, metabolism, excretion, and toxicology liabilities kill the majority of drug candidates that progress to clinical trials. The development of computational models to predict small molecule membrane permeability is therefore of considerable scientific and public health interest. Empirical qualitative structure permeability relationship models of permeability have been a mainstay in industrial applications, but lack a deep understanding of the underlying biologic physics. Others and we have shown that implicit solvent models to predict passive permeability for small molecules exhibit mediocre predictive performance when validated across experimental test sets. Given the vast increase in computer power, more efficient parallelization schemes, and extension of current atomistic simulation codes to general use graphical processing units, the development and application of physical models based on all-atom simulations may now be feasible. Preliminary results from rigorous free energy calculations using all-atom simulations indicate that performance relative to implicit solvent models may be improved, but many outstanding questions remain. Here, we review the current state-of-the-art physical models for passive membrane permeability prediction and present a prospective look at promising new directions for all-atom approaches.