Modeling nanoscale devices quantum mechanically is a computationally challenging problem where new methods to solve the underlying equations are in a dire need. In this paper, we present an approach to calculate the charge density in nanoscale devices, within the context of the nonequilibrium Green's function approach. Our approach exploits recent advances in using an established graph partitioning approach. The developed method has the capability to handle open boundary conditions that are represented by full self-energy matrices required for realistic modeling of nanoscale devices. Our method to calculate the electron density has a reduced complexity compared with the established recursive Green's function approach. As an example, we apply our algorithm to a quantum well superlattice and a carbon nanotube, which are represented by a continuum and tight binding Hamiltonian, respectively, and demonstrate significant speedup over the recursive method. Copyright © 2013 John Wiley & Sons, Ltd.