Understanding interfacial science is critical to many modern technologies. It is very common in solid-state physics for electronic properties to show novel phenomena when combining various dissimilar materials at atomically abrupt interfaces. For example, semiconductor interfaces have provided the foundation of modern electronic devices for several decades. Now with advances in growth and synthesis, controllable high quality complex oxide heterojunctions can be routinely fabricated. Since complex oxide materials exhibit a wide variety of functionalities owing to their strong coupling to the electron, lattice, orbital and spin degrees of freedom, these materials display a wide spectrum of interesting functionalities. Combining dissimilar complex oxides at interfaces allows one to explore and create intriguing phenomena that are not attainable in the lone bulk constituents. However, the key challenge has been the direct characterization of these interfaces at the nanoscale in order to understand the physical properties found at complex oxide interfaces. This requires the development of new experimental approaches. In this paper, we review the utilization of cross-sectional scanning tunneling microscopy/spectroscopy as a direct probe of these oxide interfaces at the nanoscale. This technique provides valuable insight to both structural and electronic properties of these unique systems and enables understanding of the detailed electronic structure (e.g., local electronic density of states (LDOS), charge transfer, band bending, etc.) at oxide interfaces, which is of key interest to both fundamental and applied science.