We discuss cosmological hydrodynamic simulations of galaxy formation performed with the new moving-mesh code arepo, which promises higher accuracy compared with the traditional smoothed particle hydrodynamics (SPH) technique that has been widely employed for this problem. In this exploratory study, we deliberately limit the complexity of the physical processes followed by the code for ease of comparison with previous calculations, and include only cooling of gas with a primordial composition, heating by a spatially uniform ultraviolet background, and a simple subresolution model for regulating star formation in the dense interstellar medium. We use an identical set of physics in corresponding simulations carried out with the well-tested SPH code gadget, adopting also the same high-resolution gravity solver. We are thus able to compare both simulation sets on an object-by-object basis, allowing us to cleanly isolate the impact of different hydrodynamical methods on galaxy and halo properties. In accompanying papers, Vogelsberger et al. and Sijacki et al., we focus on an analysis of the global baryonic statistics predicted by the simulation codes, and complementary idealized simulations that highlight the differences between the hydrodynamical schemes. Here we investigate their influence on the baryonic properties of simulated galaxies and their surrounding haloes. We find that arepo leads to significantly higher star formation rates for galaxies in massive haloes and to more extended gaseous discs in galaxies, which also feature a thinner and smoother morphology than their gadget counterparts. Consequently, galaxies formed in arepo have larger sizes and higher specific angular momentum than their SPH correspondents. Interestingly, the more efficient cooling flows in arepo yield higher densities and lower entropies in halo centres compared to gadget, whereas the opposite trend is found in halo outskirts. The cooling differences leading to higher star formation rates of massive galaxies in arepo also slightly increase the baryon content within the virial radius of massive haloes. We show that these differences persist as a function of numerical resolution. While both codes agree to acceptable accuracy on a number of baryonic properties of cosmic structures, our results thus clearly demonstrate that galaxy formation simulations greatly benefit from the use of more accurate hydrodynamical techniques such as arepo and call into question the reliability of galaxy formation studies in a cosmological context using traditional standard formulations of SPH, such as the one implemented in gadget. Our new moving-mesh simulations demonstrate that a population of extended gaseous discs of galaxies in large volume cosmological simulations can be formed even without energetic feedback in the form of galactic winds, although such outflows appear required to obtain realistic stellar masses.