Alterations in left ventricular mass and geometry vary along with the degree of obesity, but mechanisms underlying such covariation are not clear. In a case–control study, we examined how body composition and fat distribution relate to left ventricular structure and examine how sustained weight loss affects left ventricular mass and geometry. At the 10-year follow-up of the Swedish obese subjects (SOS) study cohort, we identified 44 patients with sustained weight losses after bariatric surgery (surgery group) and 44 matched obese control patients who remained weight stable (obese group). We also recruited 44 matched normal weight subjects (lean group). Dual-energy X-ray absorptiometry, computed tomography, and echocardiography were performed to evaluate body composition, fat distribution, and left ventricular structure. BMI was 42.5 kg/m2, 31.5 kg/m2, and 24.4 kg/m2 for the obese, surgery, and lean groups, respectively. Corresponding values for left ventricular mass were 201.4 g, 157.7 g, and 133.9 g (P < 0.001). In multivariate analyses, left ventricular diastolic dimension was predicted by lean body mass (β = 0.03, P < 0.001); left ventricular wall thickness by visceral adipose tissue (β = 0.11, P < 0.001) and systolic blood pressure (β = 0.02, P = 0.019); left ventricular mass by lean body mass (β = 1.23, P < 0.001), total body fat (β = 1.15, P < 0.001) and systolic blood pressure (β = 2.72, P = 0.047); and relative wall thickness by visceral adipose tissue (β = 0.02, P < 0.001). Left ventricular adjustment to body size is dependent on body composition and fat distribution, regardless of blood pressure levels. Obesity is associated with concentric left ventricular remodeling and sustained 10-year weight loss results in lower cavity size, wall thickness and mass.