This paper investigates seasonal cosmic ray-induced neutron variations measured over a long-term period (from 2011 to 2016) in both the high-altitude stations located in medium geomagnetic latitude and Antarctica (Pic-du-Midi and Concordia, respectively). To reinforce analysis, modeling based on ground albedo neutrons simulations of extensive air showers and the solar modulation potential was performed. Because the local environment is well known and stable over time in Antarctica, data were used to validate the modeling approach. A modeled scene representative to the Pic-du-Midi was simulated with GEANT4 for various hydrogen properties (composition, density, and wet level) and snow thickness. The orders of magnitudes of calculated thermal fluence rates are consistent with measurements obtained during summers and winters. These variations are dominant in the thermal domain (i.e., En < 0.5 eV) and lesser degree in epithermal and evaporation domains (i.e., 0.5 eV < En < 0.1 MeV and 0.1 MeV < En < 20 MeV, respectively). Cascade neutron (En > 20 MeV) is weakly impacted. The role of hydrogen content on ground albedo neutron generation was investigated with GEANT4 simulations. These investigations focused to mountain environment; nevertheless, they demonstrate the complexity of the local influences on neutron fluence rates.