We study the hydrostatic pressure response of the local environment of N in the ternary dilute nitride alloys, i.e., GaN0.021P0.979, GaN0.085As0.915, and GaN0.015Sb0.985 by Raman spectroscopy and density functional theory (DFT) of the local vibrational mode (LVM) of N. In all samples studied N is surrounded by four Ga atoms. To a first approximation the N vibrational mode can be described by the N-atom vibrating in a rigid cage formed by the four Ga nearest neighbors. However, our careful experimental and theoretical analysis reveals that, despite the same nearest neighbor environment of N in the three dilute nitride hosts under study, distinct differences between the pressure dependence of the corresponding local N-vibrational modes remain. These differences are rather due to the different chemical identities of the anions of the host than due to differences in N content and strain. Furthermore, it is shown that the LVM of N in the three alloys behaves differently under hydrostatic pressure than the optical extended phonon modes of zincblende GaN.