A Hg2+-selective chromophoric upconversion nanosystem is achieved by covalently grafting Rhodamine B hydrazide (RB-hydrazide) onto upconversion luminescent nanorod core–shell structures. The prepared β-NaYF4 nanorods are coated with silica and characterized by X-ray diffraction (XRD), transmission electron microscopy (TEM), scanning electron microscopy (SEM), energy-dispersive X-ray analysis (EDAX), Fourier-transform infrared spectroscopy (FTIR), and photoluminescence spectra. Under 980 nm excitation, the upconversion luminescence is sensitive towards Hg2+ because RB-hydrazide-Hg can efficiently absorb green upconversion emission. The prepared nanocomposites exhibit high sensitivity and selectivity towards Hg2+ over other metal ions. These results indicate that this nanosystem could be developed as a promising fluorescence probe for detecting Hg2+ ions.