In this article we designed a cyanate ester based novel hybrid nanocomposites by sol–gel approach with different weight percentages of Zirconium dioxide (ZrO2). The synthesized monomer and polymer nanocomposites were characterized by Fourier-transform infrared, 1H, and 13C nuclear magnetic resonance (NMR) spectra. Optical and thermal properties reveals the excellent optical transparency and good thermal stability of hybrid nanocomposites. Scanning electron microscope and high resolution transmission electron microscope analysis confirm the homogeneous dispersion of ZrO2 in the nanocomposites. Interestingly 7% ZrO2 nanocomposite exhibit highest k value of 9.21 which may be due to the well dispersion of ZrO2 and their strong interfacial polarization in the organic–inorganic hybrid nanocomposites. © 2014 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2014, 131, 41097.