In this study, polymer hybrid composites were synthesized by sol-gel process. 3-Amino-propyltrimethoxysilane [APTMS)/γ-Glycidoxypropyl trimethoxy-silane (GPTMS); (4, 4′-Methylene-dianiline (DDM)] and 1,4-Bis(trimethoxysilylethyl) benzene (BTB) were added to DGEBA type epoxy resin for anticipated to exhibit excellent thermal stability. Boron trifluoride monoethylamine (BF3MEA) was used as catalyst. The structure of nanocomposites was characterized by attenuated total reflectance (ATR) and solid-state 29Si NMR which suggest EP-APTMS-BTB/EP-GPTMS-BTB possesses T3; T1–T0, and T1 structures when the BTB content was lower than 10 wt % and higher 20 wt %, respectively. BF3MEA was proved to be an effective catalyst for the sol-gel reaction of APTMS, but it could not promote for GPTMS. From TEM microphotographs, EP-APTMS-BTB (10 wt %) possesses a dense inorganic structure (particle size around 5–15 nm) compare with the loose inorganic structure of EP-GPTM-/BTB (10 wt %). DSC, TGA were use to analyze the thermal properties of the nanocomposites and DMA was used to analyze the dynamic mechanical properties of hybrid composites. The Tgs of all nanocomposites decreased with the increasing BTB content. A system with BTB content lower than 10 wt % showed good dynamic mechanical property and thermal stability (Td5 increased from 336°C to 371°C, char yield increased from 27.4 to 30.2%). The structure of inorganic network affects the Td5 and dynamic mechanical properties of composite. © 2014 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2014, 131, 40984.