A novel series of composites of polyarylene ether nitrile terminated with phthalonitrile (PEN-t-Ph) filled with hybrid Fe3O4 nanospheres (h-Fe3O4) was prepared via in situ composition. Based on the cross-linking interactions between the phthalonitrile at the end of PEN-t-Ph molecular chains and the phthalonitrile on the surface of h-Fe3O4 particles to form phthalocyanine ring, it was shown that the PEN-t-Ph/h-Fe3O4 system had superior interfacial compatibility and the h-Fe3O4 particles were locked in the matrix resin. These results had been confirmed by scanning electron microscope analysis. By orthogonal experiments and statistic analysis, the optimal conditions of cure temperature, type of h-Fe3O4 and content of h-Fe3O4 had been determined. Meanwhile, the results of range analysis and variance analysis indicated that the cure temperature had great effects on the thermal properties. Thermal studies revealed that the glass transition temperature of PEN-t-Ph/h-Fe3O4 cured at 320°C was 214.7°C, increased by about 40°C compared to the PEN-t-Ph/h-Fe3O4 without heat treatment, and the temperature corresponding to the weight loss of 5 wt % was increased by about 20°C. Mechanical measurements indicated that PEN-t-Ph/h-Fe3O4 cured at 320°C possesses excellent mechanical properties with tensile strength of 93.33 MPa and tensile modulus of 2414.05 MPa, 9.91 MPa, 355.76 MPa higher than pure PEN-t-Ph film cured at 320°C, and 13.26 MPa, 397.90 MPa higher than PEN-t-Ph/h-Fe3O4 without heat treatment. Most importantly, the presence of h-Fe3O4 particles endows PEN-t-Ph/h-Fe3O4 system with good magnetic property. Thus, PEN-t-Ph/h-Fe3O4 cured at 320°C may have potential applications in field of magnetic materials. © 2014 Wiley Periodicals, Inc. J. Appl. Polym. Sci. 2014, 131, 40418.