A novel encapsulated ammonium polyphosphate (APP) with an inorganic–organic hybrid coating (MAPP) was prepared by a sol–gel method using tetraethoxysilane and octyltriethoxysilane as precursors. The properties of MAPP were investigated by water solubility, hydrophobicity, and morphological determination. The structure of MAPP was characterized by X-ray photoelectron spectroscopy. The flame retardancy of epoxy resin composite with MAPP was evaluated by limiting oxygen index, UL-94 test, and cone calorimetry test. The results showed that both tetraethoxysilane and octyltriethoxysilane hydrolyze and condense to form a dense polysiloxane layer structure on the surface of APP. The water solubility of MAPP was reduced from 0.64 to 0.13 g/100 mL of water, and the water resistance of epoxy resin composites filled with MAPP was also greatly improved. The limiting oxygen index, UL-94 test, and cone calorimeter results showed that the epoxy resin composites filled with MAPP had better flame retardancy, probably because of a synergistic effect between polysiloxane and APP. Copyright © 2013 John Wiley & Sons, Ltd.