This work reports the use of polymer-layered silicate nanoclays as potential flame retardants in unsaturated polyester resins. Preparation, characterization and flammability properties of polyester-clay hybrids have been studied. X-ray diffraction studies have provided evidence that dispersion of functionalized clays in the polymer matrix depends on the type of functional group of the organic modifier used. Flammability properties studied using cone calorimetry suggests that incorporation of nanoclays (5% w/w) reduces peak heat release rate (PHRR) by 23–27% and total heat release (THR) values by 4–11%. The fire growth rate index (FIGRA) is also reduced by 23–30% following nanoclay inclusion. While incorporation of condensed-phase flame retardants such as ammonium polyphosphate, melamine phosphate and alumina trihydrate reduce the PHRR and THR values of polyester resin, the inclusion of small amounts of nanoclay (5% w/w) in combination with these char-promoting flame retardants causes total reductions of the PHRR of polyester resin in the range 60–70%. Ammonium polyphosphate, in particular and in combination with polyester-nanoclay hybrids show the best results compared to other flame retardants. Copyright © 2006 John Wiley & Sons, Ltd.