The adequacy and utility of behavioural characters in phylogenetics is widely acknowledged, especially for stereotyped behaviours. However, the most common behaviours are not stereotyped, and these are usually seen as inappropriate or more difficult to analyze in a phylogenetic context. A few methods have been proposed to deal with such data, although they have never been tested on samples larger than six species, which limits their evolutionary interest. In the present study, we perform behavioural observations on 13 cockroach species and derive behavioural phylogenetic characters with the successive event-pairing method. We combine these characters with morphological and molecular data (approximately 6800 bp) in a phylogenetic study of 41 species. We then reconstruct ancestral states of the behavioural data to study evolution of social behaviour in these insects with regard to their social systems (i.e. solitary, gregarious, and subsocial) and diversity of habitat choice. We report for the first time that nonstereotyped behavioural data are adequate for phylogenetic analyses: they are no more homoplastic than traditional data, and support several phylogenetic relationships that we discuss. From an evolutionary perspective, we show that the solitary species Thanatophyllum akinetum does not display original behavioural interactions, suggesting phylogenetic inertia of interactive behaviours despite a radical change in social structure. Conversely, the subsocial species Parasphaeria boleiriana shows original behavioural interactions, which could result from its peculiar social system or habitat. We conclude that phylogenetic approaches in studies of behaviour are useful for deciphering evolution of behaviour and discriminating between its different modalities, even for nonstereotyped characters. © 2013 The Linnean Society of London, Biological Journal of the Linnean Society, 2014, 111, 58–77.