Output-only modal identification is needed when only structural responses are available. As a powerful unsupervised learning algorithm, blind source separation (BSS) technique is able to recover the hidden sources and the unknown mixing process using only the observed mixtures. This paper proposes a new time-domain output-only modal identification method based on a novel BSS learning algorithm, complexity pursuit (CP). The proposed concept—independent ‘physical systems’ living on the modal coordinates—connects the targeted constituent sources (and their mixing process) targeted by the CP learning rule and the modal responses (and the mode matrix), which can then be directly extracted by the CP algorithm from the measured free or ambient system responses. Numerical simulation results show that the CP method realizes accurate and robust modal identification even in the closely spaced mode and the highly damped mode cases subject to non-stationary ambient excitation and provides excellent approximation to the non-diagonalizable highly damped (complex) modes. Experimental and real-world seismic-excited structure examples are also presented to demonstrate its capability of blindly extracting modal information from system responses. The proposed CP is shown to yield clear physical interpretation in modal identification; it is computational efficient, user-friendly, and automatic, requiring little expertise interactions for implementations. Copyright © 2013 John Wiley & Sons, Ltd.