As one of the emerging technologies, the Second Life virtual world provides learners of English as a Foreign Language with a unique opportunity of learning authentic language with native and non-native speakers of English in a virtual environment. It enables them to learn the target language in a real-life-like social communication environment. To date, numerous studies have focused more on the benefits and affordances of learning in such an environment. Little has been done in terms of an empirical exploration into the barriers that impede effective spoken English learning. This study was intended to reduce the gap through investigating a group of Chinese students' experience when they carried out spoken English learning activities in Second Life. A qualitative research method was employed to unveil the challenges of Spoken English learning in this virtual world. Participant observation was made both during the training sessions and in the process of their performing learning tasks. The participants were later interviewed. The study concluded that the success of enabling effective spoken English leaning requires a timely elimination of technical problems and a systemic optimization of learning environments from both pedagogical and administrative perspectives. Limitations of this study and directions for future research were also discussed.