The association between smoking and sleep quality was investigated in a sample of 1439 Japanese civil servants. In this cross-sectional survey, the questionnaire used included items on the Pittsburgh sleep quality index (PSQI), smoking status (current, ex- and non-smoker), smoking intensity, grade of employment, physical activity, alcohol consumption, longstanding illness and work characteristics. A general linear model analysis was undertaken to assess the association between smoking status and components of sleep quality. Multiple logistic analysis was used to evaluate the relations between smoking status and smoking intensity with poor sleep in men and women, separately. Significant associations were found between smoking status and components of sleep quality, such as subjective sleep quality and sleep latency in men, as well as sleep latency and sleep medication use in women. The global PSQI score was significantly higher (lower sleep quality) in male ex-smokers versus non-smokers and in female current smokers versus non-smokers. Male current smokers smoking 21–40 cigarettes/day and ex-smokers presented a significantly increased prevalence of poor sleep: adjusted odds ratios (OR) were 1.54 (1.01–2.36) and 2.01 (1.20–3.37), respectively. But in women, only current smokers were associated with a high prevalence of poor sleep: adjusted OR was 2.68 (1.23–5.84). Sleep quality is therefore affected not only by smoking but also by the process of smoking cessation in men. These results suggest that a special attention should be paid to smoking habits in dealing with poor sleep quality.