Dynamic spectrum auction (DSA) has been considered as one of potential spectrum allocation approaches in cognitive femtocell networks. As a modified version of traditional spectrum auction, DSA should not only increase auction revenue but also improve spectrum utilization on finer time granularity. We propose a DSA algorithm based on a double optimization framework (DOF), which focuses on the optimization of auction revenue and spectrum utilization. The optimization processing consists of two stages. Firstly, a proper auction period is selected to balance the expected spectrum utilization and auction revenue. Then, the cognitive femtocell base station adjusts its reserve price with the repetition of auction to leverage over instant revenue and spectrum utilization. At the same time, the bidders can adjust their bidding price to improve utilities. Performance analysis shows that the DOF-based DSA algorithm has low complexity and can resist collusion, so it can be carried out frequently with small overhead. On the other hand, it is better than the greedy algorithm and Vickrey–Clarke–Groves auction on revenue. Simulation results show that the DOF-based DSA algorithm can keep a fine spectrum utilization and bring the cognitive femtocell base station more revenue in both single-unit award spectrum auction and multi-unit awards spectrum auction. Copyright © 2012 John Wiley & Sons, Ltd.