Effects of Mg substitution on order/disorder transition, microstructure, and microwave dielectric characteristics of Ba((Co0.6Zn0.4)1/3Nb2/3)O3 complex perovskite ceramics have been investigated. The ordered complex perovskite solid solutions are obtained in Ba((Co0.6−x/2Zn0.4−x/2Mgx)1/3Nb2/3)O3 ceramics (x = 0, 0.1, 0.2, and 0.3), and the ordering degree in the as-sintered dense ceramics increases with increasing Mg-substitution amount. The significantly improved Qf value is obtained in the present ceramics with increasing x, whereas the dielectric constant decreases slightly together with some increase of temperature coefficient of resonant frequency. The best combination of microwave dielectric characteristics is obtained in the composition of x = 0.3: εr = 33.7, Qf = 93 800 GHz, and τf = 9.6 ppm/°C. In the Mg-substituted compositions, clear domain boundaries are obtained and the domain size increases as x increases, the highest Qf value is obtained when the domain size is about 40–60 nm in the ceramics with x = 0.3. The increased ordering degree and the fine ordering domain structure are considered to primarily contribute to the significant increase of Qf value in the Mg-substituted Ba((Co0.6Zn0.4)1/3Nb2/3)O3 complex perovskite ceramics.