We present an iterative algorithm for estimating the observation-noise covariance matrix Rt for ensemble-based data assimilation. The proposed algorithm enables us to estimate parameters in Rt, the number of which may become exceedingly large for data assimilation. The optimal Rt is estimated based on the maximum likelihood and the algorithm can be described as an expectation-maximization (EM) algorithm. The key to constructing the algorithm is to obtain the optimality condition of Rt in a closed form. The proposed method estimates Rt, which is adaptive to the current observation yt. We present an application of the proposed method to a coupled atmosphere–ocean model and find that only a few iterations are necessary for convergence. We also find that Rt needs to have some kind of structure to obtain a reasonable covariance matrix using the proposed algorithm.