It has been increasingly recognized that landscape matrices are an important factor determining patch connectivity and hence the population size of organisms living in highly fragmented landscapes. However, most previous studies estimated the effect of matrix heterogeneity using prior information regarding dispersal or habitat preferences of a focal organism. Here we estimated matrix resistance of harvest mice in agricultural landscapes using a novel pattern-oriented modeling with Bayesian estimation and no prior information, and then conducted model validation using data sets independent from those used for model construction. First, we investigated the distribution patterns of harvest mice for approximately 400 habitat patches, and estimated matrix resistance for different matrix types using statistical models incorporating patch size, patch environment, and patch connectivity. We used Bayesian estimation with a Markov chain Monte Carlo algorithm, and searched for appropriate matrix resistance that best explained the distribution pattern. Patch connectivity as well as patch quality was an important determinant of local population size for the harvest mice. Moreover, matrix resistance was far from uniform, with rice and crop fields exhibiting low resistance and forests, creeks, roads and residential areas showing much higher resistance. The deviance explained by this model (heterogeneous matrix model) was much larger than that obtained by the model with no consideration of matrix heterogeneity (homogeneous matrix model). Second, we obtained distribution data from five additional landscapes that were more fragmented than that used for model construction, and used them for model validation. The heterogeneous matrix model well predicted the population size for four out of five landscapes. In contrast, the homogeneous model considerably overestimated population sizes in all cases. Our approach is widely applicable to species living in fragmented landscapes, especially those for which prior information regarding movement or dispersal is difficult to obtain.