Matrix models have been widely used to investigate the population dynamics of plant species. To make use of this method, we first divide individuals into groups and estimate transition probabilities per pair of groups. When a continuous variable, such as plant size, is used for grouping, there is often a trade-off: if the class intervals are narrow each group will only include a small number of samples, but if the intervals are wider, this may obscure some changes. This paper introduces a new matrix model in which we no longer have to divide individuals into arbitrarily defined size classes. The methodology is based on the Bayesian non-parametric binary regression. We first divide the data into ‘very fine’ intervals. For estimating transition probabilities in a ‘large’ matrix, we do not use the observed transition rate per class directly, but we smooth neighboring observed rates and select the most appropriate degree of smoothing using an information criterion called the Akaike Bayesian Information Criterion (ABIC). Our approach is illustrated using long-term forest monitoring data from an old-growth, warm-temperate evergreen forest, in which we examined the population dynamics of four evergreen subcanopy tree species. Transition probabilities allowed us to represent d.b.h.-related growth and mortality patterns graphically, and matrix analysis provided stable size distributions, reproductive values and elasticity that vary smoothly for trees of different sizes. The quantitative approach makes it possible to determine characteristic patterns of population dynamics for qualitatively similar species.