Understanding tree growth as a function of tree size is important for a multitude of ecological and management applications. Determining what limits growth is of central interest, and forest inventory permanent plots are an abundant source of long-term information but are highly complex. Observation error and multiple sources of shared variation (spatial plot effects, temporal repeated measures, and a mosaic of sampling intervals) make these data challenging to use for growth estimation. We account for these complexities and incorporate potential limiting factors (tree size, competition, and resource supply) into a hierarchical state-space model. We estimate the diameter growth of white fir (Abies concolor) in the Sierra Nevada of California from forest inventory data, showing that estimating such a model is feasible in a Bayesian framework using readily available modeling tools. In this forest, white fir growth depends strongly on tree size, total plot basal area, and unexplained variation between individual trees. Plot-level resource supply variables (representing light, water, and nutrient availability) do not have a strong impact on inventory-size trees. This approach can be applied to other networks of permanent forest plots, leading to greater ecological insights on tree growth.