Joint Evaluation of the System of USDA’s Farm Income Forecasts USDA’s farm income estimates are the official measures of the farm sector’s contributions to the national economy and play an important role in the development of agricultural policy (Schnepf

This study evaluates a system of USDA’s Net Cash Income forecasts, released as part of the farm sector’s income statement, which includes crop receipts, livestock receipts, government payments, farm-related income, and expenses over 1986-2017. We examine these forecasts jointly for bias, accuracy, efficiency, and compositional consistency. Our findings demonstrate that underestimation in early Net Cash Income forecasts stems from underestimation in crop and livestock receipts as well as expenses forecasts. While most components except government payments contribute to the improvement in 12-month ahead forecasts, improvements in 9-month out forecasts are mostly due to crop receipts and expenses forecasts, and government payment forecasts were a main source of improvement in 6-month ahead forecasts. Despite the observed biases and inefficiencies, these forecasts are compositionally consistent with the actual outcomes and represent realistic projections of the farm sector accounts.


Joint Evaluation of the System of USDA's Farm Income Forecasts
USDA's farm income estimates are the official measures of the farm sector's contributions to the national economy and play an important role in the development of agricultural policy (Schnepf 2019). These forecasts have been released by USDA since 1910 and serve as one of the main indicators of the economic well-being of the farm sector. They are widely used by policymakers and media sources to help understand developments in the agricultural economy and by lenders and other agricultural sector stakeholders seeking to understand the magnitude and drivers of farm sector well-being. Farm income forecasts are also used to inform policies on trade assistance, crop insurance, and other policies designed to offset declining revenues and boost farm incomes. Therefore, these forecasts are often used by the farm equipment industry, farm banks, and other farm-related industries to formulate business plans and account for such policies. Furthermore, state and local governments use USDA farm income forecasts for forecasting personal and real property tax receipts. Thus, the interest in USDA's farm income forecasts spans across and beyond the farming community (Dubman, McElroy, and Dodson 1993).
Net farm income is one of the most frequently cited USDA statistics with the estimates providing a retrospective view of farmers' income and financial status (McGath et al. 2009). Lucier, Chesley, and Ahearn (1986) state that while net income forecasts are still used to measure the economic well-being of the farming sector as originally intended, they are also used by federal legislators to determine performance and direction of farm policies. Kuethe, Hubbs, and Sanders (2018) argue that net farm income forecasts are used extensively by all participants in the legislative process to alert lawmakers to changing economic conditions for farmers and ranchers. The Congressional Research Service provides an annual report prepared for members and committees of the Congress that includes the USDA farm income forecast and an agricultural trade outlook to inform policy makers (Schnepf 2019). There are numerous instances where the farm income forecasts and estimates have been used to justify changes in farm policies or continuing an existing policy, particularly during Farm Bill discussions.
Furthermore, these forecasts serve as an input in various USDA models as well as U.S. GDP estimates (McGath et al. 2009). Given the important role of these forecasts, it is imperative to ensure that they are accurate and reliable.
However, despite their prominent role in the agricultural sector, very little research has been devoted to evaluating farm income forecasts. In fact, most of the previous USDA forecast evaluation literature focused on price and production forecasts (e.g., Manfredo 2002, 2003;Good 2006, 2013). Because these price and production forecasts often serve as inputs for farm income forecasts (McGath et al. 2009), it is likely that any deficiencies detected in production and price forecasting literature may be carried over into farm income forecasts. A recent study by Kuethe, Hubbs, and Sanders (2018) provides a detailed analysis of bottom line net farm income forecasts. One of their main findings suggests that there is a downward bias in the initial forecast released in February, 18 months before the official estimate is released in August of the following year. They also find that the updated forecasts, released 12 to 6 months before the official estimate, are inefficient as these forecasts tend to overreact to new information. However, their study is focused on the forecasts of total net farm income and does not evaluate the accuracy of its components that are released at the same time.
Data availability for the components of the farm income forecasts over an extended period of time is a major challenge for this type of analysis. Our study uses a new dataset that includes cash components of farm income that form net cash income forecasts 1 over 1986-2017.
Specifically, the components of the net cash income (NCI) include crop receipts, animal and animal product receipts, cash farm-related income, total direct government payments, and cash expenses. To the best of our knowledge, the accuracy and efficiency of these components of farm income accounts, as well as their contribution to the NCI forecast accuracy, has not been evaluated in previous studies. Furthermore, previous literature provides little guidance on evaluation of joint forecasts (such as net cash income forecasts and their components), as most of the forecast evaluation methodology (e.g., Nordhaus 1987;Holden and Peel 1990;Patton and Timmermann 2007) has been developed for single forecast applications, such as the net farm income analyzed by Kuethe, Hubbs, and Sanders (2018).
Only a few recent studies of the macroeconomic forecasts attempted multivariate evaluation of joint forecasts. Caunedo et al. (2013) jointly tested the rationality of the Federal Reserve's forecasts of inflation, unemployment, and output growth using the methodology developed by Komunjer and Owyang (2012). Their approach is based on deriving the weights for a multivariate utility function using forecast errors. Sinclair, Stekler, and Carnow (2015) developed an alternative approach that is based on comparison of vectors of related forecasts.
The rationale behind this approach is whether several forecasts may be substituted for one another or used in place of actual data for policy decisions in real time. We believe that this approach is most suitable for net cash farm income forecast evaluation.
1 Net cash income is a less comprehensive measure of farm income as it does not include noncash items, such as the value of inventory adjustments.
Given the limitations of the previous literature, the goal of this study is to evaluate the accuracy of NCI forecasts and its components taking into account the joint nature of these forecasts. The evaluation of NCI forecasts and its components in this study focuses on three optimality conditions: bias, improvement, and efficiency. The multivariate approach considers the joint nature of the forecasts of the NCI components and provides insights into the combined accuracy of these forecasts. This characteristic is particularly relevant for forecasts used for policy analysis, such as NCI forecasts. Our findings indicate that, similar to net farm income forecasts, net cash income forecasts are biased downward at longer forecast horizons. In contrast to previous studies, our study is able to uncover sources of bias in NCI forecasts due to individual components and reveal their relative contribution to NCI forecast errors at various forecast horizons. Furthermore, we are able to demonstrate that the accuracy of NCI forecasts significantly improves between 18-and 12-month ahead forecasts and to identify the contribution of various component forecasts to this improvement. Finally, our findings show that despite the biases and inefficiencies detected in this study, NCI forecasting systems at all forecast horizons are compositionally consistent 2 with the official estimates which makes them suitable for decision making.

Farm Income and Wealth Forecasts
USDA's Economic Research Service (ERS) agency releases U.S. farm sector income and wealth statistics data which include historical U.S.-and state-level farm income and wealth estimates, as 2 Compositional consistency is determined based on the difference between the vector of the forecasts and the vector of the official estimates. well as U.S.-level forecasts for the current calendar year. These forecasts are released within an income statement that follows an accounting equation: (1) Net cash income = (Crop receipts + Livestock receipts + Cash farm-related income + Total direct government payments) -Cash expenses = Gross cash income -Cash expenses. Net farm expenses (EXP) are production expenses related to inputs purchased for use in the production of commodities, including farm origin inputs (feed, livestock and poultry purchased, seed), manufactured inputs (fertilizer and lime, fuels and oils, electricity, pesticides), interest charges (short-term interest, real estate interest), other operating expenses (e.g. repair and maintenance, labor, machine hire and customwork), and overhead expenses (capital consumption, property taxes, net rent to non-operators). Expenses data are mainly based on information reported by farmers on Agricultural Resource Management Survey (ARMS) and the latest Census of Agriculture. ERS generates forecasts of production expenses by moving a base year estimate by the changes in the price and quantity indicators. Forecasts of prices paid indexes are used as the price indicators and they follow the prices paid indexes published by the National Agricultural Statistics Service (NASS). Finally, the difference between gross cash income forecast and cash expenses forecast results in the forecast of net cash income.
The first U.S. forecast for each calendar year t is released in February of that year, subsequently revised in August and November of year t, and in February of the following year t+1, followed by an official estimate in August of year t+1. Thus, every calendar year estimate is associated with four forecasts released 18 (February), 12 (August), 9 (November), and 6 (February) months ahead of the official estimate. Since the terminal event, the calendar year estimate, is the same across all these forecasts, they are considered fixed-event forecasts. These estimates can be updated several times after their initial release and are not considered as "final" until the NASS "final estimates" are released based on the most current Census of Agriculture.

Data and Descriptive Statistics
Our study examines the USDA's forecasts of NCI and its main components, including CR, LR, FRI, 6 GP, and EXP over 1986 through 2017. All data are obtained from USDA-ERS archives.
To facilitate comparisons across different forecast categories, we define forecasts in natural logarithm form as  Because the NCI forecasts are generated through an accounting equation, errors in NCI forecasts can be traced down to errors in its respective components. Thus, for each forecast horizon, NCI is an aggregate of errors in its components and the relative contribution of each component is estimated using the following ordinary least squares (OLS) regression: (2) , , , , , ℎ 6, 9, 12, 18 .
The results of this analysis shown in table 2 demonstrate that while errors in all components significantly contribute to NCI errors, the magnitude of their contribution differs considerably.
The largest driver of NCI errors appears to be the expenses component with estimated coefficients suggesting that a 1% error in EXP leads to about -2.5% error in NCI at all forecast horizons. Crop and livestock receipt errors have a similar impact on NCI accuracy with 1% error in these components leading to about 1.5% error in NCI. The impact of these errors appears to grow across forecasting horizon, from 1.36 at h=18 to 1.7 at h=6 for crop receipts and from 1.5 at h=18 to 1.8 at h=6 for livestock receipts. GP and FRI errors tend to have the smallest impact with a 1% error in these components leading to about 0.1% to 0.2% increase in NCI error. The signs of the estimated coefficients demonstrate the offsetting between errors in revenue and expense components as shown for h=18 in figure 2. This figure shows that as long as errors in GCI (the sum of revenue components) and expense components are made in the same direction (e.g. overestimation), they would tend to cancel out through the accounting equation (where expenses are subtracted from GCI) resulting in smaller NCI errors. On the other hand, errors of the components of GCI (which are summed to calculate GCI) should be negatively correlated to cancel out. Figure 3 shows changes in average errors of various forecast categories across the forecasting horizon illustrating potential biases in these forecasts. For unbiased forecasts, positive errors should be offset by negative errors resulting in zero average errors. Figure 3 shows that 18-month ahead NCI errors are about 11% on average and remain positive across the forecasting cycle (suggesting underestimation). Similar patterns but on a smaller scale (4% at h=18) are observed for CR forecasts. EXP and LR errors are closest to zero suggesting a lack of bias in these forecasts. On the other hand, FRI forecasts tend to have large positive errors (suggesting underestimation) in both the beginning and the end of the forecasting cycle. GP forecast errors tend to average 6.66% at h=18 and -4.57% at h=9, indicating underestimation in long-term and overestimation in shorter-term forecasts. Our study examines whether these systematic errors are statistically significant.
Forecast revisions are another specific characteristic of fixed-event forecasts that show how forecasts evolve during the forecasting cycle. The sum of all forecast revisions is equivalent to forecast error. For example, for 6-month ahead forecasts, the revision is equal to the forecast error; but for 12-month ahead forecasts, forecast error is equal to the sum of two revisions (between 12 and 6 months and between 6 and 0 months). Figure 4 shows that, on average, the first revision between 18-and 12-month horizons appears to be positive for all categories, followed by another smaller positive revision in 9-month ahead forecasts (which is consistent with the pattern for underestimation observed with forecast errors in figure 3), and forecasts revisions for some categories become negative only at 6-month horizon. These patterns in forecast revisions are further analyzed later in this study within the forecast efficiency tests.

Methodology
Our forecast evaluation approach accounts for both the joint and fixed event nature of USDA's

Tests of Bias
The basic requirement for forecast optimality is that forecasts at each horizon lack systematic error, or bias. The Mincer and Zarnowitz (1969) (2012) shows that the SUR estimator has efficiency gains over the OLS estimator when the error terms across the equations are contemporaneously correlated, and when the common regressors across equations are different by either including different regressors or having different numeric values of the regressors. In this multivariate setting, the null hypothesis for a test of bias is : , 0, ∀ , which is tested individually for each equation using Benjamini-Hochberg (1995) q-values as described in Anderson (2008) to adjust p-values for multiple testing, as well as jointly for the entire system. 7 Sinclair, Stekler, and Carnow (2015) use a vector autoregressive (VAR) model to evaluate bias in the errors of joint rolling-event forecasts (where the final event is equidistant from the forecasts, i.e. one-quarter ahead forecasts). However, our focus here is on the bias in the errors associated with fixed-event forecasts (with multiple forecasts, h=18, 12, 9 and 6, of the same final event), which makes an SUR system more appropriate than a VAR.

Improvements in Forecast Accuracy
Another requirement for fixed-event forecasts is that errors decrease (i.e., accuracy increases) across the forecasting cycle (Patton and Timmermann 2007). Changes in errors across the forecasting cycle are reflected in forecast revisions as: , , , , ℎ 6, 9, 12 , where the vectors , and , contain, respectively, percent forecast revisions and disturbance terms for each component j, and the vector , contains parameters to be estimated for each forecast category given by , , , , , , , , , , , , ′. In this multivariate setting, the null hypothesis for this test is : , 0, ∀ , which is tested individually for each equation using Benjamini-Hochberg (1995) q-values as described in Anderson (2008) to adjust p-values for multiple testing, as well as jointly for the entire system.

Evaluation of Forecast Efficiency
Weak-form efficiency of fixed-event forecasts, described by Nordhaus (1987), implies that forecast revisions should be uncorrelated with past revisions. This condition is typically examined in a single variable framework by regressing forecast revisions on previous revisions.
For joint forecasts, these regressions can be estimated jointly for all forecasted categories in an SUR model for each forecast horizon: , , , , , , ℎ 0, 6, 9 , 1, 2, 3, similar to the vector of the outcomes, it can be substituted for the actual data for decision making. Thus, it is a more general measure of what constitutes a "good" overall forecast.

Bias
The results for the test of bias reported in Additionally, joint significance test results indicate presence of bias in a system of 18-, 12-, and 9-month, but not 6-month ahead farm income forecasts.

Forecast Improvement
Our examination of improvement in forecast accuracy shown in and reduction in FRI forecasts by about 8.5%. Significant improvements in accuracy between 12-and 9-month ahead forecasts are limited to CR and EXP forecasts with average decrease in error of about 1%. These improvements appear to cancel out in the accounting equation as the reduction in error of 9-month ahead NCI forecasts is not significantly different from zero. The only category that demonstrates improvement at the 6-month horizon is GP forecasts with about 3% decrease in forecast error, however this improvement is not statistically significant according to the Wald's Chi-squared statistic that considers multiple tests. On the other hand, the results of the joint significance tests indicate significant improvements in the entire system of forecasts at each horizon, 12-, 9-, and 6-month ahead forecasts. Note that the final changes in the forecasts between 6-and 0-month ahead are equivalent to the forecast error analysis displayed in table 3.

Forecast Efficiency
Tests of forecast efficiency shown in table 5 examine the degree to which these changes are random. Based on Nordhaus' (1987) definition, if forecasts are efficient, future revisions should not be predictable using current or past revisions, i.e. subsequent revisions should be uncorrelated. If revisions are positively correlated, the forecasts are considered "smoothed" or revisions are "too slow," and information is incorporated over several consecutive revisions.
Negatively correlated revisions imply "jumpy" forecasts that are revised "too fast" and future revisions tend to correct an "overreaction" to new information that took place in a previous revision. Furthermore, a significant constant would indicate systematic increases or decreases in the forecasts.

Compositional Changes
Despite the evidence of biases and inefficiencies in the farm income forecasts, we need to determine whether forecasting system as a whole provides an overall view of the farm income that is consistent with the outcomes that actually occurred. The results of this analysis shown in that we fail to reject the null hypothesis for both level and percent forecasts at all forecast horizons. 9 Therefore, these forecasts are representative of the actual outcomes and could be used to obtain a realistic picture of the composition of the farm sector accounts. Specifically, the difference between the mean vector of forecasts, at each horizon, and the observed outcomes is sufficiently small. While the individual forecasts may contain bias or inefficiencies, the system of forecasts is compositionally consistent with the official estimate at each horizon.

Conclusions
This study seeks to evaluate USDA's net cash income forecasts and its components jointly as a system of fixed-event forecasts. While the forecasts of bottom line net farm income have been examined before, this is the first study to evaluate the forecasts of the components of farm income estimates that provide the building blocks for the total measures. A joint evaluation of the accuracy of the components and the total measures allows us to track down the sources of problems in the total measures to individual components in order to identify the opportunities for improving these forecasts by USDA forecast providers. 9 Through a Monte Carlo analysis, Hoffelder (2017) shows that Mahalanobis distance-based test statistics have a low probability of a type I error, but relatively high probability of a type II error in small samples.
Our findings demonstrate significant underestimation in these forecasts as a group at 18- Our findings reveal that forecast accuracy improves significantly at each forecast horizon with the largest improvement between 18-and 12-month ahead forecasts. Most components except government payments contribute to the significant improvement in the accuracy of 12month ahead NCI forecasts. This finding is not surprising as better information from other USDA sources becomes available in time for the 12-month ahead forecasts, such as WASDE and initial Crop Production reports. Further improvement in crop receipt and expenses forecasts is observed at 9-month horizon due to better source data, but the government payment forecasts are not significantly improved until 6-month horizon. We believe that better communication with the USDA agencies responsible for administering government payment programs may help improve these forecasts earlier, and in the meantime, policy makers should consider government payments forecast accuracy during farm policy discussions.
Tests of efficiency in forecast revisions reveal significant correlations between consecutive revisions, indicating that new information is not incorporated into these forecasts efficiently. Specifically, 9-and 6-month ahead livestock receipt revisions and 6-month ahead expenses revisions are positively correlated with previous revisions, suggesting that these forecasts are smoothed. On the other hand, negative correlation is detected between 9-month ahead and previous revisions of crop receipt forecasts and in the last expenses and farm-related income forecast revisions, indicating correction of information contained in the previous forecasts. Inefficiency in NCI forecasts is associated with crop receipts revisions at 9-month horizon and expenses revision at 6-month horizon.
Finally, the tests of compositional consistency, adopted from the macroeconomic literature, indicate that the farm income forecasting system as a whole is consistent with the conditions that actually occurred and therefore provided useful information for decision making.
These tests combine individual accuracy measures in an attempt to make conclusions about the characteristics of the entire vector of forecasts as a whole. This method for evaluating joint forecasts can be applied to other forecasting systems, such as farm income balance sheet or WASDE forecasts as it allows for a better understanding of interaction of the components that comprise these forecasting systems and their contribution to its accuracy and efficiency.
Our results have implications for both USDA forecasters and the farm sector in general.
To the extent that our results show a significant underestimation in the 18-to 9-month ahead forecasts, the USDA forecasters should consider changes to their forecast models and estimation procedures. On the other hand, farmers and other agricultural stakeholders can view these initial 18-to 9-month horizon forecasts as conservative projections when making their decisions, recognizing that the final estimates are typically higher.

Crop Receipts
Forecast (