Meta-mining: The political economy of meta-analysis

Meta-analysis studies the literature reporting estimates of one parameter, which at present is assumed positive. The purpose of the analysis is to find the best meta-average, which corrects the mean of the estimates for bias. The two main biases are: (i) Publication bias, where the correction nearly always makes the average smaller. (ii) Omitted variable bias, where the correction typically makes the average larger. Consequently, the bias is likely to increase if the correction is for the wrong bias. This allows a game of meta-mining to be played. A case study demonstrates the scope for meta-mining, and that it has been done. The game of meta-mining is surely against the purpose of meta-analysis.


| The political economy of estimating policy effects 3
Economic studies often deal with the effect of a policy. It always has an announced goal shared by many, and some theory proposes that the policy may serve this goal. However, the theory is qualitative and needs a quantitative estimate, which is provided by the β-literature. The following argues that there are good reasons to expect that such literatures have an upward bias.
It is likely that researchers with the strongest preferences for the β-goal are most attracted to the field. The policy is implemented by a public bureau with a budget. The classical model of such bureaus (Niskanen, 1994) suggests that the bureau wants to grow, so it wants estimates of the policy effect to be largetoo large is better than too small. The budget often sponsors research either directly or by rewarding loyal researchers in other ways. Obviously, the bureau prefers to sponsor researchers who are friendly towards its goal, and those sponsored by a bureau may develop loyalty. Consequently, collusion may result.
It is common that the main sponsor of research in such cases is the bureau. Thus, many researchers have both a preference for 'good' results and an interest in such results. As regressions are cheap to run once the data are in the machine, it is rational to mine the data by running many regressions and choose the best, which is likely to be too good.
The case study in section 4 is a typical case. It considers the first 141 studies of development aid effectiveness, where well-replicated meta-studies exist. Here both preferences and interests come together. We all prefer aid to work, so that poverty in the world is reduced. Aid budgets also sponsor research in development. In addition, most 2 One reason that researchers should make meta-studies is that it is sobering to discover the width of the range of results even when the results are published in perfectly decent journals. 3 A general theory of the research of the representative economist is found in Paldam (2018). At present, the analysis concentrates on the political economy. researchers in development have interests in the large aid industry, which employs many consultants, notably in project missions that are well-paid and require economists.

| The basis for meta-mining: two biases in the opposite direction
The β-literature consists of all papers with estimates (b, s) of β and its standard error. 4 The (b, 1/s)-scatter is the funnel, which shows the distribution of the estimates of the β-literature. It has a broad base at low precision, and as precision p = (1/s) rises, it narrows. When it is asymmetric, it points to bias. The meta-average is at the axes of symmetry once it is corrected for the asymmetry. The most precise estimates are likely to be close to this axis. Thus, the meta-average is found by weighting the estimates by their precision; see Appendix.
Results vary mainly because estimates include control variables in many combinations. Therefore, the perspective of the paper is one control variable, ζ (zeta), which is included in some, but not all, estimates, as indicated by the binary (0, 1) inclusion variable z. The effect of ζ on β is β z , where the estimates are b and b z respectively. The probability, π, that ζ is included in the estimating equation is taken as a function of b z , i.e., π = π (b z ).
The paper looks at the two main biases: Publication bias occurs if ζ is systematically included for its effect b z on b, i.e., ∂π/∂b z > 0. If b z is insignificant or 'wrong' (negative), the estimate has a high probability of being censored.
Thus, the publication bias is positive, and the meta-average corrected for the bias is smaller than the mean. Omitted variable bias occurs if ζ is included independently of b z , i.e., ∂π/∂b z ≈ 0. Normally, the variable improves the results.
Thus, the publication bias is negative. It follows that if a meta-study adjusts for the wrong bias, notably for a false omitted variable bias, the bias increases. This introduces the theory of meta-mining, which is the main theme of this paper.
The paper proceeds as follows: Section 2 discusses why estimating equations contain control variables. Section 3 considers the two biases and how to adjust for them. For ease of presentation, section 2 and 3 assumes that β has one true value 5 that is positive. 6 Section 4 is a case study showing the range of results that can and have been reached by meta-mining. A Net Appendix brings extra tables to this section. Section 5 concludes that meta-mining defeats the purpose of meta-analysis. The Appendix is a brief introduction the FAT-PET tool of meta-analysis and presents a table that should help the reader keep track of the variables discussed.

| WHY ARE ADD-ON VARIABLES INCLUDED IN MODELS?
This section is fueled by two observations: (i) I have looked at many meta-studies. Meta-studies normally report a funnel diagram displaying the distribution of the estimates, i.e., the b ≈ β. Published estimates have fine t-values, but funnels are still amazingly wide. As mentioned, most of this variation is due to control variables that go in and out of models. (ii) I have made many primary studies myself, and I know that estimates often react substantially to the inclusion/exclusion of controls.
Apart from variables coming from the theoretical model, most estimation models contain add-on variables. The two main types are ad hoc controls and ceteris paribus controls meant to control for sample differences. Section 2.1 discusses the concept of an add-on control, while section 2.2 looks at control variables that should be in some models 4 Three points should be made: (i) The meta-analyst should not prejudge papers. A breakthrough claimed by one paper may not replicate in the next; see section 4.4. (ii) It has often been investigated whether the impact factor of the journal matter for the resultthis is normally rejected. (iii) The typical paper reports about 10 estimates. The resulting clustering of results can be handled by several methods that are not discussed at present. 5 Meta-studies show if the literature finds a robust and unbiased estimate of β. We like to believe that such meta-results are close to the truthor at least that they are closer to truth than the mean. 6 Everything generalizes to a negative effect. The case where the theory claims that there is no effect is not discussed.
but not in others. Section 2.3 discusses ad hoc controls, which seem to be the most common add-on variables.
Section 2.4 asks if an ad hoc variable belongs.
2.1 | Theory and operationalization with one add-on variable ζ (= ζ 1 ) The theoretical model explains y by x and some other variables such as q 1 and q 2 : (1a) y = F(x, q 1 , q 2 ), which is linearized as (1b) y = α + βx + λ 1 q 1 + λ 2 q 2 The estimated equation typically looks as follows: other controls, which include q 1 and q 2 [] holds the remaining k -1 control variables. The theory includes q 1 and q 2 , so they should be included in all estimates. The remaining ζ's are add-on variables. The effect of ζ on b is b z , while the coefficient to The add-on variable ζ is used in some of the papers, but not in all. Journals economize on space, and authors often exercise self-control or are told to reduce their papers. Thus, authors may omit to mention ζ if b z ≈ 0 in the sample used. Perhaps a footnote will say that the author also tried variable ζ, but that it did not work. Such notes may be overlooked by the meta-analyst.

| Two types of variables that should be in some studies but not in others
The variables that should not be in all studies are ceteris paribus controls and alternative representatives of the same latent variable.
Ceteris paribus controls: Each dataset used in the literature differs, as it has some general and some specific traits. To reach a general estimate, it should include variables that control for the specific traits. Such controls are cpcontrols (for ceteris paribus). They do not give bias but reduce variation. The cp-controls should make the funnel leaner, but as mentioned, few funnels are lean. Hence, most controls cannot be cp-controls. Example: Many cross-country estimates contain an Africa-dummy to tackle the African problems of bad climate, low levels of education, poor infrastructure, etc. It normally gets a negative coefficient, and consequently b z is positive. It is obviously wrong to treat the Africa-dummy as an omitted variable in estimates covering other countries. It might be omitted in some papers that include African countries, but then these studies are likely to include explicit variables for the African problems. The example of section 4 includes an Africa-dummy.
Alternative variables: Many variables are confluent. If two variables, ζ 1 and ζ 2 , contain a common factor that affects b, it is likely that the papers include either ζ 1 or ζ 2 . 7 If ζ 1 is taken as an omitted variable when it is missing, and ζ 2 is taken as an omitted variable when it is missing, the effect of the common factor is counted twice, and hence the estimate of b is biased. Only if the estimate contains neither ζ 1 nor ζ 2 , is it missing a variable bias. 7 The right technique is to extract the common factor and use it as the explanatory variable. When a choice between z1 and z2 is made, the choice may be too good.
Example: Cross-country studies normally consider many confluent variables because development is a process that causes transitions in most variables, 8 so confluence is a large problem. Thus, the adjustment for omitted variables may generate rather large biases; see Paldam (2021).

| The pool of ad hoc controls: reasonable, but not inevitable controls
Most control variables are not of the types mentioned but are ad hoc controls that are added for their effect after a search by the researcher. In most fields, the literature has generated a set of K 'permissible' controls. These are controls which may be added at the author's discretion. As they have already been used in other papers, they do not need a thorough discussion. Such controls seem to be the most common ones. Section 4 considers a literature with a pool of 22 such variables; this appears to be a typical number. A researcher may try all or some of these, and if the dataset has, e.g., n = 250 observations, the estimating model may have five controls. Five controls can be chosen from the 22 possible ones in 22

5
= 26,334 ways. Each choice gives an estimate of β. No researcher tries all these estimates, but it is easy to make, e.g., 1,000 estimates in a search process lasting a couple of days, and various diagnostic statistics increase the efficiency of the search.
If the dataset had n = 1,000 observations, all 22 ad hoc variables could be thrown in and tested down so the ones working could be determined. There is always some randomness in the process, but if n is large, the random element is small. If the process is done with n = 100, the randomness may be substantial.
In addition, a paper may introduce a new control. This may even justify the paper. Thus, when a new variable is introduced, it has a big effect, but later papers often show that the effect is exaggerated; see section 4.4 for two such cases.
I have lectured many articles for my students, and in so doing, I have often presented a slide with the control variables of the article and asked the students why these controls were chosen. This always gives some discussion. I believe that it is common that the choice is directed by the desire to make the result better in three ways: (i) It gets closer to the prior of the researcher; (ii) it gets closer to the theoretical predictions, which may be the same as (i); and (iii) it becomes more significant statistically.

| Does an ad hoc variable ζ belong in the estimation equation?
If b z is significant when the sample size goes to infinity, ζ belongs in (2). If b z goes to zero, the control does not belong. If β z is a sizable effect, it is likely to be discovered early in the literature, and ζ will be included from then on, which is discussed as the learning case in section 3.3 below. If β z is small, many of the b z s will be insignificant and remain unreported. Figure 1 shows how the stylized distribution of b z should look in two cases. They look like the typical funnel graph shown in Figure 2 below, but they have different scales at both axes, and the horizontal axis is different. Figure 1a is the case where ζ does not belong in the model, but sometimes becomes positive by chance. If only significantly positive estimates are published, and it is assumed that this is a case of an omitted variable, so that the correction made assumes that the average effect of ζ is a, the meta-average becomes biased by a. Few authors write which controls are ad hoc controls and which are cp-controls. 8 A transition is a systematic change in the variable, so that it diverges from the traditional level as poor countries start to develop and converge to a different level in the modern countries. Transitions typically give correlations of 0.5 to 0.8 of socio-economic variables to income, in wide cross-country data samples; see Paldam (2021). F I G U R E 2 Stylized funnel graphs illustrating publication bias. Figure 2a is symmetric, and b = β = b M , so the FAT-PET is vertical. Figure 2b occurs when researchers and journals refrain from publishing estimates that have the wrong sign or are insignificant, so that the light gray part of the funnel is suppressed. It biases the mean. As p increases, the FAT-PET converges to b M = β < b.

| PUBLICATION BIAS AND OMITTED VARIABLE BIAS
F I G U R E 1 Funnel graphs for the effect b z of ζ on the estimate of β. Funnel of b z , where n is taken to be large. The gray shading indicates that the effect is significantly positive. Such estimates have a (much) larger chance of publication. The average of the significant b z 's is a.

| Publication bias
Publication bias means that the published results differ systematically from the true value. If the typical researcher behaves as predicted by economic theory, he will run many regressions and choose the best as discussed. This gives a publication bias, which can be detected and corrected by the FAT-PET.
Control ζ contributes to the bias if it is included as a function of b z . Most estimates where b z is negative or insignificant remain unpublished. Thus, published estimates are too large. Publication bias is corrected by giving high precision estimates of b a higher weight. This normally gives a downward correction of the mean, as illustrated in Figure 2b, where β is about half the mean, b. It is a good rule-of-thumb to expect that the mean estimate b ≈ 2β; see Ioannidis et al. (2017) and Doucouliagos et al. (2018).
If the funnel has asymmetries, which can be explained as the result of censoring of weak or wrong estimates, it suggests publication bias. The FAT-test statistic b F > 0, so the FAT-PET curve is negatively sloped, as shown by Figure 2b. This confirms the suggestion.

| Omitted variable bias
Control ζ generates an omitted variable bias if ζ is randomly included relative to b z , and b z is significant in the typical study. If b z is substantial, this may give a funnel two tops, which differ by the average of b z ; see Figure 3. Assuming that b z is positive, also at the limit β z , so that ζ should be included, then β 2 is the right estimate and β 1 is biased, and so is the mean b as shown. This is the omitted variable bias, and it is (normally) nega- tive. An omitted variable is corrected by giving estimates including ζ a larger weight. This amounts to adding b z to the estimates not including ζ, and thus it causes an upward correction of the mean, which ideally moves the metaaverage b M from b to β 2 .
In Figure 3a, it is a toss-up whether this is β 1 or β 2 . If ζ should be included and b z > 0, it means not only that b becomes larger, but also that the standard error of the estimate decreases, so that precision rises. Thus, it is likely that the β 2 -peak is higher than the β 1 -peak, and therefore the correction for publication bias will find that peak. This is especially likely if the estimates have publication bias in addition to the omitted variable bias, as shown in Figure 3b.
F I G U R E 3 Stylized funnel graphs illustrating omitted variable bias. The effect of ζ is β z = β 2 -β 1 . If ζ should be included in all estimates, β 2 is the true value and thus b < β, both on Figure 3b and especially on Figure 3a. The graph assumes that ζ has the probability of about 0.5 for being included.
The technique to handle an omitted variable, ζ, is to augment the FAT-PET with the z-inclusion variable, where the meta-average is termed b A . If ζ is randomly included, b A gives an unbiased estimate of β, and in addition, it finds the average value of b z . If the bias is negative, the augmentation with z increases the estimate.

| Omitted variable with learning
Imagine that ζ was not included from the start of the β-literature but was found to work in paper j. Researchers of new papers should read the old papers, so they would know that control ζ works after paper j, and a bit later it will be included in all papers, as illustrated by Figure 4a. In this case, the peak points to the true value, and corrections for either omitted variable bias or publication bias will find the same value, b M ≈ β 2 > b.
If learning is combined with publication bias, as shown in Figure 4b, it is likely that the correction for publication biasshown by the FAT-PET curveis better at finding β 2 than a correction for omitted variable bias. Note that the FAT-PET has a positive slope, while its slope was negative in Figure 2b. However, it is likely that the mean, b, is close to β 2 anyhow.
In the case of learning, imagine that the literature finds that ζ affects the result downward, so that the funnel shifts to the left. This will largely resemble Figure 3b, and the correction for publication bias will get close to the true value. Here the FAT-PET may have an insignificant slope. If the FAT-PET has a negative slope, it is a (strong) indication of a classical publication bias, and if the slope is positive, it points to a learning process. However, negative slopes are much more common than positive slopes in meta-studies.

| What is known if the basic level shows a publication bias?
About two thirds of all meta-studies (in economics) find a funnel asymmetry that indicates a publication bias. If β > 0, the FAT-PET will have a negative slope. Thus, a fraction of the published estimates has controls that are selected from their effect on b. The simple fact that estimates with the wrong sign (i.e., negative) are difficult to publish might be enough.
F I G U R E 4 Stylized funnel graph for omitted variable with learning. Figure 4a combines two funnels: the low is for the estimates without ζ. The peak is β 1. The upper funnel includes ζ. Here the funnel is shifted by d z = β 2 -β 1 . Between the two horizontal lines, some papers include ζ, and others do not. Figure 4b shows the situation from Figure 4a when wrong or insignificant estimates are censored. Here it is unclear where the mean, b, is relative to β 2 , and the FAT-PET is likely to be steep, as a is close to zero.
The β-literature has tried K controls, and we know that most of these controls are systematically selected for the size of b z . If one such control is treated as an omitted variable, and the estimate is adjusted accordingly, the bias increases. This is the basis for meta-mining. It is surely wrong. The case study in section 4 considers a literature with a substantial publication bias. It has K = 22, of which one third are found in the average estimate, in many combinations.
In such cases it is, of course, possible that a few of the ζs are randomly included, but it is difficult to detect these variables. The random part of the effect of ζ will cause ζ not to workand hence be excludedrandomly. This will give publication bias, not omitted variable bias.

| What happens to the FAT and the PET when the MRA is augmented?
Imagine you go ahead augmenting the FAT-PET when a publication bias has been found, i.e., β M < b. Assume further that ζ is one of the variables generating the bias, so that ζ is included when b z is positive and significant only. This is coded by the inclusion variable z that is 1 when ζ is included and otherwise 0.
both the PET and the FAT are estimated as if ζ was included in all estimates with its average effect when included. Thus, the PET increases, and the part of the publication bias caused by ζ disappears. Consequently, the FAT decreases. This is double bad. Not only does the publication bias increase, the tool used to detect the bias is also blunted.
By an augmentation with a handful of controls, one may even increase the PET to exceed the mean, so that β M < b < β A . In addition, the FAT may become insignificant. In the case study in section 4, these effects prove to be substantial and hence easy to misuse.

| A CASE STUDY OF META-MINING
The case study uses the data for a meta-study of 141 papers with 1,779 estimates of the effect of development aid on growth; see Doucouliagos and Paldam (2006;2008;2011;2015). The vast effort to find this effect is probably caused by the fact that researchers know that there is a problem: The simple correlation between aid and growth is zero; see Paldam (2022b).
Section 4.1 gives the basic meta-analysis, showing a publication bias as predicted by section 1.1. Section 4.2 shows how the PET and FAT react to augmentations (aug). Section 4.3 reports the scope for meta-mining.
Section 4.4 turns to the issue of replication by looking at the two most cited AEL papers in the 21st century. A Net-Appendix (Paldam, 2022a) reports further evidence.

| The AEL, aid effectiveness literature: The basic meta-analysis
Due to the said problem, researchers have made large efforts to find a positive result. Many methods are available to put structure on data to make them say something "more" than the correlation. The application of these methods has resulted in many nicely significant positive results, as seen from the funnel on Figure 5. The main tool used to go from zero correlation to the wild scatter of Figure 5 is to add ad hoc controls to the estimating equation. Some of the controls need an explanation: (1) Aid interacted with an institutional variable.
(2) Aid interacted with a measure for good policies.  (Doucouliagos & Paldam, 2008, 2011 for more details on the coding.

| From meta-analysis to meta-mining: Augmenting the FAT-PET
The case study looks at 22 control variables that are all assumed to be ad hoc controls. These controls are listed in Table 2, which also shows how often they are included in the estimates. Two of these variables are conceptually different from the rest. It is the Africa-dummy, which is a cp control, and the OLS dummy, which gives a basic classification of the estimator.
Many researchers have taken the relation to be simultaneous, so it should be estimated by regression adjusting for that. 10 The first half of the studies contains few such regressions, but later studies have many. However, the results reached with OLS and more advanced methods do not differ (Net Appendix). Table 3 shows what happens when the FAT-PET is augmented with 1 to 5 controls using the inclusion variable z for the variables of Table 2. The PET increases, and the FAT decreases. The augmentations produce a range of results reported in rows (3) to (7) of the table. Row (1) covers the 1,779 primary estimates displayed as the funnel of Figure 5. The meta-results from row (2) onwards differ much less than the primary estimates. If we compare rows (1) and (5), N is in the same order, but the std. is 7 times smaller in (5) than in (1). While the range of primary estimates in row (1) goes from À0.948 to 0.908, it only goes from À0.007 to 0.156 in row (7). Thus, all meta-results substantially reduce the range of primary results, but the augmentations still leave a wide range of choices.
I think that the profession believes that the partial correlation between aid and growth must be in the reasonable range from À0.02 to 0.10. It is reached with three augmentations. If we accept augmentations even when the literature has a publication bias, it gives a range of choices that include the full range of reasonable choices. In the same way, augmentations decrease the FAT, as seen in Figure 6b. The FAT is still positive, but it becomes insignificant in 32% of the cases and only rarely exceeds the basic test value. Thus, augmentations blunt the tool showing publication bias as predicted.
4.3 | Meta-mining through augmentations for 1 to 5 controls   The average PET and FAT change along smooth curves, so they are easy to project; see the Net-Appendix. The projection says that for six augmentations, the basic FAT-PET is fully within the 1% NW set, and for nine augmentations, the average FAT is insignificant. Meta-mining has a wide scope. This has been used in practice: The SE points include the choice of Mekasha and Tarp (2013). 11 They start by finding the same basic results as Doucouliagos and Paldam op cit. Then they augment and choose carefully, and conclude that aid is effective, and that the AEL is unbiased. Their paper does not suggest that they have selected an extreme end of a wide spectrum.

| The replication of the two most cited models
The data coded for a meta-study also allow the analyst to see if models with a new variable replicate. The most cited studies in the AEL in this century are the good policy model and the aid squared model. 12 Both papers introduced a new control variable that had a fine effect in the data chosen but proved hard to replicate outside that dataset.
The good policy model of Burnside and Dollar (2000) uses the interacted Aid x policy variable as the new control.
It is variable 2 in Table 2. The variable has been included in 411 estimates. Tables A2 and A3 (in the Net Appendix) show that the variable does not work. On average, the estimates including the variable give lower aid effectiveness estimates than the ones without this variable. The two augmented PETs both suggest that the variable works (a little), but when the estimates of the authors are excluded, the replication fails.
The model of Hansen and Tarp (2000) has aid squared as the new control. It is variable 3 in Table 2. The variable has been used in 333 estimates in the literature. Here the estimates with and without the variable differ, but the augmented PETs show a negative effect of aid squared on aid effectiveness, see Tables A2 and A3 (in the Net Appendix). 13 Thus, both models have failed at replication, and they have disappeared from the literature. This illustrates why results should be repeatedly replicated before they are trusted, and how meta-analysis allows replication of both the central model and specific model variants.

| CONCLUSION
Meta-analysis in economics is made to summarize the set of papers presenting regression results that claim to estimate the same parameter. It analyzes papers using the classic research strategyor at least papers that are presented as if it was followed. Such papers start with a small literature survey showing why the paper presents something new, then follows a theory, leading to a model, which is operationalized as an estimation model. After a 11 The skew reporting is likely to be due to strong preferences and interests of the authors, who are members of a research group at Copenhagen University, the DERG that is largely financed by aidmainly from the Danish aid agency Danida. Mekasha and Tarp (2019) is an update without augmenting. It neatly replicates Doucouliagos and Paldam (2015), with only a few polemic remarks. 12 The papers have 6,400 and about 2,800 citations in Google Scholar (Dec. 1st, 2021). The good policy researchers are from the World Bank, while the aid squared researchers are members of DERG (see the previous note). 13 Once again, the results are even worse when the results of the authors are excluded. When the model failed, the authors made another model (see Dalgaard et al., 2004). The new model is different, but it has the same policy implications, as predicted by the discussion of Mekasha and Tarp op cit.
brief presentation of the data, the model is estimated, and in the great majority of cases the paper concludes that the theory is confirmed, i.e., it is not rejected.
It has often been shown that this strategy contains a great deal of make-believe, as it allows priors to play a large role. Researchers canand often domake many regressions and choose the best. Thus, preferences and interests have a wide scope. Figure 7 shows the amazing range of results reached by the literature on one well-defined parameter. Replication studies are surely important. Here I wish to add that the data used for the present paper are available from the author upon reasonable request.
Meta-analysis is a method to replace strict replication and show what the literature has found. It is important that the result of the meta-study is manipulation robust. The basic FAT-PET MRA is robust. The full literature should be collected, and a list published, so that any reader can check that it is an unbiased list, and the literature should be coded. It is some effort to find the literature and a major effort to do the coding. A few random coding errors are inevitable, but they matter little for the result. Once all of this is done, the basic result follows.
If the basic result shows a publication bias, one should only use augmented FAT-PETS for controls that are randomly included, as regards their effect on β. It is quite difficult to show that this is the case. If the FAT-PET is augmented with the variables generating the publication bias, it brings back the bias. Hence, it defeats the very purpose of meta-analysis.

ACKNOWLEDGMENTS
The paper was presented at the MAER-net colloquium in Piraeus (Greece)  notably Bruno Frey. In addition, the paper owes much to discussions with Chris Doucouliagos, and comments from Erich Gundlach. I also wish to thank the referees for constructive advice. Mathias Barding has been an excellent research assistant.

DATA AVAILABILITY STATEMENT
The data used for the present paper are available from the author upon reasonable request.