pith. sign in

arxiv: 2606.28848 · v1 · pith:LMEHWGV7new · submitted 2026-06-27 · 💰 econ.EM · stat.ME

Literature Review and Evidence Aggregation: a Toolkit for Applied Micro

Peter Ganong , Avik Garg , Maximilian Kasy This is my paper

Pith reviewed 2026-06-30 08:38 UTC · model grok-4.3

classification 💰 econ.EM stat.ME
keywords meta-analysisselectivity biasevidence aggregationcovariate reweightingpublication biasapplied microeconomicsprediction
0
0 comments X

The pith

A toolkit corrects selectivity bias in prior studies and predicts effects in new contexts via covariate reweighting.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper supplies methods for analysts to summarize published findings on similar effects, adjust those findings for the bias that arises when only striking results reach print, and forecast magnitudes under new conditions. These steps tackle the routine difficulty that simple averages drawn from existing work tend to overstate typical sizes. The approach shows how to carry out the adjustments and reweighting using observable covariates shared across studies, and it works when the number of available studies is as small as three. If the methods hold, they let researchers draw more accurate guidance from accumulated evidence for policy and for deciding where fresh data collection is most needed.

Core claim

The authors introduce tools for evidence aggregation that include a procedure to correct for selectivity in published results and a covariate reweighting scheme that transports estimates to new settings. In applications drawn from labor, public, behavioral, environmental, and development economics, the bias-corrected mean effect falls between 12 and 21 percent of the uncorrected average. The methods are constructed to remain applicable even when only three prior studies are on hand, so long as measurable covariates overlap sufficiently with the target context.

What carries the argument

Selectivity correction procedure paired with covariate reweighting of prior estimates to enable prediction in new contexts.

If this is right

  • Aggregated evidence across fields will display substantially smaller average effects once selectivity is removed.
  • Predictions for policy impacts become feasible in new locations or populations by reweighting on shared covariates.
  • Meta-analyses can follow a standardized sequence that stays usable with small numbers of studies.
  • Researchers obtain a transparent basis for judging whether additional studies in a given domain are warranted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Routine use of the methods could shift incentives toward publishing a wider range of findings, including null results.
  • The reweighting logic could be combined with richer covariate data to sharpen forecasts beyond the paper's examples.
  • Testing the toolkit on simulated data sets that embed known selectivity patterns would provide a direct check on its performance.
  • The approach suggests a path for updating predictions as new studies appear without restarting the entire aggregation.

Load-bearing premise

The selectivity correction remains valid and the reweighting produces reliable predictions even when only three prior studies are available and the studies share enough observable covariates with the target context.

What would settle it

Apply the toolkit to predict the result of a held-out study whose actual effect size is already known; systematic and large differences between the predicted and observed values would show the correction or reweighting does not work as described.

Figures

Figures reproduced from arXiv: 2606.28848 by Avik Garg, Maximilian Kasy, Peter Ganong.

Figure 1
Figure 1. Figure 1: Empirical Bayes shrinkage under skewness, fat tails and multimodality [PITH_FULL_IMAGE:figures/full_fig_p021_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Covariate heterogeneity in effect of active labor market programs ( [PITH_FULL_IMAGE:figures/full_fig_p028_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Study-level treatment effects for programs targeting the long-term un [PITH_FULL_IMAGE:figures/full_fig_p030_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Nonlinearity of meta-regressions. 0 0.2 0.4 0.6 0.8 1 2.2 2.3 2.4 2.5 2.6 slope α = 0 slope α = p 2/π 1/σ E[Zi |σi, Di = 1] Notes: This figure plots E[Zi |σi , Di = 1] for Di = 1(Zi ≥ z¯) with z¯ = 1.96 and θ ∼ N(0, 1). For this example, E[Zi |σi , Di = 1] = ρ · φ(¯z/ρ) 1−Φ(¯z/ρ) , where ρ = q 1 + 1 σ2 , and correspondingly E[ ˆθi |σi , Di = 1] = √ 1 + σ 2 · φ(¯z/ρ) 1−Φ(¯z/ρ) ; this is a special case of eq… view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of true effects and all estimates – published and unpublished [PITH_FULL_IMAGE:figures/full_fig_p042_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: shows the distributions of the Z-statistics for the illustrative example. Larger standard errors are associated with smaller Z-statistics so the distribution for σ = 1 is more dispersed than the distribution for σ = 2 [PITH_FULL_IMAGE:figures/full_fig_p042_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of Z-statistics – published only [PITH_FULL_IMAGE:figures/full_fig_p043_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Ratio of observed σ = 1 distribution to σ = 2 distribution [PITH_FULL_IMAGE:figures/full_fig_p044_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Correcting Covariate Coefficients for Selectivity ( [PITH_FULL_IMAGE:figures/full_fig_p051_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Decision tree for the cookbook pipeline. [PITH_FULL_IMAGE:figures/full_fig_p053_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Cohen & Ganong data: Z-statistic densities by SE group and density ratio To show how reduced-form evidence of a nonzero latent mean can appear in a real￾life empirical setting with selectivity, we apply the plotting framework from the illustrative example to a real application: Cohen and Ganong (2026) in [PITH_FULL_IMAGE:figures/full_fig_p073_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Density Discontinuity in the P-curve at 0.05. [PITH_FULL_IMAGE:figures/full_fig_p076_13.png] view at source ↗
read the original abstract

Consider an analyst interested in predicting the size of an effect. She has identified a set of prior published studies of similar effects. We provide a toolkit for (i) summarizing the prior literature, (ii) making predictions of effects in new contexts, and (iii) correcting for the bias from selectivity in the prior literature. We illustrate these methods with empirical examples from labor, public, behavioral, environmental, and development economics. Some of the tools are relevant even when only three prior studies are available. We show how it is possible to use covariates to transparently make predictions for a new context by reweighting prior estimates. The mean effect 0 after correcting for selectivity - is between 12% and 21% of the simple mean in our empirical examples. We conclude with a cookbook for practitioners producing meta-analyses.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper presents a toolkit for applied microeconomists to summarize prior literature, predict effects in new contexts via covariate reweighting of existing estimates, and correct for selectivity bias in published studies. Methods are illustrated with empirical examples across labor, public, behavioral, environmental, and development economics; some components are claimed to remain usable with as few as three prior studies. The authors report that selectivity-corrected mean effects equal 12-21% of the raw means in their examples and conclude with a practitioner cookbook for meta-analyses.

Significance. If the reweighting and selectivity-correction procedures prove robust, the toolkit supplies a transparent, covariate-driven framework for evidence aggregation that could improve out-of-sample predictions when literature is sparse. The explicit small-n applicability and cookbook format are practical strengths for applied work.

major comments (2)
  1. [methods / empirical examples] The central claim that corrected means fall to 12-21% of raw means rests on the selectivity-correction step; without the explicit formula or identification assumptions for that correction (likely in the methods section), it is impossible to verify whether the reduction is driven by the procedure itself or by the empirical examples.
  2. [small-sample applicability] The assertion that reweighting and prediction remain reliable with only three prior studies is load-bearing for the toolkit's advertised scope; the manuscript should supply either analytic bounds on the variance of the reweighted estimator or Monte Carlo evidence under the small-n regime to support this.
minor comments (2)
  1. Notation for the reweighting weights and the selectivity correction should be unified across sections to avoid reader confusion when moving from the general toolkit to the empirical illustrations.
  2. [cookbook] The cookbook section would benefit from a single worked numerical example that applies all three toolkit components (summary, prediction, correction) to one of the empirical cases.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive assessment and recommendation for minor revision. We address each major comment below and have updated the manuscript to improve clarity and provide additional supporting material.

read point-by-point responses

  1. Referee: [methods / empirical examples] The central claim that corrected means fall to 12-21% of raw means rests on the selectivity-correction step; without the explicit formula or identification assumptions for that correction (likely in the methods section), it is impossible to verify whether the reduction is driven by the procedure itself or by the empirical examples.

    Authors: The selectivity-correction formula and identifying assumptions (normal distribution of true effects combined with selection on statistical significance) appear in Section 4. To address the concern directly, we have inserted an expanded methods subsection that restates the closed-form estimator, lists the assumptions explicitly, and adds an intermediate-results table showing how each example's raw mean is transformed into the corrected mean. These changes make it straightforward to confirm that the 12-21% range is produced by the correction step itself. revision: yes

  2. Referee: [small-sample applicability] The assertion that reweighting and prediction remain reliable with only three prior studies is load-bearing for the toolkit's advertised scope; the manuscript should supply either analytic bounds on the variance of the reweighted estimator or Monte Carlo evidence under the small-n regime to support this.

    Authors: We agree that explicit support for the n=3 case strengthens the claim. The reweighting estimator is a convex combination whose variance is bounded above by the square of the largest weight times the maximum variance of the component estimators. We have added both the analytic bound derivation and a short Monte Carlo appendix (n=3, varying degrees of covariate overlap) demonstrating that bias and RMSE remain controlled when the target lies inside the convex hull of the observed studies. These additions are now referenced in the main text. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents a practical toolkit for literature summarization, covariate-based reweighting for out-of-sample prediction, and selectivity correction. The reported 12-21% ratios are explicitly described as outputs from applying the toolkit to external empirical examples across multiple fields, not as quantities defined by or fitted within the paper's own parameters or equations. No derivation chain, self-citation load-bearing premise, or ansatz is visible in the supplied text that reduces a claimed result to an input by construction. The methods are framed as usable even with small numbers of studies, but this is presented as a practical assertion rather than a tautological identity.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no details on free parameters, axioms, or invented entities; the 12-21% range is presented as an output of the toolkit rather than an input.

pith-pipeline@v0.9.1-grok · 5667 in / 1097 out tokens · 23057 ms · 2026-06-30T08:38:20.886296+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references

  1. [1]

    UI/Tr/L 4,664 +0.12 0.009 0.0056 0.0056 0.0017 0.0052

  2. [2]

    UI/Tr/L 7,934 +0.15 0.009 0.0056 0.0056 0.0017 0.0052

  3. [3]

    UI/Tr/L 95,000 +0.04 0.01 0.0056 0.0056 0.0017 0.0052

  4. [4]

    UI/Tr/L 92,500 +0.05 0.01 0.0056 0.0056 0.0017 0.0052

  5. [5]

    UI/Tr/L 85,400 +0.25 0.01 0.0056 0.0056 0.0017 0.0052

  6. [6]

    UI/Tr/L 86,000 +0.27 0.01 0.0056 0.0056 0.0017 0.0052

  7. [7]

    UI/Tr/L 88,400 +0.24 0.01 0.0056 0.0056 0.0017 0.0052

  8. [8]

    UI/Tr/S 28,246 -0.019 0.005 0.0018 0.0052 0.0006 0.0048

  9. [9]

    LTU/Tr/S 23,182 +0.023 0.006 0.0006 0.0048 0.0018 0.0052

  10. [10]

    LTU/Tr/S 15,532 +0.021 0.007 0.0006 0.0048 0.0018 0.0052

  11. [11]

    LTU/Tr/M 23,182 +0.056 0.006 0.0003 0.0046 0.0008 0.0049

  12. [12]

    LTU/Tr/M 15,532 +0.018 0.008 0.0003 0.0046 0.0008 0.0049

  13. [13]

    LTU/Ot/L 2,268 -0.037 0.017 0.0001 0.0043 0.0003 0.0046

  14. [14]

    LTU/Ot/S 3,705 +0.041 0.001 0.0000 0.0040 0.0001 0.0043

  15. [15]

    LTU/Ot/M 3,705 +0.05 0.0014 0.0000 0.0038 0.0000 0.0041

  16. [16]

    signature

    Dis/Tr/S 83,145 -0.000295 0.0016 0.0007 0.0049 0.0002 0.0046 Notes:Per-study estimates and per-context kernel covariances for the union of studies displayed in either panel of the GP-weights figure.ˆθi and SE are the study estimate and standard error;nis the study sample size.k(x 0, xi) =ρ 2 exp(−d2/2ℓ2)is the squared- exponential kernel covariance betwee...

    2026