Open Science in Astrophysics: Citation Benefits of Open Code, Open Data, and Open Access
Pith reviewed 2026-07-02 06:00 UTC · model grok-4.3
The pith
Astrophysics papers with open data receive 32 percent more citations after statistical controls for other factors.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
After controlling for grants, code size, data repository size, programming language, number of authors, paper length, and publication date, open data is associated with a 32 percent citation increase, open access with 26 percent, and open code with 16 percent; the open-data advantage appears in every sub-field and is largest in Galaxies+Cosmology and ISM.
What carries the argument
Multivariate least-squares regression together with partial correlations and non-parametric tests that isolate the contribution of each openness variable from the other measured quantities.
Load-bearing premise
The regression and partial correlations successfully remove the effects of unmeasured paper quality, subfield citation norms, and selection biases that could link openness to higher citations.
What would settle it
A new analysis of a comparable paper sample that adds direct measures of intrinsic paper quality and still finds no citation difference for open versus closed papers would falsify the reported advantages.
Figures
read the original abstract
We analyze the relationship between open-accessibility in data, code, and paper text in astrophysics using a sample of 53,194 peer reviewed papers published between January 2021 and April 2025, drawn from NASA's Astrophysics Data System (ADS). We measure eleven quantities: open accessibility of text, open-code status, open-data status, number of grants received, code size, programming language, data repository size, citation count, number of authors, paper length, and publication date. We break down citation advantages based on six astrophysical sub-fields: Solar System, Planet, Stellar, ISM, High Energy, and Galaxies+Cosmology, determined by keywords. This is accomplished by tuning a multivariate least-squares regression model with alongside partial correlations and non-parametric tests to isolate the contribution of each facet of openness. After controlling for the aforementioned quantities, we find significant citation advantages associated with all three forms of openness: open data (+32%, p < 10^-24), open access (+26%, p < 10^-67), and open code (+16%, p = 0.003). The open-data citation advantage is present in all six sub-fields, and especially in Galaxies+Cosmology and ISM, which have the strongest cultures of sharing simulation outputs and observational data products. Open-code and open-data sharing rates are highest in Galaxies+Cosmology and HEA (~0.9% and ~2.9%), reflecting their more developed community data infrastructure, and lowest in Solar System and ISM, where data is distributed on platforms not taken into account by this study. Our findings support the long held notion that public access comes with concrete personal incentives for authors in terms of citations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes citation benefits associated with open access, open data, and open code using a sample of 53,194 peer-reviewed astrophysics papers (2021–2025) from NASA's ADS. It measures eleven variables including openness indicators, grants, authors, length, date, subfield (via keywords: Solar System, Planet, Stellar, ISM, High Energy, Galaxies+Cosmology), code size, language, and repo size. A multivariate least-squares regression is tuned alongside partial correlations and non-parametric tests to isolate openness effects after controls, yielding reported advantages of +32% (open data, p<10^{-24}), +26% (open access, p<10^{-67}), and +16% (open code, p=0.003), with subfield variations and higher sharing rates in Galaxies+Cosmology and HEA.
Significance. If the regression successfully isolates causal effects, the work supplies large-sample quantitative evidence that openness confers measurable citation gains in astrophysics, strongest for data and varying by subfield infrastructure. The combination of regression, partial correlations, and non-parametrics plus the sample size are methodological strengths that could inform open-science policy if robustness to selection is demonstrated.
major comments (2)
- [Abstract and Methods (regression specification)] The multivariate least-squares regression (described in the abstract and methods) controls for grants, authors, length, date, subfield, code size, language, and repo size but includes no proxy for intrinsic paper quality, novelty, or rigor. This omission risks confounding the reported +32%/+26%/+16% coefficients, as openness decisions and citation rates may both be driven by unmeasured quality; the central claim that controls isolate the contribution of openness therefore requires additional justification or sensitivity tests.
- [Abstract and Results (subfield breakdown)] Subfield classification is performed via keywords, yet the abstract notes that low sharing rates in Solar System and ISM partly reflect platforms not captured by the study. This raises the possibility that subfield-specific citation norms or data practices are incompletely controlled, which could affect the claim that the open-data advantage is present (and strongest) in all six sub-fields.
minor comments (3)
- [Methods] Provide explicit definitions and detection criteria for open-code status, open-data status, and open-access status, including any thresholds or external databases used.
- [Methods] Report the exact regression specification (functional form, interaction terms, variance inflation factors) and the robustness checks performed against the listed controls.
- [Results] Clarify whether the percentage advantages are derived from exponentiated coefficients or marginal effects and how they account for the citation-count distribution.
Simulated Author's Rebuttal
We thank the referee for their detailed and constructive report. We address each major comment below and outline planned revisions to strengthen the manuscript.
read point-by-point responses
-
Referee: [Abstract and Methods (regression specification)] The multivariate least-squares regression (described in the abstract and methods) controls for grants, authors, length, date, subfield, code size, language, and repo size but includes no proxy for intrinsic paper quality, novelty, or rigor. This omission risks confounding the reported +32%/+26%/+16% coefficients, as openness decisions and citation rates may both be driven by unmeasured quality; the central claim that controls isolate the contribution of openness therefore requires additional justification or sensitivity tests.
Authors: We agree this is a substantive limitation: no direct measure of intrinsic quality, novelty, or rigor is available in the ADS-derived dataset, and residual confounding remains possible. Number of grants, author count, and paper length provide partial proxies (as these correlate with perceived quality and resources), and the combination of multivariate regression, partial correlations, and non-parametric tests offers some triangulation. However, these do not fully substitute for a quality proxy. In revision we will (1) add an explicit limitations subsection discussing this issue and the direction of potential bias, and (2) report additional sensitivity checks (e.g., subfield-stratified models and robustness to alternative specifications). We will not claim full causal isolation but will qualify the language accordingly. revision: yes
-
Referee: [Abstract and Results (subfield breakdown)] Subfield classification is performed via keywords, yet the abstract notes that low sharing rates in Solar System and ISM partly reflect platforms not captured by the study. This raises the possibility that subfield-specific citation norms or data practices are incompletely controlled, which could affect the claim that the open-data advantage is present (and strongest) in all six sub-fields.
Authors: Keyword-based subfield assignment follows standard practice in ADS analyses and is the only scalable approach for 53k papers. The manuscript already flags that Solar System and ISM rates are depressed by external platforms. The regression treats subfield as a categorical control, and the open-data coefficient remains positive in every subfield. To address the referee's concern we will expand the results and discussion sections with (a) more detail on known subfield data practices and (b) explicit caveats about possible residual variation in citation norms. We will also verify that the reported advantage holds after additional interaction terms between openness and subfield. revision: partial
Circularity Check
No significant circularity: empirical regression on external citation data
full rationale
The paper's central claim consists of fitted coefficients (+32%, +26%, +16%) obtained via multivariate least-squares regression, partial correlations, and non-parametric tests applied to an external sample of 53,194 ADS papers. Controls (grants, authors, length, date, subfield via keywords, code size, language, repo size) and the target citation counts are all measured quantities independent of the openness indicators. No equations reduce the reported advantages to quantities defined in terms of themselves; no self-citations are invoked as load-bearing premises; no ansatz or uniqueness theorem is smuggled in. The derivation is therefore self-contained against external benchmarks and receives the default non-circularity finding.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Least-squares regression assumptions hold (linearity, independence, homoscedasticity) for the citation model.
Reference graph
Works this paper leans on
-
[1]
2025 , howpublished =
Anthropic , title =. 2025 , howpublished =
2025
-
[2]
and Waldner, Fran
Tennant, Jonathan P. and Waldner, Fran. The academic, economic and societal impacts of Open Access: an evidence-based review , journal =. 2016 , doi =
2016
-
[3]
Open Science now: A systematic literature review for an integrated definition , journal =
Vicente-Sa. Open Science now: A systematic literature review for an integrated definition , journal =. 2018 , doi =
2018
-
[4]
Nature , volume =
Lawrence, Steve , title =. Nature , volume =. 2001 , doi =
2001
-
[5]
IEEE Data Engineering Bulletin , volume =
Hajjem, Chawki and Harnad, Stevan and Gingras, Yves , title =. IEEE Data Engineering Bulletin , volume =. 2006 , note =
2006
-
[6]
PLoS ONE , volume =
Gargouri, Yassine and Hajjem, Chawki and Lariviere, Vincent and Gingras, Yves and Carr, Les and Brody, Tim and Harnad, Stevan , title =. PLoS ONE , volume =. 2010 , doi =
2010
-
[7]
Elevating the status of code in ecology , volume =
Mislan, KAS and Heer, Jeffrey M and White, Ethan P , journal =. Elevating the status of code in ecology , volume =
-
[8]
The state of
Piwowar, Heather and Priem, Jason and Larivi. The state of. PeerJ , volume =. 2018 , doi =
2018
-
[9]
2010 , howpublished =
Swan, Alma , title =. 2010 , howpublished =
2010
-
[10]
The Access/Impact Problem and the Green and Gold Roads to Open Access: An Update , journal =
Harnad, Stevan and Brody, Tim and Vall. The Access/Impact Problem and the Green and Gold Roads to Open Access: An Update , journal =. 2008 , doi =
2008
-
[11]
The Development of Open Access Journal Publishing from 1993 to 2009 , journal =
Laakso, Mikael and Welling, Patrik and Bukvova, Helena and Nyman, Linus and Bj. The Development of Open Access Journal Publishing from 1993 to 2009 , journal =. 2011 , doi =
1993
-
[12]
Figshare: A universal repository for academic resource sharing? , volume =
Thelwall, Mike and Kousha, Kayvan , year =. Figshare: A universal repository for academic resource sharing? , volume =. Online Information Review , doi =
-
[13]
and Day, Roger S
Piwowar, Heather A. and Day, Roger S. and Fridsma, Douglas B. , title =. PLoS ONE , volume =. 2007 , doi =
2007
-
[14]
, title =
Piwowar, Heather and Vision, Todd J. , title =. PeerJ , volume =. 2013 , doi =
2013
-
[15]
and Drachen, Thore M
Dorch, Bertil F. and Drachen, Thore M. and Ellegaard, Ole , title =. Proceedings of the International Astronomical Union , volume =. 2015 , doi =
2015
-
[16]
PLoS ONE , volume =
Colavizza, Giovanni and Hrynaszkiewicz, Iain and Staden, Isla and Whitaker, Kirstie and McGillivray, Barbara , title =. PLoS ONE , volume =. 2020 , doi =
2020
-
[17]
Maitner, Brian and Santos Andrade, Paul Efren and Lei, Luna and Kass, Jamie and Owens, Hannah L. and Barbosa, George C. G. and Boyle, Brad and Castorena, Matiss and Enquist, Brian J. and Feng, Xiao and Park, Daniel S. and Paz, Andrea and Pinilla-Buitrago, Gonzalo and Merow, Cory and Wilson, Adam , title =. Ecology and Evolution , volume =. doi:https://doi...
-
[18]
, title =
Peng, Roger D. , title =. Science , volume =. 2011 , doi =
2011
-
[19]
Proceedings of the National Academy of Sciences , volume =
Stodden, Victoria and Seiler, Jennifer and Ma, Zhaokun , title =. Proceedings of the National Academy of Sciences , volume =. 2018 , doi =
2018
-
[20]
IEEE Signal Processing Magazine , volume =
Vandewalle, Patrick and Kovacevic, Jelena and Vetterli, Martin , title =. IEEE Signal Processing Magazine , volume =. 2009 , doi =
2009
-
[21]
and Katz, Daniel S
Smith, Arfon M. and Katz, Daniel S. and Niemeyer, Kyle E. and. Software citation principles , journal =. 2016 , doi =
2016
-
[22]
and Gruenpeter, Marc and Honeyman, Tom , title =
Katz, Daniel S. and Gruenpeter, Marc and Honeyman, Tom , title =. Patterns , volume =. 2021 , doi =
2021
-
[23]
Wilkinson, Mark D. and Dumontier, Michel and Aalbersberg, IJsbrand Jan and Appleton, Gabrielle and Axton, Myles and Baak, Arie and Blomberg, Niklas and Boiten, Jan-Willem and da Silva Santos, Luiz Bonino and Bourne, Philip E. and others , title =. Scientific Data , volume =. 2016 , doi =
2016
-
[24]
2022 , howpublished =
Scientific Information Policy for the Science Mission Directorate (. 2022 , howpublished =
2022
-
[25]
2019 , howpublished =
Accelerating the transition to full and immediate. 2019 , howpublished =
2019
-
[26]
and Eichhorn, Guenther and Accomazzi, Alberto and Grant, Carolyn S
Kurtz, Michael J. and Eichhorn, Guenther and Accomazzi, Alberto and Grant, Carolyn S. and Murray, Stephen S. and Watson, Joyce M. , title =. Astronomy & Astrophysics Supplement Series , volume =. 2000 , doi =
2000
-
[27]
and Henneken, Edwin A
Accomazzi, Alberto and Kurtz, Michael J. and Henneken, Edwin A. and Grant, Carolyn S. and Thompson, Donna and Chyla, Roman and McDonald, Steven and Holachek, Alexandra and Dave, Roxana and Elliott, Jonathan , title =. Astronomical Data Analysis Software and Systems XXIV , series =. 2015 , editor =
2015
-
[28]
and Accomazzi, Alberto , title =
Henneken, Edwin A. and Accomazzi, Alberto , title =. Astronomical Data Analysis Software and Systems. 2012 , editor =
2012
-
[29]
and Eichhorn, Guenther and Accomazzi, Alberto and Grant, Carolyn S
Kurtz, Michael J. and Eichhorn, Guenther and Accomazzi, Alberto and Grant, Carolyn S. and Demleitner, Markus and Murray, Stephen S. , title =. Journal of the American Society for Information Science and Technology , volume =. 2005 , doi =
2005
-
[30]
Nature , volume =
Ginsparg, Paul , title =. Nature , volume =. 2011 , doi =
2011
-
[31]
, title =
Belter, Christopher W. , title =. Journal of the Medical Library Association , volume =. 2015 , doi =
2015
-
[32]
Journal of Informetrics , volume =
Waltman, Ludo , title =. Journal of Informetrics , volume =. 2016 , doi =
2016
-
[33]
Nature , volume =
Hicks, Diana and Wouters, Paul and Waltman, Ludo and de Rijcke, Sarah and Rafols, Ismael , title =. Nature , volume =. 2015 , doi =
2015
-
[34]
Proceedings of the National Academy of Sciences , volume =
Radicchi, Filippo and Fortunato, Santo and Castellano, Claudio , title =. Proceedings of the National Academy of Sciences , volume =. 2008 , doi =
2008
-
[35]
Journal of Informetrics , volume =
Thelwall, Mike , title =. Journal of Informetrics , volume =. 2016 , doi =
2016
-
[36]
, journal =
Casey, Andy and Mandel, Ilya and Ray, P.K. , journal =. The effect of the. 2025 , month =
2025
-
[37]
Science Editing , volume =
Lammey, Rachael , title =. Science Editing , volume =. 2020 , doi =
2020
-
[38]
Quantitative Science Studies , volume =
Hendricks, Ginny and Tkaczyk, Dominika and Lin, Jennifer and Feeney, Patricia , title =. Quantitative Science Studies , volume =. 2020 , doi =
2020
-
[39]
doi:10.25495/7GXK-RD71 , url =
Zenodo , year =. doi:10.25495/7GXK-RD71 , url =
-
[40]
bioRxiv , year =
Piwowar, Heather and Priem, Jason and Orr, Richard , title =. bioRxiv , year =
-
[41]
Science , volume =
Szalay and Gray 2001 , title =. Science , volume =. 2001 , doi =
2001
-
[42]
and others , title =
Quinn, Peter J. and others , title =. Optimizing Scientific Return for Astronomy through Information Technologies , series =. 2004 , doi =
2004
-
[43]
and Fanelli, Daniele and Ioannidis, John P
Goodman, Steven N. and Fanelli, Daniele and Ioannidis, John P. A. , title =. Science Translational Medicine , volume =. 2016 , doi =
2016
-
[44]
Nature , volume =
Baker, Monya , title =. Nature , volume =. 2016 , doi =
2016
-
[45]
2014 , eprint=
The Unified Astronomy Thesaurus , author=. 2014 , eprint=
2014
-
[46]
Nelson, Dylan and Springel, Volker and Pillepich, Annalisa and Rodriguez-Gomez, Vicente and Torrey, Paul and Genel, Shy and Vogelsberger, Mark and Pakmor, R. The. Computational Astrophysics and Cosmology , volume =. 2019 , doi =
2019
-
[47]
and Primini, Francis A
Evans, Ian N. and Primini, Francis A. and Glotfelty, Kenny J. and Anderson, Craig S. and Bonaventura, Nina R. and Chen, Judy C. and Davis, John E. and Doe, Stephen M. and Evans, Janet D. and Fabbiano, Giuseppina and others , title =. The Astrophysical Journal Supplement Series , volume =. 2010 , doi =
2010
-
[48]
and Whitney, Donald R
Mann, Henry B. and Whitney, Donald R. , title =. The Annals of Mathematical Statistics , volume =. 1947 , doi =
1947
-
[49]
and Wallis, W
Kruskal, William H. and Wallis, W. Allen , title =. Journal of the American Statistical Association , volume =. 1952 , doi =
1952
-
[50]
and White, Halbert , title =
MacKinnon, James G. and White, Halbert , title =. Journal of Econometrics , volume =. 1985 , doi =
1985
-
[51]
, title =
Kendall, Maurice G. , title =
-
[52]
and Bower, Richard G
Schaye, Joop and Crain, Robert A. and Bower, Richard G. and Furlong, Michelle and Schaller, Matthieu and Theuns, Tom and Dalla Vecchia, Claudio and Frenk, Carlos S. and McCarthy, Ian G. and Helly, John C. and Jenkins, Adrian and Rosas-Guevara, Y. M. and White, Simon D. M. and Baes, Maarten and Booth, C. M. and Camps, Peter and Navarro, Julio F. and Qu, Ya...
2015
-
[53]
Python: A dynamic, open source programming language , year =
-
[54]
and Millman, K
Harris, Charles R. and Millman, K. Jarrod and van der Walt, Stefan J. and Gommers, Ralf and Virtanen, Pauli and Cournapeau, David and Wieser, Eric and Taylor, Julian and Berg, Sebastian and Smith, Nathaniel J. and others , title =. Nature , volume =. 2020 , doi =
2020
-
[55]
Proceedings of the 9th Python in Science Conference , pages =
McKinney, Wes , title =. Proceedings of the 9th Python in Science Conference , pages =. 2010 , doi =
2010
-
[56]
, title =
Hunter, John D. , title =. Computing in Science & Engineering , volume =. 2007 , doi =
2007
-
[57]
and Haberland, Matt and Reddy, Tyler and Cournapeau, David and Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and Bright, Jonathan and others , title =
Virtanen, Pauli and Gommers, Ralf and Oliphant, Travis E. and Haberland, Matt and Reddy, Tyler and Cournapeau, David and Burovski, Evgeni and Peterson, Pearu and Weckesser, Warren and Bright, Jonathan and others , title =. Nature Methods , volume =. 2020 , doi =
2020
-
[58]
Proceedings of the 9th Python in Science Conference , pages =
Seabold, Skipper and Perktold, Josef , title =. Proceedings of the 9th Python in Science Conference , pages =. 2010 , doi =
2010
-
[59]
2015 , eprint=
Software Use in Astronomy: an Informal Survey , author=. 2015 , eprint=
2015
-
[60]
2026 , url =
Joshi and Croft , title =. 2026 , url =
2026
-
[61]
2026 , publisher =
Joshi and Croft , title =. 2026 , publisher =
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.