Pith. sign in

REVIEW 1 major objections 42 references

Real-world dirty postal addresses with ground truth expose limits of data cleaning methods.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-07-02 16:52 UTC pith:NYXNHGUV

load-bearing objection The paper releases a sizable new dirty postal address dataset with ground truth and shows existing cleaners struggle on it, but the errors are too address-specific to reliably benchmark general tabular cleaning methods. the 1 major comments →

arxiv 2606.31983 v2 pith:NYXNHGUV submitted 2026-06-30 cs.DB

Clean Me If You Can: A Large Collection of Real-World Addresses for Data Cleaning Benchmarking

Fatemeh Ahmadi , Tobias Bernhard , Mohamed Abdelmaksoud , Luca Zecchini , Tilmann Rabl , Ziawasch Abedjan This is my paper
classification cs.DB
keywords data cleaningbenchmarkingreal-world datasetpostal addressesdirty dataground truthtabular data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper collects a large dataset of real postal addresses containing errors along with their clean ground truth versions. This fills a gap where data cleaning research has lacked authentic test cases beyond controlled or synthetic data. The authors detail the challenges in gathering such data and apply existing cleaning methods to show they encounter significant difficulties. From these results, they extract guidelines to guide future development of cleaning techniques.

Core claim

We address this gap by providing a large, dirty dataset with postal entries and their corresponding ground truth. We discuss the design decisions and challenges for obtaining the dataset. We demonstrate the limitations of existing cleaning approaches when faced with our proposed datasets and derive guidelines for future research.

What carries the argument

Large collection of real-world postal address data with ground truth for benchmarking data cleaning.

Load-bearing premise

The collected postal data is representative enough of real-world dirty tabular data to expose meaningful limitations in existing cleaning approaches.

What would settle it

Existing cleaning approaches performing as well on this dataset as on prior benchmarks would indicate the dataset does not reveal new limitations.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Data cleaning methods require testing on real-world dirty data to assess true effectiveness.
  • Current approaches face challenges with the error types present in postal address entries.
  • Future research should focus on developing methods that handle realistic data distributions.
  • Guidelines from the benchmark can inform the design of more robust cleaning systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar large-scale ground truth collections could be created for other data types to advance benchmarking in the field.
  • The dataset enables reproducible experiments that compare cleaning performance across different tools on identical real data.
  • Insights from address cleaning limitations may apply to other structured text data like names or descriptions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The manuscript claims to address the lack of real-world dirty tabular datasets for data cleaning research by releasing a large collection of postal address entries paired with ground truth. It discusses design decisions and challenges in dataset collection, demonstrates limitations of existing cleaning approaches on the proposed data, and derives guidelines for future research.

Significance. If the dataset's error distributions are shown to be representative of broader tabular cleaning challenges, the release would be a valuable contribution by enabling realistic benchmarking in a field that currently relies on controlled or synthetic data. The explicit evaluation of existing methods' shortcomings and the resulting guidelines are constructive. The provision of a large-scale real-world dataset with ground truth supports reproducibility and is a clear strength.

major comments (1)
  1. [Abstract] Abstract: The central motivation—that the postal dataset exposes meaningful limitations in existing (general-purpose) cleaning approaches—depends on the representativeness of postal-specific error modes (formatting inconsistencies, zip mismatches) for general tabular issues such as cross-column dependencies or numerical outliers. The design decisions discussion must provide concrete evidence or analysis that the collected dirtiness is not narrowly address-centric; without this, the claim that the dataset addresses the documented gap in real-world tabular data remains insecure.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract and motivation. We agree that strengthening the discussion of representativeness is valuable and will revise the manuscript to include additional analysis.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central motivation—that the postal dataset exposes meaningful limitations in existing (general-purpose) cleaning approaches—depends on the representativeness of postal-specific error modes (formatting inconsistencies, zip mismatches) for general tabular issues such as cross-column dependencies or numerical outliers. The design decisions discussion must provide concrete evidence or analysis that the collected dirtiness is not narrowly address-centric; without this, the claim that the dataset addresses the documented gap in real-world tabular data remains insecure.

    Authors: We acknowledge the referee's point that postal address errors have domain-specific characteristics. However, the dataset does capture several error types relevant to general tabular cleaning, including cross-column dependencies (e.g., inconsistencies between street, city, and zip fields that violate real-world constraints), formatting variations, abbreviations, and missing values. These arise from the real-world collection process described in the design decisions section. To make the claim more secure, we will revise the manuscript to add a dedicated subsection with quantitative breakdown of error categories, their frequencies, and explicit mapping to common tabular issues (such as dependency violations), along with examples showing overlap beyond purely address-centric problems. This will be supported by statistics from the released dataset. revision: yes

Circularity Check

0 steps flagged

Dataset release paper exhibits no circularity

full rationale

The paper contributes a new real-world dirty postal address dataset with ground truth labels. Its claims rest on data collection, design decisions, and empirical evaluation of existing cleaners on this data; no equations, fitted parameters, predictions, or derivations are present that could reduce to inputs by construction. No self-citation chains, uniqueness theorems, or ansatzes are invoked as load-bearing steps. The representativeness assumption is an external validity concern, not a circularity issue.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Dataset paper with no mathematical content; no free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5641 in / 842 out tokens · 23551 ms · 2026-07-02T16:52:54.282341+00:00 · methodology

0 comments
read the original abstract

There has been extensive research on automating and scaling data cleaning, i.e., the detection and correction of erroneous values in tabular data. Yet, existing approaches often perform well only within controlled environments. One of the major bottlenecks in data cleaning research is the lack of real-world datasets. In this paper, we address this gap by providing a large, dirty dataset with postal entries and their corresponding ground truth. We discuss the design decisions and challenges for obtaining the dataset. We demonstrate the limitations of existing cleaning approaches when faced with our proposed datasets and derive guidelines for future research.

Figures

Figures reproduced from arXiv: 2606.31983 by Fatemeh Ahmadi, Luca Zecchini, Mohamed Abdelmaksoud, Tilmann Rabl, Tobias Bernhard, Ziawasch Abedjan.

Figure 1
Figure 1. Figure 1: Schema alignment between extracted addresses [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Country distribution in the full-named dataset. 4.3 Ethical Concerns In alignment with prior work that analyzed the Common Crawl archives [36], we refrained from including any explicit personally identifiable information (PII), such as SSNs, emails, phone numbers, and banking information. After extraction, we also used the under￾lying PII detection library14 to identify unintentionally captured PII. No PII… view at source ↗
Figure 3
Figure 3. Figure 3: Percentages of clean and erroneous cells (grouped [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: F1-score of different data cleaning systems for error detection and correction on subsets of [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 32 canonical work pages · 1 internal anchor

  1. [1]

    Mohamed Abdelaal, Christian Hammacher, and Harald Schöning. 2023. REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines. InProceedings of the International Conference on Extending Database Technology (EDBT). 499–511. https://doi.org/10.48786/edbt.2023.43

    work page doi:10.48786/edbt.2023.43 2023

  2. [2]

    Akcora, Mourad Ouzzani, Paolo Papotti, and Michael Stonebraker

    Ziawasch Abedjan, Cuneyt G. Akcora, Mourad Ouzzani, Paolo Papotti, and Michael Stonebraker. 2015. Temporal Rules Discovery for Web Data Cleaning. Proceedings of the VLDB Endowment (PVLDB)9, 4 (2015), 336–347. https://doi. org/10.14778/2856318.2856328

    work page doi:10.14778/2856318.2856328 2015

  3. [3]

    Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang

    Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang. 2016. De- tecting Data Errors: Where are we and what needs to be done?Proceedings of 12 the VLDB Endowment (PVLDB)9, 12 (2016), 993–1004. https://doi.org/10.14778/ 2994509.2994518

    work page arXiv 2016

  4. [4]

    Arocena, Boris Glavic, Giansalvatore Mecca, Renée J

    Patricia C. Arocena, Boris Glavic, Giansalvatore Mecca, Renée J. Miller, Paolo Papotti, and Donatello Santoro. 2015. Messing Up with BART: Error Generation for Evaluating Data-Cleaning Algorithms.Proceedings of the VLDB Endowment (PVLDB)9, 2 (2015), 36–47. https://doi.org/10.14778/2850578.2850579

    work page doi:10.14778/2850578.2850579 2015

  5. [5]

    Christopher Barrington-Leigh and Adam Millard-Ball. 2017. The World’s User- Generated Road Map Is More Than 80% Complete.PLOS ONE12, 8 (2017), e0180698. https://doi.org/10.1371/journal.pone.0180698

    work page doi:10.1371/journal.pone.0180698 2017

  6. [6]

    Divya Bhadauria, Hazar Harmouch, Felix Naumann, Divesh Srivastava, and Lisa Ehrlinger. 2026. A Catalog of Data Errors.CoRRabs/2604.09277 (2026). https://doi.org/10.48550/ARXIV.2604.09277 arXiv:2604.09277

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.09277 2026

  7. [7]

    Felix Bießmann, Tammo Rukat, Philipp Schmidt, Prathik Naidu, Sebastian Schel- ter, Andrey Taptunov, Dustin Lange, and David Salinas. 2019. DataWig: Missing Value Imputation for Tables.Journal of Machine Learning Research (JMLR)20, Article 175 (2019), 6 pages. https://jmlr.org/papers/v20/18-753.html

    2019

  8. [8]

    Alexander Brinkmann, Anna Primpeli, and Christian Bizer. 2023. The Web Data Commons Schema.org Data Set Series. InCompanion Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023, Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben (Eds.). ACM, 136–139. https://doi.org/10.1145/...

    work page doi:10.1145/3543873.3587331 2023

  9. [9]

    Ilyas, and Paolo Papotti

    Xu Chu, Ihab F. Ilyas, and Paolo Papotti. 2013. Holistic Data Cleaning: Putting Violations Into Context. InProceedings of the IEEE International Conference on Data Engineering (ICDE). 458–469. https://doi.org/10.1109/ICDE.2013.6544847

    work page doi:10.1109/icde.2013.6544847 2013

  10. [10]

    Fred Damerau. 1964. A Technique for Computer Detection and Correction of Spelling Errors.Communications of the ACM (CACM)7, 3 (1964), 171–176. https://doi.org/10.1145/363958.363994

    work page doi:10.1145/363958.363994 1964

  11. [11]

    Edith Desiree de Leeuw. 1992. Data Quality in Mail, Telephone and Face to Face Surveys. https://eric.ed.gov/?id=ED374136

    1992

  12. [12]

    Xiaoou Ding, Zekai Qian, Hongzhi Wang, Siying Chen, Yafeng Tang, Hongbin Su, Huan Hu, and Chen Wang. 2025. UniClean: A Scalable Data Cleaning Solution for Mixed Errors based on Unified Cleaners and Optimized Cleaning Workflow.Proceedings of the VLDB Endowment (PVLDB)18, 11 (2025), 4117–4130. https://doi.org/10.14778/3749646.3749681

    work page doi:10.14778/3749646.3749681 2025

  13. [13]

    Anna-Christina Glock, Christine Dominka-Kiss, Philipp Korom, and Lisa Ehrlinger. 2025. Detecting and Cleaning Errors in Personal Contact Information with Large Language Models.Proceedings of the VLDB Endowment (PVLDB) (2025)

    2025

  14. [14]

    Jean-Nicholas Hould. 2017. Craft Beers Dataset. https://www.kaggle.com/ nickhould/craft-cans. Version 1

    2017

  15. [15]

    Ykä Huhtala, Juha Kärkkäinen, Pasi Porkka, and Hannu Toivonen. 1999. TANE: An Efficient Algorithm for Discovering Functional and Approximate Dependen- cies.Comput. J.42, 2 (1999), 100–111. https://doi.org/10.1093/COMJNL/42.2.100

    work page doi:10.1093/comjnl/42.2.100 1999

  16. [16]

    Philipp Jung, Sebastian Jäger, Nicholas Chandler, and Felix Biessmann. 2025. Towards Realistic Error Models for Tabular Data.ACM J. Data Inf. Qual.17, 4 (2025), 28:1–28:27. https://doi.org/10.1145/3774914

    work page doi:10.1145/3774914 2025

  17. [17]

    Bojan Karlas, Peng Li, Renzhi Wu, Nezihe Merve Gürel, Xu Chu, Wentao Wu, and Ce Zhang. 2020. Nearest Neighbor Classifiers over Incomplete Information: From Certain Answers to Certain Predictions.Proceedings of the VLDB Endowment (PVLDB)14, 3 (2020), 255–267. https://doi.org/10.14778/3430915.3430917

    work page doi:10.14778/3430915.3430917 2020

  18. [18]

    Levenshtein

    Vladimir I. Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions, and Reversals.Soviet Physics Doklady10, 8 (1966), 707–710

    1966

  19. [19]

    Peng Li, Xi Rao, Jennifer Blase, Yue Zhang, Xu Chu, and Ce Zhang. 2021. CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks. InProceedings of the IEEE International Conference on Data Engineering (ICDE). https://doi.org/10.1109/ICDE51399.2021.00009

    work page doi:10.1109/icde51399.2021.00009 2021

  20. [20]

    Xian Li, Xin Luna Dong, Kenneth Lyons, Weiyi Meng, and Divesh Srivastava

  21. [21]

    https://doi.org/10.14778/2535568

    Truth Finding on the Deep Web: Is the Problem Solved?Proceedings of the VLDB Endowment (PVLDB)6, 2 (2012), 97–108. https://doi.org/10.14778/2535568. 2448943

    work page doi:10.14778/2535568 2012

  22. [22]

    Yi Li and Gao Cong. 2025. GeoBloom: Revisiting Lightweight Models for Geo- graphic Information Retrieval.Proceedings of the VLDB Endowment (PVLDB)18, 5 (2025), 1348–1361. https://doi.org/10.14778/3718057.3718064

    work page doi:10.14778/3718057.3718064 2025

  23. [23]

    Yiding Liu, Tuan-Anh Nguyen Pham, Gao Cong, and Quan Yuan. 2017. An Experimental Evaluation of Point-of-interest Recommendation in Location-based Social Networks.Proceedings of the VLDB Endowment (PVLDB)10, 10 (2017), 1010–1021. https://doi.org/10.14778/3115404.3115407

    work page doi:10.14778/3115404.3115407 2017

  24. [24]

    Mohammad Mahdavi and Ziawasch Abedjan. 2020. Baran: Effective Error Correction via a Unified Context Representation and Transfer Learning.Pro- ceedings of the VLDB Endowment (PVLDB)13, 11 (2020), 1948–1961. https: //doi.org/10.14778/3407790.3407801

    work page doi:10.14778/3407790.3407801 2020

  25. [25]

    Mohammad Mahdavi, Ziawasch Abedjan, Raul Castro Fernandez, Samuel Mad- den, Mourad Ouzzani, Michael Stonebraker, and Nan Tang. 2019. Raha: A Configuration-Free Error Detection System. InProceedings of the ACM In- ternational Conference on Management of Data (SIGMOD). 865–882. https: //doi.org/10.1145/3299869.3324956

    work page doi:10.1145/3299869.3324956 2019

  26. [26]

    Sedir Mohammed, Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Sina Noack, Hendrik Patzlaff, Felix Naumann, and Hazar Har- mouch. 2025. The effects of data quality on machine learning performance on tabular data.Information Systems132, Article 102549 (2025), 18 pages. https://doi.org/10.1016/j.is.2025.102549

    work page doi:10.1016/j.is.2025.102549 2025

  27. [27]

    Sedir Mohammed, Lisa Ehrlinger, Hazar Harmouch, Felix Naumann, and Divesh Srivastava. 2025. The Five Facets of Data Quality Assessment.ACM SIGMOD Record54, 2 (2025), 18–27. https://doi.org/10.1145/3749116.3749120

    work page doi:10.1145/3749116.3749120 2025

  28. [28]

    Wei Ni, Xiaoye Miao, Xiangyu Zhao, Yangyang Wu, Shuwei Liang, and Jianwei Yin. 2024. Automatic Data Repair: Are We Ready to Deploy?Proceedings of the VLDB Endowment (PVLDB)17, 10 (2024), 2617–2630. https://doi.org/10.14778/ 3675034.3675051

    work page arXiv 2024

  29. [29]

    Wei Ni, Kaihang Zhang, Xiaoye Miao, Xiangyu Zhao, Yangyang Wu, Yaoshu Wang, and Jianwei Yin. 2025. ZeroED: Hybrid Zero-Shot Error Detection Through Large Language Model Reasoning. In41st IEEE International Conference on Data Engineering, ICDE 2025, Hong Kong, May 19-23, 2025. IEEE, 3126–3139. https: //doi.org/10.1109/ICDE65448.2025.00234

    work page doi:10.1109/icde65448.2025.00234 2025

  30. [30]

    Mourad Ouzzani, Hossam Hammady, Zbys Fedorowicz, and Ahmed Elmagarmid

  31. [31]

    https://doi.org/10.1186/s13643-016-0384-4

    Rayyan—a web and mobile app for systematic reviews.Systematic Reviews 5, Article 210 (2016), 10 pages. https://doi.org/10.1186/s13643-016-0384-4

    work page doi:10.1186/s13643-016-0384-4 2016

  32. [32]

    Ilyas, and Christopher Ré

    Theodoros Rekatsinas, Xu Chu, Ihab F. Ilyas, and Christopher Ré. 2017. Holo- Clean: Holistic Data Repairs with Probabilistic Inference.Proceedings of the VLDB Endowment (PVLDB)10, 11 (2017), 1190–1201. https://doi.org/10.14778/ 3137628.3137631

    work page arXiv 2017

  33. [33]

    Valerie Restat, Gerrit Boerner, André Conrad, and Uta Störl. 2022. GouDa - Generation of universal Data Sets: Improving Analysis and Evaluation of Data Preparation Pipelines. InProceedings of the Workshop on Data Management for End-To-End Machine Learning (DEEM). Article 2, 6 pages. https://doi.org/10. 1145/3533028.3533311

    work page arXiv 2022

  34. [34]

    Aref, Ahmed K

    El Kindi Rezig, Mourad Ouzzani, Walid G. Aref, Ahmed K. Elmagarmid, Ahmed R. Mahmood, and Michael Stonebraker. 2021. Horizon: Scalable Dependency-driven Data Cleaning.Proceedings of the VLDB Endowment (PVLDB)14, 11 (2021), 2546–

    2021

  35. [35]

    https://doi.org/10.14778/3476249.3476301

    work page doi:10.14778/3476249.3476301

  36. [36]

    Sebastian Schelter, Tammo Rukat, and Felix Biessmann. 2021. JENGA - A Frame- work to Study the Impact of Data Errors on the Predictions of Machine Learning Models. InProceedings of the International Conference on Extending Database Technology (EDBT). 529–534. https://doi.org/10.5441/002/edbt.2021.63

    work page doi:10.5441/002/edbt.2021.63 2021

  37. [37]

    Shafaq Siddiqi, Roman Kern, and Matthias Boehm. 2023. SAGA: A Scalable Framework for Optimizing Data Cleaning Pipelines for Machine Learning Ap- plications.Proceedings of the ACM on Management of Data (PACMMOD)1, 3, Article 218 (2023), 26 pages. https://doi.org/10.1145/3617338

    work page doi:10.1145/3617338 2023

  38. [38]

    Jasmin Singh and Heiko Gebauer. 2024. Clean Customer Master Data for Cus- tomer Analytics: A Neglected Element of Data Monetization.Digital4, 4 (2024), 1020–1038. https://doi.org/10.3390/digital4040051

    work page doi:10.3390/digital4040051 2024

  39. [39]

    Nishant Subramani, Sasha Luccioni, Jesse Dodge, and Margaret Mitchell. 2023. Detecting Personal Information in Training Corpora: an Analysis. InProceedings of the Workshop on Trustworthy Natural Language Processing (TrustNLP). 208–220. https://doi.org/10.18653/v1/2023.trustnlp-1.18

    work page doi:10.18653/v1/2023.trustnlp-1.18 2023

  40. [40]

    The Unicode Consortium. 2024. The Unicode Standard, Version 15.1. https: //www.unicode.org/versions/Unicode15.1.0/

    2024

  41. [41]

    United States Postal Service. 2014. Undeliverable as Addressed Mail. https: //www.uspsoig.gov/reports/audit-reports/undeliverable-addressed-mail

    2014

  42. [42]

    Yangyang Wu, Chen Yang, Mengying Zhu, Xiaoye Miao, Wei Ni, Meng Xi, Xinkui Zhao, and Jianwei Yin. 2025. A Zero-Training Error Correction System with Large Language Models. InProceedings of the IEEE International Conference on Data Engineering (ICDE). 2949–2962. https://doi.org/10.1109/ICDE65448.2025.00221 13

    work page doi:10.1109/icde65448.2025.00221 2025