Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan

Alexander Chulzhanov; Arjun Mukherjee; Soeren Eberhardt

REVIEW 3 major objections 2 minor 10 references

Synthetic data from dictionaries teaches Q'eqchi' morphology and VOS order to an NMT model yet leaves a lexical gap on real sentences.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 16:18 UTC pith:TU6F6LJ4

load-bearing objection Synthetic dictionary data plus LoRA gets high in-domain BLEU on Q'eqchi' but near-zero on organic text, with the in-domain gains likely reflecting template overlap rather than productive morphology learning. the 3 major comments →

arxiv 2606.09767 v1 pith:TU6F6LJ4 submitted 2026-06-08 cs.CL cs.AIcs.LG

Data Synthesis and Parameter-Efficient Fine-Tuning for Low-Resource NMT: A Case Study on Q'eqchi' Mayan

Alexander Chulzhanov , Soeren Eberhardt , Arjun Mukherjee This is my paper

classification cs.CL cs.AIcs.LG

keywords Q'eqchi' Mayanlow-resource NMTdata synthesisLoRA fine-tuningagglutinative morphologysynthetic corpusstructural-semantic gap

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that generating parallel sentence pairs from community dictionaries and fine-tuning mT5 with LoRA adapters produces strong in-domain performance on an agglutinative language. This shows the synthetic templates successfully encode complex morphology and verb-object-subject syntax. When the same model is tested on an organic glossary, however, performance collapses because the outputs remain grammatically well-formed but fail to match natural lexical choices. The authors therefore conclude that synthetic bootstrapping supplies an effective structural foundation but cannot supply the semantic flexibility of authentic language without additional real data.

Core claim

Transforming community-sourced dictionaries into a massive synthetic corpus and applying LoRA adapters to mT5-base yields BLEU 42.02 on in-domain synthetic test data, confirming that constrained templates teach agglutinative morphology and VOS word order. Evaluation on an organic glossary drops to BLEU 0.59, exposing a structural-semantic gap in which the model preserves grammatical integrity but lacks lexical grounding. An ablation with multi-task learning produces negative transfer, indicating that auxiliary tasks over-optimize the adapters for synthetic markers at the expense of flexibility on natural inputs.

What carries the argument

Dictionary-derived synthetic corpus generated via constrained templates, paired with LoRA adapters on mT5-base for parameter-efficient adaptation.

Load-bearing premise

The constrained structural patterns in the synthetic templates are broad enough to let the model generalize to the syntactic fluidity and lexical variety of natural Q'eqchi' sentences.

What would settle it

Evaluating the trained model on a set of naturally occurring Q'eqchi' sentences with independent English translations; if BLEU scores stay near 0.59 even after further increases in template diversity, the structural-semantic gap cannot be closed by synthesis alone.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Synthetic templates alone suffice to teach complex morphology and VOS order when test data matches the training distribution.
Performance measured only on synthetic data overestimates readiness for real-language use.
Multi-task learning on top of LoRA adapters causes negative transfer by competing for limited adapter capacity.
Authentic parallel data is required after synthetic priming to refine semantic mappings via curriculum learning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The pipeline supports data-sovereignty goals by avoiding any web scraping of target-language text.
The same dictionary-to-synthetic approach could be tested on other Mayan or agglutinative languages to determine whether the observed gap is language-specific.
Increasing the diversity of generation templates might reduce the rigidity that forces organic inputs into learned patterns.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Synthetic dictionary data plus LoRA gets high in-domain BLEU on Q'eqchi' but near-zero on organic text, with the in-domain gains likely reflecting template overlap rather than productive morphology learning.

read the letter

The one or two things to know: this case study on Q'eqchi' finds that synthetic data from dictionaries plus LoRA fine-tuning produces high in-domain BLEU but almost none on organic text, and multi-task learning leads to negative transfer instead of improvement.

They convert community dictionaries into synthetic parallel sentences via templates, then apply parameter-efficient tuning on mT5. The in-domain result of 42 BLEU is taken as evidence that the constraints teach agglutinative morphology and VOS order. The organic glossary evaluation at 0.59 BLEU shows the model sticks to grammatical patterns but doesn't pick up natural lexical grounding. They also ran an ablation with multi-task learning that showed negative transfer, which they link to the adapters over-optimizing for synthetic markers.

What stands out is the focus on data sovereignty and avoiding extractive scraping for an Indigenous language. Reporting the specific gap between structural and semantic performance, along with the negative MTL outcome, adds a concrete observation to the literature on synthetic data for low-resource NMT.

The soft spots are around the evaluation design. The abstract gives no information on how the templates were generated or what the data splits look like, so it's possible the in-domain test set overlaps heavily with training patterns. That makes the claim of learning morphology rest on shaky ground, as the stress-test note suggests. If the test data doesn't include novel combinations, the high score could be memorization rather than generalization. The paper would benefit from more controls there.

This is aimed at people building translation tools for similar low-resource Indigenous languages. A reader working on synthetic data strategies or PEFT for NMT would find the ablation and the reported gap useful to think about. It deserves serious referee time because the application is timely and the negative result on MTL is worth documenting, even if the methods need fleshing out.

I'd recommend sending it for peer review, with the expectation that reviewers will ask for details on template construction and additional tests for productive rule learning.

Referee Report

3 major / 2 minor

Summary. The paper introduces a data synthesis methodology for low-resource NMT on Q'eqchi' Mayan by transforming community-sourced dictionaries into a synthetic corpus and applying LoRA-based PEFT on mT5-base. It claims that this approach achieves high structural acquisition with an in-domain BLEU of 42.02, demonstrating effective teaching of agglutinative morphology and VOS word order, but reveals a structural-semantic gap with BLEU 0.59 on organic glossary data due to overfitting to synthetic template constraints. An ablation with multi-task learning shows negative transfer.

Significance. Should the central claims hold after addressing evaluation concerns, this work would contribute to data-sovereign methods for Indigenous language NMT by showing synthetic data's utility as a structural primer. The contrast between in-domain and out-of-domain performance underscores the limitations of template-based synthesis for semantic grounding. The efficient use of LoRA adapters is noted as a practical strength for low-resource settings.

major comments (3)

[Abstract] The claim that 'synthetic constraints effectively teach complex agglutinative morphology and VOS word order' based on the in-domain BLEU score of 42.02 is undermined by the lack of evidence that the test set includes novel morphological combinations or syntactic variants absent from the training templates; high scores may result from memorization of fixed dictionary-derived templates rather than rule acquisition.
[Abstract] The structural-semantic gap interpretation of the BLEU 0.59 on the organic glossary lacks controls or details on template generation rules, data splits, baseline comparisons, or statistical significance, making it unclear whether the gap stems from lexical mismatch, domain shift, or true failure to internalize productive rules.
[Ablation study] The ablation study reporting negative transfer in the Multi-Task Learning architecture does not provide specifics on the auxiliary tasks, how they lead to competition for LoRA capacity, or quantitative metrics supporting the over-optimization conclusion.

minor comments (2)

The abstract refers to a 'massive synthetic corpus' without reporting its size, the number of templates, or the exact rules used for generation, which would aid in assessing the constrained structural variance.
More information on the organic glossary evaluation setup, including how it differs from the synthetic data in terms of vocabulary and syntax, would clarify the nature of the observed gap.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point-by-point below, agreeing where revisions are needed to strengthen the claims and providing the strongest honest defense based on the work presented.

read point-by-point responses

Referee: [Abstract] The claim that 'synthetic constraints effectively teach complex agglutinative morphology and VOS word order' based on the in-domain BLEU score of 42.02 is undermined by the lack of evidence that the test set includes novel morphological combinations or syntactic variants absent from the training templates; high scores may result from memorization of fixed dictionary-derived templates rather than rule acquisition.

Authors: We acknowledge that the in-domain BLEU of 42.02 alone does not conclusively demonstrate acquisition of productive rules versus memorization, as the test set composition relative to training templates was not explicitly analyzed for novelty. The synthetic data generation applies rule-based transformations from the dictionary that combinatorially vary morphology and syntax, but without a breakdown of unseen combinations, the interpretation remains suggestive rather than definitive. We will revise the abstract and add a dedicated analysis quantifying novel morphological and syntactic variants in the test set, reporting performance stratified by novelty to better support the claim. revision: yes
Referee: [Abstract] The structural-semantic gap interpretation of the BLEU 0.59 on the organic glossary lacks controls or details on template generation rules, data splits, baseline comparisons, or statistical significance, making it unclear whether the gap stems from lexical mismatch, domain shift, or true failure to internalize productive rules.

Authors: The manuscript interprets the low out-of-domain BLEU as evidence of a structural-semantic gap due to template overfitting, consistent with the observed maintenance of grammatical integrity but poor lexical grounding. However, we agree that without the requested controls, alternative explanations cannot be ruled out. We will revise to add explicit details on template generation rules, data split methodology, baseline model comparisons (e.g., untuned mT5), and statistical significance testing of the BLEU difference to clarify the gap's source. revision: yes
Referee: [Ablation study] The ablation study reporting negative transfer in the Multi-Task Learning architecture does not provide specifics on the auxiliary tasks, how they lead to competition for LoRA capacity, or quantitative metrics supporting the over-optimization conclusion.

Authors: The ablation demonstrates negative transfer, which we attribute to auxiliary tasks competing for limited LoRA capacity and leading to over-optimization on synthetic markers. We agree the section lacks the requested specifics. We will expand it to detail the auxiliary tasks, describe the capacity competition mechanism (e.g., via shared adapter updates), and report quantitative metrics such as per-task BLEU deltas and indicators of overfitting to support the conclusion. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical evaluations are direct metrics

full rationale

The paper reports direct BLEU scores from fine-tuning mT5 with LoRA on a synthetic corpus generated from dictionary templates, then evaluates on in-domain synthetic splits and an organic glossary. No equations, parameter fits presented as predictions, self-citations, or uniqueness theorems appear in the provided text. The high in-domain BLEU is a straightforward test-set metric rather than a quantity derived from the training objective by construction, and the structural-semantic gap discussion is interpretive rather than a load-bearing derivation. The central claims rest on external evaluation benchmarks, making the work self-contained against the defined circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The central claim rests on the unstated assumption that the synthetic template generation process produces a representative structural distribution.

pith-pipeline@v0.9.1-grok · 5799 in / 1288 out tokens · 25467 ms · 2026-06-27T16:18:01.434579+00:00 · methodology

0 comments

read the original abstract

Neural machine translation for digitally low-resource Indigenous languages is often hindered by extreme data scarcity, prompting reliance on extractive web-scraping. To ensure data sovereignty, this study introduces a data synthesis methodology to bootstrap NMT models without scraping target-language parallel text. Focusing on Q'eqchi' Mayan, we transformed community-sourced dictionaries into a massive synthetic corpus, utilizing Parameter-Efficient Fine-Tuning (PEFT) via LoRA adapters on an mT5-base model. In-domain evaluation demonstrates high structural acquisition (BLEU 42.02), proving that synthetic constraints effectively teach complex agglutinative morphology and VOS word order. However, evaluation against an organic glossary reveals a structural-semantic gap (BLEU 0.59), where the model maintains grammatical integrity but lacks the lexical grounding of natural language. The model exhibits overfitting to the constrained structural variance of the synthetic templates; despite high semantic entropy in the pipeline, it struggles with the syntactic fluidity of natural language, forcing organic inputs into rigid learned patterns. Furthermore, an ablation study utilizing a Multi-Task Learning architecture resulted in negative transfer, suggesting that auxiliary tasks competed for limited parameter capacity within the LoRA adapters, causing over-optimization for synthetic markers at the expense of organic flexibility. Ultimately, we establish that synthetic bootstrapping is a highly effective structural primer, but requires authentic data for semantic refinement via Curriculum Learning.

Figures

Figures reproduced from arXiv: 2606.09767 by Alexander Chulzhanov, Arjun Mukherjee, Soeren Eberhardt.

**Figure 1.** Figure 1: High-level overview of the synthetic sentence generator program written in Python. enriched with morphological metadata (e.g., transitivity, possession classes). These terms dynamically populate placeholders within a registry of syntactic templates. This decoupling of lexical content from structural patterns enables a single template to generate thousands of grammatically correct sentence permutations whi… view at source ↗

**Figure 2.** Figure 2: STL training dynamics showing rapid structural convergence. 0 5000 10000 15000 20000 25000 30000 Global Steps 0 2 4 6 8 10 12 14 16 Loss Raw Train Loss Smoothed Train Loss Eval Loss 0 5000 10000 15000 20000 25000 30000 Global Steps 10 1 10 2 10 3 10 4 10 5 Gradient Norm (Log Scale) Raw Grad Norm Smoothed Grad Norm Median Norm (7.13) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

10 extracted references · 4 canonical work pages

[1]

2980–2988 (2017)

Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection.In:ProceedingsoftheIEEEInternationalConferenceonComputerVision (ICCV), pp. 2980–2988 (2017)

2017
[2]

arXiv preprint arXiv:2010.11934 (2020) Data Synthesis and PEFT for Q’eqchi’ NMT 13

Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., Raffel, C.: mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 (2020) Data Synthesis and PEFT for Q’eqchi’ NMT 13

work page arXiv 2010
[3]

Scaling neural machine translation to 200 languages

NLLB Team.: Scaling neural machine translation to 200 languages. Nature 630, 841–846 (2024). https://doi.org/10.1038/s41586-024-07335-x

work page doi:10.1038/s41586-024-07335-x 2024
[4]

In: Proceedings of the Joint 25th Nordic Conference on Computa- tional Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pp

Scalvini, B., Debess, I.N., Simonsen, A., Einarsson, H.: Rethinking Low-Resource MT: The Surprising Effectiveness of Fine-Tuned Multilingual Models in the LLM Age. In: Proceedings of the Joint 25th Nordic Conference on Computa- tional Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pp. 609–621. University ...

2025
[5]

In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp

de Gibert, O., Attieh, J., Vahtola, T., Aulamo, M., Li, Z., Vázquez, R., Hu, T., Tiedemann, J.: Scaling Low-Resource MT via Synthetic Data Generation with LLMs. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 27674–27692. Association for Computational Linguistics, Suzhou, China (2025). https://aclanthology.o...

2025
[6]

In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

Zhang, X., Mao, R., Cambria, E.: SenticVec: Toward Robust and Human- Centric Neurosymbolic Sentiment Analysis. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). Association for Computational Linguistics, Bangkok, Thailand (2024). https://aclanthology.org/2024.findings-acl.289/

2024
[7]

arXiv preprint arXiv:2510.13854 (2025)

Keita, M.K., Homan, C., Diarra, S.: R2T: Rule-Encoded Loss Functions for Low-Resource Sequence Tagging. arXiv preprint arXiv:2510.13854 (2025). https://doi.org/10.48550/arXiv.2510.13854

work page doi:10.48550/arxiv.2510.13854 2025
[8]

Mayan Languages Preser- vation Project

Eberhardt, S., Scott, W.K.: Mayan Languages Glossary. Mayan Languages Preser- vation Project. https://mayanlanguages.wiki (2026)

2026
[9]

In: Proceedings of the Third Conference on Machine Translation: Research Papers, pp

Post, M.: A call for clarity in reporting BLEU scores. In: Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186–191 (2018). https://aclanthology.org/W18-6419/

2018
[10]

and Nivre, Joakim and Zeman, Daniel , year = 2021, month = may, journal =

de Marneffe, M.C., Manning, C.D., Nivre, J., Zeman, D.: Univer- sal Dependencies. Computational Linguistics 47(2), 255–264 (2021). https://doi.org/10.1162/coli_a_00402

work page doi:10.1162/coli_a_00402 2021

[1] [1]

2980–2988 (2017)

Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection.In:ProceedingsoftheIEEEInternationalConferenceonComputerVision (ICCV), pp. 2980–2988 (2017)

2017

[2] [2]

arXiv preprint arXiv:2010.11934 (2020) Data Synthesis and PEFT for Q’eqchi’ NMT 13

Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., Raffel, C.: mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 (2020) Data Synthesis and PEFT for Q’eqchi’ NMT 13

work page arXiv 2010

[3] [3]

Scaling neural machine translation to 200 languages

NLLB Team.: Scaling neural machine translation to 200 languages. Nature 630, 841–846 (2024). https://doi.org/10.1038/s41586-024-07335-x

work page doi:10.1038/s41586-024-07335-x 2024

[4] [4]

In: Proceedings of the Joint 25th Nordic Conference on Computa- tional Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pp

Scalvini, B., Debess, I.N., Simonsen, A., Einarsson, H.: Rethinking Low-Resource MT: The Surprising Effectiveness of Fine-Tuned Multilingual Models in the LLM Age. In: Proceedings of the Joint 25th Nordic Conference on Computa- tional Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pp. 609–621. University ...

2025

[5] [5]

In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp

de Gibert, O., Attieh, J., Vahtola, T., Aulamo, M., Li, Z., Vázquez, R., Hu, T., Tiedemann, J.: Scaling Low-Resource MT via Synthetic Data Generation with LLMs. In: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 27674–27692. Association for Computational Linguistics, Suzhou, China (2025). https://aclanthology.o...

2025

[6] [6]

In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

Zhang, X., Mao, R., Cambria, E.: SenticVec: Toward Robust and Human- Centric Neurosymbolic Sentiment Analysis. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024). Association for Computational Linguistics, Bangkok, Thailand (2024). https://aclanthology.org/2024.findings-acl.289/

2024

[7] [7]

arXiv preprint arXiv:2510.13854 (2025)

Keita, M.K., Homan, C., Diarra, S.: R2T: Rule-Encoded Loss Functions for Low-Resource Sequence Tagging. arXiv preprint arXiv:2510.13854 (2025). https://doi.org/10.48550/arXiv.2510.13854

work page doi:10.48550/arxiv.2510.13854 2025

[8] [8]

Mayan Languages Preser- vation Project

Eberhardt, S., Scott, W.K.: Mayan Languages Glossary. Mayan Languages Preser- vation Project. https://mayanlanguages.wiki (2026)

2026

[9] [9]

In: Proceedings of the Third Conference on Machine Translation: Research Papers, pp

Post, M.: A call for clarity in reporting BLEU scores. In: Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186–191 (2018). https://aclanthology.org/W18-6419/

2018

[10] [10]

and Nivre, Joakim and Zeman, Daniel , year = 2021, month = may, journal =

de Marneffe, M.C., Manning, C.D., Nivre, J., Zeman, D.: Univer- sal Dependencies. Computational Linguistics 47(2), 255–264 (2021). https://doi.org/10.1162/coli_a_00402

work page doi:10.1162/coli_a_00402 2021