Pith. sign in

REVIEW 2 major objections 2 minor 29 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Refined categories of how students use LLMs in paper critique tasks show associations with midterm performance.

2026-06-30 23:40 UTC pith:3V4ICJH2

load-bearing objection This paper refines LLM usage categories from self-reports in a critical thinking course and checks ties to midterm scores across two offerings, but the associations rest on unvalidated reports without confounder controls. the 2 major comments →

arxiv 2605.04534 v2 pith:3V4ICJH2 submitted 2026-05-06 cs.HC

Characterizing Students' LLM Usage Behaviors and Their Association with Learning in Critical Thinking Tasks

Minju Park , Ivan Orozco Vasquez , Cristina Conati This is my paper
classification cs.HC
keywords LLM usage behaviorscritical thinking tasksstudent learningeducational technologyusage categorizationstudent initiativemidterm performanceacademic paper critique
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper analyzes self-reported LLM usage from students in a course on reading and critiquing academic papers, developing a bottom-up set of usage types that are also labeled by how much initiative the student takes. It then checks whether those usage patterns and their frequency relate to scores on three midterms that test critical thinking. A sympathetic reader would care because the setting is an authentic, unrestricted course rather than a lab task, so the results speak to real classroom conditions where LLMs are already common. The work extends an earlier single-offering study by sharpening the usage taxonomy and adding the initiative dimension.

Core claim

LLM usage behaviors, captured through a refined bottom-up categorization of types that is cross-labeled by the extent of student initiative, are associated with student learning as measured by performance on three midterms in a research-oriented course where students read, reason about, and critique academic papers without restrictions on LLM use.

What carries the argument

A refined bottom-up categorization of LLM usage types cross-labeled by the extent of student initiative.

Load-bearing premise

Self-reported LLM usage practices accurately reflect actual student behaviors and that midterm performance validly measures learning gains in critical thinking without substantial confounding from unmeasured factors such as prior ability or course-specific effects.

What would settle it

A replication that logs actual LLM prompts and responses instead of relying on self-reports, controls for incoming student ability, and finds no remaining association between the categorized usage patterns and midterm scores would falsify the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Usage types that involve higher student initiative are linked to stronger midterm performance than lower-initiative types.
  • Frequency of LLM use interacts with usage type in its association with learning outcomes.
  • The same taxonomy and initiative labels can be applied to future offerings of the course to track changes in behavior over time.
  • Unrestricted LLM access during homework does not produce uniform effects on critical-thinking assessments across students.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The initiative-labeled taxonomy could be tested in STEM problem-solving courses to see whether the same associations hold outside humanities-style critique tasks.
  • Instructors could design prompts that nudge students toward higher-initiative usage categories if the observed performance links prove causal.
  • Replacing self-reports with direct logging of LLM sessions would strengthen the evidence by removing one source of measurement error.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper analyzes self-reported LLM usage data collected from students in two offerings of a research-oriented course on reading, reasoning about, and critiquing academic papers. Without usage restrictions, students reported practices on homework assignments. The work refines a prior bottom-up categorization of LLM usage types, cross-labeled by extent of student initiative, and examines associations between usage frequency/types and learning outcomes measured by performance on three midterms.

Significance. If the reported associations prove robust to confounder controls and self-report validation, the refined categorization and initiative labeling could usefully inform pedagogical guidance on LLM integration in critical thinking curricula. The bottom-up empirical approach from real course data is a strength relative to lab-constrained studies.

major comments (2)
  1. [Methods] Methods section: No sample size, number of participants, response rates, or inter-rater reliability statistics are reported for the bottom-up categorization process or its refinement across the two course offerings. Without these, the stability and generalizability of the usage types cannot be evaluated.
  2. [Results] Results section: Associations between categorized LLM usage behaviors and raw midterm scores are presented without statistical controls for confounders such as prior student ability, motivation, or baseline knowledge. Since the outcome is not gain scores and no pre-course measures or objective usage logs are described, unmeasured student-level factors could fully account for the observed links.
minor comments (2)
  1. [Abstract] Abstract: States that the categorization is 'refined' from prior single-offering work but provides no explicit comparison of changes or new categories introduced.
  2. The manuscript would benefit from a table summarizing the final usage types, their initiative labels, and example student reports for each.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on our manuscript. We address each major comment point by point below, indicating revisions where appropriate. The work draws on real classroom data from two course offerings to refine a bottom-up categorization of LLM usage and examine associations with learning outcomes.

read point-by-point responses

  1. Referee: [Methods] Methods section: No sample size, number of participants, response rates, or inter-rater reliability statistics are reported for the bottom-up categorization process or its refinement across the two course offerings. Without these, the stability and generalizability of the usage types cannot be evaluated.

    Authors: We agree that these details are necessary for readers to evaluate the categorization process. The current manuscript describes the bottom-up approach and its refinement but does not report the specific sample sizes, response rates for self-reports, or inter-rater reliability statistics. We will revise the Methods section to include the number of participants per course offering, response rates, and any inter-rater reliability measures (e.g., Cohen's kappa or percentage agreement) used during coding and refinement. This addition will improve transparency without altering the core findings. revision: yes

  2. Referee: [Results] Results section: Associations between categorized LLM usage behaviors and raw midterm scores are presented without statistical controls for confounders such as prior student ability, motivation, or baseline knowledge. Since the outcome is not gain scores and no pre-course measures or objective usage logs are described, unmeasured student-level factors could fully account for the observed links.

    Authors: We acknowledge that the reported associations are based on raw midterm scores without statistical controls for confounders. The dataset consists of self-reported LLM usage and midterm performance collected during the course; no pre-course measures of ability, motivation, or baseline knowledge, nor objective usage logs, were collected. As a result, we cannot introduce such controls. We will revise the Results and Discussion sections to explicitly note this limitation, emphasize the exploratory and correlational nature of the associations, and discuss the possibility that unmeasured student-level factors may explain the observed patterns. This framing aligns with the study's bottom-up, real-world data approach while avoiding overinterpretation. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical data collection and bottom-up categorization.

full rationale

The paper collects self-reported LLM usage data from homework in a research-oriented course, applies a refined bottom-up categorization of usage types (cross-labeled by student initiative), and examines associations with midterm performance across two course offerings. No equations, fitted parameters, predictions, or derivations are present. The extension of prior work on a single offering is noted but does not serve as load-bearing justification for the current claims, which rest on the new data and categorization. No self-definitional steps, fitted inputs called predictions, or uniqueness theorems reduce the analysis to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The study is observational and relies on standard assumptions of educational research such as the validity of self-report and statistical association testing; no free parameters, invented entities, or non-standard axioms are indicated in the abstract.

axioms (1)
  • domain assumption Self-reported usage data can be meaningfully categorized and correlated with performance metrics
    The analysis depends on treating reported practices as reliable inputs for bottom-up categorization and association testing.

pith-pipeline@v0.9.1-grok · 5690 in / 1181 out tokens · 27962 ms · 2026-06-30T23:40:55.584612+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are becoming increasingly embedded in students' learning practices, yet much of what is known about how students use LLMs and how this usage impacts learning comes from problem-solving domains or constrained experimental settings. We present an analysis of data on LLM usage collected during two offerings of a research-oriented course where students learn to read, reason about, and critique academic papers. Without restrictions on whether or how to use LLMs, students reported their LLM usage practices when asked to do these activities as a series of homework assignments during the course. This paper extends prior work done on data from a single offering of the same course by presenting a refined bottom-up categorization of LLM usage types, cross-labeled by the extent of student initiative these usages entail. Furthermore, we examine how LLM use impacts student learning, measured by performance on three midterms, looking at factors such as frequency and type of usage.

Figures

Figures reproduced from arXiv: 2605.04534 by Cristina Conati, Ivan Orozco Vasquez, Minju Park.

Figure 1
Figure 1. Figure 1: LLM usage rates for discussion points and critical view at source ↗
Figure 2
Figure 2. Figure 2: Temporal visualization of reported LLM use across assignments. view at source ↗
Figure 3
Figure 3. Figure 3: Counts for each LLM usage category for discussion view at source ↗
Figure 6
Figure 6. Figure 6: Comparisons of midterm exam scores between stu view at source ↗
Figure 5
Figure 5. Figure 5: Counts for each LLM usage category, with stacked view at source ↗
Figure 8
Figure 8. Figure 8: Comparisons of midterm exam scores between view at source ↗
Figure 7
Figure 7. Figure 7: Comparisons of midterm exam scores between stu view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    INTRODUCTION As large language models (LLMs) become more powerful, so- phisticated, and accessible, the educational community con- tinues to grapple with understanding what roles these mod- els can and should play in education [21, 13, 11]. While some argue that integrating LLMs into coursework offers benefits such as immediate and personalized feedback [...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    RELA TED WORK In this section, we review prior work on students’ use of LLMs in education along two dimensions: (1) educational contexts of student LLM use, with a focus on differences across domains and instructional settings, and (2) approaches to analyzing students’ LLM interactions, including both top- down and bottom-up approaches. 2.1 Educational Co...

  3. [3]

    3.1 Course Structure Our exploratory analysis was conducted in a third-year un- dergraduate course offered at a public North American in- stitution

    DA TA OVERVIEW This section provides an overview of the course in which the data were collected and the composition of the dataset used for analysis. 3.1 Course Structure Our exploratory analysis was conducted in a third-year un- dergraduate course offered at a public North American in- stitution. The course’s objective was to teach students re- search me...

  4. [4]

    Prompt Used: Here is my summary for the attached paper: [student’s draft] Revise it for any grammatical issues

    METHODS To determine whether data from Term 1 and Term 2 could be analyzed jointly, we compared the distributions of stu- dents’ backgrounds across the two terms, focusing on self- reported coding ability and disciplinary background. Al- though the cohorts were not identical, their distributions were similar enough for our purposes. For coding ability, st...

  5. [5]

    RESULTS In this section, we present our findings on how students used LLMs throughout the course and how these usage patterns relate to midterm exam performance, considering both the level of LLM use and differences in LLM usage types. 5.1 Students’ LLM Usage Trends We organize our analysis of students’ engagement with LLM into two parts: (1) whether and ...

  6. [6]

    We found that only a minority of students (23 out of 68; 34%) used LLMs for the course assignments

    DISCUSSION In this section, we summarize our results regarding stu- dents’ LLM usage and its impact on learning, as measured by midterm performance. We found that only a minority of students (23 out of 68; 34%) used LLMs for the course assignments. Among these students, most (16 students) used LLMs lightly, relying on them for approximately 5–33% of their...

  7. [7]

    CONCLUSION This work examined how students used LLMs in a research- oriented course, revealing diverse patterns of engagement and how these behaviors related to performance on an inde- pendently completed midterm exam. Our analyses showed meaningful variation in when students chose to use LLMs, the types of support they sought, and the amount of effort th...

  8. [8]

    This work was also supported in part by the In- stitute for Computing, Information and Cognitive Systems (ICICS) at UBC

    ACKNOWLEDGMENTS This work was funded by the UBC Skylight Grant PM015111 and complies with UBC policies 81 and LR11 regarding own- ership and usage of course material for pedagogical quality assurance. This work was also supported in part by the In- stitute for Computing, Information and Cognitive Systems (ICICS) at UBC

  9. [9]

    Brender, L

    J. Brender, L. El-Hamamsy, F. Mondada, and E. Bumbacher. Who’s Helping Who? When Students Use ChatGPT to Engage in Practice Lab Sessions. In International Conference on Artificial Intelligence in Education, pages 235–249. Springer, 2024

    2024

  10. [10]

    Bull and A

    C. Bull and A. Kharrufa. Generative Artificial Intelligence Assistants in Software Development Education: A Vision for Integrating Generative Artificial Intelligence into Educational Practice, Not Instinctively Defending Against It.IEEE Software, 41(2):52–59, 2023

    2023

  11. [11]

    Flower and J

    L. Flower and J. R. Hayes. A cognitive process theory of writing.College Composition and Communication, 32(4):365–387, 1981

    1981

  12. [12]

    J. Gao, S. A. Gebreegziabher, K. T. W. Choo, T. J.-J. Li, S. T. Perrault, and T. W. Malone. A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration. InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–11, 2024

    2024

  13. [13]

    Ghimire and J

    A. Ghimire and J. Edwards. Coding with AI: How Are Tools Like ChatGPT Being Used by Students in Foundational Programming Courses. InInternational Conference on Artificial Intelligence in Education, pages 259–267. Springer, 2024

    2024

  14. [14]

    L. Giray. Prompt engineering with chatgpt: a guide for academic writers.Annals of biomedical engineering, 51(12):2629–2633, 2023

    2023

  15. [15]

    Grande, N

    V. Grande, N. Kiesler, and M. A. Francisco R. Student perspectives on using a large language model (llm) for an assignment on professional ethics. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, pages 478–484. Association for Computing Machinery, 2024

    2024

  16. [16]

    An Empirical Study to Understand How Students Use ChatGPT for Writing Essays

    A. Jelson, D. Manesh, A. Jang, D. Dunlap, Y.-H. Kim, and S. W. Lee. An empirical study to understand how students use chatgpt for writing essays.arXiv preprint arXiv:2501.10551, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  17. [17]

    Kazemitabaar, X

    M. Kazemitabaar, X. Hou, A. Henley, B. J. Ericson, D. Weintrop, and T. Grossman. How Novices Use LLM-based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment. In Proceedings of the 23rd Koli calling international conference on computing education research, pages 1–12, 2023

    2023

  18. [18]

    J. Kim, S. Yu, S.-S. Lee, and R. Detrick. Students’ prompt patterns and its effects in ai-assisted academic writing: Focusing on students’ level of ai literacy. Journal of Research on Technology in Education, 58(3):638–655, 2026

    2026

  19. [19]

    Ban It Till We Understand It

    S. Lau and P. Guo. From “Ban It Till We Understand It” to “Resistance is Futile”: How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1, pages 106–121, 2023

    2023

  20. [20]

    B. Ma, L. Chen, and S. Konomi.Examining Student-ChatGPT Interactions in Programming Education, pages 97–114. Springer Nature Switzerland, Cham, 2025

    2025

  21. [21]

    MacNeil, A

    S. MacNeil, A. Tran, A. Hellas, J. Kim, S. Sarsa, P. Denny, S. Bernstein, and J. Leinonen. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. InProceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, pages 931–937, 2023

    2023

  22. [22]

    arXiv preprint arXiv:2306.10052 , year=

    E. Mollick and L. Mollick. Assigning AI: Seven Approaches for Students, with Prompts.arXiv preprint arXiv:2306.10052, 2023

    work page arXiv 2023

  23. [23]

    Orozco Vasquez, R

    I. Orozco Vasquez, R. Mahinpei, N. Elouazizi, and C. Conati. An emergent bottom-up categorization of students’ llms usage in an undergraduate research course. InInternational Conference on Artificial Intelligence in Education, pages 133–139. Springer, 2025

    2025

  24. [24]

    Rasnayaka, G

    S. Rasnayaka, G. Wang, R. Shariffdeen, and G. N. Iyer. An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project. In Proceedings of the 1st International Workshop on Large Language Models for Code, pages 111–118, 2024

    2024

  25. [25]

    S. K. K. Santu and D. Feng. TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks.arXiv preprint arXiv:2305.11430, 2023

    work page arXiv 2023

  26. [26]

    Sawalha, I

    G. Sawalha, I. Taj, and A. Shoufan. Analyzing student prompts and their effect on ChatGPT’s performance. Cogent Education, 11(1):2397200, 2024

    2024

  27. [27]

    The Prompt Report: A Systematic Survey of Prompt Engineering Techniques

    S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y. Li, A. Gupta, H. Han, S. Schulhoff, et al. The prompt report: a systematic survey of prompt engineering techniques.arXiv preprint arXiv:2406.06608, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    D. Sun, A. Boudouaia, J. Yang, and J. Xu. Investigating students’ programming behaviors, interaction qualities and perceptions through prompt-based learning in chatgpt.Humanities and Social Sciences Communications, 11(1):1–14, 2024

    2024

  29. [29]

    Vadaparty, D

    A. Vadaparty, D. Zingaro, D. H. Smith IV, M. Padala, C. Alvarado, J. Gorson Benario, and L. Porter. CS1-LLM: Integrating LLMs into CS1 instruction. In Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1, page 297–303. Association for Computing Machinery, 2024

    2024