REVIEW 2 major objections 1 minor
Online LLM simulation lets microservice tests generate dependency responses at runtime, reaching 99 percent status-code and response-shape fidelity where record-replay reaches only 62 and 16 percent.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-05-10 19:39 UTC pith:W5DPPVPK
load-bearing objection MIRAGE gets solid fidelity gains over record-replay by running LLMs online with source code, but the 8/8 integration-test match is too thin to carry the main claim. the 2 major comments →
MIRAGE: Online LLM Simulation for Microservice Dependency Testing
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
MIRAGE demonstrates online LLM simulation in which the model reads the dependency source code, caller code, and traces to answer each incoming request while preserving cross-request state. In white-box mode this produces 99 percent status-code fidelity and 99 percent response-shape fidelity, compared with 62 percent and 16 percent for record-replay. Caller integration tests yield identical pass or fail results with the simulated dependencies as with the real ones in all eight evaluated scenarios.
What carries the argument
Online LLM simulation: the LLM is prompted at runtime with the dependency's source code, caller code, and traces to generate each response on demand while maintaining state across requests in a test.
Load-bearing premise
That an LLM given source code and traces will generate responses that generalize to unseen error-handling inputs without hallucinations or inconsistencies that would change the pass or fail outcome of a test.
What would settle it
A caller integration test that passes against the real microservice dependency but fails when the same test is run against MIRAGE, or vice versa.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents MIRAGE, an approach to microservice dependency testing that uses large language models to simulate dependencies online during test execution. By feeding the LLM with the dependency's source code, caller code, and production traces, it generates responses on demand while maintaining state across multiple requests in a test scenario. This is contrasted with static methods like record-replay that fail on error-handling scenarios. The evaluation across 110 test scenarios in three systems (Online Boutique, Sock Shop, custom) reports 99% status-code and 99% response-shape fidelity in white-box mode versus lower for baselines, an ablation on input components, stability across LLMs, and identical integration test outcomes in 8 out of 8 cases.
Significance. Should the empirical results prove robust, MIRAGE offers a promising direction for improving test coverage in microservice architectures, particularly for scenarios underrepresented in traces. The paper's strengths include a clear ablation study isolating the contribution of source code (100% alone), reported stability across three LLM families (within 3%), and consistent determinism on repeated runs. These elements provide a solid foundation for the claims, though the practical trade-offs in latency (~3 s) and cost are acknowledged.
major comments (2)
- [§4 (Evaluation)] §4 (Evaluation): The reported 99% status-code and 99% response-shape fidelity metrics lack a detailed description of the exact criteria used to determine a 'response-shape match,' how the 110 scenarios were selected, or any error analysis for state inconsistencies across requests, which is load-bearing for the direct comparison to record-replay (62%/16%) and the generalization claim.
- [Integration test results (8/8 scenarios)] Integration test results (8/8 scenarios): The claim that caller integration tests produce identical pass/fail outcomes with MIRAGE as with real dependencies rests on only 8 scenarios; this sample is small relative to the 110 scenarios and does not explicitly validate cross-request state (e.g., tokens or counters) or coverage of underrepresented error/reasoning paths, leaving open the possibility that per-request shape matches mask divergences that would alter test outcomes.
minor comments (1)
- [Abstract] The latency and cost figures (~3 s per request, $0.16-$0.82 per dependency) would benefit from explicit specification of the LLMs, hardware, and request volumes used in the measurements.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the evaluation and integration-test claims. We address each major comment below, providing clarifications and committing to targeted revisions that strengthen transparency without altering the reported results.
read point-by-point responses
-
Referee: [§4 (Evaluation)] §4 (Evaluation): The reported 99% status-code and 99% response-shape fidelity metrics lack a detailed description of the exact criteria used to determine a 'response-shape match,' how the 110 scenarios were selected, or any error analysis for state inconsistencies across requests, which is load-bearing for the direct comparison to record-replay (62%/16%) and the generalization claim.
Authors: We agree that additional detail on the response-shape metric and scenario construction will improve reproducibility. In the revised manuscript we will expand §4.2 with: (i) an explicit definition stating that a shape match requires identical top-level JSON structure, key presence, and array cardinalities (value equality is required only for non-stateful fields); (ii) the scenario-selection protocol, which enumerated all documented API interactions plus trace-derived edge cases from the three systems, yielding the 110 scenarios; and (iii) a short error-analysis subsection confirming that the two observed mismatches were isolated per-request value errors with no cross-request state divergence. These additions directly support the record-replay comparison and will be marked as new text. revision: yes
-
Referee: Integration test results (8/8 scenarios): The claim that caller integration tests produce identical pass/fail outcomes with MIRAGE as with real dependencies rests on only 8 scenarios; this sample is small relative to the 110 scenarios and does not explicitly validate cross-request state (e.g., tokens or counters) or coverage of underrepresented error/reasoning paths, leaving open the possibility that per-request shape matches mask divergences that would alter test outcomes.
Authors: The eight integration-test scenarios were deliberately chosen as the complete end-to-end suites for the three systems; each exercises multiple sequential calls and stateful elements (session tokens, counters, and error paths). All eight produced identical pass/fail results with real dependencies. While we acknowledge the modest count relative to the per-request fidelity set, the 110 scenarios already quantify shape and status fidelity, and the integration tests serve only to confirm that high per-request fidelity translates to unchanged test verdicts. In revision we will add an appendix table listing the state variables exercised in each of the eight scenarios and note the limitation on sample size for future work. We do not plan to enlarge the set, as the current evidence already demonstrates equivalence on the available integration suites. revision: partial
Circularity Check
No circularity: empirical results rest on direct external benchmarks
full rationale
The paper reports measured fidelity (99% status-code, 99% response-shape) and integration-test equivalence (8/8) on 110 scenarios across three independent microservice systems, using direct comparison to real dependencies and a record-replay baseline. No equations, fitted parameters, self-definitional constructs, or load-bearing self-citations appear in the derivation chain. Claims are observational performance numbers, not reductions of outputs to inputs by construction. The evaluation is self-contained against external oracles.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Large language models can accurately simulate microservice dependency behavior from source code and production traces without additional fine-tuning
read the original abstract
Existing approaches to microservice dependency simulation--record-replay, pattern-mining, and specification-driven stubs--generate static artifacts before test execution. These artifacts can only reproduce behaviors encoded at generation time; on error-handling and code-reasoning scenarios, which are underrepresented in typical trace corpora, record-replay achieves 0% and 12% fidelity in our evaluation. We propose online LLM simulation, a runtime approach where the LLM answers each dependency request as it arrives, maintaining cross-request state throughout a test scenario. The model reads the dependency's source code, caller code, and production traces, then simulates behavior on demand--trading latency (~3 s per request) and cost ($0.16-$0.82 per dependency) for coverage on scenarios that static artifacts miss. We instantiate this approach in MIRAGE and evaluate it on 110 test scenarios across three microservice systems (Google's Online Boutique, Weaveworks' Sock Shop, and a custom system). In white-box mode, MIRAGE achieves 99% status-code and 99% response-shape fidelity, compared to 62% / 16% for record-replay. A signal ablation shows dependency source code is often sufficient (100% alone); without it, the model retains error-code accuracy (94%) but loses response-structure fidelity (75%). Results are stable across three LLM families (within 3%) and deterministic across repeated runs. Caller integration tests produce the same pass/fail outcomes with MIRAGE as with real dependencies (8/8 scenarios).
Figures
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.