pith. sign in

arxiv: 2606.29520 · v1 · pith:QVVUXSOSnew · submitted 2026-06-28 · 💻 cs.SE · cs.AI· cs.DB

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

Tiziano Santilli , Francesco Daghero , Mayhar Tourchi Moghaddam This is my paper

Pith reviewed 2026-06-30 02:15 UTC · model grok-4.3

classification 💻 cs.SE cs.AIcs.DB
keywords software architecturelarge language modelsbenchmark evaluationmultiple choice questionsarchitectural reasoningquality attributesdesign patternsLLM assessment
0
0 comments X

The pith

A benchmark of 2154 questions shows LLMs reach high overall accuracy on software architecture but differ sharply across categories.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents SAKE as a standardized benchmark of expert-curated multiple-choice questions to measure how well large language models reason about software architecture. Architectural tasks involve trade-offs among quality attributes, design patterns, and system constraints that existing benchmarks do not test. Evaluation of eleven models in zero-shot and five-shot settings across eight categories and four context lengths produces high average scores yet wide differences by category. The benchmark and all evaluation materials are released openly so the community can track progress on this capability.

Core claim

SAKE comprises 2154 expert-curated multiple-choice questions with four options each, stratified across eight architectural categories and four context-length levels. When eleven proprietary and open-weight models are tested in zero-shot and five-shot regimes, overall accuracy is high while performance varies markedly across categories, revealing competency gaps in areas central to professional practice. The benchmark, scripts, and results are released as open source.

What carries the argument

The SAKE benchmark: a collection of 2154 expert-curated four-option multiple-choice questions that directly test quality-attribute trade-offs, design patterns, and system-level constraints.

If this is right

  • Models can now be compared on a common, reproducible measure of architectural knowledge rather than only syntactic or algorithmic tasks.
  • Category-level score differences point to specific areas where current models fall short of professional requirements.
  • The open release of questions, scripts, and results supplies a fixed baseline for measuring future improvements in LLM architectural reasoning.
  • Performance gaps across context lengths indicate that longer architectural descriptions do not uniformly improve or degrade model answers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark could be extended with open-ended or scenario-based items to test whether multiple-choice performance predicts free-form architectural advice.
  • Category gaps may indicate that general pre-training data under-represents the explicit trade-off reasoning required in architecture roles.
  • If the benchmark is adopted widely, model developers may prioritize training signals that close the observed category differences.
  • The stratification by context length offers a way to study how architectural reasoning scales with problem size.

Load-bearing premise

Expert-curated multiple-choice questions with four options accurately measure the ability to reason about software architecture, including quality attribute trade-offs, design patterns, and system-level constraints.

What would settle it

A finding that models scoring low on SAKE still produce sound architectural decisions in real projects, or that high-scoring models produce poor ones, would falsify the claim that the benchmark measures relevant architectural reasoning.

Figures

Figures reproduced from arXiv: 2606.29520 by Francesco Daghero, Mayhar Tourchi Moghaddam, Tiziano Santilli.

Figure 1
Figure 1. Figure 1: SAKE Benchmark Methodology 3 Methodology This work’s methodology is organized into three sequential phases, as shown in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
read the original abstract

Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured. Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints, none of which are exercised by benchmarks that target syntactic or algorithmic tasks. We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible benchmark for assessing software architectural knowledge in LLMs. SAKE comprises 2154 expert-curated multiple-choice questions, each with four options, stratified across eight architectural categories and four context-length levels. We evaluate 11 proprietary and open-weight models in zero-shot and five-shot settings. Overall accuracy is high, but performance varies markedly across categories, revealing competency gaps in areas central to professional practice. SAKE, its evaluation scripts, and all results are released as open source to give the community a baseline for tracking architectural reasoning in LLMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper introduces SAKE, a benchmark of 2154 expert-curated four-option multiple-choice questions stratified across eight architectural categories and four context-length levels. It evaluates 11 LLMs in zero-shot and five-shot settings, reports high overall accuracy with marked variation across categories, and releases the benchmark, scripts, and results as open source.

Significance. If the questions validly elicit architectural reasoning about quality-attribute trade-offs, patterns, and constraints, SAKE would supply a needed, reproducible baseline for an area not covered by existing LLM benchmarks focused on syntax or algorithms. The open-source release of data, code, and results is a clear strength that enables community tracking of progress.

major comments (1)
  1. [Abstract and benchmark description] The central claim that category-wise accuracy differences reveal genuine competency gaps in professional architectural practice depends on the MCQs actually measuring the targeted reasoning skills. The manuscript provides no reported validation (expert agreement statistics, pilot testing with practicing architects, difficulty anchoring against real design tasks, or MCQ vs. open-ended comparison). This is load-bearing because, absent such evidence, the observed variations could arise from surface cues or construction artifacts rather than differences in architectural competence.

Simulated Author's Rebuttal

1 responses · 1 unresolved

We thank the referee for the constructive feedback emphasizing the need for evidence that the SAKE questions measure targeted architectural reasoning rather than artifacts. We address the major comment below and outline revisions to the manuscript.

read point-by-point responses

  1. Referee: [Abstract and benchmark description] The central claim that category-wise accuracy differences reveal genuine competency gaps in professional architectural practice depends on the MCQs actually measuring the targeted reasoning skills. The manuscript provides no reported validation (expert agreement statistics, pilot testing with practicing architects, difficulty anchoring against real design tasks, or MCQ vs. open-ended comparison). This is load-bearing because, absent such evidence, the observed variations could arise from surface cues or construction artifacts rather than differences in architectural competence.

    Authors: We agree that the absence of formal validation metrics is a limitation for interpreting category differences as genuine competency gaps. The questions were developed through expert curation by authors with software architecture experience, with explicit stratification across the eight categories and four context lengths to target quality-attribute trade-offs, patterns, and constraints. However, the original manuscript reports neither inter-expert agreement statistics, pilot testing with external architects, nor comparisons against open-ended tasks. We will revise the benchmark construction section to describe the curation workflow in greater detail, including how questions were reviewed to reduce surface cues, and will add an explicit limitations paragraph acknowledging the lack of these validation steps. We will also note this as an area for future community extension. These changes will be incorporated in the revised manuscript. revision: partial

standing simulated objections not resolved
  • We cannot retroactively supply quantitative validation results (expert agreement, pilot studies, or difficulty anchoring) that were not collected during benchmark creation.

Circularity Check

0 steps flagged

No circularity: benchmark construction with direct evaluation

full rationale

The paper describes creation of a 2154-question MCQ benchmark stratified by categories and context lengths, followed by zero-shot and five-shot accuracy evaluation on 11 models. No equations, parameter fitting, predictions derived from inputs, or self-citation chains appear in the provided text. The central claim (accuracy variation across categories) is a direct empirical measurement on the constructed dataset rather than a reduction to prior self-referential results. This is a standard benchmark paper with no load-bearing derivation steps that collapse by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central contribution is the new dataset and evaluation protocol; the main unverified premise is that the chosen questions validly capture architectural reasoning.

axioms (1)
  • domain assumption Multiple-choice questions with four options can effectively measure software architectural knowledge including quality attribute trade-offs and system-level constraints.
    The benchmark's claim to reveal competency gaps rests on this assumption about question validity.

pith-pipeline@v0.9.1-grok · 5695 in / 1144 out tokens · 32810 ms · 2026-06-30T02:15:27.624228+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 8 canonical work pages · 2 internal anchors

  1. [1]

    Akhtar, M., et al.: When ai benchmarks plateau: A systematic study of benchmark saturation (2026)

    2026

  2. [2]

    In: Software Architecture: 19th European Conference, ECSA 2025, Proceedings

    Amalfitano,D.,etal.:Automatedsoftwarearchitecturedesignrecoveryfromsource code using llms. In: Software Architecture: 19th European Conference, ECSA 2025, Proceedings. p. 73–89 (2025). https://doi.org/10.1007/978-3-032-02138-0_5 18 T. Santilli et al

    work page doi:10.1007/978-3-032-02138-0_5 2025

  3. [3]

    SIGSOFT Softw

    Avgeriou, P., Kruchten, P., Lago, P., Grisham, P., Perry, D.: Architectural knowl- edge and rationale: issues, trends, challenges. SIGSOFT Softw. Eng. Notes32(4), 41–46 (Jul 2007). https://doi.org/10.1145/1281421.1281443

    work page doi:10.1145/1281421.1281443 2007

  4. [4]

    In: Fifth International Confer- ence on Quality Software (QSIC’05)

    Babar,M.,Gorton,I.,Jeffery,R.:Capturingandusingsoftwarearchitectureknowl- edge for architecture-based software development. In: Fifth International Confer- ence on Quality Software (QSIC’05). pp. 169–176 (2005)

    2005

  5. [5]

    Bass, L., Clements, P., Kazman, R.: Software Architecture in Practice, 4th Edition

  6. [6]

    Capilla, R., Jansen, A., Tang, A., Avgeriou, P., Babar, M.A.: 10 years of software architecture knowledge management. J. Syst. Softw.116(C), 191–205 (Jun 2016)

    2016

  7. [7]

    https://doi.org/10.48550/arXiv.2506.22688

    Cervantes, H., Kazman, R., Cai, Y.: An llm-assisted approach to designing software architectures using add (06 2025). https://doi.org/10.48550/arXiv.2506.22688

    work page internal anchor Pith review doi:10.48550/arxiv.2506.22688 2025

  8. [8]

    Chen, M., et al.: Evaluating large language models trained on code (2021)

    2021

  9. [9]

    Journal of Systems and Software169(2020)

    Farshidi, S., Jansen, S., van der Werf, J.M.: Capturing software architecture knowl- edge for pattern-driven design. Journal of Systems and Software169(2020)

    2020

  10. [10]

    Addison-Wesley Longman Publishing Co., Inc

    Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design patterns: elements of reusable object-oriented software. Addison-Wesley Longman Publishing Co., Inc

  11. [11]

    Garlan, D., Shaw, M.: An introduction to software architecture (1994)

    1994

  12. [12]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language understanding (09 2020)

    2020

  13. [13]

    What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams , volume =

    Jin, D., Pan, E., Oufattole, N., Weng, W.H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences11(14) (2021). https://doi.org/10.3390/app11146421,https://www.mdpi.com/2076-3417/11/ 14/6421

    work page doi:10.3390/app11146421 2021

  14. [14]

    Liang, P., et al.: Holistic evaluation of language models (2023)

    2023

  15. [15]

    SIGSOFT Softw

    Perry, D.E., Wolf, A.L.: Foundations for the study of software ar- chitecture. SIGSOFT Softw. Eng. Notes17(4), 40–52 (Oct 1992). https://doi.org/10.1145/141874.141884

    work page doi:10.1145/141874.141884 1992

  16. [16]

    https://doi.org/10.48550/arXiv.2505.16697

    Schmid, L., Hey, T., Armbruster, M., Corallo, S., Fuchß, D., Keim, J., Liu, H., Koziolek, A.: Software architecture meets llms: A systematic literature review (05 2025). https://doi.org/10.48550/arXiv.2505.16697

    work page doi:10.48550/arxiv.2505.16697 2025

  17. [17]

    In: 2025 IEEE 22nd International Con- ference on Software Architecture (ICSA)

    Soliman, M., Keim, J.: Do large language models contain software architec- tural knowledge? : An exploratory case study with gpt. In: 2025 IEEE 22nd International Conference on Software Architecture (ICSA). pp. 13–24 (2025). https://doi.org/10.1109/ICSA65012.2025.00012

    work page doi:10.1109/icsa65012.2025.00012 2025

  18. [18]

    Srivastava, A., et al.: Beyond the imitation game: Quantifying and extrapolating the capabilities of language models (2023)

    2023

  19. [19]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

    Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al.: Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574 (2024) A Example Questions Table 6 gives one representative item per category. Each shows the stem, the four options, and the verifi...

    work page internal anchor Pith review Pith/arXiv arXiv 2024