Pith. sign in

REVIEW 2 major objections 1 minor 70 references

Over 80% of real-world LLM applications leak their system prompts under adversarial queries.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 20:45 UTC pith:7U3FZJUM

load-bearing objection The 80% leakage rate rests on an unverified sample of 1,200 apps with no described selection method. the 2 major comments →

arxiv 2606.18673 v1 pith:7U3FZJUM submitted 2026-06-17 cs.CR

Understanding and Mitigating Prompt Leaking Attacks in Real-World LLM-Based Applications

Yong Yang , Chong Fu , Tong Zhang , Rui Zeng , Qingming Li , Tianyu Du , Zonghui Wang , Shouling Ji
show 1 more author
Wenzhi Chen
This is my paper
classification cs.CR
keywords prompt leakingLLM applicationsadversarial attackssystem promptsattention driftdefense mechanismsLLM securityreal-world deployments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper conducts a large-scale measurement of 1,200 LLM-based applications across six commercial platforms and reports that the majority expose their system prompts when subjected to realistic adversarial inputs. It traces the failure of existing defenses to an attention-level phenomenon called attention drift, in which query-key alignment bias and softmax amplification cause the model to progressively disregard protective instructions. From this analysis the authors derive AREA, a defense that inserts an optimizable soft prompt to re-anchor attention and thereby restore resistance to leakage. The work also documents that leaked prompts can contain third-party API keys and that AREA improves usability over prior defenses while cutting optimization cost.

Core claim

The central claim is that prompt leaking attacks succeed against more than 80 percent of deployed LLM applications because attention drift—driven by query-key alignment bias and softmax amplification—causes models to ignore defensive constraints in system prompts, and that this drift can be countered by re-anchoring attention with an optimizable soft prompt (AREA) that achieves comparable leakage resistance while raising average usability by over 33 percent and cutting optimization overhead by nearly 3x.

What carries the argument

Attention drift, the progressive shift in which query-key alignment bias and softmax amplification cause the model to ignore defensive constraints; AREA counters it by inserting an optimizable soft prompt that re-anchors attention to the original system instructions.

Load-bearing premise

The 1,200 applications and chosen adversarial queries are representative of real-world LLM deployments and realistic attack scenarios.

What would settle it

A larger or differently sampled study that finds leakage rates below 20 percent under comparable queries would falsify the reported prevalence.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • System prompts cannot be treated as protected intellectual property in current LLM deployments.
  • Existing prompt-based defenses are insufficient because they do not address attention drift.
  • AREA offers a practical alternative that preserves usability while matching the leakage resistance of stronger prior methods.
  • Vendors have begun treating prompt leaks as medium-severity vulnerabilities after responsible disclosure.
  • Leaked prompts can expose third-party credentials in addition to application logic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Application developers may need to move sensitive logic out of system prompts entirely rather than rely on defenses.
  • Future LLM architectures could incorporate built-in mechanisms to stabilize attention against drift.
  • The same attention-drift analysis might apply to other instruction-following failures beyond prompt leaking.
  • Platform operators could integrate AREA-style re-anchoring as a default service for hosted models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper claims that over 80% of 1,200 real-world LLM-based applications across six commercial platforms leak system prompts under realistic adversarial queries (sometimes exposing sensitive data), identifies attention drift as the cause of existing defense failures via mechanistic analysis, and proposes AREA (an optimizable soft-prompt defense) that matches state-of-the-art leakage resistance while improving usability by over 33% and reducing optimization overhead by nearly 3x.

Significance. If the sampling and query design are representative, the prevalence measurement establishes prompt leaking as a widespread issue in deployed systems, the attention-drift analysis supplies a mechanistic explanation for defense failures, and AREA demonstrates a usable mitigation. The responsible disclosure leading to medium-severity vulnerability classifications adds concrete impact.

major comments (2)
  1. [§4] §4 (Data Collection and Experimental Setup): the >80% leakage statistic is obtained from a fixed set of 1,200 applications with no explicit sampling frame, stratification, random-selection protocol, or inclusion criteria described; without this, the result cannot support the claim that leakage is widespread across real-world LLM-based applications.
  2. [§4.2] §4.2 (Adversarial Query Construction): the paper provides no account of how the adversarial queries were generated, validated against a threat model, or shown to be representative of realistic attacks; this is load-bearing for interpreting the reported leakage rates and defense evaluations.
minor comments (1)
  1. [Abstract] Abstract: the phrase 'realistic adversarial queries' is used without a forward reference to the construction or validation method, which could be clarified.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. The two major comments both concern missing methodological details in §4 that are necessary to support the prevalence claims. We agree these details are required and will revise the manuscript to supply them. Below we respond point-by-point.

read point-by-point responses

  1. Referee: [§4] §4 (Data Collection and Experimental Setup): the >80% leakage statistic is obtained from a fixed set of 1,200 applications with no explicit sampling frame, stratification, random-selection protocol, or inclusion criteria described; without this, the result cannot support the claim that leakage is widespread across real-world LLM-based applications.

    Authors: We acknowledge that the current manuscript does not provide an explicit sampling frame, stratification, or random-selection protocol for the 1,200 applications. The applications were collected by enumerating publicly discoverable LLM-based services on the six platforms via their documented APIs and marketplaces; however, this procedure is only summarized at a high level. We will revise §4 to include a precise description of the collection process, inclusion/exclusion criteria, and any limitations on generalizability. We will also qualify the prevalence claim to reflect that the sample is a large convenience sample rather than a statistically representative one. revision: yes

  2. Referee: [§4.2] §4.2 (Adversarial Query Construction): the paper provides no account of how the adversarial queries were generated, validated against a threat model, or shown to be representative of realistic attacks; this is load-bearing for interpreting the reported leakage rates and defense evaluations.

    Authors: The manuscript currently states only that the queries are “realistic adversarial queries” without detailing their construction or validation. We will expand §4.2 to describe the query-generation procedure (including any templates, mutation rules, or sources drawn from prior literature), the threat model they instantiate, and any steps taken to validate realism (e.g., pilot testing or comparison with known attack reports). If the queries were derived from a specific set of published attacks, that provenance will be stated explicitly. revision: yes

Circularity Check

0 steps flagged

Empirical measurement study with no derivations or self-referential reductions

full rationale

The paper reports direct empirical measurements on 1,200 applications and experimental evaluations of a proposed defense (AREA). No equations, fitted parameters, or derivation chains appear in the provided text. The prevalence statistic is obtained by querying the sampled applications with adversarial prompts; this is a measurement, not a prediction derived from a model that reduces to the same inputs by construction. No self-citation load-bearing steps, uniqueness theorems, or ansatzes are invoked to justify core claims. The representativeness concern raised by the skeptic is a question of external validity, not circularity under the defined patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract only; no information available on free parameters, axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5777 in / 948 out tokens · 24530 ms · 2026-06-26T20:45:34.404704+00:00 · methodology

0 comments
read the original abstract

Large language model (LLM)-based applications rely on system prompts to encode core logic and developer-defined constraints, making these prompts important intellectual property. However, system prompts are vulnerable to prompt leaking attacks. Although prior work has shown such attacks in controlled settings, their prevalence, causes, and defenses in real-world deployments remain unclear. This paper presents a systematic study of prompt leaking in real-world LLM-based applications. We measure 1,200 applications across six major commercial platforms and find that over 80% of deployments leak system prompts under realistic adversarial queries, sometimes exposing sensitive information such as third-party API keys. We also show that existing defenses often fail to prevent leakage without degrading usability. To explain these failures, we conduct an attention-level mechanistic analysis and identify attention drift, where query-key alignment bias and softmax amplification cause LLMs to progressively ignore defensive constraints. Guided by this insight, we propose AREA, a practical defense that re-anchors the model's attention using an optimizable soft prompt. Experiments and real-world case studies show that AREA matches the leakage resistance of state-of-the-art defenses while improving average usability by over 33% and reducing optimization overhead by nearly 3x. Our responsible disclosure led two affected vendors to classify these leaks as medium-severity vulnerabilities.

Figures

Figures reproduced from arXiv: 2606.18673 by Chong Fu, Qingming Li, Rui Zeng, Shouling Ji, Tianyu Du, Tong Zhang, Wenzhi Chen, Yong Yang, Zonghui Wang.

Figure 1
Figure 1. Figure 1: Measurement pipeline for large-scale evaluation of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of sensitive information types in ap [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of DAR and FT-DAR for successful [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustrative attention maps over Transformer layers [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Stage-wise divergence (Δ) between adversarial￾query and defensive-instruction tokens across the attention computation pipeline [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Overview of AREA. AREA optimizes a soft prompt at the embedding layer to re-anchor the victim LLM’s attention [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Generalization of AREA across model scales. We compare PromptObfuscation, SysVec, and AREA on Qwen and [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effectiveness and usability of defenses across real [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance under surrogate-prompt guided adap [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Defense performance under response-only itera [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Defense degradation under oracle-selection upper [PITH_FULL_IMAGE:figures/full_fig_p013_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: TopK attention token allocation between adversarial-query and defensive-instruction tokens for suc￾cessful and failed prompt leaking attacks in the first output token generation. Here, ADV Token refers to adversarial￾query token, and DEF Token refers to defensive-instruction token. (a) Llama-2-7B-chat-hf, (b) Llama-3.1-8B-Instruct, and (c) Mistral-7B-Instruct. results in [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 14
Figure 14. Figure 14: Average per system prompt optimization time [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 16
Figure 16. Figure 16: Sensitivity of AREA to loss weightings. 50 200 350 500 650 800 Benign Train Data Size 0.00 0.50 1.00 1.50 2.00 Score 5 20 35 50 65 80 Adversarial Train Data Size 0.00 0.50 1.00 1.50 2.00 50 200 350 500 650 800 Benign Train Data Size 5.50 6.00 6.50 7.00 7.50 Score 5 20 35 50 65 80 Adversarial Train Data Size 5.50 6.00 6.50 7.00 7.50 SS ( ) PLS ( ) RUS ( ) FC ( ) [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Sensitivity of AREA to training data size. [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Attack strength on known surrogate system [PITH_FULL_IMAGE:figures/full_fig_p022_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Effect of the usability-preservation weight [PITH_FULL_IMAGE:figures/full_fig_p023_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 14 canonical work pages · 7 internal anchors

  1. [1]

    https://huggingface.co/sentence-transformers

    2020.Sentence Transformers. https://huggingface.co/sentence-transformers

    2020

  2. [2]

    https://huggingface.co/datasets/fka/awesome- chatgpt-prompts

    2023.A wesome Chatgpt Prompts. https://huggingface.co/datasets/fka/awesome- chatgpt-prompts

    2023

  3. [3]

    https://learnprompting.org/docs/prompt_hacking/ defensive_measures

    2023.Defensive Measures. https://learnprompting.org/docs/prompt_hacking/ defensive_measures

    2023

  4. [4]

    https://huggingface.co/meta-llama/Llama-2-7b-chat- hf

    2023.Llama-2-7b-chat-hf. https://huggingface.co/meta-llama/Llama-2-7b-chat- hf

    2023

  5. [5]

    https://promptbase.com/

    2023.Prompt Marketplace. https://promptbase.com/

    2023

  6. [6]

    https://nvd.nist.gov/vuln/detail/cve-2024-5184

    2024.CVE-2024-5184. https://nvd.nist.gov/vuln/detail/cve-2024-5184

    2024

  7. [7]

    https://github.com/LouisShark/chatgpt_system_ prompt/tree/066b8f9a6db9dce64f2d5d36d91f9e87d8ca2530

    2024.How to get system prompt. https://github.com/LouisShark/chatgpt_system_ prompt/tree/066b8f9a6db9dce64f2d5d36d91f9e87d8ca2530

    2024

  8. [8]

    https://huggingface.co/meta-llama/Llama-3.1-8B- Instruct

    2024.Llama-3.1-8B-Instruct. https://huggingface.co/meta-llama/Llama-3.1-8B- Instruct

    2024

  9. [9]

    https://huggingface.co/meta-llama/Llama-3.3-70B- Instruct

    2024.Llama-3.3-70B-Instruct. https://huggingface.co/meta-llama/Llama-3.3-70B- Instruct

    2024

  10. [10]

    https://github.com/friuns2/Leaked- GPTs/blob/main/gpts/MangaMikoAnimeGirlfriend.md

    2024.Manga Miko Anime Girlfriend GPTs. https://github.com/friuns2/Leaked- GPTs/blob/main/gpts/MangaMikoAnimeGirlfriend.md

    2024

  11. [11]

    https://huggingface.co/mistralai/Mistral-7B- Instruct-v0.3

    2024.Mistral-7B-Instruct-v0.3. https://huggingface.co/mistralai/Mistral-7B- Instruct-v0.3

    2024

  12. [12]

    https://platform.openai.com/docs

    2024.OpenAI Product. https://platform.openai.com/docs

    2024

  13. [13]

    https://huggingface.co/Qwen/Qwen2.5-72B-Instruct

    2024.Qwen2.5-72B-Instruct. https://huggingface.co/Qwen/Qwen2.5-72B-Instruct

    2024

  14. [14]

    https://github.com/agentscope- ai/agentscope/tree/main/examples/functionality/rag

    2024.Retrieval-Augmented Agent Example. https://github.com/agentscope- ai/agentscope/tree/main/examples/functionality/rag

    2024

  15. [15]

    https://github.com/friuns2/BlackFriday-GPTs- Prompts/blob/main/gpts/simulation-game.md

    2024.Simulation Game GPTs. https://github.com/friuns2/BlackFriday-GPTs- Prompts/blob/main/gpts/simulation-game.md

    2024

  16. [16]

    https://claude.com/pricing/enterprise

    2025.Claude Product. https://claude.com/pricing/enterprise

    2025

  17. [17]

    https://github.com/elder- plinius/CL4R1T4S

    2025.Full extracted system prompts, guidelines, and tools. https://github.com/elder- plinius/CL4R1T4S

    2025

  18. [18]

    https://embracethered.com/blog/posts/2025/windsurf-data-exfiltration- vulnerabilities/

    2025.Hijacking Windsurf: How Prompt Injection Leaks Developer Se- crets. https://embracethered.com/blog/posts/2025/windsurf-data-exfiltration- vulnerabilities/

    2025

  19. [19]

    https://genai.owasp.org/llmrisk/ llm072025-system-prompt-leakage/

    2025.LLM07:2025 System Prompt Leakage. https://genai.owasp.org/llmrisk/ llm072025-system-prompt-leakage/

    2025

  20. [20]

    https://bugcrowd.com/engagements/openai

    2025.OpenAI policy of bug bounty. https://bugcrowd.com/engagements/openai

    2025

  21. [21]

    https://huggingface.co/Qwen/Qwen3-30B- A3B-Instruct-2507

    2025.Qwen3-30B-A3B-Instruct-2507. https://huggingface.co/Qwen/Qwen3-30B- A3B-Instruct-2507

    2025

  22. [22]

    https://huggingface.co/Qwen/Qwen3-32B

    2025.Qwen3-32B. https://huggingface.co/Qwen/Qwen3-32B

    2025

  23. [23]

    https://huggingface.co/Qwen/Qwen3-4B-Instruct- 2507

    2025.Qwen3-4B-Instruct-2507. https://huggingface.co/Qwen/Qwen3-4B-Instruct- 2507

    2025

  24. [24]

    2025.Tongyi Agent Platform

    Alibaba. 2025.Tongyi Agent Platform. https://www.tongyi.com/discover

    2025

  25. [25]

    2025.AgentBuilder

    Baidu. 2025.AgentBuilder. https://agents.baidu.com/

    2025

  26. [26]

    2024.Coze

    ByteDance. 2024.Coze. https://www.coze.com/

    2024

  27. [27]

    2024.Coze Community Guidelines

    ByteDance. 2024.Coze Community Guidelines. https://www.coze.com/open/ docs/guides/coze_community_guidelines

    2024

  28. [28]

    Bochuan Cao, Changjiang Li, Yuanpu Cao, Yameng Ge, Ting Wang, and Jinghui Chen. 2025. You Can’t Steal Nothing: Mitigating Prompt Leakages in LLMs via System Vectors. InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security. 4423–4437

    2025

  29. [29]

    Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025. {StruQ}: Defending against prompt injection with structured queries. In34th USENIX Security Symposium (USENIX Security 25). 2383–2400

    2025

  30. [30]

    Sizhe Chen, Yizhu Wang, Nicholas Carlini, Chawin Sitawarin, and David Wagner

  31. [31]

    Defending against prompt injection with a few defensivetokens.arXiv preprint arXiv:2507.07974(2025)

    work page arXiv 2025

  32. [32]

    Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, and Chuan Guo. 2024. Aligning llms to be robust against prompt injection.arXiv e-prints(2024), arXiv–2410

    2024

  33. [33]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning

  34. [34]

    What does bert look at? an analysis of bert’s attention.arXiv preprint arXiv:1906.04341(2019)

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  35. [35]

    Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, et al. 2024. Agentscope: A flexible yet robust multi-agent platform.arXiv preprint arXiv:2402.14034(2024)

    work page arXiv 2024

  36. [36]

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real- world llm-integrated applications with indirect prompt injection. InProceedings of the 16th ACM workshop on artificial intelligence and security. 79–90

    2023

  37. [37]

    Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. 2025. Model context protocol (mcp): Landscape, security threats, and future research directions.arXiv preprint arXiv:2503.23278(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. 2024. Pleak: Prompt leaking attacks against large language model applications. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 3600–3614

    2024

  39. [39]

    Kuo-Han Hung, Ching-Yun Ko, Ambrish Rawat, I-Hsin Chung, Winston H Hsu, and Pin-Yu Chen. 2025. Attention tracker: Detecting prompt injection attacks in llms. InFindings of the Association for Computational Linguistics: NAACL 2025. 2309–2322

    2025

  40. [40]

    Zhifeng Jiang, Zhihua Jin, and Guoliang He. 2024. Safeguarding system prompts for llms.arXiv preprint arXiv:2412.13426(2024)

    work page arXiv 2024

  41. [41]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems33 (2020), 9459–9474

    2020

  42. [42]

    Zi Liang, Haibo Hu, Qingqing Ye, Yaxin Xiao, and Haoyang Li. 2024. Why are my prompts leaked? unraveling prompt extraction threats in customized large language models.arXiv preprint arXiv:2408.02416(2024)

    work page arXiv 2024

  43. [43]

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 61–68

    2022

  44. [44]

    Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. 2023. Prompt injection attack against llm-integrated applications.arXiv preprint arXiv:2306.05499(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  45. [45]

    Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In33rd USENIX Security Symposium (USENIX Security 24). 1831–1847

    2024

  46. [46]

    Yupei Liu, Yuqi Jia, Jinyuan Jia, Dawn Song, and Neil Zhenqiang Gong. 2025. DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks. In2025 IEEE Symposium on Security and Privacy (SP). IEEE, 2190–2208

    2025

  47. [47]

    Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, et al

  48. [48]

    The attacker moves second: Stronger adaptive attacks bypass defenses against LLM jailbreaks and prompt injections.arXiv preprint arXiv:2510.09023 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    2024.GPTs

    OpenAI. 2024.GPTs. https://chat.openai.com/gpts/

    2024

  50. [50]

    David Pape, Sina Mavali, Thorsten Eisenhofer, and Lea Schönherr. 2025. Prompt obfuscation for large language models. In34th USENIX Security Symposium (USENIX Security 25). 2323–2342. 15 Preprint, 2026, Online Yang et al

    2025

  51. [51]

    Dario Pasquini, Martin Strohmeier, and Carmela Troncoso. 2024. Neural exec: Learning (and learning from) execution triggers for prompt injection attacks. In Proceedings of the 2024 Workshop on Artificial Intelligence and Security. 89–100

    2024

  52. [52]

    Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models.arXiv preprint arXiv:2211.09527(2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  53. [53]

    Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2024. Jatmo: Prompt injection defense by task-specific finetuning. InEuropean Symposium on Research in Computer Security. Springer, 105–124

    2024

  54. [54]

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2024. Safety alignment should be made more than just a few tokens deep.arXiv preprint arXiv:2406.05946(2024)

    work page arXiv 2024

  55. [55]

    2024.Poe

    Quora. 2024.Poe. https://poe.com/

    2024

  56. [56]

    Zedian Shao, Hongbin Liu, Jaden Mu, and Neil Zhenqiang Gong. 2024. Making llms vulnerable to prompt injection via poisoning alignment.arXiv e-prints (2024), arXiv–2410

    2024

  57. [57]

    Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024. Optimization-based prompt injection attack to llm- as-a-judge. InProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security. 660–674

    2024

  58. [58]

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models.arXiv preprint arXiv:2402.17762(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  59. [59]

    Yicong Tan, Xinyue Shen, Yun Shen, Michael Backes, and Yang Zhang. 2025. On the Effectiveness of Prompt Stealing Attacks on In-The-Wild Prompts. In2025 IEEE Symposium on Security and Privacy (SP). IEEE, 392–410

    2025

  60. [60]

    2025.Yuanqi Agent Shop

    Tencent. 2025.Yuanqi Agent Shop. https://yuanqi.tencent.com/agent-shop

    2025

  61. [61]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  62. [62]

    Corban Villa, Shujaat Mirza, and Christina Pöpper. 2025. Exposing the Guardrails:{Reverse-Engineering} and Jailbreaking Safety Filters in {DALL· E} {Text-to-Image} Pipelines. In34th USENIX Security Symposium (USENIX Se- curity 25). 897–916

    2025

  63. [63]

    Willison

    S. Willison. [n. d.]. Delimiters won’t save you from prompt injection. https: //simonwillison.net/2023/May/11/delimiters-wont-save-you

    2023

  64. [64]

    Yong Yang, Changjiang Li, Qingming Li, Oubo Ma, Haoyu Wang, Zonghui Wang, Yandong Gao, Wenzhi Chen, and Shouling Ji. 2025. {PRSA}: Prompt Stealing At- tacks against {Real-World} Prompt Services. In34th USENIX Security Symposium (USENIX Security 25). 2283–2302

    2025

  65. [65]

    Jiahao Yu, Yuhang Wu, Dong Shu, Mingyu Jin, Sabrina Yang, and Xinyu Xing

  66. [66]

    Assessing prompt injection risks in 200+ custom gpts.arXiv preprint arXiv:2311.11538(2023)

    work page arXiv 2023

  67. [67]

    Collin Zhang, John Xavier Morris, and Vitaly Shmatikov. 2024. Extracting prompts by inverting llm outputs. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 14753–14777

    2024

  68. [68]

    Yiming Zhang, Nicholas Carlini, and Daphne Ippolito. 2023. Effective prompt extraction from language models.arXiv preprint arXiv:2307.06865(2023)

    work page arXiv 2023

  69. [69]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

    2023

  70. [70]

    Universal and Transferable Adversarial Attacks on Aligned Language Models

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043(2023). A Developer Responses and Remediation Efforts We conducted responsible disclosure mainly through platform-side reporting channels, because prompt leak...

    work page internal anchor Pith review Pith/arXiv arXiv 2023