Prompt Injection

A reader's summary of the security problem that follows almost mechanically from how large language models read text: if instructions and data share one channel, anything that can put text in front of the model can try to give it orders.

The idea at a glance

Prompt injection is an attack in which text supplied to a large language model as data — content it is merely asked to read, summarize, translate, or act on — contains instructions that the model ends up following instead of, or in addition to, the task its actual user gave it. The name is a deliberate echo of SQL injection, the decades-older class of attack where unescaped user input is interpreted as code by a database. The parallel is instructive and also where the resemblance runs out: SQL injection has a clean fix (parameterized queries, which enforce a hard syntactic wall between query and data) precisely because SQL and its inputs are two formal languages. A prompt and the text it contains are the same language, read by a model that has no reliable native notion of where one speaker's authority ends and another's begins.

Origins

The term was coined by independent researcher and Django co-creator Simon Willison in a September 2022 post, after security researchers showed that a GPT-3-powered Twitter bot built to auto-reply about remote work could be hijacked by tweeting it a message telling it to ignore its instructions and say something else instead. Willison's post did the naming work that let a scattered set of anecdotal “jailbreak” demonstrations consolidate into a single, citable vulnerability class with its own vocabulary — direct versus indirect injection, the confused-deputy framing, and so on — that researchers and vendors have used ever since.

History and context

Early cases through 2022–2023 were mostly direct and mostly playful: users typing “ignore all previous instructions” into chatbots to extract a hidden system prompt or make the bot say something its operator did not intend. The more consequential shift came as models gained two capabilities at once — the ability to browse and summarize arbitrary third-party content, and the ability to call tools and take actions on a user's behalf. That combination made indirect injection the dominant concern: by 2024–2026, documented proof-of-concept attacks had hidden instructions in resume PDFs targeting AI hiring screens, in webpages targeted at browser-using agents, in calendar invites and emails aimed at inbox-managing assistants, and in code comments and repository files aimed at AI coding assistants — each case using the same underlying trick, just a different delivery surface.

The mechanism

Critique

• “Solved” claims tend to be scoped, not general. Vendors periodically announce defenses that close a specific benchmark's attack set; independent researchers, Willison prominent among them, have repeatedly found follow-up bypasses within days, which is the pattern that has made much of the field skeptical of any claim to have solved injection outright rather than mitigated a known slice of it.
• The comparison to SQL injection can overstate the fix. SQL injection is tractable because query and data are two formal grammars a parser can keep apart; natural language has no equivalent hard boundary, so techniques that work by analogy — delimiter tags, “system” versus “user” role markers — raise the cost of an attack without removing the underlying ambiguity a sufficiently creative prompt can still exploit.
• Risk is genuinely proportional to permissions, not to novelty. A read-only summarization bot that gets injected mostly produces an embarrassing or wrong answer; the same vulnerability in an agent that can send emails, execute code, or move money produces real-world harm — which is why security guidance in 2025–2026 increasingly focuses on constraining what an agent isallowed to do rather than on trying to perfectly filter what it reads.
• It compounds with, but is distinct from, jailbreaking. Jailbreaking targets the model's own safety training to get it to produce disallowed content for the person talking to it directly; prompt injection targets the boundary between a legitimate user and third-party content the model processes on that user's behalf — the two are often confused in casual coverage but call for different defenses.

Impact

Prompt injection reframed how the industry talks about AI agent safety: architecture reviews for agentic products now routinely include a “what is the blast radius if this agent's input is adversarial” question, mirroring how web security reviews ask about untrusted input by default. It has driven concrete design patterns — least-privilege tool access, human-in-the-loop confirmation for consequential actions, sandboxed execution, content provenance tagging — into mainstream agent frameworks, and it sits behind most publicized real-world agent security incidents to date, from data-exfiltration demos against email assistants to hijacked browser agents completing unintended purchases.

Notable engagements

• Simon Willison, “Prompt injection attacks against GPT-3” (2022) — the post that named the attack and framed it against SQL injection.
• Greshake et al., “Not what you've signed up for” (2023) — the paper that formalized indirect prompt injection via retrieved content and demonstrated it across real applications.
• OWASP LLM Top 10 (2023–ongoing) — prompt injection has held the top-ranked position in OWASP's list of large-language-model application risks since the list's first release.
• Agent security demos (2024–2026) — a recurring wave of proof-of-concept attacks against browser-using and inbox-managing agents, widely cited in coverage of AI agent rollouts as the reason vendors ship increasingly conservative default permissions.