Prompt logging & privacy¶

Prompt + response text is one of the highest-signal artifacts you can log. It's also the most sensitive — a naïve trace pipeline will happily ship user PII, keys pasted into prompts, and full-fidelity model responses to a backend that outlives the request. This page is the guardrails.

Default this recipe to OFF for prompt bodies

The observability service ships with CHIRON_LOG_PROMPT_TEXT=false and CHIRON_LOG_RESPONSE_TEXT=false. Never flip them on without also wiring the redaction step below.

The rule¶

Log structure, not content, by default. Log content only when you've decided what to strip first and who can read the result.

Concretely:

What	Default	Rationale
Token counts (`gen_ai.usage.*`)	on	Structural, not sensitive. Needed for cost + throughput dashboards.
Latency (`gen_ai.client.duration`)	on	Not sensitive. Needed for SLO monitoring.
Model IDs, request params (temp / max_tokens)	on	Not sensitive; needed for A/B analysis.
Error class + message	on	Occasionally leaks a snippet — sanitize per rule below.
Prompt text	off	Contains user input verbatim.
Response text	off	Contains model output verbatim.
Vector embedding values	off	Reversible for a determined attacker.
Tool arguments	off	Frequently pass through user data.

Redaction — always before it leaves the process¶

If you decide you need prompt bodies for debugging, redact before the OTel exporter batches the span. Do it in a SpanProcessor — that way every span goes through the filter regardless of which call site produced it.

# examples/observability/service/redact.py
import re
from opentelemetry.sdk.trace import SpanProcessor

# Redact anything that looks like an email, SSN, US phone, API key or a bearer token.
_PATTERNS = [
    (re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+"),        "[REDACTED_EMAIL]"),
    (re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),           "[REDACTED_SSN]"),
    (re.compile(r"\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b"), "[REDACTED_PHONE]"),
    (re.compile(r"sk-[A-Za-z0-9]{16,}"),             "[REDACTED_APIKEY]"),
    (re.compile(r"(?i)bearer\s+[A-Za-z0-9._-]+"),    "[REDACTED_BEARER]"),
]

_SENSITIVE_ATTRS = {"gen_ai.prompt", "gen_ai.completion", "gen_ai.tool.arguments"}


class RedactingSpanProcessor(SpanProcessor):
    def __init__(self, inner: SpanProcessor):
        self._inner = inner

    def on_start(self, span, parent_context=None):
        self._inner.on_start(span, parent_context)

    def on_end(self, span):
        for k, v in list(span.attributes.items()):
            if k in _SENSITIVE_ATTRS and isinstance(v, str):
                for pat, sub in _PATTERNS:
                    v = pat.sub(sub, v)
                span._attributes[k] = v          # pragma: no cover - readonly bypass
        self._inner.on_end(span)

    def shutdown(self):
        self._inner.shutdown()

    def force_flush(self, timeout_millis: int = 30000):
        return self._inner.force_flush(timeout_millis)

Wire it before the vendor-specific batch processor:

provider.add_span_processor(
    RedactingSpanProcessor(BatchSpanProcessor(exporter))
)

This regex set is the floor, not the ceiling — extend for your domain (patient IDs, order numbers, whatever). The point is that redaction happens once, in one place, before the span leaves the process.

Per-cloud enforcement — belt and suspenders¶

The application-level redaction above is the primary line of defense. Each cloud also offers a backend-side scrubber for the audit trail (which sits outside your OTel pipeline):

Azure Monitor — Log filtering via a data collection rule (KQL transformation) drops or masks fields on ingest.
GCP Cloud Logging — Log exclusions drop matching entries; field-level filters mask specific keys.
AWS CloudWatch Logs — Data Protection Policies mask matched fields on ingest; use built-in managed data identifiers or custom regex.

Do both — application redaction is fast and cheap; backend enforcement is the auditor's answer to "prove nothing leaks."

What NEVER to log¶

Regardless of redaction:

API keys, connection strings, private keys — always via env / secret manager, never in a prompt attribute.
OAuth / JWT bearer tokens — same rule.
Anything a compliance framework (HIPAA, PCI-DSS, GDPR data-subject content) covers, unless you've completed the impact assessment for the sink.

The observability service's health endpoint publishes which flags are active so you can spot-check the running config:

GET /healthz
{"status": "ok", "provider": "azure", "log_prompt_text": false, "log_response_text": false, "redactor_active": true}

Tony's hard rule¶

Fail closed. If the redactor isn't loaded, the exporter doesn't start. If prompt-text logging is on and the redactor isn't loaded, /healthz returns 503 and the pod never becomes ready.

That check ships in examples/observability/service — the process refuses to serve traffic in an unsafe configuration.