Medium · 14 steps · ~45 min

Design a guardrails and moderation pipeline

Check every message before the model and every answer after it, within a latency budget, with a human review queue and a plan for when the checks fail.

A public chat assistant talks to anyone. Users send abuse, paste other people's personal data and try to trick the model with injected instructions, and the model can produce answers you must not show. You have to check every message before it reaches the model and every answer before it reaches the user, without adding more than a small slice to the response time, and let people look at the borderline cases. And you must decide what the product does when the checks themselves break.

Same problem as Public chat assistantsTrust and safety teamsModeration APIs

How it works

• Cheap rules and a verdict cache take the bulk of traffic, so the expensive classifiers only see what is new.
• A classifier fleet scores messages and answers; two thresholds turn scores into allow, block or send-to-a-human.
• A queue and workers record decisions and feed the human review queue off the critical path.
• When the classifiers fail, you must pick fail open or fail closed deliberately, per use case.
• A model you host yourself keeps the product alive if the provider has an outage.

What you'll build

1. The moderation API and its data model
2. A limiter, classifier fleet, gateway and model in a chain
3. Fast rule checks and a cache for repeated checks
4. A queue and workers for logging and human review
5. A design that survives a lost classifier, a provider outage, a cold cache and an abuse flood

Steps

1. 1Design the moderated chat API
2. 2Design the data model
3. 3Check before and after the model
4. 4Fast rules and a cache for repeated checks
5. 5Classifier fleet down: fail open or fail closed?
6. 6Queue and workers for review and logging
7. 7False positives, false negatives and thresholds
8. 8Lose a classifier server
9. 9Why input filtering alone cannot stop prompt injection
10. 10Survive a provider outage
11. 11Survive a cold cache
12. 12A flood of abuse
13. 13Logging and privacy of flagged content
14. 14Human review workload and escalation