ai-llm — Secure Multi-Model AI Gateway
A self-hosted gateway that routes a team's AI through one endpoint — local GPU inference with Anthropic/OpenAI fallback — behind PII redaction, injection defense, audit logging, and cost control.
- LiteLLM
- vLLM
- Ollama
- Presidio
- Docker
- Prometheus / Grafana
What it is
A self-hosted corporate LLM gateway. Every engineer's AI request — from the editor, an app, or a chat UI — goes through one OpenAI-compatible endpoint that runs local GPU inference first and falls back to the cloud, with security, governance, and cost control applied on the way through. It's the "secure gateway/proxy that integrates multiple frontier models and is optimized for logging and control" pattern — running, not theoretical.
One endpoint, many models
- Routes across vLLM (local GPU), Anthropic, OpenAI, and Ollama behind a LiteLLM gateway — local-first to keep cost and data in-house, cloud fallback when needed.
- Onboard a new frontier model by configuration, not code.
A security & control proxy
- An ordered guardrail pipeline on every request: PII redaction (Presidio) → prompt-injection defense → engineering-discipline policy → audit logging.
- API-key provisioning, rate limiting, and per-request cost tracking — so usage is attributable, governed, and bounded.
Observability & operations
- Prometheus + Grafana dashboards for performance and cost; health-check and load-test tooling; an admin UI for guardrail configuration and keys.
Architecture
An 11-service Docker Compose stack: the gateway, GPU inference, PostgreSQL (audit, keys, costs), Redis (cache + rate limits), the Presidio analyzer, a chat front end, admin and docs services, Prometheus, Grafana, and a reverse proxy. Self-hostable end to end — the data and the spend stay inside your boundary.
The point isn't "we use AI." It's that a team can use AI fast without giving up security, governance, or cost control — the same principle as CW DevOps, one layer down.