Enterprise Gateway Layer 2026: LiteLLM, Portkey, Cloudflare, Kong, AWS Strands & Privacy Router

Why an enterprise gateway in the first place?

Once more than one team uses LLMs in production, you need four things in one place:

1. Auth & RBAC – who may use which model with which budget?
2. Observability & logging – who asked what, when, what did it cost, what came back?
3. Routing by model/vendor – failover, cost optimization, PII awareness.
4. Rate limiting & quotas – per team, per model, per time of day.

Microsoft Scout was announced as an integrated enterprise gateway – but is not yet available. To go live in mid-2026, combine the following building blocks.

Decision flowchart: which gateway fits you?

Start: Do you need an LLM gateway?
  │
  ├─ GDPR-strict / on-premise mandate?
  │     │
  │     ├─ Yes →  Self-hosted OpenClaw + Privacy Router
  │     │         (local models, no hyperscaler)
  │     │
  │     └─ No  → continue ↓
  │
  ├─ Pure AWS stack with compliance mandate?
  │     │
  │     ├─ Yes →  AWS Strands / Bedrock AgentCore
  │     │         (IAM, CloudTrail, Bedrock models)
  │     │
  │     └─ No  → continue ↓
  │
  ├─ Regulated industry (bank, pharma, public sector)?
  │     │
  │     ├─ Yes →  Kong AI Gateway (self-hosted or Konnect EU)
  │     │         (mTLS, OAuth/OIDC, audit trails, plugin ecosystem)
  │     │
  │     └─ No  → continue ↓
  │
  ├─ Need PII redaction & prompt governance?
  │     │
  │     ├─ Yes →  Portkey AI Gateway (in front of LiteLLM)
  │     │         (guardrails, prompt versioning, A/B tests)
  │     │
  │     └─ No  → continue ↓
  │
  ├─ High volumes with cache potential & global edge?
  │     │
  │     ├─ Yes →  Cloudflare AI Gateway
  │     │         (DNS entry, 5 min, instant logs & cost caps)
  │     │
  │     └─ No  → continue ↓
  │
  └─ Default: multi-provider with quotas & spend tracking
        │
        └─→  LiteLLM Proxy (+ optional Portkey for governance)
             (OpenAI-compatible, 100+ providers, Docker in 10 min)

Deployment decision: self-hosted vs. VPC vs. managed vs. hybrid vs. air-gapped

Before you pick a product, decide on the deployment model. It drives data residency, governance effort and operational complexity more than the feature list.

Start: Where are prompts & logs allowed to live?
  │
  ├─ No data may leave the data center (public sector, hospital, defense)?
  │     │
  │     └─ Yes → Air-gapped (on-prem, no internet)
  │             Candidates: OpenClaw + Ollama/vLLM, Kong AI Gateway, LiteLLM
  │             Ops: high (manual updates, own monitoring)
  │
  ├─ Data must stay in your own cloud tenant (bank, insurance, pharma)?
  │     │
  │     └─ Yes → VPC / private cloud (EU region, customer-managed keys)
  │             Candidates: AWS Strands/Bedrock AgentCore, Kong (self-hosted in VPC),
  │                         LiteLLM/Portkey in your own EKS/AKS/GKE
  │             Ops: medium (hyperscaler handles infra)
  │
  ├─ GDPR-compliant, but a mix of sensitive & generic prompts?
  │     │
  │     └─ Yes → Hybrid (managed control plane + self-hosted data plane)
  │             Candidates: Portkey Hybrid, LiteLLM + Privacy Router,
  │                         Cloudflare AI Gateway with EU R2 + local fallback
  │             Ops: medium (two layers, clear routing policies required)
  │
  ├─ Standard SaaS, EU hosting is enough, time-to-value matters?
  │     │
  │     └─ Yes → Managed (SaaS / edge)
  │             Candidates: Portkey Cloud (EU), Cloudflare AI Gateway,
  │                         AWS Bedrock (Frankfurt)
  │             Ops: low (DPA + config, no infra)
  │
  └─ Full control, IaC pipelines, in-house SRE team?
        │
        └─ Yes → Self-hosted (Docker/K8s in your own cluster)
                Candidates: LiteLLM, Portkey OSS, Kong, OpenClaw + Ollama
                Ops: high (updates, HA, secrets rotation in your hands)

Criteria matrix:

💡 Rule of thumb: The stricter the data residency, the higher the operational complexity. Hybrid is the pragmatic middle ground when not all prompts are equally sensitive – PII/secrets local, generic in the cloud.

💡 Till Freitag stack recommendation: LiteLLM as multi-provider front door + Portkey as governance layer + Privacy Router for GDPR-critical paths. Once Microsoft Scout is GA, this config can be migrated with manageable effort – skills and MCP configs stay the same.

The six alternatives in detail – with enterprise workflows

LiteLLM Proxy – The OpenAI-compatible multi-provider front door

Setup: ~10 min (docker run litellm/litellm or pip install litellm). Hosting: self-hosted, EU hosting possible. 100+ LLMs behind a unified OpenAI API.

Concrete enterprise workflows:

• RBAC / Auth: virtual API keys per team with JWT validation. master_key → mints team_keys with their own model whitelists. SSO via OIDC (Okta, Entra ID) through a reverse proxy.
• Logging / Observability: OTLP export to Langfuse, Grafana Loki or Datadog. Every request tagged with user_id, team_id, input/output tokens, cost, latency.
• Routing by model/vendor: model aliases (gpt-4 → primary Azure OpenAI Frankfurt, fallback OpenAI US). Cost-based routing via model_list price info. Health checks every 60s.
• Rate limiting: quotas per key (rpm, tpm, max_budget_usd_per_month). Soft & hard caps with alert webhooks at 80% spend.

Portkey AI Gateway – The governance layer with guardrails

Setup: ~15 min (Docker or cloud). Hosting: self-hosted (OSS) or EU cloud. Sits typically in front of LiteLLM or directly in front of the provider.

Concrete enterprise workflows:

• RBAC / Auth: workspaces per department, RBAC with admin / developer / viewer roles. Virtual keys with per-key guardrail configs.
• Logging / Observability: built-in tracing dashboard with prompt diffs, cost attribution, PII hit rate. OTLP export for external stacks.
• Routing by model/vendor: strategies for loadbalance, fallback, conditional routing (e.g., "PII detected → on-prem Ollama"), guardrails as pre/post filters (toxicity, PII, JSON-schema validation).
• Rate limiting: per virtual key, per model, per time of day. Budget caps with auto-disable.

Cloudflare AI Gateway – The managed edge gateway

Setup: ~5 min (DNS entry or Worker binding). Hosting: managed, Cloudflare edge (EU PoPs available).

Concrete enterprise workflows:

• RBAC / Auth: Cloudflare Access (Zero Trust) as an IdP layer – employees authenticate against Entra ID / Okta before reaching the gateway. API tokens with scopes per service.
• Logging / Observability: built-in analytics console (requests, cache hit rate, tokens, cost). Logs to R2 / Logpush in EU region (S3, Splunk, BigQuery).
• Routing by model/vendor: multi-provider failover (e.g., Anthropic primary, OpenAI fallback). Caching on prompt hash saves up to 60% on recurring queries (marketing tools, classifiers).
• Rate limiting: cost caps per token, requests/min per account or per user header. Edge-near limits keep providers from being contacted at all.

Kong AI Gateway – The classic API gateway with AI plugins

Setup: ~30 min (Helm / Docker). Hosting: self-hosted or Kong Konnect EU.

Concrete enterprise workflows:

• RBAC / Auth: mTLS between services, OAuth 2.0 / OIDC to Entra ID, Keycloak, Okta. Consumer model with ACLs per route – ideal for multi-tenant platforms.
• Logging / Observability: plugins for OTLP, Prometheus, Datadog, Elastic. Audit trails on every route, request/response bodies optionally encrypted into the SIEM.
• Routing by model/vendor: AI-Proxy plugin speaks Anthropic, OpenAI, Cohere, Mistral, Azure OpenAI. AI-Request-Transformer for prompt manipulation, AI-Response-Transformer for schema enforcement.
• Rate limiting: Rate-Limiting-Advanced plugin (sliding window, redis-backed) per consumer, per route, per plan tier. AI-specific: tokens/min instead of just requests/min.

AWS Strands / Bedrock AgentCore – The AWS-native stack

Setup: ~30 min (AWS CLI + IAM + Bedrock console). Hosting: AWS cloud, Frankfurt region.

Concrete enterprise workflows:

• RBAC / Auth: IAM roles per Lambda/container, fine-grained per Bedrock model and per skill. SSO via IAM Identity Center, permission sets per department.
• Logging / Observability: CloudTrail for every Bedrock API call (compliance audit trail), CloudWatch Logs Insights for queries, X-Ray for tracing. Models have built-in invocation logs to S3/CloudWatch.
• Routing by model/vendor: inference profiles in Bedrock allow cross-region routing and model aliases. Bedrock Guardrails as a central PII/toxicity layer. Anthropic models, Llama, Mistral, Amazon Nova natively.
• Rate limiting: service quotas per model and region. Per-application Provisioned Throughput for predictable latency. Budgets via AWS Cost Anomaly Detection with auto-alerts.

Self-hosted OpenClaw + Privacy Router – The DIY enterprise gateway

Setup: ~30 min (Docker Compose + Ollama). Hosting: on-premise, no data leakage. Detailed guide: self-hosting GDPR.

Concrete enterprise workflows:

• RBAC / Auth: reverse proxy (Traefik, Nginx, Authentik) with OIDC against your own Entra ID. Per-team configs as YAML, skill whitelists per team.
• Logging / Observability: OpenTelemetry collector → Loki/Grafana or Elastic. Privacy Router logs the classification decision per request (local vs. cloud) – audit trail for the DPO.
• Routing by model/vendor: Privacy Router decides per prompt: sensitive → local (Ollama, vLLM), generic → cheap cloud (Haiku, Mini), complex without PII → top cloud (Sonnet, GPT). Rules as YAML + ML classifier.
• Rate limiting: Nginx or Traefik middlewares with per-team limits, token quotas via LiteLLM behind it (stacks compose well).

Comparison matrix: privacy, compliance, latency & deployment

Side-by-side overview of all six enterprise gateway options across the dimensions that matter most for procurement, security and platform teams.

How to read it: "Privacy" = where prompt/response data physically lives. "Compliance" = certifications and controls available out of the box. "Latency" = added gateway overhead, not the underlying model. "Deployment" = where you can actually run it today.

Real-world deployment examples

Short, practical scenarios – which deployment model fits which company type?

Quick-Select: which enterprise gateway for which profile?

Migration path to Microsoft Scout (once GA)

Once Microsoft Scout ships, it typically won't replace all of the above. Realistic picture:

• LiteLLM → Scout: if Microsoft delivers on its multi-provider claim (open question), LiteLLM can be replaced for Azure-first shops.
• Portkey stays useful if you want cross-provider governance.
• Privacy Router stays essential – Scout is Azure-native and doesn't solve on-premise data routing.
• Kong & AWS Strands stay, since they cover specific requirements (mTLS, AWS compliance) Scout doesn't replace.

Till Freitag recommendation

For 80% of enterprises today: LiteLLM (multi-provider) + Portkey (governance) + Privacy Router (GDPR routing) – fully open, productive in 1–2 days, migratable to Scout. AWS-only shops take Strands / AgentCore. Regulated industries with mTLS mandate: Kong AI Gateway. Want logs in 5 minutes: Cloudflare AI Gateway as edge front.

The full market overview lives in the master article: The best OpenClaw alternatives 2026.

More on this topic: Coding-Agent Layer · Multi-Agent Layer · Self-Hosted & Privacy Layer · Microsoft Scout as OpenClaw gateway · Privacy Router Guide · Master article