# Model Gateway & Evals > An OpenAI-compatible model gateway with Rupiah budgets, provider failover and a guard for Indonesian identifiers, plus an evaluation gate that blocks prompt and model regressions in CI. One door for every model call, one gate for every change. - URL: https://weaveward.com/work/llm-gateway-evaluation-platform - Year: 2026 - Industry: Technology - Capabilities: AI solutions, IT consulting - Technologies: TypeScript, Node.js, Hono, React, Vite, SQLite, Server-Sent Events, OpenAI, Anthropic Claude, Google Gemini, Ollama, Prometheus metrics ## The challenge When several teams adopt language models on their own, model access spreads faster than anyone can govern it. API keys end up scattered across applications, spending is discovered only on the invoice, personal data such as NIK, NPWP and phone numbers travels to outside providers, a slowing provider goes unnoticed, and a prompt or model change can quietly make answers worse. Model Gateway & Evals is built for companies in Indonesia that run language models across teams. One gateway centralises access, spending and personal-data policy, and an evaluation harness turns answer quality into a release gate that every prompt or model change has to pass. ## What we built Model Gateway & Evals combines a policy-enforcing gateway, an evaluation harness and an operator console. - **One OpenAI-compatible endpoint.** Applications send chat completions to one Hono service on Node.js, streamed over Server-Sent Events or returned whole. Adapters for OpenAI, Anthropic, Google Gemini and a local Ollama endpoint sit behind one interface - **Routing with fallback.** Model aliases map to provider chains, with fallback on timeouts, 5xx responses and rate limits, a circuit breaker per provider, jittered retries and request deadlines - **Policy per API key.** Request and token limits per minute, a list of allowed models and a monthly budget in Rupiah, with each call's cost estimated before it is sent and reconciled with actual usage afterwards - **A guard for Indonesian identifiers.** NIK, NPWP, phone numbers, email addresses, card numbers and bank account numbers are detected before a request leaves, then blocked, redacted or tokenised according to the key's policy - **Cache, metering and metrics.** An exact-match cache with a time to live answers repeated requests, usage and a request log are kept in SQLite, and the gateway records OpenTelemetry-style traces and serves metrics in Prometheus format - **A prompt registry.** Prompt templates are versioned and released through labels, so a change to a prompt becomes a new version instead of an edit in place - **Evaluations as a CI gate.** JSONL datasets, graders from exact match to an LLM judge, a paired comparison with bootstrap confidence intervals and a command-line tool that fails the pipeline when a candidate regresses - **An operator console.** A React console for keys and budgets, usage and latency charts, the request log with a redaction preview, and eval runs compared side by side ## How it works ### Check the key's policy Every application calls the gateway with its own API key. Before anything leaves, the gateway checks the key's allowed models, its per-minute request and token buckets and its remaining Rupiah budget, then runs the PII guard in the mode the key specifies. ### Route down the chain The router resolves the requested alias to a chain of provider adapters and skips any provider whose circuit breaker is open. A repeated request can be answered from the exact-match cache, and the response streams back to the application with tokenised values restored. ### Meter every call Each call is priced from the model price table before it is sent, reconciled afterwards and written to the SQLite request log. Traces follow the OpenTelemetry style, and a metrics endpoint exposes the gateway's counters and latencies in Prometheus format. ### Gate every change Prompt templates live in a versioned registry. The evaluation tool runs a JSONL dataset against the baseline and the candidate, grades both the same way, compares them with bootstrap confidence intervals and exits with an error when the candidate regresses beyond the threshold, so CI stops the release. ## Engineering insights The gateway treats every model call as a policy decision before it becomes a network call, and every prompt or model change as a release that has to pass a test. - **Aliases, chains and breakers.** Applications ask for a model alias rather than a provider, and the gateway resolves it to a chain. A timeout, a 5xx response or a rate limit moves the call to the next provider, retries back off with jitter inside the request's deadline, and a circuit breaker per provider stops sending traffic to one that keeps failing - **Budgets priced before the call.** Each call is estimated from the model price table before it is sent and reconciled with actual usage afterwards, so a request that would break the key's monthly Rupiah budget can be refused instead of discovered on the invoice. Request and token buckets per minute keep one application from crowding out the rest - **Identifiers checked by structure.** A NIK is validated by its region code and its birth date, in which 40 is added to the day for women, and a card number must pass the Luhn check, alongside patterns for NPWP, +62 and 08 phone numbers, email addresses and bank accounts. Tokenised values travel as reversible placeholders that the gateway restores in the response, so the provider never receives a detected value and the application still gets a complete answer - **Quality as a release gate.** Baseline and candidate answer the same cases and are graded by exact match, regex, JSON Schema validity, required citation markers or an LLM judge with a rubric that itself runs through the gateway. A bootstrap confidence interval on the paired difference decides whether the candidate regressed beyond the threshold, and the tool's exit code turns that decision into a CI gate - **Testable without a live model.** Every provider sits behind one adapter interface, and a deterministic fake provider and a fake judge run the test suite. Streaming, fallback through an injected failing provider and the eval gate are all exercised offline, without an API key ## Design system Operator console is a dense, keyboard-friendly interface for the platform engineers who run the gateway, with light and dark themes drawn from the same tokens. - **Colour.** Graphite and slate keep long tables of keys, requests and eval cases readable, violet is reserved for actions that call a model, and green, red and amber mean only pass, fail and warning - **Typography.** Inter for the interface and JetBrains Mono for payloads and IDs, both bundled through Fontsource, so request bodies and key IDs keep look-alike characters such as 0 and O distinct - **Components.** Dense tables for keys, budgets and the request log, usage and latency charts, a redaction preview of the prompt as it was sent, and a comparison view for eval runs, all usable from the keyboard - **Themes.** Light and dark themes map the same semantic tokens, so every colour keeps its meaning on a laptop by day and on a monitoring screen at night ## Where this fits Companies running language models across several teams can put one gateway and one release gate in front of them. Explore our [AI solutions](/services/ai-solutions) and [IT consulting](/services/it-consulting), or [talk to us](/contact) about governing model access in your company.