PROJECT STUDY
Model Gateway & Evals
One door for every model call, one gate for every change.
An OpenAI-compatible model gateway with Rupiah budgets, provider failover and a guard for Indonesian identifiers, plus an evaluation gate that blocks prompt and model regressions in CI.
Explore the build
At a glance
- Industry
- Technology
- Capabilities
- 8
- Technologies
- 12
- Year
- 2026
The challenge
When several teams adopt language models on their own, model access spreads faster than anyone can govern it.
API keys end up scattered across applications, spending is discovered only on the invoice, personal data such as NIK, NPWP and phone numbers travels to outside providers, a slowing provider goes unnoticed, and a prompt or model change can quietly make answers worse.
Model Gateway & Evals is built for companies in Indonesia that run language models across teams. One gateway centralises access, spending and personal-data policy, and an evaluation harness turns answer quality into a release gate that every prompt or model change has to pass.
What we built
Model Gateway & Evals combines a policy-enforcing gateway, an evaluation harness and an operator console.
One OpenAI-compatible endpoint
Applications send chat completions to one Hono service on Node.js, streamed over Server-Sent Events or returned whole. Adapters for OpenAI, Anthropic, Google Gemini and a local Ollama endpoint sit behind one interface
Routing with fallback
Model aliases map to provider chains, with fallback on timeouts, 5xx responses and rate limits, a circuit breaker per provider, jittered retries and request deadlines
Policy per API key
Request and token limits per minute, a list of allowed models and a monthly budget in Rupiah, with each call's cost estimated before it is sent and reconciled with actual usage afterwards
A guard for Indonesian identifiers
NIK, NPWP, phone numbers, email addresses, card numbers and bank account numbers are detected before a request leaves, then blocked, redacted or tokenised according to the key's policy
Cache, metering and metrics
An exact-match cache with a time to live answers repeated requests, usage and a request log are kept in SQLite, and the gateway records OpenTelemetry-style traces and serves metrics in Prometheus format
A prompt registry
Prompt templates are versioned and released through labels, so a change to a prompt becomes a new version instead of an edit in place
Evaluations as a CI gate
JSONL datasets, graders from exact match to an LLM judge, a paired comparison with bootstrap confidence intervals and a command-line tool that fails the pipeline when a candidate regresses
An operator console
A React console for keys and budgets, usage and latency charts, the request log with a redaction preview, and eval runs compared side by side
One OpenAI-compatible endpoint
Applications send chat completions to one Hono service on Node.js, streamed over Server-Sent Events or returned whole. Adapters for OpenAI, Anthropic, Google Gemini and a local Ollama endpoint sit behind one interface
Routing with fallback
Model aliases map to provider chains, with fallback on timeouts, 5xx responses and rate limits, a circuit breaker per provider, jittered retries and request deadlines
Policy per API key
Request and token limits per minute, a list of allowed models and a monthly budget in Rupiah, with each call's cost estimated before it is sent and reconciled with actual usage afterwards
A guard for Indonesian identifiers
NIK, NPWP, phone numbers, email addresses, card numbers and bank account numbers are detected before a request leaves, then blocked, redacted or tokenised according to the key's policy
Cache, metering and metrics
An exact-match cache with a time to live answers repeated requests, usage and a request log are kept in SQLite, and the gateway records OpenTelemetry-style traces and serves metrics in Prometheus format
A prompt registry
Prompt templates are versioned and released through labels, so a change to a prompt becomes a new version instead of an edit in place
Evaluations as a CI gate
JSONL datasets, graders from exact match to an LLM judge, a paired comparison with bootstrap confidence intervals and a command-line tool that fails the pipeline when a candidate regresses
An operator console
A React console for keys and budgets, usage and latency charts, the request log with a redaction preview, and eval runs compared side by side
How it works
Check the key's policy
Every application calls the gateway with its own API key. Before anything leaves, the gateway checks the key's allowed models, its per-minute request and token buckets and its remaining Rupiah budget, then runs the PII guard in the mode the key specifies.
Route down the chain
The router resolves the requested alias to a chain of provider adapters and skips any provider whose circuit breaker is open. A repeated request can be answered from the exact-match cache, and the response streams back to the application with tokenised values restored.
Meter every call
Each call is priced from the model price table before it is sent, reconciled afterwards and written to the SQLite request log. Traces follow the OpenTelemetry style, and a metrics endpoint exposes the gateway's counters and latencies in Prometheus format.
Gate every change
Prompt templates live in a versioned registry. The evaluation tool runs a JSONL dataset against the baseline and the candidate, grades both the same way, compares them with bootstrap confidence intervals and exits with an error when the candidate regresses beyond the threshold, so CI stops the release.
Engineering insights
The gateway treats every model call as a policy decision before it becomes a network call, and every prompt or model change as a release that has to pass a test.
Stack by layer
- Frontend
- React
- TypeScript
- Vite
- Backend
- Node.js
- Hono
- TypeScript
- OpenAI-compatible API
- Server-Sent Events
- Data
- SQLite
- JSONL datasets
- AI
- OpenAI
- Anthropic Claude
- Google Gemini
- Ollama
- LLM-as-judge grading
- Bootstrap confidence intervals
- Infrastructure
- Prometheus metrics
- OpenTelemetry-style traces
- Biome
Aliases, chains and breakers
Applications ask for a model alias rather than a provider, and the gateway resolves it to a chain. A timeout, a 5xx response or a rate limit moves the call to the next provider, retries back off with jitter inside the request's deadline, and a circuit breaker per provider stops sending traffic to one that keeps failing
Budgets priced before the call
Each call is estimated from the model price table before it is sent and reconciled with actual usage afterwards, so a request that would break the key's monthly Rupiah budget can be refused instead of discovered on the invoice. Request and token buckets per minute keep one application from crowding out the rest
Identifiers checked by structure
A NIK is validated by its region code and its birth date, in which 40 is added to the day for women, and a card number must pass the Luhn check, alongside patterns for NPWP, +62 and 08 phone numbers, email addresses and bank accounts. Tokenised values travel as reversible placeholders that the gateway restores in the response, so the provider never receives a detected value and the application still gets a complete answer
Quality as a release gate
Baseline and candidate answer the same cases and are graded by exact match, regex, JSON Schema validity, required citation markers or an LLM judge with a rubric that itself runs through the gateway. A bootstrap confidence interval on the paired difference decides whether the candidate regressed beyond the threshold, and the tool's exit code turns that decision into a CI gate
Testable without a live model
Every provider sits behind one adapter interface, and a deterministic fake provider and a fake judge run the test suite. Streaming, fallback through an injected failing provider and the eval gate are all exercised offline, without an API key
Design system
Operator console is a dense, keyboard-friendly interface for the platform engineers who run the gateway, with light and dark themes drawn from the same tokens.
Colour palette
- Graphite
#111827Text in the light theme and surfaces in the dark theme - Slate surface
#f8fafcPage, panel and table surfaces in the light theme - Violet
#6d28d9The accent, reserved for actions that call a model - Pass green
#16a34aPassing eval cases and gates, and successful requests - Fail red
#dc2626Failing eval cases and gates, errors and blocked requests - Warning amber
#d97706Warnings, such as a budget close to its limit
Colour
Graphite and slate keep long tables of keys, requests and eval cases readable, violet is reserved for actions that call a model, and green, red and amber mean only pass, fail and warning
Typography
Inter for the interface and JetBrains Mono for payloads and IDs, both bundled through Fontsource, so request bodies and key IDs keep look-alike characters such as 0 and O distinct
Components
Dense tables for keys, budgets and the request log, usage and latency charts, a redaction preview of the prompt as it was sent, and a comparison view for eval runs, all usable from the keyboard
Themes
Light and dark themes map the same semantic tokens, so every colour keeps its meaning on a laptop by day and on a monitoring screen at night
Where this fits
Companies running language models across several teams can put one gateway and one release gate in front of them. Explore our AI solutions and IT consulting, or talk to us about governing model access in your company.
KEEP EXPLORING
Another problem. A different approach.
RAG Finance
A research assistant for Indonesian finance and tax regulations that answers from indexed sources, cites article and page evidence, and says so when the evidence falls short.
Store & Workforce Vision
A privacy-aware store operations platform for occupancy, queue and zone events, with configurable alerts, manager review and aggregate reporting.
Let’s connect the dots
Let's put these ideas to work.
Share a process, product, or problem you want to improve. We will work through the possibilities together.