OpenRouter Production Inference Lab is a Python application for inspecting the operational path of an LLM request. Its single-screen interface shows streamed output alongside routing strategy, fallback outcome, cache behavior, cost, latency, provider metadata, and evaluation results.
The interface preserves incomplete telemetry as an explicit state. Missing provider metadata is shown as unavailable, and tracing remains optional.
Core capabilities
Routing strategies
Compare default, cost-oriented, latency-oriented, and custom configurations. Each run records the selected routing strategy with the response telemetry.
Fallback inspection
Trigger failure scenarios and inspect the failed primary attempt and successful fallback in the same run.
Cost, cache, and provider metadata
Inspect token usage, request cost, cache hits, cache writes, model selection, and provider information when OpenRouter returns them. Missing values keep an explicit unavailable state, preserving the distinction between absent data and a zero value.
Streamed responses
Model output arrives progressively while the application captures the request telemetry.
Checked-in evaluations
Run a checked-in suite of support-style and adversarial cases across multiple strategies. Results include pass or fail reasoning, matched or missed terms, latency, cost, and trace state.
Optional Langfuse tracing
When credentials are configured, Langfuse records traces and provides evaluation scores. The interface identifies when tracing is unavailable.
Local run comparison
SQLite stores recent runs for side-by-side comparison of fallback outcomes, latency, cost, and available metadata.
Technology
- Python 3.12 for the application
- NiceGUI for the browser interface
- OpenRouter Chat Completions API through direct HTTPS and streamed SSE responses
- httpx for asynchronous inference requests
- Langfuse for optional tracing and evaluation-score retrieval
- SQLite for local run history
- uv for reproducible Python environments
- pytest and Ruff for tests and linting
- Docker and Fly.io configuration for optional deployment