Positioning with Aibrix

Aibrix is an open-source inference gateway that handles replica selection for vLLM clusters: routing requests to the least-loaded pod, power-of-two choices, prefix-cache affinity, throughput-weighted routing. It is excellent at this.

Aibrix deliberately does not handle multi-tenant identity, API key authentication, weighted fairshare admission, or cost accounting. obleth deliberately does not handle replica selection. They compose cleanly.

Division of responsibility

Client
  ↓ HTTPS
HAProxy  (TLS + round-robin across obleth pods)
  ↓ HTTP
obleth  ←── owns this layer:
  • API key auth + tenant resolution
  • Weighted fairshare admission
  • Token budget enforcement
  • Response cache
  • MCP gateway
  • Cost accounting + telemetry
  ↓ HTTP (OpenAI-compatible)
Aibrix  ←── owns this layer:
  • Pod/replica selection
  • KV-cache affinity routing
  • Prefix-cache routing
  • Per-model RPS routing
  • Least-loaded replica
  ↓ HTTP
vLLM replicas

obleth decides who sends and at what priority. Aibrix decides which pod serves it.

Connecting obleth to Aibrix

Point OBLETH_UPSTREAM_BASE_URL at your Aibrix gateway endpoint:

OBLETH_UPSTREAM_BASE_URL=http://aibrix-gateway:8080/v1

obleth proxies the OpenAI-compatible request to Aibrix. Aibrix sees a normal inference request (the Authorization header is forwarded as-is, or overridden with the model's api_key if configured in obleth's model registry).

For per-model upstream overrides, use the model registry:

curl -X POST http://localhost:9180/api/v1/models \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "model_name": "llama-3-70b",
    "description": "Llama 3 70B instruct route behind Aibrix",
    "upstream_model": "llama-3-70b-instruct",
    "api_base": "http://aibrix-gateway:8080",
    "input_cost_per_token": 0.0000005,
    "output_cost_per_token": 0.0000015,
    "context_window": 131072,
    "enabled": true
  }'

Model admission weights

Models carry an admission_weight (default 100) that scales the tenant's weight during fairshare admission. It is a percentage, not a raw multiplier:

effective_weight = round(tenant.weight * model.admission_weight / 100)   # minimum 1

admission_weight=100 leaves the tenant's weight untouched. Raising it above 100 makes traffic to that model burn fair share more slowly, so it wins slots sooner under contention; lowering it below 100 does the reverse. Use it to give a model a larger or smaller relative share of gateway capacity when demand exceeds supply, independently of who is calling it.

Without Aibrix

obleth works with any OpenAI-compatible upstream. Point OBLETH_UPSTREAM_BASE_URL at:

• A raw vLLM service
• LiteLLM (provider abstraction without fairshare)
• Any other compatible API
• The bundled benchmark fixture backend (for dev/demo)

Aibrix is the recommended pairing for self-hosted GPU clusters, not a hard dependency.