Energy and carbon

The gateway reads power in one of two ways, and you can use both:

• Power per model. A query returns watts for each model. Each model's power is split between the requests it is serving at that moment, across every gateway replica. Nothing needs setting on the models, and the numbers follow your deployments as they scale.
• Power per node. A query returns watts for each node. Each model declares how many requests saturate a node, and a request is charged one of those slots at the average node's power. Use this when your metrics cannot be tied to a model.

A model that matches a series of the per-model query is charged from it. Any other model falls back to its per-node slot share, and a model with neither records zero.

Turn it on

1. 1

   Point the gateway at your power metrics

   In Settings › Energy, enter the Prometheus URL and a Power per model query, a Power per node query, or both. Choose Test the queries. It runs them once and shows the total power, which of your models match a per-model series, which series match no model, and the node count. A failure shows the Prometheus error as it came back.

2. 2

   Set your rates

   Enter the cost of a kWh, your grid's grams of CO₂ per kWh (0 leaves CO₂ out), and your data centre's PUE (default 1.0, IT power only). The poll interval defaults to 60 seconds. With per-model power, 15 to 30 seconds follows scaling more closely.

3. 3

   Switch it on and save

   Turn on Energy and save. The gateway picks the settings up without a restart.

4. 4

   Per node only: give each model its energy slots

   On each self-hosted model that has no per-model series, set Energy slots per node: how many of its requests one node serves at once when fully loaded, that is replicas per node × concurrent sequences per replica. Two server instances per node with 4 sequences each is 8. 0, the default, leaves the model out. A model with a per-model series ignores this field.

The Prometheus URL must be reachable from the gateway process. In the Compose stack that is http://prometheus:9090, not localhost:9090, which inside the gateway container is the gateway itself. The URL is checked against the gateway's outbound address policy when you test and when you save.

What each request records

Three values are stored on each request's usage row, next to its cost:

They are computed once, when the request completes. Changing a rate, PUE, query or slot count later changes only future requests; history is never rewritten or backfilled.

How it is attributed

Power per model

While a request is being served, it is charged its model's power divided by the number of requests that model is serving across the cluster:

energy_wh = ∫ model_watts(t) / in_flight(t) dt × PUE     (from admission to completion)

in_flight counts the model's admitted requests on every gateway replica. With shared fairshare slots the gateway reads it from the cluster-wide counts every 2 seconds. Without them, each replica assumes the others carry as many requests as it does. model_watts is the latest reading of the series, and a new reading applies from the moment it arrives.

If other clients send requests to the same model servers, the gateway cannot see them, and its own requests would be charged for theirs. Add a Requests per model query that returns how many requests each model's servers are handling, with the same label as the power query. in_flight is then the larger of the gateway's count and the servers' count, read at each poll.

For example, a model drawing 1,000 W serves request A alone for one hour, then A and B together for an hour, then B alone for an hour. A and B are each charged 1,500 Wh. Together they account for all 3,000 Wh the model drew.

Several models can share one series, such as two routes to the same server pool. Their requests then split that power between them.

Power per node

A request is charged its serving time's share of one slot, at the average power per node:

serving_ms     = total_ms − queue_wait_ms
watts_per_slot = cluster_watts / node_count / energy_slots_per_node × PUE
energy_wh      = watts_per_slot × serving_ms / 3,600,000

cluster_watts is sum() of your query and node_count is count() of it. For example, 44,800 W across 64 nodes is 700 W per node. At 8 slots per node and a PUE of 1.2 that is 105 W per slot, so a request served for 2.4 s is charged 0.07 Wh.

Limits of both

The figures are a lower bound for showback and sustainability reporting, not a metered bill:

• Idle power is not attributed to anyone. Per model, a model's power while it serves nothing is charged to nobody. Per node, a request only ever pays for its own slot.
• Per node, every node is charged at the cluster average, busy or not.
• Time spent waiting for a slot is free.
• Helper calls (vision, tool loops, drafts) record zero; their hardware time falls inside the main request's serving time.

A request that fails upstream still records energy for the time it was served, even when it is not billed.

Choosing a power query

Power per model

The query returns one series per model, named by a label (model unless you set Model label). A series matches a model when the label carries the model's name, its capacity Service, or its upstream model, checked in that order. Series that share a label value are summed.

On Kubernetes, the simplest source is a device exporter that labels each device with the pod using it, such as NVIDIA DCGM. Map the pod to a model with label_replace:

sum by (model) (
  label_replace(
    DCGM_FI_DEV_POWER_USAGE{namespace="llm"},
    "model", "$1", "pod", "vllm-(.+)-[a-z0-9]+-[a-z0-9]{5}"
  )
)

With only node power, give each pod its share of the node by the accelerators it requested, using kube-state-metrics. This charges a model for the whole node, including CPUs, memory and fans, in proportion to its accelerators:

sum by (model) (
  label_replace(
    sum by (namespace, pod, node) (
      kube_pod_container_resource_requests{namespace="llm", resource="nvidia_com_gpu"}
    )
    / on (node) group_left ()
      max by (node) (kube_node_status_allocatable{resource="nvidia_com_gpu"})
    * on (node) group_left ()
      max by (node) (node_power_watts),
    "model", "$1", "pod", "vllm-(.+)-[a-z0-9]+-[a-z0-9]{5}"
  )
)

Replace node_power_watts with your per-node gauge, relabelled so that its node label is node. If your pods carry a label naming the model, allow it in kube-state-metrics (--metric-labels-allowlist=pods=[your-label]) and join on kube_pod_labels instead of parsing pod names.

Requests per model

Most model servers export a gauge of the requests they are running. For vLLM, relabel its served model name to the label your power query uses:

sum by (model) (
  label_replace(vllm:num_requests_running, "model", "$1", "model_name", "(.+)")
)

Leave it empty when this gateway is the only client of its model servers.

Power per node

Any exporter with a power gauge works:

With a per-device metric, either aggregate it per node (sum by (node) (...)) or declare slots per device instead of per node. The arithmetic only needs cluster_watts / node_count to mean the power of the unit you counted slots against. Use label selectors to scope a mixed cluster, such as ipmi_dcmi_power_consumption_watts{cluster="prod"}.

Where the numbers appear

• Overview: an energy tile with kWh and CO₂ for the window.
• Reports: energy, CO₂ and power cost for the period, an Energy column in the breakdown, and Energy (kWh), CO₂ and energy cost columns in the export.
• Request logs: a request's panel shows its Wh and grams of CO₂.
• Management API and ClickHouse: energy sums beside cost_usd in usage queries and the daily rollup. See ClickHouse usage.

Management API

curl -X PUT http://localhost:9180/api/v1/settings/energy \
  -H "Authorization: Bearer $OBLETH_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"enabled": true, "prometheus_url": "http://prometheus:9090",
       "model_power_query": "sum by (model) (model_power_watts)", "model_power_label": "model",
       "model_requests_query": "sum by (model) (requests_running)",
       "power_query": "ipmi_dcmi_power_consumption_watts", "poll_interval_secs": 30,
       "energy_cost_per_kwh": 0.12, "carbon_g_per_kwh": 400, "pue": 1.2}'

An empty model_power_query or power_query turns that source off, and an empty model_requests_query counts only the gateway's own requests. The test call takes prometheus_url and any of power_query, model_power_query, model_power_label and model_requests_query. It returns cluster_watts and node_count for the per-node query (zero when not given) and, for the per-model query, models with total_watts, series, matched (each model, series, watts), unmatched_series, and requests_in_progress when a requests query is given. A model's slot count is energy_slots_per_node on the model create and update calls.