ehtishammubarik/truthscale

TGI collector, and what to do about the KV cache it does not expose 19 days ago

help wanted collector

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

Alerting rules and a webhook sink, each with the reason it exists 19 days ago

help wanted

ehtishammubarik/truthscale

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

Ceiling store and ageing: a stale ceiling is worse than no ceiling 19 days ago

help wanted signal correctness

ehtishammubarik/truthscale

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

A worked example: find a starved GPU in a real cluster, end to end about 1 month ago

good first issue docs

ehtishammubarik/truthscale

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

The decision function: when does adding a GPU actually help? about 1 month ago

help wanted signal correctness

ehtishammubarik/truthscale

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

Helm chart and DaemonSet, with the Prometheus exporter about 1 month ago

good first issue chart

ehtishammubarik/truthscale

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

vLLM collector: KV cache, queue depth, and token throughput about 1 month ago

help wanted collector

ehtishammubarik/truthscale

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

DCGM collector: read the real fields, and report what the driver will not give you about 1 month ago

help wanted collector needs-gpu

ehtishammubarik/truthscale

0

Your GPU is not 90% busy. truthscale measures what each node can actually deliver, reports the gap, and makes that the signal you scale on.

Go

#autoscaling#cloud-native#dcgm#finops#go#gpu#gpu-monitoring#gpu-utilization#inference-serving#kubernetes#kv-cache#llm-inference#mlops#nvidia#observability#platform-engineering#prometheus#rke2#sre#vllm

Open Issues Need Help