Skip to content
Helios Product

Training, evaluation, and observability on one LLM runtime.

Helios replaces four fragmented tools with a single, repo-aware runtime — so your platform team owns the loop from training job to production trace on the same run IDs. See it live with our solutions engineers in 30 minutes.

Book a 30-minute demo View pricing
  • SOC 2Type II since Mar 2024
  • 2.1MSDK downloads
  • 99.97%trailing 12-mo uptime
  • a16zSeries A lead
01 · Training Runtime

Training as a first-class runtime primitive, not a notebook accessory.

Helios Training Runtime replaces your experiment tracker, your custom Airflow DAGs, and your internal scheduler with a single repo-aware job system. Every run is reproducible, every artifact is versioned, and every checkpoint is reachable from the same UI your eval suite reads from.

Helios training run config terminal
Reproducibility

Repo-aware job specs with sealed environments.

Every training job is defined as code in your repository — model, dataset revision, seed, hardware shape, and dependency lock all pinned to the commit that triggered the run. No more "works on my cluster" reruns.

  • Hermetic container builds with reproducible lockfiles
  • Dataset and checkpoint versions addressed by content hash
  • One-command rehydrate of any historical run from cold storage
Training scheduler dashboard
Orchestration

A scheduler that understands preemption and quota.

Helios schedules across H100 pools with topology-aware packing, gang-scheduled multi-node jobs, and graceful preemption that checkpoints and resumes instead of killing. Teams running Notion- and Replit-scale workloads rely on it daily.

  • Topology-aware GPU packing with NVLink locality
  • Gang-scheduled multi-node jobs across regions
  • Preemptive fair-share queues with checkpoint resume
Training runtime benchmark table
Throughput

Latency-engineered for 70B-class models on commodity H100s.

Our inference-grade runtime optimizations extend into training: overlapping dataloader I/O, fused optimizer steps, and async checkpoint flushing keep utilization above 92% on standard 8×H100 nodes — without bespoke kernel work on your side.

  • Overlapped dataloader and optimizer step fusion
  • Async checkpoint writes that don't stall the loop
  • Drop-in sharding for FSDP, DeepSpeed, and Megatron configs
Experiment comparison view
Comparison

First-class experiment diffing, not a spreadsheet export.

Compare any two runs across loss curves, eval suites, hyperparameter diffs, and dataset slices in one view. Promote the winning config to staging without leaving the page. Replaces the manual "screenshot the W&B tab" ritual that slows your team down.

  • Side-by-side run diff with hyperparameter highlighting
  • Eval suite pass-rate breakdown per dataset slice
  • One-click promotion to staging with the same run ID
02 · Evaluation Suite

Evaluation as a gated pipeline step, not a spreadsheet.

Helios Evaluation Suite unifies offline datasets, online evals-on-inference, and LLM-as-judge workflows behind one versioned, CI-style contract. Every eval run is reproducible, every failure is attributable to a dataset slice, and every promotion is gated — not annotated after the fact.

Versioned eval dataset browser
Datasets

Versioned datasets with slice-level provenance.

Every eval dataset in Helios is content-addressed and versioned. Tag slices, attach licenses, and trace any eval failure back to the exact row and the run that produced the candidate. LangSmith spreadsheets and bespoke eval scripts stop being the source of truth.

  • Content-hashed dataset versions with lineage graph
  • Slice tags for topic, language, and safety category
  • License and PII metadata enforced at upload
Eval CI pipeline view
CI Integration

Evals that block the merge, not decorate the post-mortem.

Helios evals run as a first-class stage in your CI. Gate pull requests on regression thresholds, fail the deploy if safety evals drop, and stream progress into the PR. The eval contract becomes code, not a weekly Notion page.

  • Native GitHub, GitLab, and Buildkite integrations
  • Threshold-based gating with per-slice failure rules
  • PR annotations linking failures to offending dataset rows
LLM-as-judge configuration
LLM-as-Judge

Calibrated LLM-as-judge with explicit rubrics.

Define rubrics in code, score with the judge of your choice, and watch agreement metrics in real time. Helios tracks inter-rater agreement against human spot-checks so you know when your judge has drifted — instead of finding out in production.

  • Versioned rubrics with code-defined scoring prompts
  • Cohen's kappa and drift alerts against human spot-checks
  • Bring-your-own judge with cost and latency ceilings
Online evals dashboard
Online Evals

Evals-on-inference, not evals-on-staging.

Sample production traffic, score it with the same rubric suite you use offline, and watch the dashboard update in real time. When the online eval drifts, you see the slice and the model version that caused it — without rebuilding the eval pipeline from scratch.

  • Reservoir sampling of live traces with the same rubric suite
  • Real-time dashboards tied to the same run IDs as training
  • Auto-rollback hooks when online evals breach threshold
03 · Observability

Inference-grade telemetry tied to the same run IDs as training.

Helios Observability is the inference-side counterpart to Training Runtime and Evaluation Suite — tracing, p99 latency dashboards, hallucination flags, and cost telemetry, all keyed by the same run identifier you used in training. No separate vendor, no stitched-together IDs.

Distributed LLM trace view
Tracing

End-to-end traces across prompt, retrieval, and model.

Every inference call is broken into spans — prompt build, retrieval, model, post-processing — with token counts, latencies, and cache hits on each. Drill from a p99 spike to a specific span in two clicks. Helicone-grade cost telemetry ships in the box.

  • Span-level token, latency, and cache attribution
  • Trace search by prompt hash, model version, and user cohort
  • Cost telemetry broken down by token, request, and customer
Latency dashboard
Latency

p99 under 180ms for 70B on commodity H100s.

Our latency-engineered inference layer delivers sub-180ms p99 for 70B-parameter models on standard H100 clusters, with the kind of SLO dashboards your SREs actually trust. Alerts page on threshold breach with the run ID and the model version attached.

  • SLO dashboards with p50, p95, p99, and burn-rate alerts
  • Threshold paging tied to the deployment that introduced drift
  • Side-by-side latency comparison across model versions
Hallucination flagging panel
Quality

Hallucination and quality flags on the same trace.

Flag hallucinations, refusals, and low-quality completions directly on the trace that produced them. Promote a flagged span into a regression eval with one click — closing the loop between production quality issues and the next training cycle.

  • Inline hallucination and refusal flags on every trace
  • One-click promotion of a flagged span into the eval suite
  • Cohort filters by user, prompt template, and model version
Cost telemetry dashboard
Cost

Cost telemetry that maps spend back to features.

See per-feature, per-customer, and per-prompt-template cost. Set budget alerts that page the team when a feature exceeds its token allowance — and click straight through to the traces that drove the overspend. Finance gets a bill you can actually audit.

  • Per-feature and per-customer token and cost attribution
  • Budget alerts tied to deployment and feature flag
  • CSV export that matches the chart, not a different aggregation
By the numbers

The credibility gap between Helios and category incumbents.

1,840+ paying teams running Helios in production
2.1M cumulative SDK downloads across Python, TS, Rust
99.97% trailing 12-month uptime, public status page
4.7× faster time-to-ship for reliable AI features
Production users

Named AI product teams trust Helios in production.

From research-adjacent labs to consumer AI platforms shipping at scale.

notion
replit
jasper
character.ai
huggingface / enterprise
anthropic-adjacent labs
vercel / evalops
+ 1,832 teams
Read customer stories Book a 30-minute demo