[Junior / Senior / Staff] Software Engineer, Inference / Compute Infrastructure Engineering

Together AI

Where
San Francisco
Pay
$160K to $280K a year
Type
Full time

Posted 4 months ago· Checked live at Together AI 11 minutes ago

Stale: open 126 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Together AI asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Together AI's own hiring page 11 minutes ago.
What is the job?
We're looking for a software engineer to build the Kubernetes-native control plane that provisions and runs our GPU inference fleet.
What does it pay?
$160K to $280K a year, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

We're looking for a software engineer to build the Kubernetes-native control plane that provisions and runs our GPU inference fleet.

Responsibilities

  • Build the provisioning state machine: design and implement the software that models the full lifecycle of a physical host from discovery, inference bring-up to GPU driver/CUDA stack, health validation, and decommission/RMA — as explicit, versioned states and transitions.
  • Build the self-service API: design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call — no ticket, no human in the loop.
  • Automate self-healing: detect degraded or failed nodes, drain them safely, trigger repair or replacement, and reintroduce healthy capacity into the pool automatically.
  • Own reliability of the pipeline: idempotency, retries, rollback, and drift detection so the provisioning system is as dependable as any other production service.
  • Partner with the inference/ML platform team: understand the cluster shapes they need — topology, interconnect, scheduling constraints — and encode them as first-class abstractions in the platform.
  • Engineer it like software: strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code — this is a product, not a collection of Ansible playbooks.

Core requirements (all levels)

  • Strong software engineering background in Go, Python, Rust, or similar — you write and test real software for a living.
  • Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent to run long-lived, manifest-driven workflows that survive failures and resume mid-execution.
  • Experience building software control planes or orchestration systems that model state and reconcile it over time (e.g., Kubernetes controllers/operators, custom reconciliation loops, workflow engines).
  • Experience with event-driven systems — designing and building software around message queues, event streams, or pub/sub (e.g., Kafka, NATS, SQS) rather than polling or cron-driven scripts.
  • A product mindset. You’ve built internal platforms or APIs consumed by other engineering teams and care about the developer experience of what you ship.

Nice to have

  • Exposure to bare-metal provisioning (PXE/iPXE, Redfish/IPMI, BMC) and/or networking fundamentals (VLANs, BGP, fabric design), or GPU/accelerator infrastructure.
  • Experience with GPU cluster software stacks (NCCL, CUDA, InfiniBand/RoCE).
  • Prior work at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization.
  • Systems programming in Rust or Go.

Compensation

We offer competitive compensation, startup equity, health insurance and other competitive benefits. The US base salary range for this full-time position is: $160,000 - $280,000 + equity + benefits. Our salary ranges are determined by location, level and role. Individual compensation will be determined by experience, skills, and job-related knowledge.

Application Policy

Please read each role carefully and apply to the position best suited to you. Candidates are limited to 6 applications within a 90-day period, and applications beyond that limit will be automatically rejected. You may apply to additional roles 90 days after your last application.

Similar open jobs

Software engineer jobs in San Francisco (1709)Jobs in San Francisco (11750)