Software Engineer, Hardware Health

OpenAI

Where
San Francisco

Posted 6 months ago· Checked live at OpenAI 1 hour ago

Stale: open 191 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what OpenAI asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on OpenAI's own hiring page 1 hour ago.
What is the job?
The Hardware Health and Observability team owns the end-to-end health lifecycle of OpenAI’s global compute fleet. Our mission is to maximize healthy, usable compute across accelerator vendors, generations, cloud providers, and regions through reliable health signals, automated remediation, and scalable operational tooling.
What does it pay?
The posting doesn't say.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

The Hardware Health and Observability team owns the end-to-end health lifecycle of OpenAI’s global compute fleet.

Our mission is to maximize healthy, usable compute across accelerator vendors, generations, cloud providers, and regions through reliable health signals, automated remediation, and scalable operational tooling.

In this role, you will

  • Define and maintain health signals across GPUs, CPUs, networking, and platform infrastructure.
  • Build and evolve health checks that detect, remediate, and verify failures at scale.
  • Ensure critical health checks execute with minimal latency to maximize workload uptime.
  • Investigate hardware failures and system-level issues across large-scale compute environments.
  • Own node lifecycle workflows including drain, quarantine, repair, RMA, and return-to-service processes.
  • Build automation and tooling that enables global cluster management with minimal manual intervention.
  • Partner with workload, reliability, and provider teams to integrate health signals into training and inference systems.

You might thrive in this role if you have

  • 7+ years of industry experience in software or infrastructure engineering.
  • Strong proficiency with Python and shell scripting.
  • Experience building large-scale distributed systems or infrastructure platforms.
  • Comfort digging into noisy operational data using SQL, PromQL, or similar tooling.
  • Experience building reproducible analyses and operational tooling.
  • Strong systems debugging and operational instincts with an ownership mindset.

Bonus if you have

  • Experience with low-level hardware systems and Linux tooling (e.g. PCIe, InfiniBand, RoCE, networking, power management, kernel performance tuning, FW/SW debugging).
  • Experience operating or debugging large-scale GPU or accelerator clusters.
  • Expertise in network operations, observability, or systems telemetry.
  • Experience with automated remediation systems or fleet lifecycle management.
  • Experience improving reliability, utilization, or workload uptime in distributed compute environments.

Similar open jobs

Software engineer jobs in San Francisco (1709)Jobs in San Francisco (11750)