Staff ML Engineer, Frontier AI

Ambiencehealthcare

Where
San Francisco

Posted 8 months ago· Checked live at Ambiencehealthcare 1 hour ago

Stale: open 247 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Ambiencehealthcare asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Ambiencehealthcare's own hiring page 1 hour ago.
What is the job?
Here at Ambience, we never set out to be just another scribe. We’re building the AI intelligence platform that restores humanity to healthcare and drives meaningful ROI for health systems across the country. Our technology helps providers focus on delivering great care by removing the administrative burden that pulls them away from patients and away from their most impactful work.
What does it pay?
The posting doesn't say.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Here at Ambience, we never set out to be just another scribe. We’re building the AI intelligence platform that restores humanity to healthcare and drives meaningful ROI for health systems across the country.

Our technology helps providers focus on delivering great care by removing the administrative burden that pulls them away from patients and away from their most impactful work.

What you’ll own

  • Define the AI Quality and Evaluation Strategy: Establish how we measure production AI quality across LLM and agentic systems, including automated graders, regression testing, human evaluation, failure taxonomies, and offline-to-online validation.
  • Drive Production Model Improvement: Identify high-impact failure modes across products and lead improvements across prompting, retrieval, context, routing, data, and post-training. Design experiments that determine the right intervention and prove that improvements hold in production.
  • Advance Post-Training Capabilities: Shape how Ambience uses techniques such as supervised fine-tuning, preference optimization, reinforcement learning, distillation, and synthetic data to improve model behavior.
  • Architect Agentic AI Systems: Set technical direction for production systems involving tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
  • Build Self-Improving AI Loops: Design the systems that turn production failures and user feedback into better datasets, evaluations, and model behavior, creating scalable improvement flywheels across products.
  • Shape the AI Roadmap: Co-develop the 12-month technical roadmap, balancing near-term product wins with longer-term research and platform investments.
  • Raise the Technical Bar: Lead technical reviews, mentor engineers, establish best practices, and create reusable systems and frameworks that improve the effectiveness of the broader AI team.
  • Stay at the Cutting Edge: Distill insights from recent research in LLMs, agents, post-training, NLP, speech, and multimodal AI, and drive experiments that keep Ambience at the forefront of clinical AI.

Who you are

  • Expert in Production AI Systems

7+ years in production ML, research engineering, or applied AI.

Deep expertise with modern LLMs, transformers, and complex production AI systems.

Have led or materially shaped consequential AI systems with real production impact.

Able to reason across models, data, orchestration, evaluation, serving, and product behavior.

  • Deep Evaluation Rigor

Significant experience designing evaluations for LLMs, agents, or other complex AI systems.

Can turn ambiguous product-quality problems into robust metrics, datasets, and experiments.

Experienced with grader bias, leakage, contamination, misleading aggregate metrics, regression detection, and offline-online mismatch.

Knows how to establish whether a model or system improvement actually translates into better user outcomes.

  • Agentic Systems Expertise

Have built or architected production systems involving multiple models, tools, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.

Understand how to evaluate and debug complex agent behavior across model, tool, and system boundaries.

Comfortable making architectural decisions that span multiple products or teams.

  • Post-Training and Model Improvement Expertise

Have materially improved model behavior through post-training or model adaptation techniques such as supervised fine-tuning, preference optimization, reinforcement learning, distillation, synthetic data, or related approaches.

Understand how data quality, objective design, evaluation, and post-training choices affect downstream model behavior.

Know when post-training is the right intervention versus prompting, retrieval, context optimization, routing, or changes to the surrounding system.

Have taken model improvements from experimentation through rigorous evaluation and production deployment.

  • Production-Grade Software Engineer

Proficient in Python and modern ML frameworks; PyTorch preferred.

Comfortable with modern MLOps, CI/CD, observability, and containerized deployments.

Remains deeply hands-on: writes code, inspects traces, analyzes failures, and prototypes solutions.

  • Data-Centric AI Developer

Skilled at building large, high-quality datasets and feedback loops.

Experienced using production failures, user feedback, and active learning to improve model and system quality.

Able to design data and evaluation infrastructure that creates leverage across multiple teams.

  • Effective Technical Leader

Able to work closely with clinicians, product managers, researchers, and engineers.

Strong communicator who can simplify complex AI concepts and build alignment around technical decisions.

Comfortable operating in ambiguity, identifying the right problems to solve, and driving execution across multiple workstreams.

  • Mission-Aligned

Passion for healthcare or other mission-driven industries.

Thrives in a fast-paced, early-stage environment.

Takes broad ownership of technical outcomes and raises the effectiveness of the people around them.

Nice-to-Haves

  • Experience with realtime voice, conversational AI, or multimodal systems.
  • Prior work in healthcare, clinical AI, or other regulated, high-stakes industries.
  • Experience interviewing or hiring ML engineers.
  • Open-source contributions to ML, agent, evaluation, or post-training tooling.

Life at Ambience

Working at Ambience means opting into a high-ownership, high-trust environment built for people who want to grow fast, operate decisively and focus on work that matters. This could be the right place for you if you want to

  • Work on mission-critical AI technology that directly improves clinicians’ day-to-day lives and health system financial health across some of the most complex, high-stakes workflows in the world.
  • Join a “dream team” culture where we hire exceptional people, expect exceptional outcomes and invest deeply in feedback and continuous growth. We operate as a championship team, and that means being ok with hard, uncomfortable, ambiguous problems that lead to real greatness.
  • Operate with real ownership and accountability in an environment where there are no bystanders: If something is broken, we fix it! You will have meaningful autonomy and be expected to drive work to completion.

To help you do your best work, we pair these expectations with benefits intentionally designed to help you feel supported and safe at Ambience and beyond. Some of our key benefits include

  • Comprehensive medical, dental, and vision coverage for you and your dependents
  • 401(k) with a company match of up to 3% of base salary
  • A remote-friendly culture (with a San Francisco HQ) and full equipment provisioning to ensure you can work effectively from wherever you’re based.
  • Parental leave to support your family needs
  • Annual company-wide off-sites, team off-sites and regular team lunches and all-hands gatherings, with travel, lodging and meals covered
  • Flexible time off with no annual cap, company-wide holidays and an annual holiday shutdown from December 24–January 1 designed to support real rest and long-term sustainability.

Ambience Healthcare is an equal opportunity employer and is committed to building a diverse and inclusive workplace. We do not discriminate on the basis of race, color, religion, sex, gender identity or expression, sexual orientation, national origin, age, disability, veteran status, genetic information, or any other legally protected status. We encourage applicants from all backgrounds to apply.

Ambience is committed to supporting every candidate’s ability to fully participate in our hiring process. If you need any accommodations during your application or interviews, please reach out to our Recruiting team at accommodations@ambiencehealthcare.com recruiting@ambiencehealth.com. We’ll handle your request confidentially and work with you to ensure an accessible and equitable experience for all candidates.

Ambience Healthcare has become aware of scams targeting jobseekers with fake jobs and even interviewing people. Our emails will always come from @ambiencehealthcare.com. We would never our ask candidates to download apps or make any form of payment(s). If you are contacted through WhatsApp, Telegram, similar but fake email domains, or asked to make a payment, these contacts are not legitimate. Report the issue immediately to LinkedIn and the FBI https://www.fbi.gov/how-we-can-help-you/scams-and-safety.

Similar open jobs

Machine learning engineer jobs in San Francisco (584)Jobs in San Francisco (11741)