Research Engineer, Evals

Intrinsic Safety

Where
San Francisco
Pay
$100 an hour

Posted 7 months ago· Checked live at Intrinsic Safety 2 hours ago

Stale: open 233 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Intrinsic Safety asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Intrinsic Safety's own hiring page 2 hours ago.
What is the job?
At Variance, we are teaching machines to make the hardest judgment calls at scale. That means building AI agents for the high-stakes gray area of risk investigations, fraud, and identity reviews. We’re a small, talent-dense team in San Francisco working on a problem at the edge of what AI systems can reliably do: making good decisions in messy, adversarial, real-world environments.
What does it pay?
$100 an hour, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

At Variance, we are teaching machines to make the hardest judgment calls at scale. That means building AI agents for the high-stakes gray area of risk investigations, fraud, and identity reviews.

We’re a small, talent-dense team in San Francisco working on a problem at the edge of what AI systems can reliably do: making good decisions in messy, adversarial, real-world environments.

What you’ll do

  • Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows
  • Design and run offline and online evals that measure model performance on real customer tasks, not just abstract benchmarks
  • Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling
  • Study where models and agents break, and turn those failures into better evals, better datasets, and better training loops
  • Build reusable evaluation tools and quality building blocks that can be used across different product surfaces and workflows
  • Partner closely with research, engineering, product, and design to improve system quality through rigorous experimentation
  • Help create a strong culture of scientific experimentation, clear measurement, and continuous iteration
  • Push the boundary of how AI systems are evaluated in regulated, adversarial, and high-consequence environments

Preferred background

  • Experience training, evaluating, or improving modern ML systems
  • Strong programming skills and comfort working in research-heavy codebases
  • Experience building benchmarks, datasets, evaluation pipelines, or quality systems
  • Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas
  • Ability to design clean experiments and draw reliable conclusions from noisy results
  • Strong engineering judgment and a bias toward building
  • Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains

What we offer

  • Competitive salary and meaningful equity
  • Platinum-level medical, dental, and vision insurance
  • Unlimited PTO, sick leave, and parental leave
  • Up to $100 per month in reimbursement for personal health and wellness expenses
  • 401(k) plan

Similar open jobs

Machine learning engineer jobs in San Francisco (586)Jobs in San Francisco (11750)