Member of Technical Staff, Evals

Magic Dev

Where
Remote (San Francisco)
Pay
$200K to $550K a year

Posted 8 days ago· Checked live at Magic Dev 48 minutes ago

Live.Open on the employer's own site.

See how you score for this job

One sentence and your resume. We score it against what Magic Dev asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Magic Dev's own hiring page 48 minutes ago.
What is the job?
Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone.
What does it pay?
$200K to $550K a year, as the posting states.
Our verdict
Live. Open on the employer's own site.

What the job is

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone.

What you'll work on

  • Design and validate new evaluations and benchmarks across model training and deployment
  • Investigate model behavior and identify gaps in existing evaluations
  • Measure dataset quality and identify opportunities to improve training data
  • Improve evaluation correctness, reproducibility, and reliability
  • Audit and improve upon public benchmarks, research papers, and open-source implementations
  • Define metrics that accurately reflect model quality
  • Build tools that make evaluation results accessible and useful across Magic

What we're looking for

  • Experience developing model evaluations, benchmarks, or ML experiments
  • Strong programming skills and experience working with machine learning systems
  • Track record of owning technical projects end-to-end
  • Skepticism toward results that cannot be reproduced, validated, or explained
  • Experience analyzing model performance and interpreting experimental results
  • Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
  • Commitment to raising the standard of rigor in AI evaluations

Compensation, benefits, and perks (us)

  • Annual salary range between $200K - $550K depending on experience
  • Equity is a significant part of total compensation, in addition to salary
  • 401(k) plan with 6% salary matching
  • Generous health, dental, and vision insurance for you and your dependents
  • Unlimited paid time off
  • Visa sponsorship and relocation support for candidates moving to San Francisco
  • A small, fast-moving, highly collaborative team working on frontier AI systems

Similar open jobs

Jobs in San Francisco (11750)