Member of Technical Staff, Evals

Handshake

Where
San Francisco, CA
Type
Full time

Posted 51 days ago· Checked live at Handshake 2 hours ago

Stale: open 51 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Handshake asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Handshake's own hiring page 2 hours ago.
What is the job?
Handshake's mission is to organize expert human knowledge to advance the AI economy. Handshake AI works directly with frontier labs on their most consequential data, evaluation, and post-training challenges, building the systems that turn expert human knowledge into the data and evaluations that make frontier models better.
What does it pay?
The posting doesn't say.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Handshake's mission is to organize expert human knowledge to advance the AI economy. Handshake AI works directly with frontier labs on their most consequential data, evaluation, and post-training challenges, building the systems that turn expert human knowledge into the data and evaluations that make frontier models better.

What you'll do

  • Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments.
  • Develop reward models, programmatic verifiers, graders, and other feedback systems that make model behavior measurable and improvable.
  • Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking.
  • Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques.
  • Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and turn learnings into the next benchmark or system.
  • Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing.

What we're looking for

  • PhD in ML/AI, computer science, data science, or related fields (or equivalent research experience in industry).
  • Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM.
  • Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets (e.g. a strong GitHub profile / OSS contributions, or product portfolio).
  • Strong Python skills, experience building scalable software, working with agents.
  • Strong knowledge of frontier AI: benchmarks, eval techniques, agent harnesses, post-training recipes, data shapes.
  • Comfort operating in an ambiguous, fast-moving environment with substantial ownership.

Why join

  • Work at the very frontier of AI with most major AI labs, researching some of the most important problems in Data and Evaluations.
  • Publish results and work in public through open-source benchmarks and software, papers, and blogs.
  • Join a rapidly growing company whose data business grew from zero to nearly $1B run rate in a year.
  • Help build an early technical organization where your work shapes the roadmap, standards, and culture.
  • Attend (and publish at) conferences like NeurIPS, ICML, ICLR, COLM.

Perks

Handshake delivers benefits that help you feel supported—and thrive at work and in life.

The below benefits are for full-time US employees.

🎯 Ownership: Equity in a fast-growing company

💰 Financial Wellness: 401(k) match, competitive compensation, financial coaching

🍼 Family Support: Paid parental leave, fertility benefits, parental coaching

💝 Wellbeing: Medical, dental, and vision, mental health support, $500 wellness stipend

📚 Growth: $2,000 learning stipend, ongoing development

💻 Office: Commuting support, free lunch, and gym in our SF office

🏝 Time Off: Flexible PTO, 15 holidays + 2 flex days

🤝 Connection: Team outings & referral bonuses

Similar open jobs

Jobs in San Francisco (11741)Jobs in New York (19148)Jobs in San Jose (7798)