Member of Technical Staff, Enterprise Evals Platform

Mercor

Where
San Francisco
Pay
$20 an hour

Posted 3 months ago· Checked live at Mercor 28 minutes ago

Stale: open 96 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Mercor asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Mercor's own hiring page 28 minutes ago.
What is the job?
Mercor’s mission is to organize human intelligence for the AI economy. We build the infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. More than five million experts have joined our network, and we pay out over $5 million daily for their knowledge and expertise.
What does it pay?
$20 an hour, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Mercor’s mission is to organize human intelligence for the AI economy. We build the infrastructure that enables frontier AI labs and enterprises to train, evaluate, and improve AI on real-world professional work. More than five million experts have joined our network, and we pay out over $5 million daily for their knowledge and expertise.

Responsibilities

  • Define golden sets: decompose real tasks and encode the expert quality bar.
  • Build verifiers over agent trajectories and outputs, calibrated and hard to game.
  • Build the eval platform that runs offline environments, task suites, and grading at scale.
  • Run loss analysis over production trajectories and turn failure modes into regression tests.
  • Run the optimization loop across models, prompts, skills, and harnesses.
  • Own the rollout gates that decide whether an agent change ships.
  • Partner with the Enterprise Platform team and the Applied AI engineers embedded with customers.

What we're looking for

  • Professional, academic, or research experience in agent engineering and evaluation, including how agent runtimes and harnesses produce a trajectory and where it fails.
  • Bachelor’s Degree in Computer Science or a related technical degree such as Electrical Engineering, Physics, Mathematics, Computer Engineering, Information Systems or Statistics
  • 1–5 years in industry level software engineering
  • Experience building evaluation suites for LLM or agent systems, and familiarity with how benchmarks such as terminal-bench, tau-bench, and APEX are constructed and where they get gamed.
  • Judgment about task and rubric design: turning a fuzzy notion of quality into something measurable, with agent or model improvements to show for it.
  • Strong software engineering fundamentals, and the ability to work independently on ambiguous, loosely specified problems.
  • Bonus: experience with Harbor environments and RL environments.

Benefits

  • Up to $15k relocation bonus
  • $10K housing bonus (if you live within 0.5 miles of our office)
  • $1.5K monthly stipend for meals
  • Generous equity grant vested over 4 years
  • Free Equinox membership
  • $200 monthly laundry reimbursement
  • $200 monthly personal wellness reimbursement
  • Health, Dental, Vision insurance

Mercor is an Equal Opportunity Employer and is committed to providing equal opportunity to all employees and applicants. Mercor does not discriminate based on actual or perceived race, color, creed, religion, national origin, ancestry, citizenship status, age, sex or gender (including pregnancy, childbirth, related medical conditions and lactation), gender identity or gender expression, sexual orientation, marital status, military service and veteran status, disability, protected medical condition as defined by California or local law, genetic information, or any other characteristic protected by applicable federal, California, or local laws and ordinances. Further, Mercor prohibits discrimination, harassment, and retaliation based on any combination of protected characteristics. This Policy applies to all areas of employment, including recruitment, hiring, training, promotion, compensation, benefits, discipline, termination, and all other privileges, terms, and conditions of employment. This Policy and the law prohibit employment discrimination against any employee or applicant based on any legally protected status.

Similar open jobs

Jobs in San Francisco (11741)