Principal Research Scientist

G2

Where
Remote (US)

Posted 23 days ago· Checked live at G2 19 minutes ago

Stale: open 23 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what G2 asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on G2's own hiring page 19 minutes ago.
What is the job?
G2 is the world's largest and most trusted software marketplace. When you join G2, you’re joining the industry’s leading team that helps businesses reach their peak potential by powering decisions and strategies with trusted insights from real software users.
What does it pay?
The posting doesn't say.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

G2 is the world's largest and most trusted software marketplace. When you join G2, you’re joining the industry’s leading team that helps businesses reach their peak potential by powering decisions and strategies with trusted insights from real software users.

In this role, you will

  • Own the evaluation methodology. Set the standard for how G2 evaluates AI agents, defend it to skeptical audiences inside and outside the company, and keep it current as models change.
  • Contribute to today's product. Work closely with engineers and product managers to build and improve the evaluations behind G2's current offerings. Help scope features, turn methods into production-ready components, and iterate on what customers and the business need.
  • Design realistic evaluations. Build environments and task sets that reflect the work businesses actually give AI agents, and that stay useful as models improve.
  • Develop sound measurement. Create scoring approaches, including automated checks, LLM-based judges and human review, that are accurate, reproducible and hard to game. Measure consistency and reliability, not just average performance.
  • Validate the evaluations themselves. Audit tasks and graders, quantify noise, and establish when differences between agents are real.
  • Represent the results. Present and defend evaluation findings to buyers, customers and partners, and to the research community. Be the credible voice behind the numbers when the audience has a stake in the outcome.
  • Mentor and multiply. Manage and guide researchers and engineers on evaluation design and measurement, review their work, and write the documentation and standards that let the team run evaluations without you in the room.
  • Set and pursue a research agenda. Choose which evaluation problems matter most and carry long-running projects through to something that works, favoring simple ideas that generalize over ones tuned to this year's models. Directions include simulated-user and environment-based evaluation in the tradition of benchmarks like τ-bench; realistic user and persona simulation; LLM-as-judge reliability; evaluation of voice and multimodal agents; assessing agents in open-ended, long-horizon workflows; and protecting test sets from contamination.
  • Stay current. Follow new benchmarks, papers and evaluation practices across the research community. Translate them into product and roadmap decisions, and communicate them clearly to the team.
  • Share and collaborate. Write methodology documentation and, where appropriate, papers and public posts. Partner with data and go-to-market teams on how results are presented and used.

Minimum qualifications

We realize applying for jobs can feel daunting at times. Even if you don’t check all the boxes in the job description, we encourage you to apply anyway.

  • Experience working with evaluation design or dataset curation in multi-agentic systems
  • 5+ years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents.
  • 2+ years experience managing small to medium teams preferred.
  • A graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics or a related field, or equivalent applied research experience.
  • A track record of original ideas in machine learning, or clear improvements on existing ones, shown through first-author publications or projects you led end-to-end.
  • The ability to own a research agenda: pick the problems worth solving, and carry long-running projects without being directed.
  • A track record of shipping ML work in a product setting, working closely with engineers and product managers.
  • Experience setting technical direction for others: mentoring researchers or engineers, reviewing methodology, and raising the quality of work you did not do yourself.
  • Familiarity with how AI systems are benchmarked, and a clear view of how benchmark scores can mislead.
  • Sound statistical judgment: experimental design, variance, significance and agreement between human and automated raters.
  • Good judgment about trade-offs, including when a practical solution now serves customers better than a more complete one later.
  • Clear writing, and the ability to explain methodology to non-specialists and to defend it to skeptical audiences.

Similar open jobs

Remote machine learning engineer jobs (519)