Senior AI Product Manager, Evals & Annotations

Arize AI

Where
United States (Remote)
Pay
$200K to $250K a year

Posted 8 days ago· Checked live at Arize AI 2 hours ago

Live.Open on the employer's own site.

See how you score for this job

One sentence and your resume. We score it against what Arize AI asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Arize AI's own hiring page 2 hours ago.
What is the job?
AI is rapidly transforming the world. As generative AI reshapes industries, teams need powerful ways to monitor, troubleshoot, and optimize their AI systems. That’s where we come in. Arize AI is the leading AI & Agent Engineering observability and evaluation platform, empowering AI engineers to ship high-performing, reliable agents and applications.
What does it pay?
$200K to $250K a year, as the posting states.
Our verdict
Live. Open on the employer's own site.

What the job is

AI is rapidly transforming the world. As generative AI reshapes industries, teams need powerful ways to monitor, troubleshoot, and optimize their AI systems. That’s where we come in. Arize AI is the leading AI & Agent Engineering observability and evaluation platform, empowering AI engineers to ship high-performing, reliable agents and applications.

What You'll Do

  • Own Arize's Evals & Annotations product area end-to-end, including eval creation and execution, human review, annotation workflows, and the broader evaluation lifecycle.
  • Set the product strategy and roadmap, balancing customer needs, foundational platform investments, UX improvements, and emerging opportunities in AI and agent development.
  • Drive a best-in-class evaluation experience. Develop strong opinions about how workflows should work, independently prototype solutions, and partner with Design to turn those ideas into intuitive, polished experiences.
  • Shape the future of evaluations as AI development evolves toward autonomous and managed agents, including opportunities within the broader Dynatrace ecosystem.
  • Work closely with Engineering on technically complex problems spanning evaluation execution, backend systems, orchestration, and data infrastructure. Contribute to technical discussions, implementation approaches, and tradeoffs.
  • Spend significant time with customers—typically multiple conversations each week—to understand how they evaluate and improve AI systems, uncover underlying problems, and validate product direction.
  • Collaborate directly with our cofounders and cross-functional teams on product direction, design exploration, and prioritization, bringing independent judgment and turning collaborative thinking into execution.
  • Make clear prioritization decisions across competing demands and drive products from problem definition through launch, adoption, and iteration.

What We're Looking For

  • Significant product management experience independently owning technically complex developer-facing products. We're flexible on exact years of experience for candidates with exceptional technical depth, product judgment, and ownership.
  • Strong understanding of LLM evaluations and AI development workflows, including automated evals, human feedback, experimentation, and how developers iterate on AI applications and agents.
  • Experience with developer tools, AI infrastructure, ML platforms, or adjacent technical products, with a solid understanding of backend systems and software architecture.
  • Strong technical depth and hands-on engineering ability. We don't expect you to ship large features as an IC engineer, but we do expect you to:
  • Debug issues independently and navigate code when needed.
  • Put up small fixes or PRs.
  • Build prototypes and technical proofs of concept.
  • Propose technical implementations and reason through tradeoffs.
  • Hold credible technical discussions with engineers.
  • Excellent product and UX instincts.You can independently define and prototype core product experiences, not just write requirements, while partnering effectively with designers and design engineers to refine and ship them.
  • Strong ownership, initiative, and bias toward action. You're comfortable navigating ambiguity, making tradeoffs, and getting hands-on without waiting for a fully defined roadmap or direction.
  • A collaborative, low-ego approach to working with founders, engineers, and designers. You bring strong opinions, engage in constructive debate, and thrive in an environment where responsibilities overlap.
  • Experience working closely with technical customers, with strong communication skills and the ability to move comfortably between strategy and execution.

Bonus Points, But Not Required

  • Direct experience with LLM evaluation platforms, annotation systems, human-in-the-loop workflows, or agent orchestration.
  • Strong backend or platform engineering background, particularly with distributed systems, workflow orchestration, or data infrastructure.
  • Hands-on programming experience and comfort working across backend and frontend technologies.
  • Experience with LLM-as-a-judge, custom evaluators, experiment tracking, datasets, or feedback-driven improvement workflows.
  • Previous engineering experience or substantial hands-on software development experience.
  • Early-stage or high-growth startup experience, particularly in founder-led environments.

The estimated annual salary for this role is between $200,000-$250,000, plus a competitive equity package. Actual compensation is determined based upon a variety of job related factors that may include: transferable work experience, skill sets, and qualifications. Total compensation also includes a comprehensive benefit package, including: medical, dental, vision, 401(k) plan, unlimited paid time off, generous parental leave plan, and others for mental and wellness support.

While we are a remote-first company, we have opened offices in New York City and the San Francisco Bay Area, as an option for those in those cities who wish to work in-person. For all other employees, there is a WFH monthly stipend to pay for co-working spaces.

More About Arize

Arize’s mission is to make the world’s AI work—and work for people.

Our founders came together through a shared frustration: while investments in AI are growing rapidly across every industry, organizations face a critical challenge—understanding whether AI is performing and how to improve it at scale.

Similar open jobs

Remote product manager jobs (732)