Sr. Legal AI Prompter, Evaluator

USA018 Thomson Reuters Holdings Inc.

Where
Hybrid in United States of America, Eagan, Minnesota
Pay
$138K a year

Posted 11 days ago· Checked live at USA018 Thomson Reuters Holdings Inc. 4 days ago

Live.Open on the employer's own site.

See how you score for this job

One sentence and your resume. We score it against what USA018 Thomson Reuters Holdings Inc. asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on USA018 Thomson Reuters Holdings Inc.'s own hiring page 4 days ago.
What is the job?
Sr. Legal AI Prompter / Evaluator Job Description
What does it pay?
$138K a year, as the posting states.
Our verdict
Live. Open on the employer's own site.

What the job is

Sr. Legal AI Prompter / Evaluator

Job Description

Sr. Legal AI Prompters / Evaluators are responsible for improving CoCounsel Legal by writing and refining prompts, skills, and agent configurations; experimenting with large language models (LLMs), including Thomson, our proprietary LLM; and evaluating the results.

Key Responsibilities

  • Create, test, and refine prompts, skills, and agent configurations for CoCounsel Legal to ensure accurate, legally sound, and contextually appropriate responses
  • Experiment with Thomson and other LLMs on legal tasks, comparing results to inform which model best fits each use case
  • Build gold-standard datasets with SMEs, coordinate evaluations, and design evaluation schemes, including rubrics, metrics, success criteria, and LLM-as-judge prompts, to test model outputs and inform launch readiness decisions
  • Write and run scripts, experiments, and automated tests that ensure quality and consistency over large datasets and time periods
  • Monitor production performance, analyze failure modes and identify where in the pipeline they originate (prompting, tooling, retrieval, or content), and provide strategic feedback to software engineers and data scientists
  • Collaborate with software engineers, data scientists, and cross-functional teams as they integrate LLMs into multi-step systems, and evaluate the feasibility and potential impact of new product ideas
  • Engage with legal professionals, industry leaders, and internal stakeholders to gather insights and validate product concepts, and stay informed about developments in legal technology and LLMs
  • Mentor colleagues and draft guidance materials, reusable prompts, and best-practice documentation on getting the most out of LLMs

Required Qualifications

  • Juris Doctor degree or equivalent law degree and bar admission in a US or Canadian jurisdiction
  • 5-7 years of combined experience, including at least 2-3 years of legal practice, with the remaining years in related work such as collaborating with software engineers, building or testing products, applying generative AI, or writing code
  • Proven expertise applying generative AI, including prompt engineering and evaluating model outputs, and an understanding of how LLMs work and where they fail
  • Proficiency working in a Python code base and using Git, including writing, modifying, and debugging Python scripts, calling LLMs through code (e.g., OpenAI, Anthropic, Gemini), and using coding agents to build and iterate on that work
  • Excellent communication and collaboration skills, with the ability to work effectively with legal SMEs and across functions

Preferred Qualifications

  • Have substantial experience working with and testing legal products and technology, including online legal research and practice tools
  • Have hands-on experience designing evaluations, such as rubrics, human grading frameworks, or LLM-as-judge prompts
  • Are familiar with agentic and RAG systems, including how retrieval, tool use, and prompting each affect output quality
  • Are a self-starter who is comfortable with ambiguity and learns new tools and models quickly

Replacement: This position is open due to an existing vacancy to support our evolving business needs.

Similar open jobs

Legal jobs in New York (546)Legal jobs in Toronto (18)Jobs in New York (19130)Jobs in Toronto (828)