Applied AI Software Engineer

Canvasmedical

Where
Hybrid in San Francisco, CA / Remote

Posted 17 months ago· Checked live at Canvasmedical 6 minutes ago

Stale: open 533 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Canvasmedical asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Canvasmedical's own hiring page 6 minutes ago.
What is the job?
Canvas Medical is the electronic medical records (EMR) and payments development platform for healthcare. We build modern, elegant front- and back-end tooling to enable new ways for developers and clinicians to collaborate to solve healthcare’s toughest challenges.
What does it pay?
The posting doesn't say.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Canvas Medical is the electronic medical records (EMR) and payments development platform for healthcare. We build modern, elegant front- and back-end tooling to enable new ways for developers and clinicians to collaborate to solve healthcare’s toughest challenges.

Who You Are

  • You have extensive hands-on experience evaluating LLM-based systems, including multi-agent architectures and prompt-based pipelines.
  • You are deeply familiar with foundation model APIs (OpenAI, Claude, Gemini, etc.) and how to systematically benchmark agent performance using those models in applied settings.
  • You care about correctness and reproducibility and have built or contributed to frameworks for automated evals, annotation pipelines, and experiment tracking.
  • You bring structure to ambiguity and know how to define “correctness” in complex, nuanced domains.
  • You are comfortable collaborating across engineering, product, and clinical subject matter experts.
  • You are not afraid of complexity and are energized by the rigor required in healthcare deployments.

What You’ll Do

  • Design and execute large-scale evaluation plans for LLM-based agents performing clinical documentation, scheduling, billing, communications, and general workflow automation tasks.
  • Build end-to-end test harnesses that validate model behavior under different configurations (prompt templates, context sources, tool availability, etc.).
  • Partner with clinicians to define accurate expected outcomes (gold standard) for performance comparisons in domains of clinical consequence, and partner with other subject matter experts in other non-clinical domains.
  • Run and replicate experiments across multiple models, parameters, and interaction types to determine optimal configurations.
  • Deploy and maintain ongoing sampling for post-deployment governance of agent fleets.
  • Analyze results and summarize tradeoffs in clarity for product and engineering stakeholders, as well as for technical stakeholders among our customers and the broader market.
  • Take ownership over internal eval tooling and infrastructure, ensuring speed, rigor, and reproducibility.
  • Identify and recommend candidates for reinforcement fine-tuning or retrieval augmentation based on gaps identified in evals.

What Success Looks Like at 90 Days

  • An expanded set of robust evaluation suites exists for all major AI features currently in development and in production.
  • We have well-defined correctness criteria for each workflow and a reliable source of expert-determined outcome objects.
  • Product and engineering teams have integrated your evaluation tools into their daily workflows.
  • Evaluation results are clearly documented and reproducible, enabling trust in the performance trajectory.
  • Your have effectively engaged your marketing counterparts to translate your work into key messages to the market and to Canvas customers.

Qualifications

  • 5+ years of experience in applied machine learning or AI engineering, with a focus on evaluation and benchmarking.
  • Proficiency with foundation model APIs and experience orchestrating complex agent behaviors via prompts or tools.
  • Experience designing and running high-throughput evaluation pipelines, ideally including human-in-the-loop or expert-labeled benchmarks.
  • Superlative Python engineering skills and familiarity with experiment management tools and data engineering toolsets in general including, yes, SQL and database management.
  • Familiarity with clinical or healthcare data is a strong plus.
  • Experience with reinforcement fine-tuning, model monitoring, or RLHF is a plus.
  • Research shows that women and other minority groups might avoid applying if they don’t meet 100% of the qualifications. We encourage you to apply even if you don’t meet everything listed in the job posting.

Canvas Medical provides equal employment opportunities to all employees and applicants for employment without regard to race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

Similar open jobs

Machine learning engineer jobs in San Francisco (586)Jobs in San Francisco (11750)