Senior Manager, Engineering - AI Validation

Sonatus

Where
Sunnyvale, CA
Pay
$240K to $290K a year

Posted 4 months ago· Checked live at Sonatus 2 hours ago

Stale: open 125 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Sonatus asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Sonatus's own hiring page 2 hours ago.
What is the job?
At Sonatus, we’re driving the transformation to AI-enabled software-defined vehicles. Traditional automotive software methods can’t keep pace with consumer expectations shaped by the mobile industry—where features evolve rapidly, update seamlessly, and improve continuously. That’s why leading OEMs trust Sonatus to accelerate this shift.
What does it pay?
$240K to $290K a year, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

At Sonatus, we’re driving the transformation to AI-enabled software-defined vehicles. Traditional automotive software methods can’t keep pace with consumer expectations shaped by the mobile industry—where features evolve rapidly, update seamlessly, and improve continuously. That’s why leading OEMs trust Sonatus to accelerate this shift.

Role and Responsibility

  • Define and drive Sonatus's AI validation strategy across embedded, in-vehicle, cloud-connected, and cloud-native AI systems, identifying gaps in model development, testing, deployment, and governance.
  • Lead, hire, mentor, and grow a high-performing AI Validation organization, establishing scalable engineering processes, technical direction, and execution excellence.
  • Own the end-to-end validation strategy for AI/ML models, LLMs, RAG pipelines, and agentic AI workflows—from data pipelines and model training through cloud services and in-vehicle deployment.
  • Architect and operationalize scalable evaluation frameworks and benchmarking platforms for AI systems, including multi-step agentic workflows, using deterministic metrics, LLM-as-a-Judge methodologies, automated regression testing, and production feedback loops.
  • Design and maintain evaluation harnesses for RAG and agentic systems that measure retrieval quality, grounding, citation accuracy, factual consistency, context relevance, safety, latency, reliability, and execution correctness.
  • Evaluate, integrate, and optimize open-source and commercial AI validation technologies, driving build-versus-buy decisions for Sonatus's AI quality platform.
  • Establish AI governance, Responsible AI practices, model lineage, safety guardrails, and compliance processes appropriate for automotive safety-critical systems and enterprise AI products.
  • Act as the quality gatekeeper for AI-enabled releases, partnering with engineering, product, safety, and OEM stakeholders to identify risks, define release criteria, and ensure production readiness.
  • Collaborate across engineering teams to define validation strategies for emerging AI capabilities, rapidly prototype new evaluation approaches, and standardize successful practices into reusable frameworks.
  • Drive continuous improvement by tracking industry advances in AI evaluation, agentic AI, LLM validation, and RAG systems, translating them into scalable validation capabilities across Sonatus.

Qualifications

  • Bachelor's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field required (MS preferred).
  • 10+ years of experience in software or systems engineering—including embedded, cloud, networking, security, or automotive domains—with 3+ years leading high-performing engineering or QA organizations.
  • Hands-on experience developing, deploying, testing, or operating AI/ML systems, with strong expertise in modern ML workflows, neural networks, and MLOps.
  • Deep understanding of LLMs, RAG architectures, vector databases, embeddings, retrieval optimization, and agentic AI frameworks such as LangGraph or equivalent orchestration platforms.
  • Proven experience designing and implementing scalable evaluation frameworks for AI systems, including multi-step agentic workflows, regression testing, benchmarking, and automated quality scoring.
  • Strong expertise with hybrid evaluation methodologies combining deterministic validation (citation grounding, structural validation, exact matching) and probabilistic LLM-as-a-Judge techniques (faithfulness, answer relevance, context precision, task completion).
  • Practical experience with RAG evaluation frameworks such as RAGAS, including evaluation tuning, embedding optimization, retrieval quality improvement, and production-scale LLM evaluation pipelines.
  • Experience validating hallucination, grounding, citation accuracy, bias, fairness, toxicity, and factual consistency in production LLM applications.
  • Experience designing systems that verify external knowledge claims and ensure responses are grounded in traceable citations and trusted data sources.
  • Strong experience testing cloud-native platforms and cloud-managed embedded products, including end-to-end system validation.
  • Experience establishing AI governance, safety, compliance, and Responsible AI practices for enterprise or safety-critical systems.
  • Proficiency in Python, Linux, shell scripting, modern test frameworks (PyTest, Playwright, Behave), and engineering productivity tools such as Jenkins and JIRA.

Sunnyvale HQ Benefits & Perks Offered

  • Health care plan (Medical, Dental & Vision)
  • Flexible and Dependent Care Expense program
  • Retirement plan (401k)
  • Life Insurance (Basic, Voluntary & AD&D)
  • Unlimited paid time off per year, 14+ paid holidays
  • Hybrid office work arrangement
  • Complimentary lunches, snacks, and beverages during on-site working days
  • Wellness benefit allowance
  • Phone & Internet reimbursement
  • Computer Accessory Allowance

The posted salary range is a general guideline and represents a good faith estimate of what Sonatus ("Company") could reasonably expect to pay for a base salary for this position. The pay offered to a selected candidate will be determined based on factors such as (but not limited to) the scope and responsibilities of the position, the qualifications of the selected candidate, departmental budget availability, geographic location and external market pay for comparable jobs. The Company reserves the right to modify this range in the future, as needed, as market conditions change.

Base Salary Pay Range

$240,000 - $290,000 USD

Similar open jobs

Engineering manager jobs in San Jose (133)Jobs in San Jose (7796)