Senior AI Test Automation Engineer

USA GPC Genuine Parts Company

Where
Hybrid in Birmingham, AL, USA

Posted 2 months ago· Checked live at USA GPC Genuine Parts Company 4 days ago

Talent poolAn always-open posting that collects resumes. There may be no real opening behind it.

See how you score for this job

One sentence and your resume. We score it against what USA GPC Genuine Parts Company asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on USA GPC Genuine Parts Company's own hiring page 4 days ago.
What is the job?
Senior AI Test Automation Engineer Summary
What does it pay?
The posting doesn't say.
Our verdict
Talent pool. An always-open posting that collects resumes. There may be no real opening behind it.

What the job is

Senior AI Test Automation Engineer

Summary

The Senior AI Test Automation Engineer designs, develops, maintains, and executes automated testing and evaluation solutions for both traditional software and LLM-powered applications.

Required Experience

  • 5+ years of experience in test automation engineering, software quality assurance, or a related role.
  • 1–2+ years of hands-on experience testing or evaluating LLM-powered applications, including building eval datasets, defining pass/fail criteria for nondeterministic outputs, and using LLM-as-judge or heuristic evaluators.
  • Hands-on experience with LangSmith for tracing, evaluation, and observability of LLM applications, including creating datasets, running experiments, and configuring evaluators.
  • Hands-on experience with LangGraph, including graph-based agent workflows, state and context management, and tool calling.
  • Hands-on experience building and maintaining automated test suites using Playwright, preferably with TypeScript.
  • Working proficiency in Python and/or TypeScript sufficient to author custom evaluators, tracing instrumentation, and test code.
  • Experience automating UI and API testing, and experience with API testing tools or frameworks.
  • Familiarity with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or equivalent, including integrating eval runs as pipeline gates.
  • Working knowledge of source-code version control, including Git.
  • Understanding of Agile/Scrum delivery practices and participation in Agile ceremonies.
  • Ability to analyze requirements and user feedback, identify test and eval scenarios, and create maintainable automated coverage.
  • Strong problem-solving, troubleshooting, communication, and collaboration skills, including comfort engaging directly with end users to gather feedback in a forward-deployed capacity.

Preferred / Nice to Have

  • Ability to understand and document end-to-end business processes, user journeys, and operational workflows.
  • Partner with business stakeholders and end users to translate process requirements into test scenarios, acceptance criteria, and LLM evaluation datasets.
  • Identify process exceptions, edge cases, dependencies, and risks that may affect application or agent behavior.
  • Validate that automated workflows and LLM-powered features produce outcomes aligned with defined business rules and user needs.
  • Use production feedback and observed user behavior to continuously refine process coverage, test automation, and evaluation frameworks.
  • Experience with online evaluations, production monitoring, and quality drift detection for LLM applications.
  • Experience with agent trajectory evaluation, RAG evaluation (retrieval relevance, groundedness, hallucination detection), or guardrails validation.
  • Familiarity with prompt engineering and prompt versioning workflows, and evaluating the impact of prompt or model changes.
  • Understanding of statistical approaches to nondeterministic testing (multiple-run sampling, confidence thresholds, summary metrics across datasets).
  • Experience authoring BDD/Gherkin scenarios using Cucumber or a similar framework.
  • Experience with contract testing tools.
  • Familiarity with SAFe and Agile Release Train (ART) practices.
  • Experience with AI-assisted engineering practices (e.g., using AI coding agents to accelerate test and eval development).
  • Experience with performance, accessibility, mobile, or security test automation.
  • Experience with test management and defect-tracking tools, such as Azure DevOps.

Not the right fit? Let us know you're interested in a future opportunity by joining our Talent Community on jobs.genpt.com or create an account to set up email alerts as new job postings become available that meet your interest!

GPC conducts its business without regard to sex, race, creed, color, religion, marital status, national origin, citizenship status, age, pregnancy, sexual orientation, gender identity or expression, genetic information, disability, military status, status as a veteran, or any other protected characteristic. GPC's policy is to recruit, hire, train, promote, assign, transfer and terminate employees based on their own ability, achievement, experience and conduct and other legitimate business reasons.