Certification Assessment Specialist

OpenAI

Where
Hybrid in San Francisco

Posted 1 hour ago· Checked live at OpenAI 40 minutes ago

Live.Open on the employer's own site.

See how you score for this job

One sentence and your resume. We score it against what OpenAI asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on OpenAI's own hiring page 40 minutes ago.
What is the job?
Customer Education helps customers build the practical skills and confidence to use AI and OpenAI products safely and effectively. We bring together customer insight, learning and enablement expertise, product signals, and measurement to help organizations move from access to durable adoption.
What does it pay?
The posting doesn't say.
Our verdict
Live. Open on the employer's own site.

What the job is

Customer Education helps customers build the practical skills and confidence to use AI and OpenAI products safely and effectively. We bring together customer insight, learning and enablement expertise, product signals, and measurement to help organizations move from access to durable adoption.

In this role, you will

  • Translate assessment blueprints into realistic scenarios, candidate instructions, and hands-on tasks. Work with subject matter experts to check that the tasks reflect the job and elicit the appropriate evidence for each rubric.
  • Define assessment environment requirements and build or configure the environments candidates will use. Establish repeatable provisioning and reset workflows for accounts, tools, data, and permissions so candidates start from consistent conditions.
  • Build and maintain pipelines that capture candidate submissions and other scoring evidence, apply scoring rules, and route results for review. Make scores traceable to the underlying evidence and protect sensitive data under approved controls.
  • Build and test model-assisted scoring with qualified human reviewers using established validation criteria. Compare model scores with calibrated expert judgments, investigate disagreements, implement agreed rules for routing cases to human review, and rerun validation tests after material changes to the model, task, or rubric.
  • Test assessments end to end, from environment setup to evidence capture and scoring. Use dry runs and pilots to check instructions, timing, accessibility, and failure recovery; fix defects and verify the fixes before release. Test scoring against multiple valid approaches, partial and incorrect submissions, and failures in the assessment environment, so a platform issue isn’t mistaken for a candidate error.
  • Translate assessment needs into requirements for delivery integrations. Partner with engineering and delivery providers to test candidate access, submission and results handoffs and troubleshoot issues.
  • Prepare assessments for launch with platform and operations partners. Document setup, version dependencies, and troubleshooting so delivery teams can run assessments consistently and respond when something breaks.
  • Maintain versioned tasks, environments, scoring configurations, and test fixtures. Use candidate feedback and delivery and scoring data to identify issues, update or retire tasks, and check that changes preserve the intended skills and difficulty.
  • Investigate technical issues raised during delivery or candidate review. Reproduce defects where possible, document what was affected, and test fixes with the relevant delivery partners.

You might thrive in this role if you

  • Have personally designed and built performance-based, hands-on assessments, labs or simulations for technical roles, such as software developers or solution architects, and taken them through testing, launch and maintenance. Experience creating question-and-answer tests alone does not meet this requirement.
  • Have used scripting and automation to build and troubleshoot technical assessment environments. You can work with technical subject matter experts to translate realistic solutions into assessments; deep AI-engineering expertise or prior experience with a particular language or delivery platform is not required.
  • Have experience working at a technology company in a fast-paced environment, setting priorities and delivering work amid changing requirements and evolving business needs.
  • Can use a blueprint and rubric to design realistic tasks and specify what evidence distinguishes competent performance from task completion.
  • Have built or maintained workflows that capture submissions, process evidence, and produce traceable scores or reviewer outputs.
  • Have implemented or validated model-assisted scoring or evaluation against expert human judgments, and can explain when a score should go to human review.
  • Have worked directly with external technical experts or delivery providers and can translate requirements into integrations you can test end to end.
  • Have used pilot results, candidate feedback, and scoring or delivery data to improve assessment quality.
  • Understand how to build for accessibility and protect candidate evidence and confidential assessment content.
  • Communicate technical tradeoffs clearly to subject matter experts, measurement specialists, platform engineers, and operations teams.

It’s a plus if you have experience

  • Building hands-on assessments for AI application development, agentic workflows, or customer deployments.
  • Working with psychometricians or measurement specialists on rubric calibration, scoring consistency, or validation of performance assessments.
  • Launching performance-based certification assessments across both live and remote delivery.

Similar open jobs

Jobs in San Francisco (11741)