Research Engineer, Benchmarks & Evaluation

Ramp

Where
Hybrid in New York, NY (HQ)
Pay
$275 an hour
Type
Full time

Posted 16 days ago· Checked live at Ramp 38 minutes ago

Stale: open 16 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Ramp asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Ramp's own hiring page 38 minutes ago.
What is the job?
Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends. We automate how over $275B in annualized spend flows in and out of 85,000+ companies: authorizing payments, flagging risk, categorizing spend, and closing books. The problems are high-stakes, data-dense, and unforgiving.
What does it pay?
$275 an hour, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends. We automate how over $275B in annualized spend flows in and out of 85,000+ companies: authorizing payments, flagging risk, categorizing spend, and closing books.

The problems are high-stakes, data-dense, and unforgiving.

We hire people with high agency and high urgency.

What you’ll do

  • Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment
  • Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on
  • Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks
  • Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship
  • Publish external benchmarks and write-ups on how models perform at real finance work

What you need

  • Hands-on experience building LLM benchmarks or evals that people used to make real decisions
  • Strong Python and software engineering; your eval code is reproducible and fast
  • Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise
  • Working knowledge of post-training (SFT, RL with verifiable rewards) and how evals feed into it
  • Comfort with SQL and large datasets; you'll work directly from production traces
  • Taste for what is worth measuring

Nice-to-have

  • Industry experience on an eval, benchmarking or post-training team
  • Built RL environments or verifiers for agentic tasks
  • Published benchmark or evaluation research (NeurIPS Datasets & Benchmarks, ICML, ACL)
  • Built or contributed to eval frameworks (verifiers, lm-evaluation-harness, HELM or similar)
  • Familiarity with accounting, finance or other expert-domain work
  • Public work we can look at: a benchmark, an eval suite, a failure-analysis write-up

Benefits available to all full-time ramp employees (global)

  • Flexible PTO
  • Centralized home-office equipment ordering
  • Health and wellness stipend
  • Budget for intra-office travel
  • Weekly coffee stipend

Similar open jobs

Machine learning engineer jobs in San Francisco (586)Machine learning engineer jobs in New York (479)Jobs in San Francisco (11750)Jobs in New York (19157)