Software Engineer, Knowledge Engine – Data Infrastructure

Exa

Where
San Francisco, California

Posted 5 months ago· Checked live at Exa 59 minutes ago

Stale: open 157 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Exa asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Exa's own hiring page 59 minutes ago.
What is the job?
The Knowledge Platform team builds Exa's knowledge engine. We continuously synthesize information from across the web and beyond into one of the world's largest open-domain knowledge graphs, with hundreds of billions of evidence-backed claims. The work has two halves: construction pipelines that decide what to trust, and retrieval engines that decide how best to search it.
What does it pay?
The posting doesn't say.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

The Knowledge Platform team builds Exa's knowledge engine. We continuously synthesize information from across the web and beyond into one of the world's largest open-domain knowledge graphs, with hundreds of billions of evidence-backed claims. The work has two halves: construction pipelines that decide what to trust, and retrieval engines that decide how best to search it.

What you'll do

  • Continuously synthesize the live web crawl, vendor deliveries and APIs, and other off-the-web sources into one typed graph: extraction that turns pages and records into ontology-shaped claims, each carrying its provenance, timestamp and confidence.
  • Bring every new source in cleanly: versioned, append-only history with schema evolution, late-arriving and corrected records, and replayable cursors, so every claim traces back to the exact delivery that produced it.
  • Decide what to trust: entity resolution across billions of records (blocking, pairwise scoring, clustering) into canonical people, companies, places and publications, then truth discovery that weighs source reliability, recency and corroboration to reconcile conflicting claims.
  • Keep that picture current and convergent: streams arrive continuously, out of order and at very different rates, so the graph is eventually consistent by design. You'll build incremental computation over change data feeds, where each update is a diff of what changed and every job is idempotent and resumable, so the graph converges to the same state however runs interleave, fail or replay, and readers only ever see committed, consistent versions.
  • Measure precision and completeness: labeled evals for extraction and resolution, per-source coverage and correctness metrics, and lineage that pinpoints which source or stage broke and why.

What we're looking for

  • You've built and operated large-scale distributed data pipelines in production over petabytes of structured and unstructured data, with Spark over a lakehouse format (Delta Lake, Iceberg, Hudi).
  • You understand incremental processing deeply: change data capture, idempotent writes, backfills and reprocessing.
  • Strong in Python and at least one systems language; you can read and tune Spark query plans.
  • You obsess over correctness, reliability and data quality as much as scale.

Similar open jobs

Data engineer jobs in San Francisco (87)Jobs in San Francisco (11750)