Senior Software Engineer - Model Performance

Inference

Where
San Francisco
Pay
$220K to $320K a year

Posted 10 months ago· Checked live at Inference 2 hours ago

Stale: open 301 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Inference asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Inference's own hiring page 2 hours ago.
What is the job?
Help us make inference blazingly fast. If you love squeezing every last drop of performance out of GPUs, diving deep into CUDA kernels, and turning optimization techniques into production systems, we'd love to meet you. Inference.net http://Inference.net trains and hosts specialized language models for companies that need frontier-quality AI at a fraction of the cost.
What does it pay?
$220K to $320K a year, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Help us make inference blazingly fast. If you love squeezing every last drop of performance out of GPUs, diving deep into CUDA kernels, and turning optimization techniques into production systems, we'd love to meet you.

Inference.net http://Inference.net trains and hosts specialized language models for companies that need frontier-quality AI at a fraction of the cost.

Key Responsibilities

  • Implement and productionize optimization techniques including quantization, speculative decoding, KV cache optimization, continuous batching, and LoRA serving
  • Deep dive into inference frameworks (vLLM, SGLang, TensorRT-LLM) and underlying libraries to debug and improve performance
  • Profile and optimize CUDA kernels and GPU utilization across our serving infrastructure
  • Add support for new model architectures, ensuring they meet our performance standards before going to production
  • Experiment with novel inference techniques and bring successful approaches into production
  • Build tooling and benchmarks to measure and track inference performance across our fleet
  • Collaborate with applied ML engineers to ensure trained models can be served efficiently

Requirements

  • 2+ years of experience in ML systems, inference optimization, or GPU programming
  • Strong proficiency in Python and familiarity with C++
  • Hands-on experience with LLM inference frameworks (vLLM, SGLang, TensorRT-LLM, or similar)
  • Deep understanding of GPU architecture and experience profiling GPU workloads
  • Familiarity with LLM optimization techniques (quantization, speculative decoding, continuous batching, KV cache management)
  • Experience with PyTorch and understanding of how models execute on hardware
  • Track record of measurably improving system performance

Nice-to-Have

  • Experience with CUDA programming
  • Familiarity with serving non-LLM models (TTS, vision, embeddings)
  • Experience with distributed inference and multi-GPU serving
  • Contributions to open-source inference frameworks
  • Experience with Docker and Kubernetes

You don't need to tick every box. Curiosity and the ability to learn quickly matter more.

Compensation

We offer competitive compensation, equity in a high-growth startup, and comprehensive benefits. The base salary range for this role is $220,000 - $320,000, plus equity and benefits, depending on experience.

Similar open jobs

Software engineer jobs in San Francisco (1709)Jobs in San Francisco (11750)