Senior Staff LLM Inference Engineer

D Matrix

Where
Hybrid in Santa Clara

Posted 3 months ago· Checked live at D Matrix 14 minutes ago

Stale: open 93 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what D Matrix asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on D Matrix's own hiring page 14 minutes ago.
What is the job?
Join d-Matrix and help build the future of AI inference. We’ve pioneered a memory-centric architecture that brings compute directly into memory, making AI faster, more energy efficient, and accessible on a global scale. From chips and software to rack-scale systems, our team is tackling some of AI infrastructure’s hardest challenges — and shipping breakthrough technology today.
What does it pay?
The posting doesn't say.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Join d-Matrix and help build the future of AI inference. We’ve pioneered a memory-centric architecture that brings compute directly into memory, making AI faster, more energy efficient, and accessible on a global scale. From chips and software to rack-scale systems, our team is tackling some of AI infrastructure’s hardest challenges — and shipping breakthrough technology today.

Responsibilities

  • Identify and prototype emerging LLM inference use cases suited to heterogeneous hardware deployments.
  • Build compelling proof-of-concept systems that demonstrate D-Matrix capabilities to customers, partners, and internal stakeholders.
  • Develop and tune custom kernels and operator-level optimizations to maximize throughput and minimize latency.
  • Drive quantization, sparsity, and batching strategies tailored to the D-Matrix computational model.
  • Build and maintain inference runtimes, serving frameworks, and evaluation tooling.
  • Contribute to distributed inference systems: tensor/pipeline parallelism, disaggregated prefill/decode, KV-cache management.
  • Work closely with hardware architects to provide firmware and compiler teams with actionable inference workload insights.
  • Partner with product and business development to translate POCs into customer-facing demonstrations.
  • Contribute to technical publications, whitepapers, and open-source projects that advance D-Matrix visibility.

Required Qualifications

  • Bachelor’s degree in Computer Science, Electrical Engineering, or a related field, and 10+ years of relevant engineering experience or equivalent demonstrated experience.
  • Master’s or PhD in Computer Science, Electrical Engineering, or a related field preferred, with 6+ years of relevant industry experience.
  • Strong proficiency in Python and C/C++.
  • Hands-on experience optimizing LLM inference — attention kernels, KV cache, batching strategies, quantization (INT8/FP8/INT4).
  • Experience with at least one major inference framework (vLLM, SGLang, TensorRT-LLM, ONNX Runtime, or similar) at a contributor level.
  • Familiarity with GPU kernel programming (CUDA/Triton) and performance profiling tools.

Preferred Qualifications

  • Experience with heterogeneous compute deployments — scheduling inference workloads across dissimilar hardware (accelerators, CPUs, GPUs).
  • Familiarity with custom silicon or ASIC-based inference (beyond GPU-only environments).
  • Experience with distributed inference: tensor parallelism, pipeline parallelism, disaggregated serving.
  • Contributions to open-source inference or ML systems projects.
  • Experience with production inference serving at scale (latency SLOs, continuous batching, multi-model serving).
  • Familiarity with speculative decoding, mixture-of-experts routing, or long-context-serving techniques.
  • Working familiarity with the material in the JAX Scaling Book or equivalent systems-level understanding of modern LLM training and inference.

Similar open jobs

Machine learning engineer jobs in San Jose (445)Jobs in San Jose (7796)