Software Engineer, Performance Instrumentation
NVIDIA
- Where
- US, CA, Santa Clara
Posted 20 days ago· Checked live at NVIDIA 3 days ago
Stale: open 20 daysUp for more than two weeks. It may already be filled.
See how you score for this job
One sentence and your resume. We score it against what NVIDIA asks for, line by line.
See my score- Is this job still open?
- Yes. We checked it live on NVIDIA's own hiring page 3 days ago.
- What is the job?
- NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world.
- What does it pay?
- The posting doesn't say.
- Our verdict
- Stale. Up for more than two weeks. It may already be filled.
What the job is
NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world.
NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what’s never been done before takes vision, innovation, and the world’s best talent. As an NVIDIAN, you’ll be immersed in an inclusive, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world.
NVIDIA is looking for a Software Engineer, Performance Instrumentation to join our outstanding team in Santa Clara, CA. This is an outstanding opportunity to be part of a world-class organization, driving innovation at the intersection of AI and accelerated computing. As a part of our team, you will work on groundbreaking projects and have a direct impact on the future of technology.
What you’ll be doing
- Define and maintain the performance data requirements from various sources including profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
- Identify and prove visibility gaps with concrete evidence, demonstrating which diagnoses are not possible with current data. Collaborate with relevant teams to address these gaps or build solutions where necessary.
- Inform the build of low-overhead instrumentation and annotation APIs from the perspective of the consumer, ensuring the necessary information is conveyed efficiently and effectively.
- Ensure data quality at ingestion by implementing validation, sanitization, checks, and alerting mechanisms, so that any gaps or errors are detected before impacting users or automated analysis.
- Measure and manage the overhead and observer effects of instrumentation on real workloads, ensuring they remain within acceptable limits.
- Collaborate with multiple partner teams simultaneously, managing commitments and ensuring continuous progress.
What we need to see
- Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
- 8+ years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
- Proficiency in CUDA, and working knowledge of machine learning frameworks like PyTorch or JAX, as well as annotation or tracing APIs such as NVTX.
- Demonstrated experience in maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
- Proven ability to measure instrumentation overhead and make trade-offs among overhead, fidelity, sampling, and aggregation.
- Proven history of implementing modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and ensuring completion through to delivery.
- Strong debugging skills across component boundaries and the ability to validate changes through code review, tests, and independent measurements, including using automated development tools.
- Excellent communication skills, especially in conveying evidence, uncertainty, and next steps with engineers and data users.
- Proficiency in Python and C++.
NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what’s never been done before takes vision, innovation, and the world’s best talent. As an NVIDIAN, you’ll be immersed in an inclusive, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world.
NVIDIA is looking for a Software Engineer, Performance Instrumentation to join our outstanding team in Santa Clara, CA. This is an outstanding opportunity to be part of a world-class organization, driving innovation at the intersection of AI and accelerated computing. As a part of our team, you will work on groundbreaking projects and have a direct impact on the future of technology.
What you’ll be doing
- Define and maintain the performance data requirements from various sources including profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
- Identify and prove visibility gaps with concrete evidence, demonstrating which diagnoses are not possible with current data. Collaborate with relevant teams to address these gaps or build solutions where necessary.
- Inform the build of low-overhead instrumentation and annotation APIs from the perspective of the consumer, ensuring the necessary information is conveyed efficiently and effectively.
- Ensure data quality at ingestion by implementing validation, sanitization, checks, and alerting mechanisms, so that any gaps or errors are detected before impacting users or automated analysis.
- Measure and manage the overhead and observer effects of instrumentation on real workloads, ensuring they remain within acceptable limits.
- Collaborate with multiple partner teams simultaneously, managing commitments and ensuring continuous progress.
What we need to see
- Bachelor’s degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
- 8+ years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
- Proficiency in CUDA, and working knowledge of machine learning frameworks like PyTorch or JAX, as well as annotation or tracing APIs such as NVTX.
- Demonstrated experience in maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
- Proven ability to measure instrumentation overhead and make trade-offs among overhead, fidelity, sampling, and aggregation.
- Proven history of implementing modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and ensuring completion through to delivery.
- Strong debugging skills across component boundaries and the ability to validate changes through code review, tests, and independent measurements, including using automated development tools.
- Excellent communication skills, especially in conveying evidence, uncertainty, and next steps with engineers and data users.
- Proficiency in Python and C++.
Ways to stand out from the crowd
- Experience establishing telemetry conventions or schemas adopted by other teams.
- Background in storage, checkpointing, or data-loading performance.
- Expertise in crafting low-overhead event collection.
- Familiarity with network and collective-communication telemetry, including NCCL, NICs, and fabric counters.
Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary range for this position is 168,000 USD – 270,250 USD.
You will also be eligible for equity and benefits.
Applications for this job will be accepted at least until November 3, 2026.
This posting is for an existing vacancy.
NVIDIA uses AI tools in its recruiting processes.
NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.
Similar open jobs
- Staff Engineer, CPU Load Store VerificationTenstorrent · Austin, Texas, United States · $100K to $500K a year
- System Software Lab Integration EngineerIntuitive · Sunnyvale, CA, United States
- Software Engineer, AI TransformationGoodleap · Remote, Austin, TX
- Staff Electrical EngineerCoreWeave · New York, NY / Sunnyvale, CA / Bellevue, WA · $207K to $275K a year
- Senior Software Engineer, Cloud PaymentsMongoDB · Atlanta · $126K to $248K a year
- Staff Engineer, Design VerificationSamsung Semiconductor · San Jose, California, United States · $163K to $253K a year
Software engineer jobs in Austin (367)Software engineer jobs in San Jose (1462)Jobs in Austin (4551)Jobs in San Jose (7796)