Principal Software Engineer - Observability & Telemetry Data

Smartsheet

Where
Bellevue, WA, USA
Pay
$223K to $258K a year
Type
Full time

Posted 2 months ago· Checked live at Smartsheet 1 hour ago

Stale: open 78 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what Smartsheet asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on Smartsheet's own hiring page 1 hour ago.
What is the job?
For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking.
What does it pay?
$223K to $258K a year, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

For over 20 years, Smartsheet has empowered teams to manage work seamlessly and scale solutions smarter. Now, in our most ambitious chapter yet, we are uniting human teams with AI agents. By orchestrating the work agents do best, automating manual tasks and uncovering insights at scale, we create the space for people to focus on what truly matters: judgment, creativity, and big thinking.

You Have

  • 10+ years of experience building and operating large-scale distributed systems, data platforms, or observability infrastructure, including time at Principal or Staff level.
  • Deep Observability Expertise: Hands-on production ownership of metrics, logs, and distributed tracing at scale, including at least one major backend (Datadog, or comparable) and a working understanding of cardinality and cost mechanics.
  • Telemetry Data Engineering: Demonstrated depth in large-scale data architecture, including open table formats (Delta Lake, Apache Iceberg), Spark or comparable distributed processing, streaming ingestion, partitioning and schema evolution, and query performance and cost tuning over very large datasets.
  • Databricks Depth: Practical experience with the Databricks platform (jobs, clusters, Unity Catalog, Delta) and with MLflow for model and agent telemetry.
  • OpenTelemetry Depth: Practical experience with OTel collectors, semantic conventions, context propagation, and sampling strategy, including tail-based sampling.
  • Pipeline Engineering: Experience with high-volume log and telemetry pipelines, including FluentBit or Fluentd, streaming transport such as Kinesis or Kafka, and search backend index and mapping design.
  • Advanced AWS & Kubernetes Expertise: EKS, ECS Fargate, EC2, Lambda, and CloudWatch in production.
  • 10+ years of programming experience with modern languages such as Go, Java, Python, or Scala, and strong SQL.
  • Infrastructure as Code: Terraform, and GitOps workflows such as Flux or ArgoCD.
  • Architectural Influence at Scale: A track record of setting technical direction that multiple teams and organizations adopted, including the written artifacts (architecture decisions, standards, RFCs) that made it durable after you moved on.
  • Strong incident response instincts, with experience improving mean-time-to-resolution through better instrumentation and better data rather than more heroics.
  • A degree in Computer Science, Engineering, or a related field, or equivalent practical industry experience.
  • Legally eligible to work in the U.S. on an ongoing basis

Nice to Have

  • Experience instrumenting LLM or agentic systems, including OTel GenAI semantic conventions and tracing agent tool-call workflows.
  • Data reliability engineering practice: freshness, quality, and lineage SLOs for production data pipelines
  • Telemetry cost engineering or FinOps at scale.
  • Experience in regulated environments (FedRAMP, GovCloud) and designing telemetry that remains useful under redaction.
  • Prometheus, Grafana, and Alertmanager.
  • Snowflake, or experience operating across more than one lakehouse or warehouse platform.
  • Service catalog or internal developer platform work (Backstage or similar).

Current US Perks & Benefits

  • Employer subsidized medical/vision and dental coverage for full-time employees
  • 401k Match to help you save for your future (50% of your contribution up to the first 6% of your eligible pay)
  • Monthly stipend to support your work and productivity
  • Flexible Time Away Program, plus Sick Time Off
  • US employees are automatically covered under Smartsheet-sponsored life insurance, short-term, and long-term disability plans
  • US employees receive 12 paid holidays per year
  • Up to 24 weeks of Parental Leave
  • Personal paid Volunteer Day to support our community
  • Opportunities for professional growth and development including access to Udemy online courses
  • Company Funded Perks, including a counseling membership, local retail discounts, and your own personal Smartsheet account
  • Teleworking options from any registered location in the U.S. (role specific)

Smartsheet provides a competitive base salary range for roles that may be hired in different geographic areas we are licensed to operate our business from. Actual compensation is determined by several factors including, but not limited to, level of professional, educational experience, skills, and specific candidate location. In addition, this role will be eligible for a market competitive incentive opportunity.

US Base Salary Pay Range

$222,500 - $257,500 USD

Similar open jobs

Software engineer jobs in Seattle (980)Jobs in Seattle (5777)