Staff Security Data Engineer

ADUS-Adobe Inc.

Where
San Jose
Pay
$159K to $302K a year

Posted 52 days ago· Checked live at ADUS-Adobe Inc. 3 days ago

Stale: open 52 daysUp for more than two weeks. It may already be filled.

See how you score for this job

One sentence and your resume. We score it against what ADUS-Adobe Inc. asks for, line by line.

See my score
Is this job still open?
Yes. We checked it live on ADUS-Adobe Inc.'s own hiring page 3 days ago.
What is the job?
Adobe’s Security Data Platform team builds and operates a petabyte-scale security data lakehouse that turns enterprise telemetry into trusted data for threat detection, investigations, compliance, and security analytics. As a Staff Security Data Engineer, you will own full-system designs and lead critical engineering initiatives across ingestion, storage, governance, orchestration, and data serving.
What does it pay?
$159K to $302K a year, as the posting states.
Our verdict
Stale. Up for more than two weeks. It may already be filled.

What the job is

Adobe’s Security Data Platform team builds and operates a petabyte-scale security data lakehouse that turns enterprise telemetry into trusted data for threat detection, investigations, compliance, and security analytics.

As a Staff Security Data Engineer, you will own full-system designs and lead critical engineering initiatives across ingestion, storage, governance, orchestration, and data serving.

Core responsibilities

Platform architecture and engineering

  • Own end-to-end designs that shape the Security Data Platform’s fundamental architecture on Databricks and AWS, using Spark, Delta Lake, and Unity Catalog. Translate ambiguous needs into design proposals, implementation plans, and production capabilities.
  • Build and evolve distributed batch and streaming pipelines for EDR telemetry, network logs, identity events, cloud audit logs, and new security data sources. Design for throughput, fault tolerance, replay, schema evolution, and sub-minute freshness where use cases require it.
  • Write maintainable Python and SQL, develop complex platform features, and refactor unreliable code. Establish reusable engineering patterns, lead code and design reviews, and own test strategy and automation choices across pipelines and infrastructure.
  • Build and maintain tested, reusable Terraform modules for AWS and Databricks infrastructure, including S3, IAM, VPC networking, and Amazon Managed Workflows for Apache Airflow (MWAA). Automate delivery through version control and CI/CD.

Data modeling governance and operations

  • Design layered lakehouse data models and reusable normalization patterns, including Open Cybersecurity Schema Framework (OCSF) mappings where appropriate. Preserve source fidelity and make datasets useful for investigations, analytics, and detection workloads.
  • Establish data contracts, quality checks, cataloging, classification, lineage, access controls, and retention policies with data owners. Use Unity Catalog to make data ownership and governance traceable from raw events to analyst-facing datasets.
  • Engineer and operate Airflow and MWAA orchestration, including dependencies, retries, backfills, alerting, and automated recovery. Define service level objectives with consumers and instrument data freshness, completeness, pipeline health, and query performance.
  • Lead diagnosis and resolution of complex production issues across Spark, storage, orchestration, and cloud infrastructure. Improve incident response, runbooks, recovery testing, and operational readiness to prevent recurring failures.

Performance and technical leadership

  • Profile workloads and lead measurable cost and performance improvements through query and Spark tuning, storage lifecycle policies, partitioning, and compute right-sizing. Establish baselines and dashboards that show savings alongside reliability, latency, and capacity.
  • Lead cross-team technical discussions, reconcile competing requirements, and build agreement on architecture and delivery priorities without formal management authority. Communicate decisions, landmarks, risks, and business impact proactively to your manager and partners.
  • Partner with detection and security analytics engineers to deliver the data interfaces, governance, and pipeline capabilities needed for SIEM modernization and automated detection deployment.
  • Mentor engineers through pairing, technical reviews, documentation, and knowledge sharing. Help others lead technical work, promote inclusive debate, and contribute to hiring decisions through technical assessment.
  • Evaluate and introduce useful platform innovations, including AI-assisted engineering and operational automation. Validate outputs, protect sensitive security data, measure improvements, and help partner teams adopt proven approaches.

Required qualifications

  • Substantial experience building and operating production data platforms, typically 10 or more years in data or software engineering, with demonstrated Staff-level scope and impact. Equivalent depth and outcomes are welcome; a prior Staff or Principal title is not required.
  • A record of personally delivering complex distributed systems and leading full-system design and implementation across team boundaries. Ability to turn general, unstructured assignments into clear technical approaches and working software.
  • Deep production expertise in Spark's traditional and real-time data workloads, Databricks, Delta Lake, and lakehouse architecture; strong Python and advanced SQL skills for data modeling, profiling, and query optimization.
  • Strong AWS experience across storage, compute, networking, and IAM, plus production Terraform experience designing, testing, and maintaining reusable infrastructure modules.
  • Practical understanding of distributed systems, including partitioning, replication, consistency, back pressure, scalability, and fault tolerance, with experience diagnosing failures across system boundaries.
  • Experience with Airflow, MWAA, or equivalent orchestration, and with production observability, data quality, automated testing, CI/CD, and reliable deployment and recovery practices.
  • Demonstrated ability to improve platform cost, performance, and reliability with measurable results; sound judgment in balancing these goals against security and consumer requirements.
  • Effective technical communication and multi-functional influence, including leading design reviews, resolving disagreements, mentoring engineers, and building adoption of shared engineering practices.

Preferred qualifications

  • Experience with security telemetry, OCSF or comparable security schemas, and data requirements for SIEM, SOAR, threat detection, or investigations.
  • Experience with Unity Catalog governance, data classification and retention, regional data controls, or multi-region platform design.
  • Experience with Cribl Stream, Vector, or similar telemetry routing and enrichment tools, and platform integrations supporting automated threat detection rollout.

Expected Pay Range

Our compensation reflects the cost of labor across several  U.S. geographic markets, and we pay differently based on those defined markets. The U.S. pay range for this position is $159,200 - $301,600 annually. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience. Your recruiter can share more about the specific salary range for the job location during the hiring process.

In California, the pay range for this position is $208,300 - $301,600

At Adobe, for sales roles starting salaries are expressed as total target compensation (TTC = base + commission), and short-term incentives are in the form of sales commission plans. Non-sales roles starting salaries are expressed as base salary and short-term incentives are in the form of the Annual Incentive Plan (AIP).

In addition, certain roles may be eligible for long-term incentives in the form of a new hire equity award.

Similar open jobs

Data engineer jobs in San Francisco (87)Data engineer jobs in San Jose (49)Jobs in San Francisco (11750)Jobs in San Jose (7796)