Senior Cloud Observability Engineer
Job Summary
The Senior Cloud Engineer (Observability) is responsible for engineering and operating the enterprise observability platform across hybrid cloud and containerized environments. This hands-on role involves instrumenting services, managing Datadog dashboards and alerts, and utilizing telemetry to resolve complex production performance and reliability issues. The engineer will provide technical leadership for APM strategy and drive data-driven operational decision-making in a 24x7x365 environment.
Responsibilities
- Engineer/operate enterprise observability stack (Datadog)
- Build/tune dashboards, SLOs/SLIs, and alerting policies
- Instrument services using OpenTelemetry and language-specific tracers
- Perform performance engineering and complex problem solving
- Monitor Cloud (Azure/AWS) and Container (K8s/OpenShift) platforms
Required Skills
- Datadog or comparable observability platforms
- Distributed tracing and APM expertise
- K8s/OpenShift and Cloud monitoring (Azure/AWS)
- Infrastructure-as-Code (Terraform, Bicep, ARM)
- Database performance tuning and analytics
Job Details
PRIMARY RESPONSIBILITIES
Observability Platform Engineering
- Engineer and operate the enterprise observability stack (Datadog or comparable), including metrics, logs, traces, APM, RUM, synthetic monitoring, and network performance monitoring.
- Build, tune, and maintain dashboards, monitors, SLOs/SLIs, and alerting policies that produce actionable signal and minimize noise.
- Instrument services, infrastructure, and containerized workloads using agents, OpenTelemetry, and language-specific APM tracers (Java, .NET, Python, Node.js, Go) with consistent span tagging, W3C TraceContext propagation, and unified service tagging.
- Develop and maintain integrations between observability platforms, ITSM (ServiceNow), CI/CD pipelines, and on-call/paging workflows.
- Define and enforce a unified tagging standard across metrics, logs, and traces; manage tag cardinality and governance to keep telemetry queryable, attributable, and cost-controlled.
Cloud and Container Monitoring Engineering
- Design and deliver monitoring coverage for Microsoft Azure and AWS workloads, including PaaS, serverless, networking, identity, and managed databases.
- Engineer managed database observability across AWS RDS/Aurora, Azure SQL/PostgreSQL/MySQL, and NoSQL/cache services (DynamoDB, Cosmos DB, Redis), including query-level performance analytics, lock/deadlock/wait analysis, and session monitoring.
- Engineer container-platform observability for OpenShift/Kubernetes, covering cluster health, control plane, pods, ingress, and service mesh.
- Build standardized, reusable monitoring modules deployable via infrastructure-as-code (Terraform, Bicep, ARM).
Performance Engineering and Problem Solving
- Lead data-driven investigation and resolution of complex performance, latency, saturation, and reliability issues.
- Use APM distributed traces, continuous code profiling, database query analytics, and RUM-to-backend trace correlation to isolate bottlenecks.
- Partner with engineering teams to implement remediation, tuning, and architectural improvements based on telemetry evidence.
- Define and implement trace-based SLOs and deployment tracking to identify performance regressions.
- Provide senior technical leadership during major incidents, delivering impact analysis and root-cause analysis.
Capacity, Reliability, and Continuous Improvement
- Analyze operational telemetry to identify capacity risks and efficiency opportunities.
- Build and maintain capacity and performance dashboards for technical and leadership stakeholders.
- Define capacity thresholds and alert baselines for scaling and resource reallocation.
- Drive continuous improvement of observability coverage, alert quality, and operational maturity.
REQUIRED QUALIFICATIONS
- Citizenship/Work Authorization: Must meet contract requirements.
- Clearance: Ability to obtain and maintain SEC Public Trust.
- Experience: Minimum 8 years in IT infrastructure or platform engineering, with 5+ years in observability, performance engineering, or SRE.
- Demonstrated experience engineering/operating an enterprise observability platform (Datadog preferred).
- Proven experience building APM and distributed tracing coverage for production multi-tier applications.
- Hands-on experience monitoring Kubernetes/OpenShift clusters and containerized workloads.