
Senior DevOps Engineer, Observability and FinOps
Required skills
- TS/SCI with polygraph
- Grafana Alloy
- Mimir
- Loki
- Tempo
- OpenShift
- Elastic
- Dynatrace
- Splunk
- OpenTelemetry
Nice to have
- OpenCost
- Kubecost
- CloudHealth
- Apptio Cloudability
- AWS Cost Explorer
- Istio
Role summary
The Senior DevOps Engineer, Observability and FinOps role focuses on making mission systems visible and their costs accountable within air-gapped networks. This position blends platform engineering (Kubernetes/OpenShift, IaC, CI/CD) with full-stack development to create essential dashboards, APIs, and tools for operators and financial stakeholders. Responsibilities include deploying and operating observability stacks, instrumenting applications and infrastructure, building cost visibility and optimization strategies, and automating deployments. The role requires proficiency in Python or Go, SQL, PromQL, and experience with secure system hardening and air-gapped network operations. A TS/SCI with polygraph clearance is mandatory.
The Opportunity:
This engineer makes mission systems visible and their costs accountable, on networks that cannot call out to the internet. The role blends platform engineering (Kubernetes/OpenShift, infrastructure as code, CI/CD) with enough full stack development to build the dashboards, APIs, and small internal tools that operators and financial stakeholders use every day.
Responsibilities:
- Deploy, configure, and operate the observability stack (for example Grafana Alloy, Mimir, Loki, and Tempo on OpenShift, or Elastic or Dynatrace where selected) on each network
- Instrument applications and infrastructure with OpenTelemetry and Prometheus exporters; standardize labels, tags, and naming
- Build real-time mission status dashboards and service health views for operators, engineers, and leadership
- Implement cost visibility and allocation across AWS, OpenShift, and on-premises resources using tagging standards and showback or chargeback models
- Build cost and utilization dashboards that join billing exports with observability metrics (unit cost per mission application, per namespace, per program)
- Identify and execute optimization: rightsizing, idle resource cleanup, storage lifecycle policies, Reserved Instance and Savings Plan coverage analysis
- Automate infrastructure and stack deployment with Terraform, Ansible, and Helm or Operators; manage configuration through GitOps (Argo CD or similar)
- Build and maintain CI/CD pipelines (GitLab CI, Jenkins, or Tekton) including image scanning, signing, and promotion into air-gapped registries
- Package software and dependencies for offline transfer and cross-domain delivery
- Develop lightweight web apps, APIs, and Grafana plugins or panels for custom status views and cost reports
- Write data integration scripts (Python, Go) that pull from billing exports, CMDBs, and ticketing systems
- Document runbooks, architecture diagrams, and user guides; train operators and mentor junior engineers
Qualifications:
Required:
- TS/SCI with polygraph
- Bachelor's degree in Computer Science, Information Technology, Software Engineering, or a related field.
- 3+ years deploying and operating at least one enterprise stack: Grafana/Prometheus ecosystem, Elastic (ELK), Dynatrace, or Splunk
- Metrics, logs, and traces instrumentation; working knowledge of OpenTelemetry or Prometheus exporters
- Built operational dashboards and alert rules used by real operators; understands SLI/SLO concepts
- 3+ years with Kubernetes in production; Red Hat OpenShift strongly aligned
- 2+ years with AWS (EC2, S3, EKS, IAM, CloudWatch, Cost and Usage Reports)
- Demonstrated experience analyzing cloud spend, applying tagging strategies, and delivering at least one cost optimization outcome
- Terraform and Ansible; Helm charts or Operators
- Built and maintained pipelines in GitLab CI, Jenkins, or equivalent; Git workflows
- Proficient in Python or Go plus scripting (Bash); able to write and query SQL and PromQL or equivalent
- RHEL administration and troubleshooting
- Hardening to DISA STIGs; familiarity with RMF and NIST SP 800-53 controls
- Worked in air-gapped or disconnected networks, including offline package and image management
Desired:
- Hands-on with Grafana Alloy, Mimir, Loki, and Tempo at scale
- Production time on two or more of Grafana, Elastic, and Dynatrace, enabling side-by-side judgment
- Store-and-forward pipelines, edge collectors, Kafka or similar buffering over intermittent links
- Moved data or software through cross-domain solutions (CDS) or approved transfer processes
- OpenCost or Kubecost, CloudHealth, Apptio Cloudability, or AWS Cost Explorer and Cost and Usage Report analytics
- Istio or a comparable service mesh, including mesh-level metrics and tracing
- Cluster monitoring and user workload monitoring, Operators, OpenShift GitOps and Pipelines
What You Can Expect:
A culture of integrity.
At CACI, we place character and innovation at the center of everything we do. As a valued team member, you’ll be part of a high-performing group dedicated to our customer’s missions and driven by a higher purpose – to ensure the safety of our nation.
An environment of trust.
CACI values the unique contributions that every employee brings to our company and our customers - every day. You’ll have the autonomy to take the time you need through a unique flexible time off benefit and have access to robust learning resources to make your ambitions a reality.
A focus on continuous growth.
Together, we will advance our nation's most critical missions, build on our lengthy track record of business success, and find opportunities to break new ground — in your career and in our legacy.
Pay Range:
There are a host of factors that can influence final salary including, but not limited to, geographic location, Federal Government contract labor categories and contract wage rates, relevant prior work experience, specific skills and competencies, education, and certifications. Our employees value the flexibility at CACI that allows them to balance quality work and their personal lives. We offer competitive compensation, benefits and learning and development opportunities. Our broad and competitive mix of benefits options is designed to support and protect employees and their families. At CACI, you will receive comprehensive benefits such as; healthcare, wellness, financial, retirement, family support, continuing education, and time off benefits.
Since this position can be worked in more than one location, the range shown is the national average for the position.
The proposed salary range for this position is:
$98,500-$206,800CACI is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, age, national origin, disability, status as a protected veteran, or any other protected characteristic.
Similar roles
Cloud/SecDevOps EngineerLeidos · Clarksburg, WV · Hybrid
Science & Technology - HPC DevOps Engineer Intern - Summer 2027Corning · Corning, NY · Onsite
DevOps EngineerDANE, LLC · Washington, DC · Hybrid- DevOps Engineer 1Qu · Remote · Hybrid
DevOps EngineerWafer · San Francisco, CA · Onsite
DevOps EngineerChameleon Integrated Services · Washington, DC · Onsite
