Senior Applied AI Engineer (Observability)

We Breathe Life Into Data
At Komodo Health, our mission is to reduce the global burden of disease. And we believe that smarter use of data is essential to this mission. That’s why we built the Healthcare Map — the industry’s largest, most complete, precise view of the U.S. healthcare system — by combining de-identified, real-world patient data with innovative algorithms and decades of clinical experience. The Healthcare Map serves as our foundation for a powerful suite of software applications, helping us answer healthcare’s most complex questions for our partners. Across the healthcare ecosystem, we’re helping our clients unlock critical insights to track detailed patient behaviors and treatment patterns, identify gaps in care, address unmet patient needs, and reduce the global burden of disease.
As we pursue these goals, it remains essential to us that we stay grounded in our values: be awesome, seek growth, deliver “wow,” and enjoy the ride. At Komodo, you will be joining a team of ambitious, supportive Dragons with diverse backgrounds but a shared passion to deliver on our mission to reduce the burden of disease — and enjoy the journey along the way.
The Opportunity At Komodo Health
Healthcare in the U.S. is fragmented and inefficient. Komodo Health is working to fix that—with data.
By building the most complete view of patient journeys across the U.S., Komodo enables life sciences companies, payers, and providers to make better decisions that directly improve patient outcomes.To unlock the full potential of this data, Komodo is investing in AI-native infrastructure—systems that make AI reliable, scalable, and deeply embedded into how products are built and used.
Mission Of The Role
We’re hiring Senior Applied AI Engineers to join a newly formed AI Platform / Observability team focused on building the foundation that makes AI systems trustworthy at scale. This is a backend-heavy, systems-focused role in a greenfield environment. You’ll own problems end-to-end—from early exploration through production deployment—while helping define how AI systems are built, evaluated, and operated across Komodo.
Your starting point will be AI observability, evaluation, and production reliability. From there, your scope expands into broader platform ownership: agent systems, orchestration layers, and shared infrastructure.
Observability is the entry point—not the end state.
Looking back on your first 12 months at Komodo Health, you will have accomplished…

• Build the observability and reliability foundation for AI systems across Komodo (logging, tracing, evaluation pipelines, feedback loops)
• Define how the organization measures LLM performance and quality in production (hallucinations, drift, latency, failure modes)
• Ship production-grade AI systems that improve platform reliability, scalability, and performance
• Lead design and architecture for complex applied AI systems (multi-agent workflows, tool-calling systems, model pipelines)
• Establish evaluation frameworks and experimentation practices (A/B testing, offline + online evaluation)
• Contribute to reusable infrastructure, patterns, and standards adopted across teams

What You'll Own
AI Observability & Reliability (Initial Focus)

• Design and implement:

• Logging, tracing, and request visibility for LLM systems
• Evaluation pipelines and benchmarking frameworks
• Feedback loops for continuous system improvement

• Define metrics for:

• Output quality and correctness
• Latency and system performance
• Tool usage and agent behavior

• Detect and debug: hallucinations, model drift and system degradation and failure modes

Applied AI Systems & Platform (Expanded Scope)

• Architect and deploy end-to-end AI systems agent-based workflows, prompt chains and tool integrations and scalable LLM-powered services
• Transition prototypes into reliable, production-grade systems
• Contribute to shared AI infrastructure and orchestration patterns
• Partner with product, data, and platform teams to shape AI-driven solutions

What You Bring To Komodo Health (required)

• Experience building production AI systems end-to-end (not just prototypes)
• Strong expertise with LLMs and prompt systems and agent orchestration and tool/function calling
• Hands-on experience with:

• AI observability, evaluation, or monitoring systems
• Debugging and improving production AI behavior

• Strong backend engineering skills: Python, APIs, distributed systems, or platform architecture
• Experience designing evaluation frameworks and experiments (A/B testing, benchmarking)
• Ability to operate in ambiguous, fast-moving environments
• Strong communication and mentorship skills
• You will drive experimentation across the organization, set best practices, and integrate new AI techniques into Komodo’s broader engineering ecosystem.

Additional skills and experience we’d prioritize (nice to have)…

• Healthcare data expertise.
• Experience with distributed computing frameworks (e.g., Spark, Snowflake, Databricks) for large-scale data processing.
• Experience building internal observability platforms and LLM evaluation or monitoring systems
• Familiarity with request tracing, replay systems, or model diagnostics
• Location flexible to NYC or SF hybrid, and remote

The pay range for each job posting reflects a minimum and maximum range of annual base pay that we reasonably expect to pay for this position within the US. We carefully consider multiple business-related factors when determining compensation, including job-related skills, work experience, geographic work location, relevant training and certifications, business needs and market demands.
The starting annual base pay for this role is listed below. This position may be eligible for performance-based bonuses as determined in the Company’s sole discretion and in accordance with a written agreement or plan. This role may also be eligible for equity awards. In addition, this role is eligible for benefits including, but not limited to, comprehensive health, dental, and vision insurance; flexible time off and holidays; 401(k) with company match; disability insurance and life insurance; and leaves of absence in accordance with applicable state and local laws and regulations and company policy.
San Francisco Bay Area And New York City
$230,000—$270,000 USD
All Other US Locations:
$200,000—$235,000 USD
Komodo's AI Standard
At Komodo, we're not just witnessing the AI revolution – we're leading it. This is a pivotal moment in time, where being first to market with AI transforms industries and sets the bar. We've already established industry leadership in leveraging AI to revolutionize healthcare, and we expect every team member to contribute. AI here isn't optional; it's foundational. We expect you to integrate AI into your daily work – from summarizing documents to automating workflows and uncovering insights. This isn't just about efficiency; it's about making every moment more meaningful, building on trust in AI, and driving our collective success.
***Join us in shaping the future of healthcare intelligence.*
Where You’ll Work**
Komodo Health has a hybrid work model with hubs in San Francisco, New York City, and Chicago. Roles vary — some can be performed from anywhere in the country, others are scoped to a specific region, and some are based near one of our hubs. For hub-based Dragons, we're building intentional in-office rhythms alongside the flexibility that's core to how we work. Whatever your setup, expectations will always be clear before you join.
Equal Opportunity Statement
Komodo Health provides equal employment opportunities to all applicants and employees. We prohibit discrimination and harassment of any type with regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws.
By submitting your application, you acknowledge that you have read and understand Komodo Health’s Privacy Notice for Employees and Contractors.
*This notice explains how we collect, use, and retain applicant data.*