AI Engineer, Agent Platform

Location: Remote

Department: Information Technology

Reports To: Director of Engineering

## Position Overview

## We’re seeking a Mid-Level AI Engineer to help build, evaluate, and harden production AI agents that power real clinical workflows. This role is execution-focused and hands-on: you’ll work closely with Staff and Senior engineers to optimize prompts, build evaluation harnesses, instrument observability, and support customer-facing deployments.

## You’ll operate at the intersection of AI behavior, workflow reliability, and real-world usage, helping ensure that agents are explainable, measurable, safe, and continuously improving in production. This is an ideal role for an engineer who wants to deepen their expertise in agent systems, evaluation, and applied AI in regulated environments.

## This role is critical to turning AI experimentation into a repeatable, trustworthy production system. You’ll be close to real users, real data, and real outcomes, learning how high-impact AI systems are built, validated, and scaled responsibly.

## Key Responsibilities

• Develop, refine, and test prompts to improve accuracy, consistency, and cost efficiency.
• Collaborate with Staff engineers to align prompts with safety policies, workflow constraints, and clinical acceptance criteria.
• Treat prompts as production artifacts: versioned, reviewed, tested, and rolled out through controlled releases.
• Implement and maintain offline evaluation suites, including golden datasets and regression tests.
• Write evals that measure workflow-level outcomes (accuracy, escalation rate, acceptance rate, error categories).
• Help wire evaluations into CI/CD pipelines to prevent regressions before release.
• Instrument workflows with structured logging, metrics, and traceability.
• Analyze eval results and partner with senior engineers to recommend improvements.
• Work closely with product, clinical, and customer teams to support real deployments.
• Assist with workflow configuration, prompt tuning, and debugging in customer environments.
• Translate customer feedback and production learnings into concrete platform improvements.
• Help bridge the gap between prototype behavior and production-grade reliability.
• Contribute to shared agent libraries, tooling, and utilities under the guidance of senior engineers.
• Follow established reference architectures, safety patterns, and release discipline.
• Participate in design reviews, code reviews, and post-incident retrospectives.

## Required Qualifications

• 5-7 years of experience building and shipping production software systems.
• Strong proficiency in Python (experience with C# or multi-language systems is a plus).
• Hands-on experience working with LLMs, prompts, and tool-calling workflows.
• Experience writing tests or evaluation logic for complex systems (AI or non-AI).
• Familiarity with logging, monitoring, and debugging distributed or asynchronous systems.
• Comfort working in fast-moving environments with real users and production constraints.
• Driven to succeed
• Eligible to work in the U.S. without relying solely on a student visa or a visa sponsored by a third-party employer.

## Preferred Qualifications

• Experience with MLOps tooling, evaluation frameworks, experiment tracking, or data analysis tooling.
• Familiarity with vector databases, embeddings, or retrieval-augmented generation (RAG).
• Exposure to regulated or high-stakes domains (healthcare, fintech, security).
• Experience supporting customer deployments or working in a forward-deployed / solutions role.
• Interest in AI safety, auditability, and explainable systems.

## Core Competencies:

• Flexible, Detail Oriented, Customer Focus, Team Working, Initiative, Problem Solving, Organized, Self-Motivated

## General:

• Adopt the OnPoint culture of respect, integrity, and accountability that contributes to an internal environment of teamwork and promotes a positive brand image to our external customers.
• Incorporate a data-centric leadership mindset to your role.
• Comply with OnPoint procedures, policies, and regulations relevant to your role.
• Successfully completes all OnPoint training requirements (i.e. HIPAA, compliance, etc.)
• Responsible for compliance with all regulatory requirements and/or guidelines. These requirements/guidelines include, but are not limited to: HIPAA, Federal Fraud and Abuse laws.
• Able to communicate effectively with others, both verbally and in writing.
• Proven ability to manage time, meet deadlines and prioritize.
• Able to maintain standards and professionalism during periods of fluctuating workloads.
• Provide professional service to direct customers and vendors/delivery partners of OnPoint in all interactions.
• Build effective working relationships with other team members.
• Manage daily tasks to ensure business needs are consistently met.
• Excellent communication and customer service skills.
• Able to work independently and as part of a team.
• Able to handle multiple tasks and prioritize work effectively.

## Physical Demands:

The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. While performing the duties of this job, the employee is regularly required to speak, hear, read, and type. This is largely a sedentary role; however, some shipping may be required. This position requires the ability to occasionally lift office products and supplies up to 40 pounds.

## Work Environment:

To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed above are representative of the knowledge, skill and/or ability required.

Please note this job description is not designed to cover or contain a comprehensive listing of activities, duties, or responsibilities that are required of the employee for this job. Duties, responsibilities, and activities may change at any time, with or without notice.