AI Operations Specialist

AI Operations Specialist

Remote, U.S.

What This Role Is

We are looking for a hands-on AI Operations Specialist to own customer AI implementations from discovery through launch and ongoing optimization.

You will configure and improve LLM-powered workflows, evaluate production behavior, troubleshoot issues, and work directly with customers and internal teams to make sure implementations work reliably in the real world.

This is a customer-facing technical IC role. You should be comfortable moving between AI behavior, implementation details, customer workflows, and production troubleshooting.

What You’ll Do

• Own customer AI implementations from discovery through launch and ongoing optimization.

• Translate customer workflows and requirements into clear AI behavior, implementation plans, and testing criteria.

• Design and improve prompts and workflow logic for customer-facing AI agents.

• Build and run evaluations, test plans, and regression checks to improve reliability and consistency.

• Investigate production issues across prompts, workflows, integrations, data, APIs, and application behavior.

• Determine whether problems can be resolved directly or require product or engineering support, and escalate with clear evidence and reproduction steps.

• Facilitate customer testing and troubleshooting, and communicate status, risks, blockers, and required decisions clearly.

• Drive open issues to resolution across customers and internal teams, including when ownership or next steps are ambiguous.

• Use recurring customer and production issues to improve implementation practices and product behavior.

What You Bring

• Hands-on experience working with LLM-powered products or workflows in production.

• Experience building evaluations and improving unreliable or nondeterministic AI behavior.

• Strong technical troubleshooting skills and the ability to investigate complex systems without immediately handing problems to engineering.

• Experience owning customer implementations from discovery through launch and optimization, including working directly with customers throughout the process.

• Experience with healthcare technology and/or conversational or voice AI is helpful but not required.

Benefits

• Health, dental, and vision insurance

• 401(k) with company match

• Flexible PTO and company-paid holidays

• Fully remote work within US hours

• Company-provided laptop