Calltree Verified
Software, SaaS, Incident Management, DevOps
AI Operations Analyst
San Francisco, California, United StatesOnsiteFull Time$70,000–$100,000 /yrPosted 2 months agoHidden Gem · YC Startup
Role summary
Calltree, a YC W25 company, is seeking an AI Operations Analyst to act as a human supervisor for their AI agents. This hybrid role involves monitoring live AI-customer interactions in real-time, ensuring quality, intervening when necessary to resolve complex issues, and de-escalating frustrated customers. The analyst will identify performance patterns, QA new AI experiences, and collaborate with engineering to improve AI capabilities. This position requires strong judgment, customer support skills, and a passion for both AI technology and helping people. It offers a launchpad for career growth into AI Team Lead or Product roles.
**About Calltree:** We're a YC W25 company with multiple enterprise customers, including public retail brands you'd recognize. We're building AI that actually works for complex, real-world use cases. For example, our AI handles 30+ minute technical troubleshooting calls—the kind everyone else avoids.
**The Role:** You'll be the human supervisor behind our AI agents, working with live customer interactions in real-time. You'll monitor how AI performs, ensure quality, and when needed, jump in to rescue conversations and ensure great customer outcomes. You'll also work directly with engineering to make the AI smarter every day.
**What You'll Do:**
* Monitor live AI-customer conversations and ensure quality standards
* Make real-time judgment calls on when to intervene
* Step in as the escalation point when customers need a human, de-escalating frustrated customers and resolving complex issues
* Identify patterns in AI performance and document what's working and what's not
* QA new AI experiences before release—interact with them as customers would and flag issues
* Work with engineering to improve the AI based on what you observe
* Build the playbooks and processes for this function as we scale
**What Success Looks Like:**
* Measurable improvement in AI performance metrics AND customer satisfaction
* Good judgment on when to intervene vs. let the AI continue
* Catching issues before they hit production
* Building institutional knowledge about what makes conversations succeed or fail
**What We're Looking For:**
* Recent college graduate (any field)
* Comfortable being on the phone with customers, including frustrated ones
* Someone who's excited about AI technology AND genuinely enjoys helping people
* Strong judgment and decision-making skills
* Ability to work one weekend day
**Honest Expectations:** This role is a balance of two things: operational monitoring work (watching, reading, documenting) and live customer interaction. Some days will feel repetitive. But you're also accountable for real customer outcomes—when you jump in, you're the person who saves the experience. If you love technology but never want to talk to customers, this isn't the role. If you want both—mastering AI systems AND helping real people—you'll thrive here.
**Career Growth:** This isn't just a job—it's a launchpad. Top performers have clear paths to become AI Team Leads, move into Product roles, or specialize in AI training and development. You'll gain hands-on experience with cutting-edge AI technology that will set you apart in your career.
**To Apply:** Send your resume with a short video (2-3 minutes). For the video: pick an appliance or product you have in your house, then roleplay a customer service scenario where you're helping a frustrated customer troubleshoot a problem with it.
**The Role:** You'll be the human supervisor behind our AI agents, working with live customer interactions in real-time. You'll monitor how AI performs, ensure quality, and when needed, jump in to rescue conversations and ensure great customer outcomes. You'll also work directly with engineering to make the AI smarter every day.
**What You'll Do:**
* Monitor live AI-customer conversations and ensure quality standards
* Make real-time judgment calls on when to intervene
* Step in as the escalation point when customers need a human, de-escalating frustrated customers and resolving complex issues
* Identify patterns in AI performance and document what's working and what's not
* QA new AI experiences before release—interact with them as customers would and flag issues
* Work with engineering to improve the AI based on what you observe
* Build the playbooks and processes for this function as we scale
**What Success Looks Like:**
* Measurable improvement in AI performance metrics AND customer satisfaction
* Good judgment on when to intervene vs. let the AI continue
* Catching issues before they hit production
* Building institutional knowledge about what makes conversations succeed or fail
**What We're Looking For:**
* Recent college graduate (any field)
* Comfortable being on the phone with customers, including frustrated ones
* Someone who's excited about AI technology AND genuinely enjoys helping people
* Strong judgment and decision-making skills
* Ability to work one weekend day
**Honest Expectations:** This role is a balance of two things: operational monitoring work (watching, reading, documenting) and live customer interaction. Some days will feel repetitive. But you're also accountable for real customer outcomes—when you jump in, you're the person who saves the experience. If you love technology but never want to talk to customers, this isn't the role. If you want both—mastering AI systems AND helping real people—you'll thrive here.
**Career Growth:** This isn't just a job—it's a launchpad. Top performers have clear paths to become AI Team Leads, move into Product roles, or specialize in AI training and development. You'll gain hands-on experience with cutting-edge AI technology that will set you apart in your career.
**To Apply:** Send your resume with a short video (2-3 minutes). For the video: pick an appliance or product you have in your house, then roleplay a customer service scenario where you're helping a frustrated customer troubleshoot a problem with it.
