AI Operations Specialist – LLM, Prompt Engineering & Conversation Systems | Remote
Position Type: Full-Time, Remote
Working Hours: U.S. Business Hours
Industry: Artificial Intelligence / SaaS / Telehealth Technology
Department: AI Operations / Conversational AI
Focus: Production LLM Systems, AI Conversation Optimization & Conversion Performance
About the Role
At Pavago, one of our clients is hiring a highly skilled AI Operations Specialist to manage, monitor, troubleshoot, and continuously optimize production AI conversation systems.
Our client operates an AI-powered SMS lead recovery platform serving telehealth brands , processing millions of messages every month. These AI systems communicate directly with customers, making conversation quality, reliability, and conversion performance critical to business success.
This is a hands-on, execution-focused role for someone with proven experience operating Large Language Model (LLM) systems in production , rather than simply experimenting with AI tools.
You’ll be responsible for onboarding new clients, configuring AI conversation workflows, managing knowledge bases and integrations, monitoring live customer interactions, identifying AI failure modes, optimizing prompts, and measuring the impact of changes on conversion performance.
The ideal candidate combines technical AI knowledge, prompt engineering expertise, analytical thinking, and operational ownership with the ability to work independently in a fast-moving environment.
You’ll collaborate directly with the founder to ensure AI systems consistently deliver accurate, effective, and conversion-focused customer conversations.
What You’ll Own
Client Onboarding & AI System Configuration
Onboard new clients onto the AI-powered conversation platform.
Configure client-specific AI conversation systems and workflows.
Set up and troubleshoot webhook integrations, APIs, and knowledge bases .
Configure SMS campaigns and conversation workflows.
Ensure AI systems are aligned with client requirements and objectives.
Test conversation flows and integrations before deployment.
Identify and resolve configuration issues during onboarding.
Verify that new client systems are functioning correctly before going live.
Maintain accurate documentation of client configurations and setup processes.
Production AI Operations & Monitoring
Monitor live AI-generated conversations and customer interactions .
Review conversation transcripts to identify errors, inconsistencies, and missed conversion opportunities.
Diagnose AI behavior issues and determine their underlying causes.
Identify patterns affecting conversation quality, accuracy, and performance.
Detect unexpected AI responses, workflow failures, and system inconsistencies.
Investigate issues involving conversation logic, prompts, knowledge bases, or integrations.
Escalate technical issues when necessary and follow through until resolution.
Maintain consistent quality standards across customer-facing AI systems.
Proactively identify opportunities to improve production reliability and conversation effectiveness.
Prompt Engineering & LLM Optimization
Write, edit, and optimize system prompts and customer-facing AI messaging instructions .
Improve AI conversation logic based on real customer interactions.
Identify weaknesses in existing prompts and system behavior.
Adjust instructions to improve response quality, accuracy, and consistency.
Test prompt changes systematically before deployment.
Evaluate AI behavior before and after implementing changes.
Document prompt updates, testing results, and system modifications.
Continuously refine AI-generated responses to improve customer engagement.
Ensure prompt changes support both conversation quality and conversion objectives.
AI Conversation Analysis & Quality Assurance
Review large volumes of AI-generated conversation transcripts.
Identify recurring response errors and behavioral patterns.
Analyze missed opportunities within customer conversations.
Evaluate…