About the Role
We at Innovaccer are looking for a Principal Engineer to build and own the most amazing product experience for our contact center. You’ll get to work with other engineers to build delightful feature experiences to understand and solve our customers’ pain points. Our backend platform powers a healthcare contact center and AI agent product used by health systems and clinics, running as a set of async, containerized services across multiple live customer deployments, with a growing AI agent stack spanning voice, chat, and SMS. Beyond core ticketing and scheduling workflows, the platform is investing in two strategic capabilities: standalone AI voice/chat/SMS agents that can operate with minimal platform dependency, and an in-call "next best action" layer that turns today's reactive AI assistant into a proactive one.
A Day in the Life
Own the architecture for one or more of: the AI voice agent platform (conversational AI pathway orchestration, call routing, inbound/outbound flows), next-best-action AI assistant guidance, or standalone AI agents that operate across voice, chat, and SMS with minimal dependency on the core platform UI
Design real-time, low-latency systems for in-call intent detection and next-best-action suggestion (sub-second latency budgets across transcription, speaker separation, and inference)
Design an orchestration layer that exposes AI agent workflows as reusable, callable actions consumable by other systems (e.g. a broader automation/journey engine), including output contracts and versioning
Define patient/customer identification, context retrieval, and escalation contracts for agents operating with minimal identifiers, ensuring safe failure and handoff to a human with a full audit trail
Drive the AI agent evaluation and reliability framework (task completion rate, containment rate, transfer rate, conversion/booking rate, webhook success rate, P50/P95/P99 latency) and set SLOs for agent-facing services
Lead technical design reviews (high- and low-level designs) across backend services (core ticketing, workqueue/case management, scheduling, transcription and speech processing) and drive architectural consistency across them
Design and build backend services in Python (FastAPI async, some Django), model data across MongoDB and PostgreSQL, and build Kafka-based event streams for cross-service and agent-to-core communication
Work hands-on with LLM/AI agent tooling (e.g. LangChain, pydantic-ai, MLflow or equivalents) for scheduling agents, in-call recommendations, and automated documentation
Set the bar for testing, observability, and production triage practices across the team; mentor senior and mid-level engineers on distributed-systems and AI-agent design problems
Partner with product on scoping and sequencing ambiguous initiatives, translating them into phased technical plans
Apply security and compliance practices relevant to healthcare data (PHI handling, audit logging, OAuth/JWT auth, "minimum necessary" data scoping for agent context retrieval)
What You Need
8+ years of backend engineering experience, including demonstrated technical ownership of a complex, multi-service system (not just feature delivery)
Strong proficiency in Python, including async programming (asyncio), with production experience in FastAPI (or Django)
Experience designing distributed, multi-service architectures: event-driven systems (Kafka or similar), service-to-service auth, webhook-based integrations, and idempotency/retry design
Experience with MongoDB or another NoSQL database, and working knowledge of relational databases (PostgreSQL)
Direct experience building or operating conversational AI / voice agent systems in production (telephony, IVR, or conversational voice platforms), including latency-sensitive real-time pipelines (speech-to-text, speaker separation, text-to-speech)
Experience designing systems that make real-time decisions from streaming context (e.g. intent detection, recommendation surfacing,…