SENIOR DATA ENGINEER, PLATFORM
Salary range: $180,000 - $220,000 per year
ABOUT THE ROLE
Two kinds of data decide whether we can do our job. What a health plan tells us happened to a member, in eligibility files, claims, pharmacy and hospital event feeds. And what our own software records about what we did for them. When a file is late, malformed, or quietly incomplete, care teams work from a stale list. When the two do not line up, nobody can say whether the work is helping.
You will own the infrastructure our data runs on: where and how our jobs run and whether they run efficiently, including compute choices, serverless versus clusters, cost and performance; how work is orchestrated and deployed; how we connect to partners and systems, over SFTP, S3, private networking and partner feeds; and whether all of it is reliable and observable. You will set the technical direction for that platform, make the architecture calls, and build it as software, with tests, version control and infrastructure as code. This is a senior individual contributor role, with a clear path to leading data engineering as the team grows.
This role is complementary to analytics engineering. Analytics owns the questions and the answers, analytics engineering builds the gold tier (curated tables) they work from, and you own the platform underneath it, so those tables arrive complete, on time and at a sensible cost.
We are a small, AI-forward team. The whole team builds with AI coding agents, and part of this platform's job is to make our data usable by AI. We build and evaluate AI systems on real data, where a wrong answer has consequences, so evaluation is part of the work rather than something bolted on at the end.
We want the person who looks at a 384 GB Spark driver and asks why. We are adding markets and data quickly, and we want a platform that grows by adding machines rather than by buying bigger ones. You will know what belongs on a single machine and what does not, and you will keep our pipelines fast, predictable and economical to run as they grow.
Our partners send and expect files on fixed monthly schedules. Eligibility, claims and attribution files come in; engagement, roster and service files go out. Part of the job is making those exchanges boring: replacing recurring manual work with automation, so routine exchanges run themselves and people are pulled in only when something needs judgment. And you will not become the person everything depends on. We treat that as a design flaw, not a badge.
IN YOUR FIRST 90 DAYS, YOU MIGHT TAKE ON ONE OR TWO OF THESE:
- Take over the performance work already underway on our heaviest batch jobs, carry it from single-machine processing onto distributed compute, and prove the output did not change
- Move our legacy Databricks jobs onto our newer platform, with the orchestration, deploys and runbooks to operate them
- Connect a new payer data feed end to end, from the first file over SFTP or S3 to members live in our care platform, monitored from day one
- Review where each of our scheduled jobs runs, serverless or cluster, and right-size the ones that cost more than they should
You will also be close to the people whose work this data describes: coaches, clinicians and researchers who spend their days with our members. When you cannot tell what a field means, or whether a number is plausible, you can go and ask the person who generated it. Most data engineers never get to do that. Our members are among the most underserved people in American healthcare, and these data problems are not abstract.
ABOUT VANNA HEALTH
At Vanna Health, we believe that a sense of belonging and engagement in the community is the foundation for improving overall health and wellbeing for individuals with serious mental illness. We help people living with serious mental illness thrive by strengthening the community ecosystem around them. Through innovative payment models and technology, Vanna Health promotes collaboration across…