Basic Function
The Senior Data Engineer is responsible for designing, building, and optimizing the foundational data infrastructure and tools that support Lumin Digital’s data ecosystem. With a strong focus on the Databricks data platform, this role develops scalable, reliable, and cost-effective solutions that enable engineering and analytics teams to accelerate development and delivery. The Senior Data Engineer collaborates with cross-functional stakeholders to identify platform needs, develop integrations, improve performance, and implement reusable tools and capabilities that enhance developer productivity. This role also contributes to technical planning, architectural decisions, and continuous improvement initiatives to ensure the data platform remains scalable, sustainable, secure, and aligned with evolving business needs.
Essential Functions and Responsibilities:
- Design, develop, and maintain scalable, reliable, and cost-effective data platform architecture and foundational capabilities, with a primary focus on Databricks.
- Build and optimize data pipelines, integrations, and reusable tools that streamline data processing, improve developer productivity, and accelerate delivery across engineering and analytics teams.
- Monitor and optimize data platform performance, reliability, and resource utilization, identifying opportunities to improve efficiency and reduce operational costs.
- Collaborate with security and engineering teams to ensure data platform solutions align with established security standards, data governance practices, and applicable regulatory and compliance requirements.
- Partner with cross-functional stakeholders to identify shared platform needs, prioritize foundational improvements, and contribute to technical planning and architectural decisions.
- Perform other duties as assigned.
Physical Demands:
These are standard across Lumin Digital, if you have unique requirements, add them to the Other bullet below.
- While performing the duties of this Job, the employee is regularly required to sit; use hands to type, handle, or feel and talk or hear
- Specific vision abilities required by this job include close vision
- Ability to occasionally lift/move up to 25 pounds
- Individuals with a disability who are otherwise able to perform the essential functions of the job may request reasonable accommodation through the Human Resources department.
- Other (Please specify below)
Position Specifications
Education:
- Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.
Experience:
- 5+ years of experience in software engineering, data engineering, or related technical disciplines.
- 5+ years of hands-on Databricks experience designing, developing, administering, and maintaining scalable cloud-based data platforms.
- Experience building data pipelines, integrations, automation, and reusable tools that support data processing and improve engineering productivity.
- Experience configuring, monitoring, troubleshooting, and optimizing data platform infrastructure for performance, reliability, scalability, and cost efficiency.
- Experience working with cloud infrastructure and services, preferably AWS, with experience in Azure or Google Cloud also valued.
- Experience developing and maintaining solutions using Scala, Python, SQL, Apache Spark, or similar data engineering technologies.
- Experience implementing automated infrastructure provisioning, configuration management, and deployment processes using infrastructure-as-code and CI/CD practices.
- Experience supporting production environments, resolving complex technical issues, and implementing improvements that increase platform stability and operational efficiency.
- Experience collaborating with cross-functional engineering teams to deliver shared technical capabilities and platform improvements.
- Familiarity with streaming data platforms and event-driven architectures, such as Apache Kafka…