Job Description:
Department Overview
The Data Platform Department (DPD) at Rakuten Group develops and maintains a comprehensive data platform, empowering over 70 Rakuten services with solutions for data ingestion, discovery, governance, analytics, and querying. We support data-driven decision-making across one of Japan's largest data ecosystems, providing the tools and infrastructure to support key domains such as Data Lakes, Data Warehouses, and Business Intelligence.
Position:
Position Details
We are seeking a DevOps Engineer to manage the end-to-end operations of our enterprise Databricks platform hosted on Azure and GCP. The role will focus on platform administration, automation, CI/CD, monitoring, security, incident management, and ensuring a secure, reliable, and highly available platform.
Key Responsibilities
Manage day-to-day operations of Azure and GCP Databricks environments
Automate infrastructure provisioning and platform configuration using Terraform
Build and maintain CI/CD pipelines for Databricks deployments
Monitor platform health, performance, availability, and costs
Troubleshoot production issues, perform root cause analysis, and drive operational improvements
Manage platform security, access controls, secrets, and audit logging
Collaborate with engineering, cloud, networking, and security teams to support platform operations
Mandatory Qualifications:
3+ years of hands-on experience administering Azure Databricks in production environments
Strong knowledge of Unity Catalog, workspace administration, cluster policies, compute management, Jobs & Workflows, permissions, secret scopes, and audit logs
Experience managing multiple Databricks workspaces across development, test, and production environments
Strong hands-on experience with Microsoft Azure and working knowledge of Google Cloud Platform (GCP)
Experience with cloud networking concepts including VNet/VPC, Private Link/Private Endpoints, firewall rules, DNS, VPN/Peering, and secure connectivity
Hands-on experience with Terraform for provisioning and managing cloud infrastructure
Experience automating infrastructure deployment, configuration management, and environment standardization
Experience building and maintaining CI/CD pipelines using Azure DevOps, GitHub Actions, GitLab CI, or Jenkins
Strong knowledge of Git, branching strategies, release management, and deployment automation
Experience implementing monitoring, logging, and alerting using Azure Monitor, Log Analytics, GCP Cloud Monitoring, or similar tools
Strong experience in production support, incident management, root cause analysis (RCA), and operational runbook creation
Experience implementing RBAC, Microsoft Entra ID (Azure AD), IAM, service principals, managed identities, secret management, and audit logging
Good understanding of cloud security best practices and compliance requirements
Proficiency in Python and Bash (PowerShell is an advantage)
Experience developing automation scripts for operational and deployment tasks
Strong understanding of DevOps principles, Infrastructure as Code, CI/CD, GitOps, change management, and platform reliability engineering
Experience managing multi-cloud Databricks platforms (Azure & GCP)
Knowledge of Delta Sharing, Delta Live Tables, Databricks Asset Bundles, and Lakehouse architecture
Experience with Docker, Kubernetes, Grafana, Prometheus, Splunk, or Datadog.
Experience with cloud cost optimization (FinOps)
Azure, GCP, Databricks, or Terraform certifications
Preferred Qualifications:
Strong troubleshooting and analytical skills
Experience supporting mission-critical production platforms
Excellent communication and documentation skills
Automation-first mindset with a focus on continuous improvement
#engineer #infrastructureenginee #technologyplatformdiv