Senior Dataflow Software Engineer (Software Engineer Level 3 – SWE3)
Wyetech is seeking an experienced Senior Dataflow Software Engineer (SWE3) to design, develop, and optimize scalable data processing solutions supporting mission-critical analytical operations. This position focuses on dataflow architecture, data transport mechanisms, Apache Spark-based distributed processing, and the development of efficient data ingress and egress pathways.
The ideal candidate will possess strong Python and Java development experience, expertise in Apache Airflow and Apache Spark, and a solid understanding of data engineering, workflow orchestration, and large-scale data processing. The Software Engineer will collaborate with data engineers, system engineers, and mission stakeholders to deliver reliable, high-performance data solutions.
Required Skills/Experience:
Experience using Linux command-line interfaces (CLI), system utilities, and troubleshooting tools.
Experience developing Bash scripts to automate manual processes and improve operational efficiency.
Recent hands-on software development experience using Python and Java.
Experience with Apache Airflow, including Directed Acyclic Graph (DAG) design, scheduling, operators, and sensors to orchestrate and monitor complex workflows.
Experience developing and optimizing distributed big data processing solutions using Apache Spark.
Familiarity with SQL database technologies such as MySQL, MariaDB, and PostgreSQL for querying, joining, and aggregating large datasets.
Experience using Jupyter Notebook for analytical development, experimentation, and data exploration.
Experience with data wrangling, transformation, and preprocessing using pandas, NumPy, or comparable technologies.
Experience working with structured, semi-structured, and unstructured data, including Parquet, JSON, CSV, and XML formats.
Familiarity with data quality practices, validation techniques, and anomaly detection.
Proficiency with Git for source code management, version control, and collaborative software development.
Desirable Skills:
Familiarity with High-Performance Computing (HPC) job scheduling technologies, including Slurm.
Experience using Atlassian collaboration and development tools, including Jira and Confluence.
The Software Engineer designs, develops, tests, deploys, documents, maintains, and enhances complex software systems based on documented requirements. These systems may include processing-intensive analytics, large-scale data repositories, real-time applications, data transport services, and distributed computing solutions.
The engineer applies software engineering methodologies, configuration management practices, and system integration principles while supporting software quality, performance optimization, and technical leadership.
Responsibilities and Capabilities:
Design, develop, and optimize dataflow architectures and data transport mechanisms supporting large-scale analytical workloads.
Implement and maintain efficient data ingress and egress pipelines to ensure reliable, scalable data processing.
Develop, schedule, and monitor complex workflows using Apache Airflow, including DAGs, operators, and sensors.
Build and optimize distributed data processing applications using Apache Spark.
Develop and maintain Python and Java applications supporting data engineering and analytical operations.
Create Bash scripts and automation tools to streamline data processing and operational workflows.
Integrate relational databases and diverse data formats into distributed analytical pipelines.
Perform data preprocessing, transformation, validation, and quality assurance to support accurate analytical results.
Analyze system requirements and develop software solutions that meet performance, scalability, and resource constraints.
Troubleshoot software defects, optimize application performance, and support system integration and deployment.
Collaborate with software engineers, data engineers, and technical…