ASRC Federal is searching for a Staff HPC Engineer to support ASRC Federal Technology Solution out of Manhattan, KS. ASRC Federal Technology Solution proudly supports the USDA Animal and Plant Health Inspection Service (APHIS) Marketing and Regulatory Programs – Information Technology Division (MRP-ITD). The USDA APHIS mission requires rapid, secure, and reliable access to large-scale scientific data for genomics, diagnostics, modeling, and analytics to support decision-making, research, and emergency outbreak responses.
The Scientific Computing (SciComp) HPC system and the Marketplace for Data Analytics (MDA) form the computational, data analytics, and data-sharing backbone for APHIS’s laboratory network as well as other MRP Program areas. These systems facilitate high-speed data transfer, AI modeling, data visualization, and reproducible scientific outcomes through the integration of on-premises HPC clusters, cloud-based analytics platforms, and secure data transfer via Internet2 (I2). Our employees embrace innovation and are committed to a culture of continuous, standards-driven process improvement and assimilation of the industry’s best practices.
Summary: The successful candidate will be an active supporting member of the ASRC Federal team. The HPC Systems Engineer/Administrator is responsible for the administration, maintenance, optimization, and lifecycle management of a high-performance computing (HPC) environment supporting scientific research. This position ensures the availability, security, and performance of compute, storage, networking, and authentication infrastructure while providing technical support to USDA APHIS researchers and collaborating with on-site facilities, networking, and cybersecurity teams.
Primary Roles, Duties and Responsibilities
HPC Infrastructure Administration
Administer and maintain a Linux-based HPC environment consisting of compute, login, management, and GPU nodes.
Manage Slurm workload manager configuration, partitions, queues, scheduling policies, and resource allocation.
Perform operating system installation, provisioning, upgrades, and lifecycle management.
Develop and maintain automation using Ansible, shell scripting, and other administrative tools (NVIDIA Base Command Manager).
Monitor overall cluster health, capacity, utilization, and performance.
Storage Administration
Administer and optimize a multi-petabyte Lustre parallel file system (HPE E1000).
Configure and maintain storage quotas, project directories, tiering policies, and data placement strategies.
Troubleshoot Lustre performance, metadata operations, changelog processing, and client connectivity.
Monitor storage utilization, capacity growth, and system performance using enterprise monitoring platforms (Starfish Storage).
High-Speed Networking
Administer and troubleshoot NVIDIA InfiniBand fabrics, including Subnet Manager, routing, and multi-rail configurations.
Maintain NVIDIA Base Command Manager and Cumulus Linux infrastructure supporting the HPC environment.
Optimize MPI, UCX, NCCL, and RDMA communications for distributed scientific workloads.
Support Internet2 connectivity and high-performance data transfers (Globus Data Transfer Nodes).
GPU Computing Support
Support NVIDIA GPU compute nodes and associated software stacks.
Install, configure, and maintain CUDA, NCCL, GPU drivers, and related scientific software.
Benchmark and validate GPU performance following infrastructure changes.
Software Support
Install, configure, and maintain scientific software packages and development environments.
Build and install software from source using Make, CMake, Ninja, and other build systems.
Configure and troubleshoot software compilation using GCC, G++, and other compiler toolchains.
Resolve library dependencies, compiler compatibility issues, and software linkage problems.
Develop and maintain environment modules supporting multiple software versions.
Support Conda, R, Apptainer, compilers, MPI implementations, CUDA toolkits,…