Senior Machine Learning Infrastructure Engineer

We are looking for an ML Infra engineer to help identify and build new machine learning products within our multi-terabyte data lake. You’ll be part of a cross-functional team that includes deep learning experts, software engineers, and officers from all branches of the military. We also do something common in Silicon Valley tech companies but radical in the defense space: we develop and test products side-by-side with our users, including on US military bases and remote locations abroad. Our engineering culture is about empathizing with users and building software products that deliver maximum mission impact.

As a founding member of the team your decisions today will have a large impact on the company today and into the future. We believe that simple systems are easier to understand, maintain, and scale - we prefer pragmatic, product-centric solutions. You are innovative: willing to take risks, think critically about novel solutions, and advocate for your ideas while seeking feedback and collaboration. You will be making trade-offs as you work to ensure that our systems are prepared to scale — being able to effectively communicate these trade-offs with your teammates is a must.

What You’ll Do

• Write data pipeline workflows for building features and training models
• Build robust and scalable machine learning infrastructure for performing model inference in batch and real-time
• Optimize LLM deployment and inference
• Design and build software to support ML model iteration and production deployment such as model repositories and model monitoring software
• Work with development teams and product managers to ideate user experience, product and software solutions
• Design server-side architecture and data model changes to support data products that integrate machine learning
• Develop and manage well-functioning databases and applications
• Write effective APIs
• Test software to ensure responsiveness and efficiency
• Write technical documentation
• Collaborate with analysts to improve software

Qualifications

• Experience engineering data heavy applications/building large data pipelines from scratch
• Experience on an engineering team supporting MLOps software, or similar
• 4+ years as a ML Infra, Backend Engineer or similar role
• Experience with Python
• Familiarity with relational databases
• Excellent communication and teamwork skills
• Ability to learn new technologies quickly and work around the stack

Nice to haves

• Experience with AWS or similar
• Experience with Docker
• Experience with Kubernetes
• Experience with Vector databases
• Experience with Ray
• Experience with Temporal
• Experience with LLM deployment, inference, or prompt engineering
• Experience with model training and development