Senior Software Engineer- Data Acquisition
Luma AI
This job is no longer accepting applications
See open jobs at Luma AI.See open jobs similar to "Senior Software Engineer- Data Acquisition" General Catalyst.Software Engineering
Palo Alto, CA, USA
Posted on May 16, 2024
We are looking for people with experience gathering data from the web at scale. You’ll be part of Luma’s infrastructure team, working cross-functionally with Applied AI and Research on mission critical work-streams.
Responsibilities
- Own / lead engineering projects for data acquisition.
- Work with Research and Applied AI to ensure their data and technical needs are met.
- Develop and deploy highly scalable distributed systems capable of handling petabytes of data.
- Build and gather high quality datasets by collaborating closely with the research team.
- Build and maintain backend services for data storage.
- Architect and implement algorithms for data management including indexing & search.
- Conduct and analyze system performance experiments.
Experience
- Proficiency in Python.
- Experience of 5+ years in web-scraping at-scale.
- Experience building and working efficiently in very large distributed systems.
- Experience working with Kubernetes, proxies and other relevant tools.
- Experience working with research datasets.
- Please note this role is not meant for recent grads.
All your applications are reviewed by real people.
This job is no longer accepting applications
See open jobs at Luma AI.See open jobs similar to "Senior Software Engineer- Data Acquisition" General Catalyst.