Data Engineer (L4)
Elevate K-12
This job is no longer accepting applications
See open jobs at Elevate K-12.See open jobs similar to "Data Engineer (L4)" General Catalyst.Data Science
Bengaluru, Karnataka, India
Posted on Jan 18, 2024
Job Description for Data Engineer
Skills Required:
- Good understanding of cloud technologies - Azure/AWS (primarily - Azure)
- 4+ years of experience in Data Warehousing projects with working experience on data modeling, data normalization techniques, slow changing dimensions, star & snowflake schemas.
- Strong capability on advanced SQL.
- Experience on both structured and unstructured data (RDBMS and NoSQL databases, and vector database would be a plus)
- 4+ years of experience on developing and maintaining data pipelines (ETL).
- Should have experience with handling big data - Spark and Cloud hosted relational databases like Snowflake, Redis, SQL Servers.
- Experience on performance optimization of both data pipelines and complex SQL queries
- 2-3 years of experience on programming language python is must, should have knowledge on modular & OOP programming.
- Technical know-how to convert jupyter notebooks or python code written by data scientists to sophisticated production ready code, preferrable MLOps platform would be Azure Machine Learning services, Spark/Databricks, MLflow.
Roles & Responsibilities –
- Collaborate with business analysts, data scientists and other stakeholders to understand their data needs and requirements.
- Build data pipelines on Azure Data Factory while architecting data pipelines which are easy to maintain.
- Write Documentation (Technical Specifications Documents) for the ETL pipelines developed by you.
- Have ownership of various data pipelines.
- Review and audit datapipelines built by peers.
- Debug the datapipelines at the earliest when they fail.
- Respond quickly to ad-hoc requests to generate any data reports based on SQL queries.
- Think critically when building data pipelines, explicit and easily comprehensible is always better than complex data transformations.
- Optimize datapipelines accordingly based on their resource utilization and execution time.
- Write production ready code for the code written by data scientists preferably on Azure ML layer.
This job is no longer accepting applications
See open jobs at Elevate K-12.See open jobs similar to "Data Engineer (L4)" General Catalyst.