We're currently seeking a Data Engineer to build, maintain, and optimise robust data pipelines. This role develops efficient ETL/ELT workflows using both Microsoft Fabric tools and the Databricks Lakehouse Platform to deliver clean, production-ready data for enterprise analytics. • Strong proficiency in PySpark, Databricks SQL, T-SQL, and handling Delta Lake formats.
• Hands-on experience building production pipelines inside Databricks workspaces and Fabric capacities.
• Practical knowledge of structural data formats including Parquet, JSON, and CSV.
• Experience with orchestration tools, scheduling dependencies, and error handling in a cloud environment.
• Familiarity with unit testing data pipelines and implementing automated data quality checks.


• Construct scalable data ingestion pipelines using Fabric Data Factory, Dataflow Gen2, and Databricks Workflows.
• Implement Delta Live Tables (DLT) to automate batch and real-time streaming data processing.
• Build and maintain the Medallion Architecture silver & gold layer across shared storage.
• Write optimized transformations in Fabric Notebooks and Databricks clusters using PySpark and SQL.
• Configure data access, masking, and governance policies within Databricks Unity Catalog.
• Optimize Lakehouse table performance using Z-Order indexing, liquid clustering, and file compaction techniques.
• Deploy data pipelines using CI/CD frameworks, Git integration, and Databricks Asset Bundles (DABs).
• Prepare finalized gold layer datasets to support real-time Power BI Direct Lake reporting model