Job Summary
The Architect role involves designing and implementing scalable data solutions using advanced technologies like MS Data Fabric Power BISpark OneLake SQL PySpark and Have hands- on Experience in Power BI. The candidate will work in a hybrid model collaborating with cross-functional teams to deliver high-quality data architectures that drive business insights and innovation.
Key Responsibilities: Data Engineer
Pipeline Development: Design, develop, and maintain robust, scalable ETL/ELT data pipelines using Azure Data Factory (ADF) and Azure Databricks.
Data Transformation: Utilize PySpark and SparkSQL to perform complex data transformations, cleaning, and aggregation.
Azure Data Lake Management: Work with Azure Data Lake Storage (ADLS) Gen2 and Azure Blob Storage for efficient data handling.
Data Modeling: Create and optimize data models (conceptual/logical/physical) for Lakehouse architecture (Medallion: Bronze, Silver, Gold).
Optimization: Perform advanced performance tuning for PySpark jobs and ADF pipelines to ensure reliability and cost efficiency.
CI/CD & DevOps: Implement CI/CD pipelines for data engineering code using Git, Azure DevOps, or GitHub Actions.
Collaboration: Work with stakeholders, data scientists, and analysts to translate business requirements into technical solutions
Required Skills: Microsoft Fabric, ADF, Azure Databricks, PySpark, Python, ADLS, SQL, ADO
Certifications Required
Certified Spark Developer Python Data Engineering Certification SQL Database Certification MS Power BI
