Job Type: Contract

Job Category: IT

Job Description

Job Title: Data Pipeline Engineer
Location: Remote
Contract

Job Summary:

We are seeking a highly skilled Data Pipeline Engineer to design, build, and optimize robust data pipelines and platforms supporting large-scale data analytics and processing workloads. The ideal candidate will have hands-on experience with Databricks, Azure SQL Server, PySpark, and modern ETL/ELT frameworks, coupled with a solid understanding of data modeling, containerization, and scalability best practices.

Key Responsibilities:

·         Design, develop, and maintain data pipelines using Databricks (Python and PySpark) to support ingestion, transformation, and storage of large datasets.

·         Implement ETL/ELT processes to extract data from various sources, ensuring data integrity, consistency, and accuracy across systems.

·         Develop and optimize data models to support analytical and operational data needs.

·         Build and manage Azure SQL Server instances including Managed Instances, Azure SQL Database, and SQL Server VMs for data storage and access.

·         Apply data engineering best practices for performance tuning, scalability, and cost optimization across the data pipeline infrastructure.

·         Utilize Docker and Azure Kubernetes Service (AKS) for containerized deployment and orchestration of data pipeline components.

·         Collaborate with Data Scientists, Analysts, and Business Stakeholders to translate business requirements into technical solutions.

·         Monitor and troubleshoot data pipeline performance, ensuring high availability, reliability, and data quality.

·         Maintain thorough documentation of data architecture, data flows, and pipeline configurations.

·         Drive automation initiatives to streamline deployment, testing, and data operations.

Required Skills & Qualifications:

·         Proven experience with Databricks using Python, SQL, and PySpark.

·         Strong knowledge of Azure SQL Server ecosystem — including Managed Instance, Azure SQL DB, and SQL Server VMs.

·         Expertise in data modeling, ETL/ELT design patterns, and data integration frameworks.

·         Solid understanding of containerization (Docker) and Kubernetes (AKS) for data engineering workloads.

·         Experience in performance optimization and scalability in cloud data environments.

·         Strong SQL development skills, including query optimization and stored procedure development.

·         Hands-on experience with Azure Data Services (ADF, Synapse, Storage Accounts) is a plus.

·         Familiarity with CI/CD pipelines for automated deployment of data solutions.

Required Skills