Job Type: Contract

Job Category: IT

Job Description

Job Title:  Databricks Engineer
Location: Chicago, IL
Contract

Responsibilities:

• Data Pipeline Engineering: Design, develop, and deploy robust and scalable data pipelines using Databricks, incorporating data extraction from diverse sources (databases, APIs, streaming platforms), transformation and cleansing using Spark, and loading into target systems (data lakes, data warehouses, etc.). Optimize pipeline performance for efficiency and cost-effectiveness.
• Databricks Ecosystem Expertise: Utilize the full capabilities of the Databricks platform, including Databricks SQL, Delta Lake, Databricks Runtime, and Databricks Workflows, to orchestrate complex data workflows and ensure data quality and pipeline reliability.
• Cloud Integration: Seamlessly integrate Databricks with cloud services (Azure, AWS, or GCP) for storage, compute, and security, leveraging services like Azure Data Lake Storage, AWS S3, Google Cloud Storage, and more.
• Data Architecture and Modeling: Contribute to the design and implementation of robust data models and architectures for data lakes and data warehouses, ensuring data consistency, integrity, and accessibility for various use cases.
• Data Quality Assurance: Implement rigorous data quality checks and validation procedures throughout the data pipeline to maintain high accuracy and reliability. Adhere to data governance policies and best practices.
• Collaboration and Communication: Work closely with data scientists, analysts, business intelligence specialists, and business stakeholders to understand their data requirements and provide technical expertise on Databricks and data engineering best practices.
• Continuous Learning: Stay up to date with the latest Databricks features, cloud technologies, and data engineering trends to continuously improve our solutions and deliver innovative results for our clients.

#DatabricksEngineer #DataEngineering #DataPipelines #DataScience #DataAnalytics #BigData #DataStorage #DataRetrieval #ETL #DataQuality #InnovativeTechnology #USJobs #TechJobs #EngineeringJobs #ITJobs

Required Skills