Job Type: Full Time

Job Category: IT

Job Description

Role: Python\PySpark Engineer

Location: Jersey City, NJ

FTE only

Job Description

Must Have Technical/Functional Skills

Technical Skills

• Strong proficiency in Python, packaging, dependency management, and virtual environments.
• Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization).
• Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic.
• Solid unit and integration testing practices, including data quality validations.
• Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning.
• Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks).
• Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems.
• Secure coding practices including secrets management, PII protection, and compliance-aware development. 

Functional Skills 

• Strong documentation discipline for frameworks, reusable components, and best-practice patterns. 
• Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines. 
• Clear communication of technical ideas and solution approaches. 
• Comfort working in Agile environments with iterative development and frequent releases.

Roles & Responsibilities

• Design and develop scalable Python/PySpark ingestion and transformation pipelines.
• Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms.
• Optimize Spark jobs for performance, cost efficiency, and production readiness.
• Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance.
• Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability.
• Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions

Required Skills