Role: Data Engineer
Location: Charlotte, NC – Onsite
Fulltime – Permanent
Job Description
Skill: IT Security Engineer
Must Have Technical/Functional Skills:
Objective of Role:
The candidate is expected to have knowledge in the cyber security domain.
Primary Level of Engagement: Works as a team member under supervision from a more senior domain expert.
Primary Interactions:
Business partners and stakeholders
Product Owner
Scrum Master
Development Team
Feature Lead
Architect
Domain Community of Practice
Data Scientists
Required Skills:
- Minimum 8+ years hands-on data engineer/Software development exp (at least 5 years of onsite exp)
- Experience with big data processing systems like Hadoop, Cloudera etc
- Experience with one or more of the programming languages like Python (Modular python programming exp)
- Strong SQL and UNIX experience
- Hadoop – Hive, Spark , Pispark
• Proven experience with technology leadership guiding junior engineers towards project completion and deliveries
• Experience with cyber security domain, preferably vulnerability data management
• 8+ years of hands-on data engineer/software development
• Ability to switch from technology leadership to hands on coding when required
• Experience with big data processing systems like Hadoop, Cloudera, etc.
• Experience with Cloud Services\Technologies like AWS, Azure, etc.
• Experience with one or more of the programming languages like Python
• Strong SQL and UNIX experience
• Experience with data stores like HIVE, HBase Elastic
• Experience with code versioning systems like Bitbucket
• Experience with CICD pipelines
• Good knowledge on Data engineering principles and SDLC process
• Ability to detect early and articulate with business partners the challenges and gaps in requirements
Experience: Minimum 8+ years
Roles & Responsibilities:
• Ensure that the design and engineering approach for complex data solutions is consistent across multiple flows and systems.
• Conduct research, designs prototyping and other exploration activities such as evaluating new toolsets.
• Utilize multiple architectural components in design and development of client requirements.
• Maintain, improve, clean and manipulate data for the operational and/or analytics data systems.
• Constantly looking for better ways of solving technical problems and designing the solution without being afraid of challenging the status quo.
• Collaborate with development teams to understand data requirements and ensure the data architecture is feasible to implement and subsequently implemented accurately.
• Assemble large, complex data sets that meet functional/ non-functional requirements.
• Build processes supporting data transformation, data structures, metadata, data quality controls, dependency and workload management.
• Define and build data pipelines that enable faster, better, data-informed decision-making within the business.
• Contribute to existing test suites (integration, regression, performance), analyze test reports, identify any test issues/errors, and triage the underlying cause.
• Document and communicate required information for deployment, maintenance, support, and business functionality.
• Adhere to team delivery/release process and cadence pertaining to code deployment and release
• Identify gaps in data management standards adherence and work with appropriate partners to develop plans to close gaps.
• Lead concept and experimentation testing, and synthesize the results to validate and improve analytical solution.
• Define key performance indicators and internal controls.
• Mentor more junior Data Engineers and coach team members in the delivery/release CI-CD events – (for example, branching timelines, pull requests, issue triage, merge/conflict resolution, and release notes).