Senior Big Data Engineer

Role Description : Senior Big Data Engineer – Scala/Spark

Location: [CLT/NY/NJ/or Remote/Hybrid]

Type: [Full-time/Contract]

Experience Level: Senior level only

Role Overview

We are looking for a highly skilled Senior Big Data Engineer with strong expertise in Scala and Apache Spark to support our data engineering team. The ideal candidate should be able to read and refactor existing Scala Spark code, develop enhancements, create robust unit tests, and help migrate code into a modern cloud-native environment.

Key Responsibilities:

• Analyze and refactor existing Scala Spark code to support migration to new environments.
• Write and enhance Scala Spark code with a focus on performance and maintainability.
• Develop comprehensive unit tests to ensure code quality and stability.
• Work with data stored in Amazon S3 (CSV and Parquet formats).
• Integrate and manipulate data from MongoDB.
• Independently set up environments to validate and test code changes.
• Execute both manual and automated testing procedures.
• Collaborate within an Agile team and contribute to sprint planning, story grooming, and reviews.
• Maintain clear and proactive communication with team members, project managers, and stakeholders.

Technical Skills & Qualifications:

• Proficiency in Scala and Apache Spark for large-scale data processing.
• Working knowledge of Python for scripting and data transformation.
• Hands-on experience with Amazon Web Services (AWS), including:
• S3
• AWS Glue
• Step Functions
• Experience working with MongoDB and structured/unstructured data.
• Familiarity with SQL and BDD frameworks.
• Experience with Terraform for infrastructure-as-code is a plus.
• Strong understanding of software development lifecycle (SDLC), from development to production deployment.
• Experience using tools like Git, Azure DevOps/TFS, and JIRA.
• Comfortable working in Agile/Scrum environments.

Core Technologies:

1. Scala
2. Spark
3. AWS Glue
4. AWS Step Functions
5. Maven
6. Terraform

Soft Skills:

• Strong problem-solving abilities and attention to detail.
• Ability to work independently and as part of a collaborative Agile team.
• Excellent written and verbal communication skills.