Senior Spark Developer (Python, AWS, SQL)
We are seeking a highly skilled Spark Developer with strong experience in Python, AWS, and SQL to join our team. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing solutions, ensuring data quality, scalability, and performance. This role requires a solid background in distributed computing, cloud environments, and data engineering best practices.
Compensation for NYC: 120-150000 USD Gross per year and based on interview results.
Responsibilities
Design, develop, and maintain scalable data pipelines using Apache Spark (batch and/or streaming).
Build, optimize, and manage ETL/ELT workflows integrating multiple data sources.
Develop data solutions in Python for data transformations, automation, and orchestration.
Leverage AWS services (S3, EMR, Glue, Lambda, Redshift, Kinesis, etc.) to implement cloud-native data platforms.
Write efficient SQL queries for data extraction, transformation, and reporting.
Ensure data quality, lineage, and governance across pipelines.
Collaborate with data engineers, architects, and analysts to deliver end-to-end data solutions.
Troubleshoot performance bottlenecks and optimize Spark jobs for speed and cost-efficiency.
Skills
Must have
8+ years of experience in data engineering or backend development.
Hands-on experience with Apache Spark (PySpark) in large-scale data environments.
Strong proficiency in Python programming.
Expertise in SQL (including advanced queries, performance tuning, and optimization).
Experience working with AWS services such as S3, Glue, EMR, Lambda, Lake Formation.
Understanding of data warehousing concepts and ETL best practices.
Strong problem-solving skills and ability to work in an agile, collaborative environment.
Nice to have
Experience with Databricks or similar Spark-based platforms.
Knowledge of streaming frameworks (Kafka, Flink).
Familiarity with CI/CD pipelines, Docker, Kubernetes, Terraform.
Exposure to data modeling (star schema, snowflake, data vault).
Experience in financial services / capital markets .
Other
Languages
English: C1 Advanced
Seniority
Senior
Recommended Jobs
Barista
About the Role: We’re looking for enthusiastic baristas to join the Lazy Sundaes team! You’d be a great fit if you are passionate about sweet treats, care about quality customer service, and want to…
Ui/visual designer
Job Description The role involves investigating user experience design requirements across digital assets and formulating a comprehensive UI/UX strategy aligned with the pre-paid phone brand yo…
Program Manager - MSP
POSITION TITLE : Program Manager LOCATION : Rochester, NY COMPENSATION : $70,000 - $75,000 GENERAL JOB DESCRIPTION : The Program Manager is responsible for delivering operatio…
Class A OTR No Touch Driving Job
Class A OTR No Touch Driving Job We are a long-haul (OTR) trucking company located in Long Island, NY. We are looking for a CDL-A driver to grow with, driving reefer and dry van work for long runs (2…
Emergency Medicine Advanced Practice Clinician in Brooklyn, NY
Are you looking for some extra income and have current or previous emergency medicine (EM) experience as an advanced practice clinician (APC)? If so, come grow your career with TeamHealth and this ex…
Senior Platform Engineer
About Polymarket Polymarket is the world’s largest prediction market. It’s half "put your money where your mouth is," half search engine for the future. We're growing fast – both in terms of volu…