Databricks Senior Data Engineer – Pyspark
- Location
- New York, New York, United States
- Salary
- Not specified
- Company
- Akash pawar
Job description
Role & Company / Project Overview: We are seeking a talented Databricks Senior Data Engineer – Pyspark for an onsite role in New York. In this role, you will lead the design and delivery of scalable, cloud-native data pipelines for a major enterprise, driving high performance and reliability across AWS and GCP environments. Key Responsibilities & Daily Duties: • Build and optimize Databricks pipelines using Spark/PySpark across AWS and GCP platforms. • Design and implement ETL/ELT flows that integrate diverse data sources into Delta Lake. • Tune Spark jobs, optimize clusters, and partition storage for maximum throughput. • Develop monitoring, alerting, and observability solutions for production reliability. • Author complex BigQuery SQL for cross-platform analytics. • Collaborate with architecture leads on design reviews and standards enforcement. • Produce comprehensive documentation, runbooks, and implementation guides. • Participate in code reviews and mentor junior team members. • Build automated testing and data validation frameworks. Required Qualifications, Skills & Experience: • Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field. • Total Experience: 12+ years of total data engineering experience, including 4+ years of hands-on experience with Databricks/Spark. • Deep expertise in Spark, PySpark, Delta Lake, and SQL performance tuning. • Proven experience deploying Databricks on AWS (S3/Glue) and GCP (GCS/BigQuery). • Advanced Python programming skills (Pandas/NumPy) and familiarity with pytest testing. • Strong knowledge of cloud services including AWS S3/Glue and GCP BigQuery/Dataflow. • Consulting background with end-to-end project delivery. • Proficiency with Git version control and CI/CD pipelines. • Excellent communication skills for presentations and technical documentation. • Eligibility: Must be based in the United States and fully authorized to work in the US. Working Hours, Shifts & Location Details: • Location: Onsite in New York, NY, United States. Application Instructions: • Submit your resume for review if you meet the qualifications outlined above.