Lead Data Engineer (Hands-On)
- Location
- Cary, North Carolina, United States
- Salary
- Not specified
- Company
- Akash pawar
Job description
• Role & Company / Project Overview We are looking for an experienced Lead Data Engineer with strong expertise in Azure Databricks, large-scale data engineering, and Generative AI to build an enterprise-scale, AI-first Data Hub for a leading financial services and insurance client. We are seeking a hands-on technical leader who can design scalable data platforms, write production code, optimize pipelines, and mentor engineering teams. • Key Responsibilities & Daily Duties - Design scalable enterprise data platform architecture and provide technical leadership - Write robust production code and build large-scale data pipelines - Build and deploy production-grade LLM-based systems and AI-powered data pipelines - Optimize data pipelines and implement Medallion Architecture with Delta Lake - Implement data quality frameworks, anomaly detection, and data governance standards - Mentor engineering teams and drive CI/CD best practices • Required Qualifications, Skills & Experience - Experience: 12–18 Years of overall experience - Must Have: 3+ years of hands-on experience building and deploying production-grade LLM-based systems, along with strong data engineering and architecture expertise - Strong proficiency in Python, PySpark, Scala, and SQL - Deep expertise in Azure Databricks, Delta Lake, and Medallion Architecture - Proficiency with Azure Data Factory (ADF) and ADLS Gen2 - Generative AI, LLMs, RAG, and AI-powered data pipelines - Hands-on experience with LangChain, LangGraph, or LlamaIndex - Data quality, anomaly detection, and data governance - Azure DevOps, Terraform, and CI/CD - Enterprise data platform architecture and technical leadership • Working Hours, Shifts & Location Details - Employment Type: Full-Time - Location: Cary, NC (On-site) • Compensation & Benefits - Not specified • Application Instructions - Interested candidates, please submit your updated resume for consideration.