Lead Data Engineer – Azure Databricks + AI/LLM
- Location
- Cary, North Carolina, United States
- Salary
- Not specified
- Company
- Akash pawar
Job description
Role & Company / Project Overview: Hirelly is hiring for a Lead Data Engineer to lead the architecture and engineering of an AI-first enterprise Data Hub for a global Insurance & Financial Services organization. This is a full-time role requiring hands-on technical leadership to design, review, and troubleshoot production systems, owning the end-to-end delivery of a modern Azure Databricks Lakehouse supporting 150+ data feeds and 35+ downstream systems. Key Responsibilities & Daily Duties: • Own the end-to-end technical design and delivery of an enterprise Azure Databricks Lakehouse architecture. • Support and integrate 150+ data feeds and 35+ downstream systems combining data engineering, cloud, and AI/LLM at scale. • Design enterprise Medallion Architecture and implement data governance, security, and PII protection. • Build and deploy production-grade LLM and GenAI systems, vector retrieval, and agentic workflows. • Implement streaming and batch data pipelines, data quality checks, and automated reconciliation. • Lead technical delivery, communicate architecture decisions to stakeholders, and conduct production code reviews and troubleshooting. Required Qualifications, Skills & Experience: • 12–18 years of Data Engineering / Data Platform experience. • Expert programming skills in Python, PySpark, and Scala. • Strong hands-on expertise with Azure Databricks, Lakehouse architecture, Delta Lake, and Unity Catalog. • Proven experience designing enterprise Medallion Architecture. • 3+ years building and shipping LLM/GenAI systems in production. • Strong experience in RAG, embeddings, vector retrieval, and agentic workflows using LangChain, LlamaIndex, or LangGraph. • Hands-on technical knowledge of Azure Data Factory, ADLS Gen2, Azure Event Hubs / Kafka, Spark Structured Streaming, Databricks Asset Bundles, and Databricks Model Serving. • Experience with Azure OpenAI, OpenAI, Vector & Hybrid Search, Semantic Layer / Text-to-SQL, and AI-driven data mapping & anomaly detection. • Strong data governance, security, and PII compliance experience. • Proficiency in Infrastructure as Code using Terraform and CI/CD with Azure DevOps. • Excellent architecture communication and stakeholder management skills. • Financial Services / Insurance industry experience is highly preferred. • Eligibility: US Citizens Only. Working Hours, Shifts & Location Details: • Location: Cary, NC • Work Mode: On-site / Hybrid • Employment Type: Full-Time Application Instructions: Interested US Citizen candidates should submit their updated resume referencing the role title.