Principal AI/GPU Performance & Validation Architect

Location

Plano, Texas, United States

Salary

Not specified

Company

Akash pawar

Job description

Role & Company / Project Overview: We are hiring for a Principal AI/GPU Performance & Validation Architect. The role consists of two critical halves: making the platform useful by enabling distributed training and production inference on custom-built clusters, and owning the infrastructure validation offering—an independent, evidence-based assessment ensuring multi-million dollar GPU estates perform according to reference architecture specifications. Key Responsibilities & Daily Duties: • Workload Engineering: - Enable and tune distributed training: PyTorch DDP and FSDP, DeepSpeed/ZeRO, tensor and pipeline parallelism, NCCL tuning, MFU measurement and improvement - Architect inference serving (Triton, NVIDIA NIM, vLLM, TensorRT-LLM) with quantization, continuous batching, KV-cache, paged attention, and autoscaling to latency SLOs - Size inference platforms backward from customer SLOs: time-to-first-token, inter-token latency, concurrency, context length - Support enterprise GenAI patterns including RAG, fine-tuning, agentic pipelines, and vector database integration • Validation Offering: - Build and own GPU infrastructure validation suite: fabric validation, NCCL scaling curves, GPU burn and thermal/power soak, storage throughput, GPUDirect verification, reference-workload MFU baselines - Deliver acceptance validation engagements and author signed acceptance reports used to hold vendors accountable to commitments - Design continuous-validation service: post-downtime revalidation, driver and firmware matrix certification, performance drift detection against baseline - Produce published benchmark methodologies and reports to establish technical credibility in the market - Own PoC workload design and benchmark evidence to close engagements Required Qualifications, Skills & Experience: • 5+ years ML engineering, MLOps, or performance engineering with production GPU workloads • Hands-on distributed training on multi-node GPU clusters (not single-GPU or notebook-scale work) • Production LLM inference serving: Triton, vLLM, or TensorRT-LLM • Rigorous benchmarking discipline with ability to profile systems, explain GPU time allocation, and defend performance metrics • Strong Python skills, containers, and Kubernetes-native workflows (Kubeflow, Ray, Argo Workflows) • Clear technical writing skills to produce high-standard customer-deliverable validation reports • Nice to have: NVIDIA NIM and NeMo; MLPerf or formal benchmark program experience; test-and-validation engineering background; fine-tuning at scale (LoRA/QLoRA); customer-facing PoC delivery Working Hours, Shifts & Location Details: • Location: Plano, TX • Work Model: Hybrid Application Instructions: • Interested candidates can forward their resume to the specified recruiter email.