Lead Data Engineer with AI experience

Lead Data Engineer with AI Experience

3Pillar is an AI transformation partner on a mission to help enterprises build the AI-native products and intelligent agents that will define the next era of business. We are looking for a Lead Data Engineer to build, operate, and continuously improve the data pipelines, retrieval infrastructure, and ML/LLMOps foundations that power our AI initiatives. You will work on turning reference architectures and data contracts into robust, production-grade implementations that serve conversational AI assistants, dashboard copilots, autonomous agents, RAG applications, and predictive ML models.

Key Responsibilities

• Data Pipeline Engineering: Build, test, and maintain production pipelines (batch & real-time) on Snowflake, PySpark, Delta Lake, and Kafka. Implement data quality checks, schema validation, and alerting at every pipeline stage. Migrate legacy ETL/DWH to cloud-native AWS/Azure architectures with measurable latency and cost improvements. Maintain CI/CD pipelines: automated testing, deployment, rollback, and IaC (Terraform, GitHub Actions).
• RAG, Vector & Retrieval Infrastructure: Build end-to-end retrieval infrastructure: document ingestion, embedding pipelines, vector store management (Pinecone, FAISS, ChromaDB, OpenSearch), and hybrid retrieval layers. Implement chunking, metadata filtering, and re-ranking — tuning for precision, recall, and latency. Maintain data freshness and index consistency; instrument with context relevance and faithfulness metrics.
• Semantic Layer & Knowledge Infrastructure: Implement and maintain business entity mappings, ontologies, and knowledge graphs (Neo4j) per Architect design. Build and version the feature store and semantic data contracts serving both ML models and LLM applications. Manage metadata, data lineage, and audit trail instrumentation across the platform.
• ML/LLMOps Pipeline Support: Build ML data infrastructure: training curation, feature engineering, MLflow experiment tracking, dataset versioning. Support LLM fine-tuning workflows — corpus curation, quality filtering, dataset formatting. Implement automated evaluation pipelines: factual accuracy, hallucination detection, regression tracking. Maintain production monitoring dashboards for pipeline health, model metrics, and alerting.
• Agentic Data Infrastructure: Build and maintain data APIs, tool schemas, and memory/state stores that autonomous agents depend on. Implement agent observability: capture inputs, retrieved context, tool calls, reasoning traces, and outputs. Maintain text-to-SQL layers, semantic query interfaces, and context APIs for conversational AI consumers.
• Governance, Security & Data Quality: Implement RBAC, attribute-based access, PII detection/masking, data classification, and audit logging. Enforce data contracts and schema governance with automated breaking-change detection and versioned migrations. Build data quality monitoring (completeness, freshness, consistency) with automated alerting and root-cause tooling. Support compliance readiness: audit trails, data provenance, and regulatory documentation.

Qualifications

• 7+ years data engineering using Cloud services
• 2+ years production AI/ML or LLM-era data infrastructure. Proven experience building production pipelines at scale — batch and streaming, Snowflake, AWS/Azure.
• Deep expertise: Python, PySpark, Snowflake, Delta Lake, Kafka, Spark Structured Streaming.
• Hands-on with vector stores, embedding pipelines, and retrieval infrastructure.