Databricks Engineer - Senior/ Lead

Accountabilities:

• • Design, develop, and maintain scalable data pipelines and data engineering solutions using Azure Databricks.
  • Build robust ETL/ELT workflows using Python, PySpark, Spark SQL, and Apache Spark.
  • Develop complex SQL queries and transformations to process and prepare large datasets for analytics and machine learning.
  • Optimize Spark jobs and distributed processing workloads for performance, scalability, reliability, and cloud cost efficiency.
  • Work with structured and semi-structured data formats including Parquet, Delta, JSON, and CSV.
  • Design, build, and maintain Delta Lake tables, leveraging capabilities such as ACID transactions, time travel, and schema evolution.
  • Integrate Databricks workloads with Azure Data Lake Storage Gen2 and other Azure data services.
  • Apply distributed computing principles to efficiently process large-scale datasets.
  • Implement data quality, validation, monitoring, and error-handling processes across data pipelines.
  • Collaborate with data scientists, analysts, and business stakeholders to understand requirements and deliver reliable data solutions for analytics and ML use cases.
  • Apply Azure security, access control, governance, and RBAC best practices across data engineering environments.
  • Use Git-based version control and follow collaborative software development practices.
  • Contribute to CI/CD pipelines and automated deployment processes using tools such as Azure DevOps or GitHub Actions.
  • Support data modeling initiatives and help ensure data structures are optimized for downstream analytics and reporting.
  • Contribute to streaming data solutions using technologies such as Spark Structured Streaming, Azure Event Hubs, or Kafka.
  • Identify opportunities to improve pipeline architecture, engineering standards, performance, and operational efficiency.
  • Provide technical guidance and contribute to engineering best practices appropriate for a Senior or Lead-level position.

  Requirements:

  • 6+ years of professional experience in Data Engineering or a closely related field.
  • Strong hands-on experience designing and developing solutions with Azure Databricks.
  • Advanced proficiency in Python for data processing, transformation, and pipeline development.
  • Strong SQL skills, including complex joins, window functions, data transformations, and query performance optimization.
  • Hands-on expertise with Apache Spark and PySpark, including distributed data processing and Spark job optimization.
  • Solid experience working with Delta Lake, including ACID transactions, time travel, and schema evolution.
  • Strong knowledge of Azure Data Lake Storage Gen2 and cloud-based data architecture.
  • Understanding of distributed computing concepts and the ability to design solutions for large-scale data workloads.
  • Experience using Git for version control and collaborative development.
  • Strong understanding of modern ETL/ELT patterns, data pipeline architecture, and data engineering best practices.
  • Experience with Azure Data Factory is preferred.
  • Exposure to CI/CD practices and tools such as Azure DevOps or GitHub Actions is an advantage.
  • Basic understanding of data modeling and its application to analytical workloads.
  • Familiarity with Azure security principles, RBAC, access control, and data governance.
  • Exposure to real-time and streaming data technologies such as Spark Structured Streaming, Event Hubs, or Kafka is a plus.
  • Strong analytical and problem-solving abilities, with a structured approach to debugging and performance optimization.
  • Strong communication and collaboration skills, with the ability to work effectively with technical teams, data scientists, analysts, and business stakeholders.
  • Ability to take ownership of complex engineering initiatives and provide technical direction at a senior or lead level.

  Benefits:

  • Opportunity to work on large-scale data engineering and analytics initiatives within an AI and technology-focused environment.
  • Hands-on experience with Microsoft Azure, Azure Databricks, Apache Spark, Delta Lake, and modern cloud data platforms.
  • Opportunity to design and optimize production-grade data pipelines handling complex and large-scale datasets.
  • Exposure to advanced distributed data processing, cloud architecture, and modern ETL/ELT practices.
  • Opportunity to contribute to analytics and machine learning use cases in collaboration with data scientists and analysts.
  • Exposure to CI/CD, cloud security, data governance, and streaming data technologies.
  • Senior/Lead-level scope with opportunities to influence engineering standards and technical architecture.
  • Collaborative environment involving data engineering, analytics, data science, and business stakeholders.
  • Opportunities for continued technical growth across cloud data engineering, AI, and emerging data technologies.