Senior Specialist - Data Engineering

Location: Pune

Experience: 7–11 Years

About the Role

We are seeking an experienced Data Engineer with 8+ years of expertise in Big Data Engineering, Databricks, Apache Spark, PySpark, Scala, and SQL. The role focuses on building and delivering scalable data solutions using Databricks, Delta Lake, Lakehouse architecture, AWS, and Power BI in a fast-paced Agile environment.

Job Description

The role involves designing, developing, and supporting modern data engineering solutions using Databricks and Apache Spark, with strong emphasis on PySpark, Scala, Python, SQL, and data modeling. The candidate will work with Delta Lake and Lakehouse architecture to build scalable distributed data processing solutions.

The position requires hands-on experience with AWS Big Data services, including S3, Glue, EMR, DMS, Athena, RDS, Redshift, and Kinesis, as well as experience managing EMR clusters and working with relational databases such as PostgreSQL, Oracle, and SQL Server.

The candidate will also contribute to data orchestration, streaming, CI/CD, source control, data cataloguing, and Power BI solutions. Experience with Agile at Scale, Continuous Integration, Test-Driven Development, and Infrastructure as Code practices will support successful delivery.

Key Responsibilities

• Design and develop scalable data engineering solutions using Databricks, Apache Spark, PySpark, Scala, and Python.
• Build and maintain data pipelines using Delta Lake and Lakehouse architecture.
• Develop complex SQL queries and implement effective data models across relational databases.
• Work with AWS Big Data services and manage EMR clusters and associated services.
• Develop data visualization and reporting solutions using Power BI.
• Implement data orchestration, streaming, CI/CD, and source control practices.
• Work with data cataloguing solutions such as Unity Catalog.
• Collaborate with stakeholders in Agile environments and take ownership of data engineering deliverables.

Required Skills

• Databricks
• Apache Spark, PySpark, and Scala
• Python
• SQL and Data Modelling
• Power BI
• Delta Lake and Lakehouse architecture
• AWS cloud platform
• AWS services: S3, Glue, EMR, DMS, Athena, RDS, Redshift, Kinesis
• EMR cluster management
• Data orchestration tools
• Kafka / streaming frameworks
• RDBMS: PostgreSQL, Oracle, SQL Server
• Complex SQL queries, particularly in PostgreSQL
• Git and CI/CD pipelines
• Azure DevOps / GitHub
• Unity Catalog
• Distributed computing principles
• Agile, Continuous Integration, Test-Driven Development, and Infrastructure as Code practices

Preferred / Good to Have

• Data warehousing concepts
• Experience supporting Machine Learning pipelines in Databricks
• MLflow experiment tracking and model registry
• Databricks Mosaic AI, Vector Search, and embeddings pipelines
• Generative AI concepts including LLMs, embeddings, and RAG
• Terraform or CloudFormation
• Debezium or other CDC tools
• Jenkins-based workflow orchestration

Mandatory Skills

• Databricks

Candidate Profile

The ideal candidate will have 8+ years of Data Engineering experience, including strong hands-on expertise in Databricks, Spark, Python, SQL, AWS, and modern data architecture. The candidate should possess strong analytical and problem-solving abilities, communicate effectively with stakeholders, and demonstrate ownership while working in a fast-paced Agile environment.

Benefits Package

• Competitive salary and benefits package
• Opportunities for professional growth and development
• A collaborative and inclusive work environment
• The opportunity to work on exciting and challenging projects with leading clients

Originally posted on Himalayas