The Opportunity

Design, develop, and optimize high-performance batch and streaming data pipelines using Databricks, Apache Spark, and Python
Optimize Spark jobs for performance, scalability, and cost efficiency
Implement data ingestion frameworks to ingest data from multiple structured and semi-structured sources
Develop data curation and consumption layers that clean, transform, and enriched raw data for analytics and reporting
Enforce data quality rules, validation checks, and schema management throughout the pipeline lifecycle
Create scalable data solutions on AWS leveraging S3, Glue, Lambda, and related services
Ensure strong data governance , including data lineage, metadata management, and access controls.
Implement data security best practices and compliance standards; Monitor pipelines and proactively resolve data quality or reliability issue
Implement CI/CD pipelines using GitHub Actions and Jenkins for seamless deployment and version control
Use Terraform to provision and manage cloud infrastructure following Infrastructure-as-Code best practices
Extensive working experience on SparkSQL and Databricks -Data Ingestion, Data Quality frameworks
Advanced working knowledge of analytical programming- Python. SQL is must.
Exposure to machine learning model implementation in production using MLOps would be added advantage
Extensive working experience in Data curation and analysis
Extensive working experience on Big data delivery DevOps lifecycle including CI / CD processes
Extensive working experience on code refactoring
Good Knowledge in AWS cloud tech stack
Good Knowledge on Airlines Industries /E-commerce platform

AWS Data Engineer

The Opportunity

About Tata Consultancy Services