IT Delivery

Technology->Cloud Platform->Databricks AI Services,Technology->Data Engineering->Databricks

Key Responsibilities:

• Lead the design and development of scalable data pipelines and ETL/ELT workflows using Databricks on Azure.
• Architect and implement data lakehouse solutions, optimizing storage, compute, and query performance for large datasets.
• Develop, optimize, and maintain Databricks notebooks, jobs, and workflows for batch and near real-time data processing.
• Define and enforce coding standards, best practices, and reusable frameworks for Databricks-based data engineering.
• Collaborate with data architects, analysts, and business teams to understand requirements and translate them into technical designs.
• Implement robust data quality checks, validation rules, and monitoring to ensure reliability and trust in data products.
• Optimize Spark jobs for performance and cost efficiency, including cluster sizing, caching strategies, and partitioning.
• Ensure security, governance, and compliance of data solutions, including access control and data lineage within Databricks and Azure.
• Troubleshoot complex data issues, perform root cause analysis, and drive long-term fixes and improvements.
• Mentor junior engineers, provide technical guidance, conduct code reviews, and support knowledge sharing within the team.
• Contribute to roadmap planning, effort estimation, and delivery management for data engineering initiatives.
• Knowledge of more than one technology
• Basics of Architecture and Design fundamentals
• Knowledge of Testing tools
• Knowledge of agile methodologies
• Understanding of Project life cycle activities on development and maintenance projects
• Understanding of one or more Estimation methodologies, Knowledge of Quality processes
• Basics of business domain to understand the business requirements
• Analytical abilities, Strong Technical Skills, Good communication skills
• Good understanding of the technology and domain
• Ability to demonstrate a sound understanding of software quality assurance principles, SOLID design principles and modelling methods
• Awareness of latest technologies and trends
• Excellent problem solving, analytical and debugging skills