Location : Bangalore or Chennai, with no travel

Full-time for 1 year on loan to client

Qualification: B.Tech/BE/MCA

3-5 years experience

At one of the world’s largest retail conglomerate, there are massive scale data pipelines that operate to produce mission-critical insights on a weekly basis every week throughout the year. These data pipelines pick up data from Teradata warehouse (in their data center), populate the Hadoop data lake (in their data center), and then extract to Azure environment, where there are multiple Machine Learning and Data Science models written in Spark that are executed using Databricks. Airflow is the scheduling engine used throughout to control the pipelines.

- Monitor the data pipelines for failures as L1 support

- Implement alerting mechanisms to ensure failures are communicated

- Troubleshoot and identify root cause, including data-related problems and big data system-related problems

- Map dependencies and take action on downstream dependent tasks in case of failure of a task in the pipelines, including understanding, changing and triggering Airflow bundles
- Handle customer communication across timezones

- Work with development team for issue resolution
- Manage ticketed query system and ensuring comprehensive database of queries and resolutions is kept up to date
- Maintain and update technical documents and procedures

- Coordinate with platform teams

- Be willing to work night shift, Saturdays, holidays and/or Sunday on-call support.