Responsibilities

<ul>
<li>Create and maintain optimal data pipeline architecture; assemble large, complex data sets that meet functional / non-functional requirements.</li>
</ul>

<ul>
<li style="text-align:justify;">Design the right schema to support the functional requirement and consumption patter.</li>
</ul>

<ul>
<li style="text-align:justify;">Design and build production data pipelines from ingestion to consumption.</li>
</ul>

<ul>
<li>Create necessary preprocessing and postprocessing for various forms of data for training/ retraining and inference ingestions as required.</li>
</ul>

<ul>
<li>Create data visualization and business intelligence tools for stakeholders and data scientists for necessary business/ solution insights.</li>
</ul>

<ul>
<li>Identify, design, and implement internal process improvements: automating manual data processes, optimizing data delivery, etc.</li>
</ul>

<ul>
<li>Ensure our data is separated and secure across national boundaries through multiple data centers </li>
</ul>

Requirements and Skills

<ul>
<li>You should have a bachelors or master’s degree in computer science, Information Technology or other quantitative fields</li>
</ul>

<ul>
<li>You should have at least 8 years working as a data engineer in supporting large data transformation initiatives related to machine learning, with experience in building and optimizing pipelines and data sets</li>
</ul>

<ul>
<li>Strong analytic skills related to working with unstructured datasets.</li>
</ul>

<ul>
<li>Experience with Azure cloud services, ADF, ADLS, HDInsight, Data Bricks, App Insights etc</li>
<li style="text-align:justify;">Experience in handling ETL’s using Spark.</li>
</ul>

<ul>
<li>Experience with object-oriented/object function scripting languages: Python, Pyspark, etc.</li>
</ul>

<ul>
<li style="text-align:justify;">Experience with big data tools: Hadoop, Spark, Kafka, etc.</li>
</ul>

<ul>
<li style="text-align:justify;">Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.</li>
</ul>

<ul>
<li style="text-align:justify;">You should be a good team player and committed for the success of team and overall project.  </li>
</ul>

Lead Data Engineer in Delhi (UID-9716)

Job Description

Job Details

Skills Required for Lead Data Engineer

Job Description for Lead Data Engineer

Explore More Opportunities