Responsibilities

<ul>
<li>Design, build, and optimize ETL pipelines using AWS Glue 3.0+ and PySpark.</li>
</ul>

<ul>
<li>Implement scalable and secure data lakes using Amazon S3, following bronze/silver/gold zoning.</li>
</ul>

<ul>
<li>Write performant SQL using AWS Athena (Presto) with CTEs, window functions, and aggregations.</li>
</ul>

<ul>
<li>Take full ownership from ingestion → transformation → validation → metadata → documentation → dashboard-ready output.</li>
</ul>

<ul>
<li>Build pipelines that are not just performant, but audit-ready and metadata-rich from the first version.</li>
</ul>

<ul>
<li>Integrate classification tags and ownership metadata into all columns using AWS Glue Catalog tagging conventions.</li>
</ul>

<ul>
<li>Ensure no pipeline moves to QA or BI team without validation logs and field-level metadata completed.</li>
</ul>

<ul>
<li>Develop job orchestration workflows using AWS Step Functions integrated with EventBridge or CloudWatch.</li>
</ul>

<ul>
<li>Manage schemas and metadata using AWS Glue Data Catalog.</li>
</ul>

<ul>
<li>Take full ownership from ingestion → transformation → validation → metadata → documentation → dashboard-ready output.</li>
</ul>

<ul>
<li>Ensure no pipeline moves to QA or BI team without validation logs and field-level metadata completed.</li>
</ul>

<ul>
<li>Enforce data quality using Great Expectations, with checks for null %, ranges, and referential rules.</li>
</ul>

<ul>
<li>Ensure data lineage with OpenMetadata or Amundsen and add metadata classifications (e.g., PII, KPIs).</li>
</ul>

<ul>
<li>Collaborate with data scientists on ML pipelines, handling JSON/Parquet I/O and feature engineering.</li>
</ul>

<ul>
<li>Must understand how to prepare flattened, filterable datasets for BI tools like Sigma, Power BI, or Tableau.</li>
</ul>

<ul>
<li>Interpret business metrics such as forecasted revenue, margin trends, occupancy/utilization, and volatility.</li>
</ul>

<ul>
<li>Work with consultants, QA, and business teams to finalize KPIs and logic.</li>
</ul>

<ul>
<li>Build pipelines that are not just performant, but audit-ready and metadata-rich from the first version.</li>
</ul>

<ul>
<li>Integrate classification tags and ownership metadata into all columns using AWS Glue Catalog tagging conventions.</li>
</ul>

<ul>
<li>This is not just a coding role. We expect the candidate to think like a data architect within their module – designing pipelines that scale, handle exceptions, and align to evolving KPIs.</li>
</ul>

Essential Skills

<ul>
<li>Strong hands-on experience with AWS: Glue, S3, Athena, Step Functions, EventBridge, CloudWatch, Glue Data Catalog.</li>
</ul>

<ul>
<li>Programming skills in Python 3.x, PySpark, and SQL (Athena/Presto).</li>
</ul>

<ul>
<li>Proficient with Pandas and NumPy for data wrangling, feature extraction, and time series slicing.</li>
</ul>

<ul>
<li>Strong command over data governance tools like Great Expectations, OpenMetadata / Amundsen.</li>
</ul>

<ul>
<li>Familiarity with tagging sensitive metadata (PII, KPIs, model inputs).</li>
</ul>

<ul>
<li>Capable of creating audit logs for QA and rejected data.</li>
</ul>

<ul>
<li>Experience in feature engineering – rolling averages, deltas, and time-window tagging.</li>
</ul>

<ul>
<li>BI-readiness with Sigma, with exposure to Power BI / Tableau (nice to have).</li>
</ul>

<ul>
<li>Personal</li>
</ul>

<ul>
<li>Excellent communication and collaboration skills with data scientists, QA, and business users.</li>
</ul>

<ul>
<li>Self-starter with strong problem-solving and critical thinking abilities.</li>
</ul>

<ul>
<li>Ability to translate business KPIs and domain requirements into technical implementations.</li>
</ul>

<ul>
<li>Detail-oriented with a high standard of data quality and compliance.</li>
</ul>

<ul>
<li>Demonstrated accountability, confidentiality, and ethical standards in handling data.</li>
</ul>

Preferred Skills

<ul>
<li>Experience in domains such as logistics, supply chain, enterprise finance, or B2B analytics.</li>
</ul>

<ul>
<li>Knowledge of ML pipelines and I/O formats like Parquet/JSON.</li>
</ul>

<ul>
<li>Familiarity with data modeling and KPI interpretation.</li>
</ul>

<ul>
<li>Working knowledge of Agile methodology.</li>
</ul>

Personal

<ul>
<li>Ability to multitask and handle fast-paced delivery environments</li>
</ul>

<ul>
<li>Proactive mindset with a collaborative and mentoring approach</li>
</ul>

<ul>
<li>Demonstrates ownership, initiative, and a commitment to high performance</li>
</ul>

Other Relevant Information

<ul>
<li>Bachelor's or master’s degree in computer science, Data Engineering, or a related field</li>
</ul>

<ul>
<li>Minimum 6–9 years of hands-on data engineering with at least 3 years in AWS-native data pipelines and governance.</li>
</ul>

Senior Data Engineer in Gurugram (UID-10118)

Job Description

Job Details

Skills Required for Senior Data Engineer

Job Description for Senior Data Engineer

Explore More Opportunities