Roles and Responsibilities:

<ul>
<li>Work on tasks related to implementation of Cloudera Data Platform Cloudera Data Platform on-premises and be a part of planning, installation, configuration, and integration with existing systems.</li>
</ul>

<ul>
<li>Infrastructure Management: Manage and maintain the Cloudera-based infrastructure, ensuring optimal performance, high availability, and scalability. This includes monitoring system health, and performing routine maintenance tasks.</li>
</ul>

<ul>
<li>Strong troubleshooting skills and operational expertise in areas such as system capacity, bottlenecks, memory, CPU, OS, storage, and networking.</li>
</ul>

<ul>
<li>Creating Runbooks and automating them using scripting tools like Shell scripting, Python etc.</li>
</ul>

<ul>
<li>Working knowledge with any of the configuration management tools like Terraform, Ansible or SALT</li>
</ul>

<ul>
<li>Data Security and Compliance: Implement and enforce security best practices to safeguard data integrity and confidentiality within the Cloudera environment. Ensure compliance with relevant regulations and standards (e.g., GDPR, HIPAA, DPR).</li>
</ul>

<ul>
<li>Performance Optimization: Continuously optimize the Cloudera infrastructure to enhance performance, efficiency, and cost-effectiveness. Identify and resolve bottlenecks, tune configurations, and implement best practices for resource utilization.</li>
</ul>

<ul>
<li>Capacity Planning: Planning and performance tuning of Hadoop clusters, Monitor resource utilization trends and plan for future capacity needs. Proactively identify potential capacity constraints and propose solutions to address them.</li>
</ul>

<ul>
<li>Collaborate effectively with infrastructure, network, database, application, and business intelligence teams to ensure high data quality and availability.</li>
</ul>

<ul>
<li>Work closely with teams to optimize the overall performance of the PhonePe Hadoop ecosystem.</li>
</ul>

<ul>
<li>Backup and Disaster Recovery: Implement robust backup and disaster recovery strategies to ensure data protection and business continuity. Test and maintain backup and recovery procedures regularly.</li>
</ul>

<ul>
<li>Develop tools and services to enhance debuggability and supportability.</li>
</ul>

<ul>
<li>Patches &amp; Upgrades: Routinely apply recommended patches and perform rolling upgrades of the platform in accordance with the advisory from Cloudera, InfoSec and Compliance.</li>
</ul>

<ul>
<li>Documentation and Knowledge Sharing: Create comprehensive documentation for configurations, processes, and procedures related to the Cloudera Data Platform. Share knowledge and best practices with team members to foster continuous learning and improvement.</li>
</ul>

<ul>
<li>Collaboration and Communication: Collaborate effectively with cross-functional teams including data engineers, developers, and IT operations personnel. Communicate project status, issues, and resolutions clearly and promptly.</li>
</ul>
 Skills Required:

<ul>
<li>Bachelor's degree in Computer Science, Engineering, or related field.</li>
</ul>

<ul>
<li>Proficiency in Linux system administration, shell scripting, and networking concepts  including IPtables, and IPsec.</li>
</ul>

<ul>
<li>Strong understanding of networking, open-source technologies, and tools.</li>
</ul>

<ul>
<li>3-5 years of experience in the design, set up, and management of large-scale Hadoop clusters, ensuring high availability, fault tolerance, and performance optimization.</li>
</ul>

<ul>
<li>Strong understanding of distributed computing principles and experience with Hadoop ecosystem technologies (HDFS, MapReduce, YARN, Hive, Spark, etc.).</li>
</ul>

<ul>
<li>Experience with Kerberos and LDAP.</li>
</ul>

<ul>
<li>Strong Knowledge of databases like Mysql,Nosql,Sql server</li>
</ul>

<ul>
<li>Hands-on experience with configuration management tools (e.g., Salt,Ansible, Puppet, Chef).</li>
</ul>

<ul>
<li>Strong scripting skills (e.g., PERL,Python, Bash) for automation and troubleshooting.</li>
</ul>

<ul>
<li>Experience with monitoring and logging solutions (e.g., Prometheus, Grafana, ELK stack).</li>
</ul>

<ul>
<li>Knowledge of networking principles and protocols (TCP/IP, UDP, DNS, DHCP, etc.).</li>
</ul>

<ul>
<li>Experience with managing *nix based machines and strong working knowledge of quintessential Unix programs and tools (e.g. Ubuntu, Fedora, Redhat, etc.)</li>
</ul>

<ul>
<li>Excellent communication skills and the ability to collaborate effectively with cross-functional teams.</li>
</ul>

<ul>
<li>Excellent analytical, problem-solving, and troubleshooting skills..</li>
</ul>

<ul>
<li>Proven ability to work well under pressure and manage multiple priorities simultaneously.</li>
</ul>

Good To Have:

<ul>
<li>Cloudera Certified Administrator (CCA) or Cloudera Certified Professional (CCP) certification preferred.</li>
</ul>

<ul>
<li>Minimum 2 years of experience in managing and administering medium/large hadoop based environments (&gt;100 machines), including Cloudera Data Platform (CDP) experience is highly desirable.</li>
</ul>

<ul>
<li>Familiarity with Open Data Lake components such as Ozone, Iceberg, Spark, Flink, etc.</li>
</ul>

<ul>
<li>Familiarity with containerization and orchestration technologies (e.g. Docker, Kubernetes, OpenShift) is a plus</li>
</ul>

<ul>
<li>Design,develop and maintain Airflow DAGs and tasks to automate BAU processes,ensuring they are robust,scalable and efficient. </li>
</ul>

Big Data Engineer in Bengaluru (UID-10275)

Job Description

Job Details

Skills Required for Big Data Engineer

Job Description for Big Data Engineer

Explore More Opportunities