Middle/Senior Data Engineer for Support (#3970)

Role Overview:

As a Data Engineer on our Support Team, you will be responsible for maintaining robust data pipelines to support our products and services. You will work closely with cross-functional teams to understand data requirements and implement efficient solutions using PySpark. The ideal candidate will have a strong background in data engineering, exceptional problem-solving skills, and a passion for working with large-scale datasets.

Key Responsibilities:

• Perform ongoing maintenance and optimization of existing data pipelines built with PySpark.
• Monitor pipeline performance and proactively identify areas for improvement to enhance efficiency and reliability.
• Maintain scalable data pipelines using PySpark to support various business needs.
• Ensure that all data is getting refreshed automatically daily as per SLA and data quality is being permanently monitored
• Troubleshoot and resolve complex issues of Foundry users within SLAs
• Optimize and tune existing data pipelines for improved performance and reliability.
• Troubleshoot and resolve data-related issues on time to ensure smooth operation of production systems.
• Document processes, procedures, and troubleshooting steps to facilitate knowledge sharing and maintain system integrity.
• Ensure immediate communications in case of Foundry outages and delays, but also for upcoming changes and other support information to end users.
• Implement continuous improvement of existing processes

Requirements:

• 3-4 years of experience working as a Data Engineer, strongly focusing on building data pipelines using PySpark.
• Proficiency in Python programming and experience with relevant libraries and frameworks.
• Solid understanding of distributed computing principles and experience with Apache Spark.
• Experience working with large-scale datasets and designing efficient data processing workflows.
• Strong analytical and problem-solving skills, with the ability to troubleshoot complex issues in production environments.
• Excellent communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.
• Intermediate+ English level

Nice to have:

• Readiness to work with TypeScript
• Experience with Palantir Foundry or similar data integration and analytics platforms is a plus