Lead Big Data Engineer (#3873)

Role Overview:
As a Lead Big Data Engineer, you will combine hands-on engineering with technical leadership. You’ll be responsible for designing, developing, and optimizing Spark-based big data pipelines in Palantir Foundry, ensuring high performance, scalability, and reliability. You will also mentor and manage a team of engineers, driving best practices in big data engineering, ensuring delivery excellence, and collaborating with stakeholders to meet business needs. While our project uses Palantir Foundry, prior experience with it is a plus, but not mandatory.

Key Responsibilities:

• Lead the design, development, and optimization of large-scale, Spark-based (PySpark) data processing pipelines.
• Build and maintain big data solutions using Palantir Foundry
• Ensure Spark workloads are tuned for performance and cost efficiency.
• Oversee and participate in code reviews, architecture discussions, and best practice implementation.
• Maintain high standards for data quality, security, and governance.
• Manage and mentor a team of Big Data Engineers, providing technical direction
• Drive continuous improvement in processes, tools, and development practices.
• Foster collaboration across engineering, data science, and product teams to align on priorities and solutions.

Requirements:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 6+ years in Big Data Engineering, with at least 1-2 years in a lead (tech/team lead) role.
• Deep hands-on expertise in Apache Spark (PySpark) for large-scale data processing.
• Proficiency in Python and distributed computing principles.
• Experience designing, implementing, and optimizing high-volume, low-latency data pipelines.
• Strong leadership, communication, and stakeholder management skills.
• Experience with Palantir Foundry is a plus, but not required.
• Familiarity with CI/CD and infrastructure as code (Terraform, CloudFormation) is desirable.