Senior/Lead Big Data Engineer + AI (#4061)

We are looking for Senior/Lead Big Data Engineer to join our team for a long-term cooperation. 

Role Overview:
As a Lead Big Data Engineer, you will combine hands-on engineering with technical leadership. You’ll be responsible for designing, developing, and optimizing Spark-based big data pipelines in Palantir Foundry, ensuring high performance, scalability, and reliability. You will also mentor and manage a team of engineers, driving best practices in big data engineering, ensuring delivery excellence, and collaborating with stakeholders to meet business needs. 

In addition to core data engineering responsibilities, you will play a critical role in enabling AI and machine learning initiatives by building and optimizing the data pipelines that power model training, inference, and deployment. While our project uses Palantir Foundry, prior experience with it is a plus, but not mandatory.

Key Responsibilities:

• Build and maintain solutions using Palantir Foundry.
• Design and implement data pipelines to support AI and ML use cases, including data preparation, feature engineering, and real-time model serving.
• Collaborate with data scientists to productionize AI/ML models, ensuring seamless integration into scalable data workflows.
• Oversee and participate in code reviews, architecture discussions, and best practice implementation.
• Maintain high standards for data quality, security, and governance, with a focus on ethical and compliant use of data in AI applications.
• Manage and mentor a team of engineers, providing technical direction.
• Drive continuous improvement in processes, tools, and development practices.
• Foster collaboration across engineering, data science, and product teams to align on priorities and solutions.

Requirements:

• 5+ years in Big Data Engineering.
• Experience in a lead (tech/team lead) role for 1-2 years is a plus, but not required.
• Deep hands-on expertise in Apache Spark (PySpark) for large-scale data processing.
• Proficiency in Python and distributed computing principles.
• Experience designing, implementing, and optimizing high-volume, low-latency data pipelines.
• Experience supporting AI/ML projects (e.g., enabling model training pipelines, feature engineering, real-time inference, or MLOps workflows).
• Strong leadership, communication, and stakeholder management skills.
• Familiarity with CI/CD and infrastructure as code (Terraform, CloudFormation) is desirable.
• Experience with Palantir Foundry is a plus, but not required.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.