Data Engineer in Test (#3042)

We are seeking a proactive and experienced Data Engineer in Test to join our vibrant team. In this role, you will be instrumental in ensuring the reliability, accuracy, and performance of our data pipelines and backend systems. Working within Palantir Foundry, you will design, implement, and maintain robust automated testing frameworks tailored for data validation, transformation, and API performance across complex data workflows.

You will collaborate closely with data engineers and stakeholders to embed testing into the development lifecycle, ensuring our data infrastructure is trustworthy, scalable, and aligned with industry best practices.

Key Responsibilities:

• Design, develop, and maintain automated testing frameworks for data pipelines and backend systems, ensuring high data quality and integrity.
• Implement automated regression, integration, and performance testing for ETL/ELT processes using Python, PySpark, SQL, and TypeScript.
• Build and execute API test automation focusing on data validation, consistency, performance, and error handling.
• Work with Palantir Foundry tools such as Contour, Workbook, Ontology Manager, Slate, and Workshop to test data workflows, Ontologies, and Foundry Functions.
• Collaborate with data engineers to define test strategies and identify data quality risks across data ingestion, transformation, and delivery layers.
• Monitor data pipeline executions and build alerts for quality regressions or anomalies.
• Monitor and improve test coverage across critical data processes, ensuring continuous integration and delivery through CI/CD pipelines.
• Proactively identify automation opportunities and contribute to test architecture improvements.
• Stay current with the latest tools and techniques in both data engineering and test automation, applying innovations to strengthen data quality assurance.

Required Qualifications:

• 3+ years of experience in data engineering.
• Expertise in automated testing of data pipelines, ETL/ELT flows, and APIs.
• Proficiency in Python, SQL, and test frameworks like pytest or Jest.
• Experience with PySpark for data transformations on distributed systems.
• Exposure to cloud-based data services (e.g., AWS Glue, GCP Dataflow, Azure Data Factory).
• Familiarity with data governance, data privacy, and compliance standards.
• Strong understanding of data modeling, data quality metrics, and error-handling patterns.
• Ability to build automated test solutions from the ground up and troubleshoot data issues effectively.
• Comfortable working in a CI/CD and version-controlled environment.
• Upper-intermediate or higher English proficiency.

Nice to Have:

• Hands-on experience with Palantir Foundry components is a plus.
• Prior work in pharma/life sciences or other data-regulated industries is a plus.