Lead Big Data Engineer with AI/ML experience (#4508)

N-iX is looking for a Lead Big Data Engineer with AI/ML experience. As a Lead Engineer, you’ll be responsible for designing, developing, and optimizing data and AI pipelines (including Spark-based and RAG-style workflows), ensuring performance, scalability, and reliability. You’ll also be working on technical discovery and solution assessment: helping the client understand existing AI tooling (currently a RAG-based solution used for document analysis), evaluating scalability, quality, cost, and risks, and advising on future architecture options. 

Responsibilities:

• Design and implement data pipelines to support AI and ML use cases, including data preparation, feature engineering, and real-time model serving.
• Support and enable RAG-based and NLP use cases, including document ingestion, data preparation, feature extraction, and structured data generation.
• Collaborate with AI engineers to productionize AI/ML solutions and integrate them into reliable data workflows.
• Participate in and lead discovery phases to analyze existing AI solutions, clarify how they work, and identify architectural gaps, risks, and improvement opportunities.
• Assess scalability and multilingual support of AI/data solutions (including feasibility, configuration needs, cost, and expected quality).
• Contribute to evaluations of vendor lock-in, platform alternatives, and potential migration paths (e.g., toward Palantir Foundry or other modern data/AI platforms).
• Ensure high standards of data quality, security, governance, and compliance.
• Drive continuous improvement in development processes, tooling, and engineering practices.
• Foster collaboration across engineering, data science, and product/business stakeholders.

Requirements:

• 6+ years of experience in Data Engineering
• Strong hands-on experience with PySpark and large-scale data processing.
• Proficiency in Python and distributed computing concepts.
• Experience designing and optimizing scalable data pipelines for high-volume data.
• Experience supporting AI/ML projects (e.g., enabling model training pipelines, feature engineering, real-time inference, or MLOps workflows).
• Ability to perform technical discovery and solution assessment, including explaining complex systems to non-technical stakeholders.
• Strong leadership, communication, and stakeholder management skills.
• Experience with Palantir Foundry is a plus, but not required.
• Familiarity with CI/CD and infrastructure as code (e.g., Terraform, CloudFormation) is desirable.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.