Middle Data Engineer (6 months’ engagement) (#5895)

N-iX is looking for a Middle Data engineer for 6 months' engagement. 

You will act as the independent hands-on engineer and the technical owner of data readiness. The role combines pipeline and query engineering with investigation of source, mapping, hierarchy, and validation issues; coordination of market and provider validation cycles; reconciliation of outputs; and delivery of technically ready datasets to downstream consumers.

The Data Engineering team builds and maintains pipelines, while mappings and configurations determine cell/model scope; Snowflake extraction jobs run SQL through Databricks, with Blob Storage, validation, Medallion transformations, Power BI review, and Gold delivery to Ekimetrics. The operating model also requires end-to-end triage across MDFA, CDF, WPP, Data Foundation, Redmill, and market stakeholders rather than treating every discrepancy as a coding defect.

Key responsibilities:

• Independently design, develop, maintain, test, deploy, and optimize ingestion, validation, transformation, aggregation, data-quality, anomaly-detection, and extraction workflows
• Develop and tune Python/PySpark packages, SQL extraction queries, Databricks jobs, ADF pipelines, Delta tables, and source-to-target contracts
• Own technical readiness for new cells and refreshes: clarify filters and expected scope, implement or update queries, review mappings/configuration, reconcile outputs, and confirm readiness for business validation
• Investigate source, NCID, taxonomy, hierarchy, mapping, naming-convention, aggregation, missing-week, outlier, and performance issues across CDF, MDFA/PFME, APIs, manual files, and specific sources
• Coordinate technical validation cycles with Product, CMIA/markets, WPP/MDFA, CDF, other data providers, and Ekimetrics; convert reported discrepancies into actionable owners and technical evidence
• Drive complex incident resolution across pipelines, data contracts, access, service principals, secrets, networking, Power BI refreshes, and downstream delivery
• Review pull requests and test evidence; guide the Junior engineer and delegate scoped engineering/support work without becoming a people manager
• Maintain architecture documentation, interface contracts, repository documentation, runbooks, deployment procedures, and support/escalation guidance
• Recommend practical automation, reliability, performance, and maintainability improvements while respecting platform standards and business-validation ownership.

Must-have technical competencies:

• 5–6 years’ professional engineering experience with independent production ownership
• Strong Python, PySpark, and SQL, including complex transformations, query optimization, reusable packages, debugging, tests, and reconciliation
• Strong ETL/ELT and batch-pipeline engineering across relational warehouses, APIs, object storage, and file-based ingestion
• Experience with a cloud data lake/lakehouse, Medallion patterns, schema/interface contracts, data-quality controls, orchestration, observability, and incident recovery
• Solid Git engineering practices: branching, pull requests, reviews, automated tests, deployment controls, and documentation
• Proven ability to translate business/data requirements into filters, mappings, transformations, validation rules, and operational workflows
• Stakeholder-facing problem solving: explain discrepancies, challenge incomplete requirements, establish technical owners, and drive issues to closure.

Nice-to-have:

• Strong preference for Azure Data Factory, Azure Databricks, Databricks Jobs/API, Delta Lake, Snowflake, Azure Blob Storage/Data Lake, Power BI, GitHub, and Azure Key Vault
• Valuable experience with Pandera or equivalent schema-validation frameworks, Streamlit, Managed Identity/service principals, Azure Communication Services, Managed VNet/Private Endpoints, and Dev/Prod release practices
• Experience with PFME/media data, syndicated sales, marketing hierarchies, MMM inputs, or multi-market data onboarding is preferred but can be learned.