N-iX is a global company with Ukrainian roots that helps businesses across the world develop successful software products. Founded in 2002, N-iX has come a long way and increased its presence in eight countries spanning Europe, the US, and Latin America. Today, we are a strong community of 2,000+ professionals and a reliable partner for global industry leaders and Fortune 500 companies.
N-iX is looking for a Senior Data Engineer who can become a part of our team.
Our Client is a financial technology company based in California, one of the leading companies in this sector, with a buy now, pay later service.
Requirements:
- 4+ years of Python programming experience
- Experience with writing Spark SQL and working with DataFrames, debugging Spark applications via metrics, history server, etc
- Understanding of shuffling, re-partitioning concepts, off-heap vs on-heap memory usageUnderstanding of join in a distributed context; eg sort-merge vs broadcast joins will be a plus
- Kubernetes (understanding of spark driver pod vs. executor pods, cluster, experience using kubectl to inspect issues, experience with docker)
Responsibilities:
- Moving batch activity from MySQL to the S3 data lake and enhancing database operability
- Evaluate and tune batch pipelines (e.g., Airflow DAGs, Luigi, Apache Spark)
- Hands-on support for batch pipeline migration from MySQL DB activity to the data lake, including testing, performance tuning, and cut-over support
- Batch pipeline migration strategy and best practices support
We offer:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
Ukraine, Europe, Colombia
