15 Oct
Intelligent Resourcing Solutions
London
Job Title: Lead Data Engineer(AWS)
Location: Remote/Hybrid
Employment Type: Full-time
Reports to: Architect Owner
Company Overview:
We are recruiting on behalf of a leading cloud solutions provider specialising in digital transformation and data engineering services. The company's mission is to empower organisations to unlock the full potential of their data through innovative cloud-based solutions. As they continue to grow, they are seeking a talented Lead Data Engineer to support and guide the data engineering team in designing and implementing cutting-edge solutions for complex migration projects.
Job Overview:
As the Lead Data Engineer,
you will play a critical role in overseeing and guiding the design and implementation of data engineering solutions. You will lead a team of Data Engineers in migrating Legacy Oracle databases to AWS managed databases, ensuring the delivery of high-quality, scalable data solutions. Your technical expertise in AWS services such as AWS Glue, AWS Managed Flink, S3, AWS RDS, and Lambda functions will enable you to mentor the team, troubleshoot complex issues, and deliver optimized data processing solutions. This position is ideal for a seasoned professional who is passionate about cloud-based data engineering and enjoys leading technical teams.
Key Responsibilities:
- Lead the design and implementation of ETL processes using AWS Glue to extract, transform, and load data from Legacy Oracle databases into AWS managed databases, document databases, and data lakes.
- Oversee the development and maintenance of scalable data pipelines and data processing workflows across the data engineering team.
- Provide technical leadership and mentorship to Data Engineers,
ensuring best practices are followed in data migration and integration efforts.
- Collaborate with solution architects, data scientists, and client stakeholders to ensure that data engineering solutions meet project goals and client requirements.
- Optimize and troubleshoot data workflows using AWS services such as AWS Glue, AWS Managed Flink, S3, AWS RDS, and Lambda functions, providing guidance to resolve complex issues.
- Ensure the team adheres to data quality, security, and governance standards while implementing data engineering solutions.
- Automate and monitor data pipelines, implementing proactive alerting mechanisms for data quality, accuracy, and performance.
- Drive continuous improvement initiatives,
staying current with emerging AWS services and data engineering best practices to enhance team capabilities.
- Lead technical discussions with cross-functional teams and ensure alignment on solution design and implementation strategies.
- Provide regular updates on project status, risks, and technical challenges to senior leadership and stakeholders.
Qualifications:
- Extensive experience as a Lead Data Engineer or Senior Data Engineer with a focus on AWS cloud services.
- Proven expertise in designing and developing ETL processes using AWS Glue.
- Experience in migrating Legacy Oracle databases to AWS managed databases (eg, RDS, Aurora).
- Strong knowledge of AWS services including S3, AWS Managed Flink, Lambda, and CloudWatch.
- Expertise in building and maintaining data lakes, document databases,
and relational databases in a cloud environment.
- Hands-on experience with data pipeline architecture, design, development, and maintenance.
- Proficiency in SQL and Python for data processing, manipulation, and automation.
- Strong understanding of data governance, data quality, and data security principles.
- Leadership and mentoring experience, with a track record of guiding technical teams to deliver high-quality solutions.
- AWS certifications (eg, AWS Certified Data Analytics, AWS Certified Solutions Architect) are preferred.
Preferred Skills:
- Experience with big data processing technologies such as Apache Spark or PySpark.
- Knowledge of DevOps practices and tools for automating data infrastructure deployment and monitoring.
- Familiarity with data warehousing, dimensional modelling, and data architecture best practices.
What We Offer:
- Competitive salary and benefits package.
- Leadership role in a dynamic and fast-growing cloud solutions company.
- Access to cutting-edge AWS technologies and opportunities for continuous professional growth.
- A collaborative, innovative, and supportive work environment.
- Flexibility to work remotely or in a hybrid setup.
How to Apply:
If you are an experienced Data Engineer with strong leadership skills and are excited about leading data migration projects in the cloud, we encourage you to apply. Please submit your resume and a cover letter detailing your relevant experience and leadership capabilities.