Data Engineer

Job Purpose

As a Data Engineer, you will play a critical role in building and optimizing our infrastructure and services to support the company’s Business Intelligence needs. This position is essential in ensuring that our data infrastructure meets the demands of our stakeholders and end-users. By developing robust data pipelines and architectures, you will rely on your expertise in AWS, data engineering and development practices. Your responsibility is to align technical possibilities with business requirements, ensuring our data systems are reliable, scalable, and efficient.

Main Duties and Responsibilities

• Design, develop, and maintain scalable data pipelines and architectures to support Business Intelligence and data analytics.
• Ensure the integration of data from various sources, transforming raw data into usable formats for analysis and reporting.
• Collaborate with business stakeholders to understand their data requirements and ensure their needs are met through effective data solutions.
• Optimize data delivery, re-designing infrastructure for greater scalability and performance where necessary.
• Implement data quality processes to ensure the accuracy and reliability of the data.
• Maintain relationships with stakeholders across various departments, educating them on technological advancements in data engineering and exploring potential opportunities.
• Serve as the main link between stakeholders and internal teams, ensuring clear and effective communication.
• Establish testing and sandboxing environments to ensure reliable and stable product development and deployment.
• Establish proper logging, monitoring, and alerting for our data systems to maintain high system reliability and performance.
• Enable practices of multi-instance environments with load balancing, ensuring redundancy, stability, and scalability.
• Ensure that our systems and processes adhere to the highest industry standards and best practices in data engineering and continuous delivery.
• Stay informed about technology trends and requirements, collaborating with the team to adapt and improve based on this knowledge.

Qualifications

• Extensive experience with data engineering methodologies and best practices, including ETL (Extract, Transform, Load) processes.
• Advanced skills in AWS services such as S3, Redshift, RDS, Glue, and other related AWS tools.
• Strong knowledge of data modelling, data warehousing, and big data technologies.
• Solid understanding of IT infrastructure, including fundamental troubleshooting capabilities and familiarity with common software suites.
• A talent for identifying and addressing data requirements through a user-growth perspective, ensuring scalability and user satisfaction.
• Proficiency in various analytic tools to monitor, measure, and improve system performance and reliability.
• Good understanding of the infrastructure and data lifecycle, from planning and development to monitoring and maintenance.
• Exceptional ability to prioritize tasks efficiently, managing multiple priorities and projects effectively.
• Minimum of 3 years of experience in a data engineering role, preferably with a focus on AWS technologies.
• Solid knowledge of Python, with experience in data-related libraries and frameworks.
• Excellent English oral and written communication skills, facilitating clear and concise dialogues across all levels of the organization.

Not Mandatory Qualifications (Will Be Considered an Advantage)

• Experience with network traffic monitoring tools, firewall configurations, load balancers, etc.
• Good working experience with databases such as MySQL, PostgreSQL, and NoSQL databases.
• Awareness of information security best practices, and their implementations.
• Experience with BI tools like PowerBI and Tableau.
• Management of queue-based systems for efficient task processing and communication.
• Experience with web frameworks such as Django and Flask, REST APIs.