Data Engineer

• Minimum Qualification :

About the Role

As a Data Engineer, you will play a key role in building the data infrastructure that powers our data lake on AWS. Youll be responsible for designing, deploying, and maintaining data pipelines and ensuring a seamless, scalable, and reliable data flow to support our analytics efforts. Leveraging your experience with AWS, Airbyte or similar ELT/ETL tool, and custom solutions in Go, youll work closely with our team to turn data into a strategic asset for our company.

Key Responsibilities

• Data Infrastructure Setup: Architect and maintain robust, scalable, and secure data infrastructure on AWS.
• Data Pipeline Development: Design, develop, and maintain data pipelines, primarily using tools like Airbyte and custom-built services in Go, to automate data ingestion and ETL processes.
• Data Lake Management: Oversee the creation and maintenance of the data lake, ensuring efficient storage, high data quality, and effective partitioning, organization, performance, monitoring and alerting.
• Integration and Customization: Integrate tools like Airbyte with various data sources and customize data flows to align with specific business needs. Where necessary, build custom connectors in Go to support unique data requirements.
• Performance and Scalability: Optimize data pipelines and data lake storage for performance and scalability, ensuring low latency and high availability.
• Data Governance and Security: Implement best practices for data governance, security, and compliance in AWS, including access control, encryption, and monitoring.
Collaboration and Documentation: Work closely with platform engineers, data analysts and other stakeholders to understand data requirements and document infrastructure, processes, and best practices.

Required Qualifications

• Experience in Data Engineering: 3+ years of experience as a Data Engineer, with a focus on data lake architecture and ETL pipeline development.
• AWS Proficiency: Strong experience with AWS services including but not limited to S3, Glue, Lambda, Redshift, and IAM.
• ETL Expertise: Hands-on experience with Airbyte or similar ETL tools for data ingestion and transformation.
• Proficiency in Go: Experience writing services and connectors in Go, particularly for data pipeline automation.
• SQL and Data Modeling: Solid understanding of data modeling, SQL, and database concepts.
• Strong Scripting Skills: Familiarity with Python, Bash, or other scripting languages for automation and data manipulation.
Data Governance and Security: Experience implementing security and governance best practices in cloud environments.

Preferred Skills

• Containerization and Orchestration: Experience with Docker and Kubernetes is a plus.
• Knowledge of Data Analytics Tools: Familiarity with data analytics tools such as Tableau, Looker, or QuickSight.
Experience with Data Warehouses: Familiarity with data warehouse solutions, particularly in the AWS ecosystem (e.g., Redshift, Athena).

<

Information :

• Company : Loubby AI
• Position : Data Engineer
• Location : Nigeria
• Country : NG