Data Engineer

As a Data Engineer at IBM, your primary responsibility will be designing, developing, and managing data infrastructure on AWS, with a strong focus on data warehousing solutions. You will be tasked with ensuring that our clients can efficiently extract, transform, and load (ETL) data, enabling data-driven decisions and insights.

Responsibilities

1. Data Infrastructure Management. You will design, develop, and manage data infrastructure using AWS services, with a focus on creating and optimizing data warehousing solutions. You will work closely with AWS technologies to ensure the most efficient and scalable data systems are in place.
2. SQL Query Development. Writing complex SQL queries for the extraction, transformation, and loading (ETL) of data. You will be expected to utilize your expertise in SQL to solve intricate data challenges, ensuring smooth data flow and integration.
3. Data Modeling and Transformation. Utilize DBT for data modeling and transformation tasks, ensuring data is clean, accurate, and structured for easy consumption by business intelligence tools and analytics teams.
4. Data Engineering in Python. You will leverage Python for a range of data engineering tasks, from automation to scripting. Your experience with Python will be crucial to creating efficient and scalable data pipelines, integrating with multiple data systems and platforms.
5. Automation of Data Workflows. Implementing scheduling tools such as Airflow, Control M, or shell scripting to automate data processes and workflows. Your ability to optimize these workflows will ensure a more efficient and reliable data pipeline.
6. Agile Methodology. Participate actively in an Agile environment, working with dynamic teams to adjust to changing priorities and continuously deliver results. Your ability to adapt quickly and stay focused on high-priority tasks will help drive success in this role.

Required Technical and Professional Expertise

1. Proven Expertise in AWS Technologies. A strong understanding of AWS services such as S3, EC2, Lambda, Redshift, and others. Your expertise in utilizing these services will be key to building scalable and effective data infrastructure.
2. Data Warehousing and SQL Proficiency. Extensive experience in data warehousing, with a solid understanding of how to design, implement, and manage databases that support large-scale data processing. You will be expected to write complex SQL queries to extract, transform, and load data efficiently.
3. Python for Data Engineering. Strong work experience in Python programming for data engineering tasks. You should be proficient in writing clean, optimized, and scalable Python scripts for automation, data manipulation, and ETL processes.
4. Scheduling Tools Expertise. Experience using scheduling tools like Airflow, Control M, or shell scripting to automate and optimize data workflows.
5. Excellent Communication Skills. Strong communication skills are crucial, as you will be collaborating with cross-functional teams, stakeholders, and clients. You should be able to articulate technical concepts clearly and concisely.
6. Willingness to Learn. A passion for learning and continuously improving your technical skill set. IBM encourages you to explore new technologies and methodologies to stay ahead in a rapidly evolving field.

Preferred Technical and Professional Experience

1. DBT Knowledge. Experience using DBT (Data Build Tool) for data modeling and transformation is a plus. DBT’s role in data engineering is becoming increasingly important, and familiarity with it can give you an edge.
2. Experience with PySpark or Spark. PySpark or Spark expertise is highly desirable, as it would help you handle large-scale data processing and distributed computing tasks effectively.
3. Familiarity with DevOps and CI/CD. Experience with DevOps methodologies and CI/CD pipelines will benefit you as we continuously strive for faster, more efficient development cycles and automated deployment.
4. Agile Methodologies. Familiarity with Agile methodologies and experience working in an Agile environment is a nice-to-have. Your ability to collaborate and adapt to the iterative, fast-paced nature of Agile will ensure success in this role.

Why IBM?

Working at IBM offers a unique blend of career growth, continuous learning, and innovative challenges. We offer opportunities to engage with the latest technologies and collaborate with talented professionals globally. You’ll be part of a culture that fosters creativity and challenges you to think differently, driving impact across industries.

If you are passionate about data engineering, cloud technologies, and automation, and you thrive in an environment where innovation and collaboration are prioritized, we would love to have you on our team. Apply today and become part of IBM Consulting's dynamic environment where you will grow your career while making a meaningful impact on our clients and the world.