We are on an exciting journey to migrate big data workloads from on-premises environments to a cloud-based solution. This large-scale initiative involves hundreds of use cases transitioning to Google Cloud Platform (GCP) for a more efficient, scalable, and high-performance data ecosystem.
As a Big Data Engineer, you will play a crucial role in designing, developing, and optimizing data platforms while ensuring seamless migration of workloads. This is an opportunity to work with cutting-edge technologies and collaborate with cross-functional teams to build robust, cloud-native solutions.
What You’ll Do
- Design, develop, and maintain scalable and high-performance data platforms to support business intelligence, analytics, and operational data needs.
- Collaborate closely with product teams, data scientists, and business stakeholders to understand use case requirements and ensure smooth migration to GCP.
- Leverage cloud-based data tools such as BigQuery, Data Proc, Data Flow, and Cloud SQL to implement data solutions.
- Automate processes and build tools to simplify data transformations, ensuring minimal manual intervention.
- Develop, optimize, and maintain ETL workflows using low-code/no-code platforms.
- Write and optimize complex SQL queries for various RDBMS systems, including Hive, MySQL, and PostgreSQL.
- Work with Spark and other big data technologies to process large datasets efficiently.
- Ensure best practices in version control, CI/CD, DevOps, and automated testing.
- Maintain high code hygiene through Git management, including PR reviews and documentation.
- Monitor system performance, troubleshoot issues, and continuously improve the reliability and scalability of data solutions.
- Stay up-to-date with industry trends and emerging technologies in cloud computing, big data, and machine learning.
What We’re Looking For
- Education. Bachelor’s degree in Computer Science, Engineering, or a related field. A Master’s degree is a plus.
- Experience. 3+ years of software development experience with hands-on expertise in Java, Python, or Scala.
- Cloud Expertise. At least 2+ years of experience working with GCP cloud tools such as BigQuery, Data Proc, Data Flow, Cloud SQL, and Pub/Sub.
- ETL & Data Processing. Experience in setting up ETL jobs using low-code UI platforms.
- SQL & Database Knowledge. Strong RDBMS skills with expertise in writing complex SQL queries for Hive, MySQL, PostgreSQL, and NoSQL databases.
- Big Data Processing. Familiarity with Spark and Hive for processing large datasets efficiently.
- DevOps & CI/CD. Hands-on experience with CI/CD pipelines, automated testing frameworks, and source code management tools such as Jenkins, Git, Stash, Jira, Confluence, and Splunk.
- Agile Development. Experience working within Agile/SAFe frameworks to drive iterative and incremental development.
- Certifications (Preferred). GCP Professional Data Engineer or Cloud Architect certification.
- Soft Skills. Strong problem-solving abilities, effective communication, and a collaborative team player mindset.
Why Join American Express?
We believe in backing our employees just as much as we back our customers. When you join American Express, you gain access to a range of benefits and programs designed to support your overall well-being, career development, and work-life balance.
Perks & Benefits
- Competitive base salary with performance-based bonuses.
- Comprehensive medical, dental, vision, life insurance, and disability benefits.
- Flexible work arrangements with hybrid, onsite, or remote options depending on role and business needs.
- Generous parental leave policies to support work-life balance.
- Retirement and financial well-being programs.
- Free and confidential mental health support through the Healthy Minds program.
- Career development, training, and mentorship opportunities.
- Access to on-site wellness centers (location-dependent) staffed with medical professionals.
- A diverse, inclusive, and collaborative work culture that fosters personal and professional growth.
At American Express, we are committed to creating an environment where every employee feels seen, heard, and valued. We celebrate diversity and offer equal employment opportunities to all individuals, regardless of race, color, religion, gender, sexual orientation, national origin, disability, veteran status, or any other legally protected status.