We are seeking an experienced Data Engineering Lead to design, build, and optimize large-scale data pipelines that drive business-critical insights. As a key member of the team, you will lead the end-to-end development of data solutions, integrate AI/ML components, and enforce best practices in data governance, quality, and operations. You will collaborate with cross-functional teams including Data Scientists, Data Engineers, and Business Analysts to ensure seamless data flow, high performance, and scalability.
Key Responsibilities
- Data Pipeline Architecture & Development. Lead the design and development of large-scale, reliable, and scalable data pipelines. Ensure reusability, efficiency, and adherence to best practices in data modeling, storage, and transformation techniques.
- Data Analysis & Metric Definition. Engage with stakeholders to understand business requirements, define key metrics, and map them to appropriate data sources. Conduct deep data analysis to derive meaningful insights.
- AI/ML Integration. Develop AI/ML pipeline integrations, enabling efficient model deployment and lifecycle management within the metric repository. Enhance ML Ops components to support ongoing AI-based initiatives.
- Data Governance & Quality Assurance. Enforce data quality standards, ensuring accuracy, integrity, and consistency across all deliverables. Implement proactive monitoring and alerting mechanisms to detect and address data discrepancies.
- Code Review & Best Practices. Establish and uphold best practices for development, deployment, and maintenance of data pipelines. Conduct regular code reviews, ensure adherence to CI/CD processes, and maintain comprehensive technical documentation.
- Operations & Client Deliverables. Support the generation and delivery of high-quality insight feeds for clients. Ensure timely execution and adherence to service-level agreements.
- Team Collaboration & Leadership. Mentor and lead a team of Data Engineers, fostering a culture of continuous improvement. Collaborate with various teams to enhance operational efficiency and leverage existing frameworks where applicable.
Technical Expertise Required
- Strong experience in designing and building globally applicable data platforms using best-in-class data engineering practices.
- Deep understanding of data modeling, storage, and data flow techniques to support complex business use cases.
- Expertise in building scalable data pipelines using PySpark, Hive, and Apache Airflow.
- Hands-on experience with scheduling tools such as Airflow and Oozie, with a focus on building robust data processing workflows.
- Proficiency in writing and optimizing SQL queries in a Big Data environment.
- Strong background in Hadoop ecosystem, including data ingestion, processing, and storage.
- Experience in working with Linux/Unix environments and command-line utilities.
- Knowledge of CI/CD frameworks and version control systems such as Git.
- Familiarity with solution architectures involving APIs and microservices.
- Hands-on experience in machine learning model inference, validation, deployment, and ongoing management within a production environment.
Strategic & Functional Excellence
- Strong business acumen, with the ability to align data engineering solutions to key business needs.
- Experience in the payments industry or a strong interest in financial services.
- Ability to translate complex data and technical concepts into actionable business insights.
- Exposure to Agile methodologies and a solid understanding of program management best practices.
- Problem-solving mindset with the ability to propose feasible solutions and communicate effectively with senior leadership.
- Excellent communication, collaboration, and presentation skills, with the ability to engage with stakeholders across various departments and levels.
- Willingness to continuously learn new tools, technologies, and paradigms as the field of data engineering evolves.
Qualifications
Basic Qualifications
- 8+ years of relevant work experience with a Bachelor’s degree in Computer Science, Information Science, Data Engineering, or a related field.
- 7+ years of relevant work experience with a Master’s degree in the same fields.
Preferred Qualifications
- 5+ years of experience in developing centralized data repositories.
- Experience working in shared services, consulting, or financial services is a strong advantage.
Why Join Visa?
- Opportunities to work on cutting-edge technologies in a global environment.
- A diverse and inclusive workplace where your ideas and contributions are valued.
- Competitive compensation and benefits designed to support your professional and personal growth.
- Professional development programs and learning opportunities to advance your career.
- An opportunity to make an impact by working on meaningful projects that shape the future of digital payments.
Equal Employment Opportunity Statement
Visa is an Equal Employment Opportunity Employer. Qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability, or protected veteran status. Visa also considers qualified applicants with criminal histories in accordance with applicable local laws and EEOC guidelines.
If you are passionate about data engineering, AI/ML, and driving innovation in the payments industry, we encourage you to apply today and become part of our mission to create a more connected and secure world of payments!