We are looking for an experienced Senior Data Engineer to join our team and play a crucial role in building a scalable data collection, storage, and distribution platform. This platform will be the foundation for ingesting and processing data from multiple sources, including vendors, research providers, exchanges, prime brokers (PBs), and web scraping. The goal is to make high-quality data accessible to systematic and fundamental portfolio managers, as well as key enterprise functions such as Operations, Risk, Trading, and Compliance.
The ideal candidate will have a strong background in SQL, Python, and AWS, along with expertise in containerization technologies like Docker and Kubernetes. You will be working on data pipeline enhancements, onboarding new data providers, and supporting data migration projects. If you have experience in market data projects or capital markets, it will be a significant plus.
Key Responsibilities
- Develop and implement web scraping solutions using scripts, APIs, and tools to extract data from various sources.
- Contribute to the design, development, and maintenance of a greenfield data platform based on Snowflake and AWS.
- Analyze and enhance existing data pipelines to meet new business requirements.
- Collaborate with stakeholders to onboard new data providers efficiently.
- Work on data migration projects, ensuring seamless transfer of large datasets.
- Implement best practices in DevOps, leveraging Kubernetes (K8s), Docker, and Jenkins for efficient deployment and scaling of applications.
- Optimize database queries and ETL processes to ensure high performance and reliability.
- Ensure data security, compliance, and governance across the entire data pipeline.
- Monitor system performance and troubleshoot issues in a Linux-based environment.
Required Skills & Experience
- 8+ years of experience in data engineering with a strong focus on ETL, data pipelines, and data infrastructure.
- Proficiency in SQL for data querying, optimization, and transformation.
- Strong programming skills in Python, particularly for data processing, automation, and scripting.
- Experience working with Linux-based systems for server management and automation.
- Hands-on expertise in containerization technologies like Docker and Kubernetes (K8s).
- Strong knowledge of AWS services such as S3, Lambda, EC2, and RDS.
- Good understanding of DevOps principles and experience with tools like Jenkins for CI/CD.
- Excellent communication skills to collaborate effectively with cross-functional teams.
Nice-to-Have Skills
- Prior experience working on market data projects or within capital markets.
- Hands-on experience with Snowflake, particularly in data warehousing and analytics.
- Experience using Apache Airflow for workflow orchestration and pipeline automation.
Why Join Us?
- Opportunity to work on a high-impact, cutting-edge data platform with a global reach.
- Collaborative and innovative work environment with exposure to the latest cloud and data technologies.
- Competitive salary, benefits, and career growth opportunities in a fast-paced organization.
If you are passionate about data engineering, cloud technologies, and building scalable data platforms, we’d love to hear from you!