We are looking for a highly skilled Senior Data Engineer to play a crucial role in building a scalable data collection, storage, and distribution platform. This platform will be designed to aggregate data from multiple sources, including vendors, research providers, exchanges, prime brokers (PBs), and web scraping. The goal is to make high-quality data accessible to key stakeholders such as Systematic & Fundamental Portfolio Managers (PMs), Operations, Risk, Trading, and Compliance teams.
As part of this role, you will be responsible for developing internal data products and analytics, ensuring the efficient movement, transformation, and utilization of data across the enterprise. If you are passionate about big data, cloud computing, and scalable infrastructure, this is a fantastic opportunity to work on cutting-edge data engineering solutions.
Key Responsibilities
Web Scraping & Data Acquisition
- Develop and maintain web scraping scripts, APIs, and automation tools to collect data from various sources.
- Ensure the accuracy, efficiency, and reliability of extracted data while complying with relevant policies and regulations.
Data Pipeline Development & Enhancement
- Work on greenfield data platform development leveraging Snowflake and AWS.
- Understand existing data pipelines, identify bottlenecks, and enhance them to support new business requirements.
- Ensure data integrity and optimize data flow for scalability and performance.
Onboarding New Data Providers
- Collaborate with data vendors and internal teams to seamlessly onboard new data sources.
- Standardize and integrate structured and unstructured data into the platform.
Data Migration Projects
- Lead and support data migration initiatives, ensuring minimal downtime and high data quality.
- Design and implement solutions for migrating large datasets between different storage systems and databases.
DevOps & Infrastructure Management
- Utilize containerization technologies (Docker, Kubernetes) to deploy and manage applications efficiently.
- Implement CI/CD pipelines using Jenkins to automate deployment and testing processes.
- Work closely with the DevOps team to ensure a smooth data engineering workflow in cloud environments.
Mandatory Skills & Experience
- 10+ years of experience as a Data Engineer, working on large-scale data processing and storage solutions.
- Strong SQL expertise, ability to write optimized queries, stored procedures, and work with large datasets efficiently.
- Proficiency in Python for data manipulation, automation, and backend processing.
- Experience with Linux environments for scripting, automation, and system management.
- Expertise in containerization technologies, including Docker & Kubernetes, to deploy and scale applications.
- Cloud experience with AWS, including EC2, S3, Lambda, and other data services.
- Hands-on experience in DevOps practices, including Kubernetes (K8s), Docker, and Jenkins for CI/CD.
- Excellent communication skills to work effectively with cross-functional teams and stakeholders.
Nice-to-Have Skills
- Experience working on Market Data Projects or Capital Markets.
- Familiarity with Apache Airflow for workflow automation.
Why Join Us?
- Work on a high-impact data engineering project in the financial sector, supporting key business functions.
- Gain hands-on experience with cutting-edge cloud and big data technologies.
- Collaborate with a dynamic and highly skilled team to solve complex data challenges.
- Opportunity to work on greenfield projects with ample room for innovation and learning.
If you are an experienced Data Engineer with a strong background in SQL, Python, AWS, and containerization, we encourage you to apply!