We are seeking a highly experienced Senior Data Engineer to play a critical role in developing and maintaining a scalable data collection, storage, and distribution platform. This platform will aggregate data from multiple sources, including vendors, research providers, exchanges, prime brokers (PBs), and web-scraping. The goal is to ensure that systematic and fundamental portfolio managers, along with enterprise functions such as Operations, Risk, Trading, and Compliance, have seamless access to high-quality data. Additionally, the role involves developing internal data products and analytics that will enhance decision-making across the organization.
Key Responsibilities
- Implement web scraping solutions using scripts, APIs, and various automation tools to collect structured and unstructured data.
- Play a key role in designing, developing, and maintaining a greenfield data platform on Snowflake and AWS, ensuring scalability, security, and efficiency.
- Analyze existing data pipelines, identify performance bottlenecks, and enhance them to support new business requirements.
- Onboard new data providers by integrating their feeds into the platform, ensuring seamless data ingestion and transformation.
- Lead data migration projects, ensuring a smooth transition of data from legacy systems to the modern cloud-based infrastructure.
- Work closely with DevOps and engineering teams to optimize deployment processes using Kubernetes, Docker, and Jenkins.
- Collaborate with stakeholders, including data scientists, analysts, and developers, to ensure data solutions align with business needs.
- Maintain high standards of data quality, reliability, and compliance by implementing best practices in data engineering and governance.
Required Skills and Qualifications
- A minimum of 10 years of experience as a Data Engineer, with a proven track record of designing and building robust data platforms.
- Expertise in SQL for data querying, transformation, and optimization.
- Proficiency in Python for data processing, automation, and analytics.
- Strong Linux skills, including scripting and system administration.
- Hands-on experience with containerization technologies such as Docker and Kubernetes for scalable deployment.
- Solid understanding of cloud platforms, specifically AWS, including services like S3, Lambda, EC2, and RDS.
- Strong DevOps knowledge with experience in Kubernetes (K8s), Docker, Jenkins, and CI/CD pipelines.
- Excellent communication skills, with the ability to explain technical concepts to both technical and non-technical stakeholders.
Preferred Skills and Experience
- Experience working with market data projects or in capital markets, providing insights into financial data processing and analysis.
- Familiarity with Apache Airflow for workflow orchestration and automated data pipelines.
- Knowledge of advanced data engineering concepts such as distributed computing, parallel processing, and ETL optimization.
Why Join Us?
This role offers an exciting opportunity to work on cutting-edge data engineering projects within a dynamic and innovative environment. You will be instrumental in building a state-of-the-art data platform that supports critical business functions across the organization. If you are passionate about data engineering, cloud technologies, and financial data solutions, we encourage you to apply and be part of our growing team.