Your Responsibilities
- Design, build, and maintain robust data pipelines for real-time trading applications.
- Manage and optimise Data Lakes and data warehousing solutions.
- Support development and deployment of ML/AI Model, using MLOps best practices
- Oversee data collection processes and ensure high standards of data integrity.
- Employ automation to enhance data processes and system efficiency.
- Develop and maintain scalable data pipelines to support real-time decision-making.
- Manage the storage and accessibility of data in Data Lakes and warehousing systems.
- Implement automation strategies to improve data integrity and processing.
- Collaborate with Software Engineers and other Data Engineers on diverse projects to enhance trading systems.
- Deep knowledge of programming languages such as Java, C# and/or C++ and developing application in Python
- Significant experience working with modern data technologies, like Postgres, Timescale dB, Quest dB, MongoDB, Kafka/Event Hub
- Acquaintance in deploying and managing microservices with docker, Kubernetes and bash scripting
- Expert knowledge of Azure, AWS or GCP and management of large-scale data pipeline system
- Desired capabilities with MLOps Frameworks like Kubeflow, MLFlow, Databrick ML tools, etc
- Experience with Databricks, Data Lakes, and workflow orchestration tools like Dagster or Airflow.
- Problem solver oriented with a keen interest in environmental sustainability and technological innovation.
- Familiarity with automated trading systems and real-time data streaming for high-frequency trading, is beneficial but not mandatory.