Job description
Big Data Leader
Responsibilities
1. Overall Planning of Data Warehouse Architecture
Own the overall architecture design, iterative upgrades, and implementation of the core data warehouse and data middle platform for the exchange. Lead the construction of real-time data warehouse, offline data warehouse, streaming-batch integration, and lake-warehouse integration architecture. Coordinate the overall planning of offline and real-time data systems to support data needs across trading, risk control, user, funds, marketing, and quantitative scenarios, ensuring high availability, high concurrency, high real-time performance, scalability, and security of the data architecture.
2. Core Construction of Real-time Data Warehouse
Lead the construction and optimization of real-time and offline data pipelines for the exchange, utilizing core components such as Flink, Kafka, and ClickHouse. Complete the collection, cleaning, computation, modeling, and output of real-time data including trading matching data, user behavior data, fund flows, market data, and risk control logs. Support core scenarios such as real-time risk control alerts, real-time fund monitoring, real-time market data, quantitative data push, and real-time monitoring of abnormal trading, ensuring second-level and minute-level data timeliness and accuracy.
3. Data Modeling and System Specification Construction
Responsible for the overall design of the layered architecture of the data warehouse (ODS/DWD/DWS/ADS) and thematic domain models. Establish exclusive data modeling specifications for the exchange, ETL/ELT development standards, real-time task development specifications, and data metric system specifications. Unify data standards across the company, building five core data thematic domains: trading, users, assets, risk control, and operations, addressing core issues such as data redundancy, inconsistent standards, and data latency.
4. Data Quality and Data Governance
Establish a comprehensive data quality monitoring, lineage analysis, permission control, and data security system for the exchange. Conduct targeted governance on issues such as real-time data latency, data loss, data corruption, and duplicate calculations. Implement real-time task alert mechanisms, fault review mechanisms, and data iteration review mechanisms to ensure the accuracy, consistency, completeness, and timeliness of financial-grade data, meeting compliance and risk control requirements of the exchange.
5. Team Management and Project Implementation
Lead the data warehouse development team, responsible for daily management, task breakdown, progress control, technical challenges, talent development, and team building. Collaborate with business, product, risk control, operations, and quantitative teams to deeply analyze business requirements, coordinate various data development, data iteration, and data platform optimization projects, and deliver data analysis reports, data dashboards, and underlying data services to empower business decisions.
6. Technical Optimization and Cost Control
Continuously optimize real-time data warehouse tasks, offline computing tasks, storage resources, and computing resources to resolve issues such as data throughput bottlenecks, task accumulation, excessive delays, and resource wastage in high-concurrency trading scenarios. Optimize data warehouse architecture performance, reduce server and operational costs, and ensure stable operation of data systems during promotions, market fluctuations, and high-concurrency trading scenarios.
Requirements
1. Education and Work Background
Bachelor's degree or above in Computer Science, Big Data, Software Engineering, Statistics, or related fields, with over 5 years of experience in big data data warehouse development and over 2 years of team management experience in exchanges/financial quant/ payment industries. Experience in building real-time data warehouses from scratch is required, familiarity with cryptocurrency exchange operations is a plus.
2. Core Technical Skills (Mandatory)
Proficient in real-time data warehouse, offline data warehouse architecture, and streaming-batch integration technology systems. Skilled in using core big data components such as Flink, Spark, Kafka, ClickHouse, Hive, HBase, and Doris. Proficient in real-time data pipeline development and real-time computing engine optimization, with practical experience in processing high-concurrency, large-volume real-time trading data. Skilled in data modeling, layered design, and metric system construction, with expertise in financial-grade data governance and data quality control solutions.
3. Business Acumen
Deep understanding of the core business logic of exchanges, familiar with trading in the cryptocurrency space, matching mechanisms, fund clearing, user account systems, risk control and anti-fraud measures, market data, quantitative trading, and marketing activities. Ability to independently break down complex business requirements into standardized, reusable data models and data services.
4. Project and Problem-Solving Skills
Experience in implementing large-scale financial/exchange real-time data warehouse projects, capable of independently resolving online issues such as real-time task delays, data skew, task avalanches, and data anomalies. Experience in leading architecture iterations, technical restructuring, and targeted governance projects, able to withstand high-intensity, high-concurrency, and high-stability financial-grade technical requirements.
5. Comprehensive Qualities
Excellent team management, communication, coordination, requirement control, and project management skills. Clear logic, strong sense of responsibility, and strong ability to work under pressure and conduct problem reviews. Familiar with financial data compliance and data security standards.
Preferred Qualifications
1. Experience in building and maintaining real-time data warehouses for leading exchanges such as OKX, Gate, Binance, and Coinbase is preferred;
2. Proficient in lake-warehouse integration architecture, with experience in self-developing big data platforms and building data metric platforms is preferred;
3. Experience in constructing quantitative data, high-frequency trading data, and real-time risk control data systems is preferred;
4. Experience in building teams from scratch, cultivating technical teams, and standardizing process systems is preferred.
Work Location
Remote