from the first second, the rushing party to the order payment, to complete the real-time calculation and delivery to the full screen of the media. Second level response. In the face of increasing traffic, real-time data is getting faster and faster. Behind the hold of the data peak is a comprehensive upgrade of Alibaba’s streaming computing technology.
Stream computing application scenario
The Data Technology and Products Department is positioned in the Alibaba Data Center. In addition to offline data, the real-time data it produces also serves multiple data scenarios within the group. Including this year’s(in fact, any previous year) double 11, Media large-screen real-time data, real-time data-oriented business consultant, As well as various live studio products for internal executives and small staff, covering the entire Alibaba Group’s Big Data Division.
At the same time, with the continuous development of the business, so far,the daily real-time processing peak value exceeds 40 million/s , the total number of processing records per day has reached the trillion level, and the total processing data volume has reached the PB level.
In the face of real-time data of massive data, we have successfully achieved data delay control in the range of seconds. In terms of calculation accuracy, high precision and 0 error have been achieved to achieve accurate processing. For example, this year’s double 11 day, the first record of the double eleven media screen is calculated from the transaction table, and the calculation process reaches the media big screen is a second response.
Data link in the practice of streaming computing in data
After years of experience in data flooding, our stream computing team has accumulated a wealth of experience in engine selection, optimization performance, and development flow computing platforms. We have also formed a stable and efficient data link architecture. The following figure is a schematic diagram of the entire data link:
There are many sources of business data, and the incremental data is acquired in real time through two tools (DRC and middleware logagent) and synchronized to DataHub (a PubSub service).
The real-time calculation engine Flink job is processed in real time by subscribing to these incremental data , and after the ETL processing , the detailed layer is returned to the Datahub again . All the business parties will define the real-time data onto multi-dimensional aggregation , and the aggregated data will be put . In a distributed database or a relational database (Hbase , Mysql ) , and provide real-time data services through a public data service (One Service ) .
In the past year , we have done a lot of work on computer engines and computer optimization , achieving an increase in computing power and development efficiency .
Calculation engine upgrades and optimizations
In 2017, we performed a comprehensive upgrade on the real-time computing architecture, migrating from Storm to Blink, and doing a lot of optimization on the new technology architecture, real-time peak processing capacity increased by 2 times, and smooth processing capacity is more than 5 times higher:
Optimized state management
A lot of state is generated in the real-time computing process. It was previously stored in HBase and will now be stored in RocksDB. Local storage reduces network overhead and can greatly improve performance and meet fine-grained data statistics. (Now the number of keys can be raised to the billion level, is it awesome?
Optimize checkpoint and compaction
State will become larger and larger as time goes by. If you do a full checkpoint every time, the pressure on the network and disk is very large; Therefore, for the scenario of data statistics, by optimizing the configuration of rocksdb and using incremental checkpoint, network transmission and disk read and write can be greatly reduced.
Asynchronous Sink
Changing the sink to an asynchronous form maximizes CPU utilization and provides a large TPS.
Abstract public component
In addition to engine-level optimization, the data center also develops its own aggregation components based on Blink (currently all real-time public layer tasks are implemented through this component). This component provides the functionality commonly used in data statistics to abstract the topology and business logic into a json file. This only needs to be controlled by parameters in the json file to realize development and configuration, which greatly reduces the development threshold and shortens the development cycle. Let’s take another example: Before we did the development work for 10 people/day, now that componentization has reduced the workload to 0.5 person/day, it is good news for both the demand side and the developer side. The components improve job performance.
According to the above ideas and functional precipitation, the flow computing development platform [Red Rabbit] was finally polished.
The platform generates real-time tasks in a simple “support,drag,haul” format, without writing a single line of code, providing regular data statistics components, and integrating metadata management and reporting system access. As a team supporting the Group’s real-time computing business, our special features in the [Red Rabbit Platform] that have been precipitated after the live ammunition of the double 11 have become its unique highlights:
First, size dimension merging
such as many real-time statistical operations need to do the calculation of the day granularity and hourly granularity, previously calculated separately by two tasks, the aggregation component will combine these tasks, and the intermediate state to share, reduce network transmission More than 50% will also streamline the calculation logic and save CPU.
Two, compact storage
For Keyvalue stored in RocksDB, we designed an encoding mechanism that uses an index to effectively reduce the state storage more than half. This optimization can effectively reduce the stress on the network, CPU, and disk.
Third, high performance sorting
Sorting is a very common scenario in real time. The top component takes advantage of the Priorityqueue in memory and the new MapState feature in blink ,which greatly reduces the number of serializations and improves performance about 10 times.
Fourth, bulk transfer and write operations
When we finally write the results table HBase and Datahub, if we write a library every time we write a record, it will greatly limit our throughput. Our components use a time-triggered or record-triggered mechanism (timer function) to implement bulk transfer and mini-batch sink, and can be flexibly configured according to service delay requirements, improving task throughput while reducing service. End pressure.
Data protection
For high-priority applications (24-hour uninterrupted service), you need to implement disaster recovery across the equipment room. When there is a problem with a link, you can switch to other links in seconds. The following figure shows the link of the entire real-time public layer. Security architecture diagram:
From data collection, data synchronization, data calculation, data storage, and data services, the entire link is independent. Through dynamic configuration in Oneservice, link switching can be implemented to ensure that data services are not terminated.
The above content is the secret weapon of stream computing technology that guarantees the peak of this year’s double 11 traffic – we are not only innovating, but also hope to precipitate and reuse technology and optimize it to the daily life.
As the technology of the flow calculation is constantly changing, we will continue to optimize the upgrade flow calculation technology based on the Ali rich business scenario:
- Platformization, servitization, Stream Processing as a service
- The semantic layer is unified, Apache Beam, Flink’s Table API, and finally Stream SQL are very hot projects.
- Real-time intelligence, the deep learning of the current fire may cause sparks in the future
- Real-time offline unification, this is also a big trend. Compared with the current real-time set, the off-line set of practices, real-time offline unification is also what all major engines try to achieve.
The article is selected from the WeChat public Ali technology:
For more articles you can follow us on:
