Erik Rydow

Abstract:

Many software and hardware applications generate an increasing volume of data and logs in real-time. Visual analytics is essential to support system monitoring and analysis of such data. For example, the world's largest radio telescope, the Square Kilometer Array (SKA), is expected to generate an estimated 160 TB a second of raw data captured from different sources. Transporting large amounts of data from distributed sources to a web browser for visualization is time-consuming due to data transport latencies. In addition, visualizing real-time data in the browser is challenging and limited by the data rates which a web browser can handle. We propose a novel low latency data streaming architecture, which uses a messaging system for real-time data transport to the web browser. Based on this architecture, we propose techniques and provide a tool for analyzing the performance of serialization protocols and the web-visualization rendering pipeline. We empirically evaluate the performance of our architecture using three visualizations use cases relevant to the SKA. Our system proved extremely useful in streaming high-volume data in real-time with low latency and greatly enhanced the web-visualization performance by enabling streaming an optimal number of data points to different visualizations.