Data Access and Caching

Panopticon assumes in general that data is never at rest and too big to be simply loaded into memory. The data can either be subscribed against or polled (automatically refreshed on a defined period).

This means either:

q  Load Subset of Data in Memory

q  Load Summary and Parameterized Detail Views

q  ROLAP (Dynamically explore datasets)

Consequently, for direct access, Panopticon is only as fast as the underlying data platform, or the refreshing of result set caches.

 

When data is not changing on a timely basis, such as a daily updated data warehouse, there is the additional option of retrieving data into a data extract.

 

Consequently:

q  Only required data is retrieved. Majority of the data stays in the underlying data sources

q  Typically aggregated, conflated, filtered data is retrieved

q  Behind each dashboard part (visualization) is a micro-cube

q  Each cube is designed for streaming real time updates

q  Behind each cube is a real-time data table (also powering filters)

q  Behind each data table is a resultset cache

q  Behind the cache is the underlying data repository

q  Caches can be loaded on the fly, or pre-loaded on a periodic basis

q  All caching is optional

q  Consequently, data access is either:

·         Work Directlyagainst underlying sources (either Exploratory Analysis (ROLAP), Or Pre-Defined Parameterised Views)

·         Extract & Cache Data from slower underlying sources.  And query this data extract locally. (Similar to competitor products).

In reality, usage is typically Hybrid.  Based on the characteristics of the underlying data, you choose whether to extract and load, or query directly.

This is to cater for real world data landscapes, where different data has different data retrieval latency characteristics, and different timeliness; and where there is too much data to simply load all into memory.