Download PDF

ClickHouse > Case Studies > ClickHouse: The Backbone of Dassana's Security Data Lake

ClickHouse: The Backbone of Dassana's Security Data Lake

Technology Category

Application Infrastructure & Middleware - Event-Driven Application
Cybersecurity & Privacy - Cloud Security

Applicable Industries

Electrical Grids
National Security & Defense

Use Cases

Cybersecurity
Tamper Detection

Services

System Integration
Training

The Challenge

Modern enterprises are investing heavily in security products due to the increasing cyber risks and their impact on businesses. A typical large enterprise today uses more than a dozen security technologies, which emit data in various shapes and sizes, making it difficult to make sense of the data. Security Information and Event Management (SIEM) systems, designed for immutable time series event data, struggle with the mutable nature of security data. For instance, the state of an alert could change from 'open' to 'closed', and SIEMs cannot update this change. The solution is to re-insert the updated data and query the most recent data, which is challenging on append-only systems like SIEMs. Additionally, SIEM companies have stopped innovating and investing in solving basic problems such as data normalization. Dassana, a security data lake, aims to address these challenges.

About The Customer

Dassana is a security data lake founded by a team of successful serial entrepreneurs and cloud security veterans. They aim to consolidate disparate data sources and provide contextualized data insights. Their goal is to simplify data access at scale without compromising performance and optimizing costs, enabling customers to focus on strategic business priorities. Dassana ingests data from various security sources like Cloud Security Posture Management (CSPM) tools, Intrusion Detection Systems (IDS), etc., and normalizes it, allowing for schema-less querying and visualization.

The Solution

Dassana chose ClickHouse after evaluating more than a dozen different big data systems. ClickHouse offers flexibility with different table engines for storing data according to the use case, the ability to insert data frequently using asynchronous inserts, an in-built accounting system to track query costs, and the ability to automatically move data to a different storage tier. Dassana uses the ReplacingMergeTree table engine to store mutable data and the AggregatingMergeTree with a materialized view for near real-time deduplicated results. To query duplicated data, they used a 'group by' aggregation query. As their data grew, they started using the dictionary feature of ClickHouse for better performance. They also experimented with the 'Join' table engine but found the dictionary-based approach more performant.

Operational Impact

ClickHouse has provided Dassana with a flexible and cost-effective solution for managing their security data. The different table engines allow Dassana to store data in a format that suits their use case, and the ability to insert data frequently using asynchronous inserts has simplified their operations. The in-built accounting system helps them track query costs, and the automatic data movement to different storage tiers has saved them development time. The advanced external dictionary options have enabled them to achieve better performance as their data grew. However, they also learned that ClickHouse needs to be handled with care and attention to detail, and they recommend using ClickHouse Cloud for managing the complexity of scaling and replicating data.

Quantitative Benefit

ClickHouse costs a fraction of what other big data systems would cost Dassana.
ClickHouse's in-built storage tier system has saved Dassana precious development time.
Dassana can store hundreds of millions of assets (distinct record_id) data on a 32GB RAM box.