ClickHouse: Powering Darwinium's Security and Fraud Analytics

• Application Infrastructure & Middleware - Database Management & Storage
• Infrastructure as a Service (IaaS) - Cloud Databases

• National Security & Defense
• Oil & Gas

• Logistics & Transportation
• Product Research & Development

• Cybersecurity
• Fraud Detection

• Cloud Planning, Design & Implementation Services
• System Integration

Darwinium, a digital risk platform, was facing several challenges in the security and fraud domain. The platform needed to ingest and process data at a high throughput, deal with large volumes of data, and have capabilities to analyze data in a complex way. The database backend needed to handle high-speed writes and serve data for analysis as soon as it was ingested. Darwinium's real-time engine continuously profiles and monitors a digital asset, resulting in large volumes of data. The database needed to be capable of analyzing data at scale, and potentially process an entire year's worth of data. Technical types of fraud and security challenges required storing most digital datapoints for future investigations. The nature of analyzing fraudulent data required complex interactive analysis, and a database system that could respond in timeframes of 1 second or less, while providing a feature-rich functional toolbox.

Darwinium is a digital risk platform that supports real-time journey orchestration and continuous adaptive trust for digital user authentication. It was built to tackle complex business problems as they happen, adapting to adversaries regardless of how quickly they attack. The platform is designed for developers and data scientists to test, model, and deploy with ease, regardless of business processes or organizational constraints. Darwinium integrates with your Content Delivery Network (CDN) or Proxy as a Darwinium-hosted solution, or optional one-click install onto a new or existing Kubernetes cluster.

Darwinium chose ClickHouse as their database engine after assessing several solutions. ClickHouse's mutable engine simplified the data ingestion pipeline complexity, and its choice of table engine led to simpler data pipelines. ClickHouse could easily handle a few thousand writes per second, with multiple data pipeline writers writing to a single ClickHouse server at any given instant. It also supported complex data types, which was a fundamental requirement for Darwinium to build upon for interactive analytics. ClickHouse's recent addition of JSON type support further met the analytics requirements of Darwinium. ClickHouse also supported the concept of tumbling data retention windows, where “hot” data could be initially placed on a fast access medium like a local SSD, with the ability to subsequently move the data to a relatively slower but cheaper storage system like S3. ClickHouse was also cloud-native, and could be run from a low-end laptop, an on-premise cluster of nodes, or on any of the myriad hardware configurations possible even on a single cloud provider like AWS.

• ClickHouse's mutable engine simplified the data ingestion pipeline complexity, and its choice of table engine led to simpler data pipelines. It also supported complex data types, which was a fundamental requirement for Darwinium to build upon for interactive analytics. ClickHouse's recent addition of JSON type support further met the analytics requirements of Darwinium. ClickHouse also supported the concept of tumbling data retention windows, where “hot” data could be initially placed on a fast access medium like a local SSD, with the ability to subsequently move the data to a relatively slower but cheaper storage system like S3. ClickHouse was also cloud-native, and could be run from a low-end laptop, an on-premise cluster of nodes, or on any of the myriad hardware configurations possible even on a single cloud provider like AWS. This made ClickHouse a cloud native offering for Darwinium use cases.

• ClickHouse could easily handle a few thousand writes per second, with multiple data pipeline writers writing to a single ClickHouse server at any given instant.
• ClickHouse server side metrics showed a consistent and very low (<5%) user space and system space core usage while the write operations alone were being executed.
• ClickHouse can utilise all of the cores available on a single node that it is running on to process a query.