Download PDF
Harnessing ClickHouse and Materialized Views for High-Performance Analytics: A Case Study of Inigo
Technology Category
- Application Infrastructure & Middleware - Database Management & Storage
- Infrastructure as a Service (IaaS) - Cloud Databases
Applicable Industries
- Equipment & Machinery
- National Security & Defense
Applicable Functions
- Maintenance
- Product Research & Development
Use Cases
- Real-Time Location System (RTLS)
- Time Sensitive Networking
Services
- Cloud Planning, Design & Implementation Services
- System Integration
The Challenge
Inigo, a pioneering company in the GraphQL API management industry, was in search of a database solution that could handle a high volume of raw data for analytics. They explored various alternatives, including SQLite, Snowflake, and PostgreSQL, but none of these options met their needs. Snowflake was too slow and costly for their needs, especially when handling real-time customer data within a product. PostgreSQL, while an excellent transactional database, was unsuitable for large analytic workloads. The company was able to get it to work with around 100K rows, but past that, the indexes were growing out of control and the cost of running a PostgreSQL cluster didn’t make sense. There was significant performance degradation once they hit the 100K - 1M rows mark. Inigo needed a solution that could handle billions of GraphQL API requests, create aggregated views on high cardinality data, generate alerts, and create dashboards based on extensive data.
About The Customer
Inigo is a leading company in the GraphQL API management industry, specializing in comprehensive security and governance at scale. Their platform empowers API and platform teams to fully control, manage, and secure GraphQL deployments by providing immediate visibility, security, access control, compliance, and continuous delivery. In addition to its security features, Inigo offers key developer experience enhancements and workflow efficiencies, ultimately enabling organizations to expedite GraphQL API development, lower costs, and improve end-user experiences. They process and analyze billions of GraphQL API requests, creating aggregated views on high cardinality data, generating alerts, and creating dashboards based on extensive data.
The Solution
Inigo chose ClickHouse, a columnar database specifically designed to handle high volumes of data while providing fast query execution. They paired ClickHouse with Materialized Views, which allow datasets to be pre-aggregated, significantly improving the performance of analytical queries. ClickHouse's ability to automatically manage and update materialized views as new data is ingested simplified the maintenance process. The open-source nature of ClickHouse and the ability to run it as a simple binary or docker on their laptops were major attractions. Inigo's current ClickHouse setup is a single database where nearly every Inigo service has its own table, with some tables containing billions of rows of data. They consolidated the data into several materialized views, each grouping the data into distinct time window buckets. By creating materialized views with different intervals, they could enhance query performance and enable more efficient data analysis across various timeframes.
Operational Impact
Quantitative Benefit
Related Case Studies.
Case Study
Smart Water Filtration Systems
Before working with Ayla Networks, Ozner was already using cloud connectivity to identify and solve water-filtration system malfunctions as well as to monitor filter cartridges for replacements.But, in June 2015, Ozner executives talked with Ayla about how the company might further improve its water systems with IoT technology. They liked what they heard from Ayla, but the executives needed to be sure that Ayla’s Agile IoT Platform provided the security and reliability Ozner required.
Case Study
IoT enabled Fleet Management with MindSphere
In view of growing competition, Gämmerler had a strong need to remain competitive via process optimization, reliability and gentle handling of printed products, even at highest press speeds. In addition, a digitalization initiative also included developing a key differentiation via data-driven services offers.
Case Study
Predictive Maintenance for Industrial Chillers
For global leaders in the industrial chiller manufacturing, reliability of the entire production process is of the utmost importance. Chillers are refrigeration systems that produce ice water to provide cooling for a process or industrial application. One of those leaders sought a way to respond to asset performance issues, even before they occur. The intelligence to guarantee maximum reliability of cooling devices is embedded (pre-alarming). A pre-alarming phase means that the cooling device still works, but symptoms may appear, telling manufacturers that a failure is likely to occur in the near future. Chillers who are not internet connected at that moment, provide little insight in this pre-alarming phase.
Case Study
Premium Appliance Producer Innovates with Internet of Everything
Sub-Zero faced the largest product launch in the company’s history:It wanted to launch 60 new products as scheduled while simultaneously opening a new “greenfield” production facility, yet still adhering to stringent quality requirements and manage issues from new supply-chain partners. A the same time, it wanted to increase staff productivity time and collaboration while reducing travel and costs.
Case Study
Integration of PLC with IoT for Bosch Rexroth
The application arises from the need to monitor and anticipate the problems of one or more machines managed by a PLC. These problems, often resulting from the accumulation over time of small discrepancies, require, when they occur, ex post technical operations maintenance.
Case Study
Robot Saves Money and Time for US Custom Molding Company
Injection Technology (Itech) is a custom molder for a variety of clients that require precision plastic parts for such products as electric meter covers, dental appliance cases and spools. With 95 employees operating 23 molding machines in a 30,000 square foot plant, Itech wanted to reduce man hours and increase efficiency.
