Download PDF
Leveraging ClickHouse Kafka Engine for Enhanced Data Collection and Analysis: A Case Study of Superology
Technology Category
- Analytics & Modeling - Machine Learning
- Application Infrastructure & Middleware - Event-Driven Application
Applicable Industries
- Buildings
- Construction & Infrastructure
Applicable Functions
- Quality Assurance
Use Cases
- Experimentation Automation
- Time Sensitive Networking
Services
- System Integration
- Testing & Certification
The Challenge
Superology, a leading product tech company in the sports betting industry, was faced with the challenge of effectively collecting and analyzing quantitative data to improve customer experience and business operations. The company needed to gather metrics such as app or site visits, customer clicks on specific pages, number of comments and followers in their social section, and various conversion events and bounce rates. The data collected varied in structure, requiring a dynamic approach to data collection and analysis. Superology was using Google Protocol Buffers (Protobuf) to collect this data, but needed a more efficient and scalable solution to handle the large volume of data and its dynamic nature.
About The Customer
Superology is a seasoned product tech company that has been innovating in the sports betting industry since 2012. Acquired by Superbet group in 2017, it has become a leading force in the industry, with its platforms being used by hundreds of thousands of people and processing millions of transactions daily. Superology uses a data-informed approach at every level of work to satisfy user needs and accomplish business goals. The company values personal growth as much as company growth, and empowers its people to deploy their talents and own their work end-to-end.
The Solution
Superology adopted the ClickHouse Kafka Engine and Protocol Buffers to enhance their data collection and analysis process. ClickHouse, with its built-in Kafka connector and Protobuf input type, provided a fast and reliable solution. The company was able to easily scale their ClickHouse implementation both horizontally and vertically. The data collected was ingested into a 'big' origin table, with the ClickHouse columnar structure offering great extensibility. Superology also implemented a system to allow changes to their proto scheme, ensuring backward compatibility. The data was then filtered and transformed using a materialized view, enabling a focused analysis of specific aspects of customer behavior. ClickHouse also facilitated efficient AB testing and other experiments, with its extensive set of statistical functions. Superology plans to further enrich their ClickHouse architecture by coupling it with MindsDB, to create a Machine learning architecture at the database level.
Operational Impact
Quantitative Benefit
Related Case Studies.
Case Study
Energy Saving & Power Monitoring System
Recently a university in Taiwan was experiencing dramatic power usage increases due to its growing number of campus buildings and students. Aiming to analyze their power consumption and increase their power efficiency across 52 buildings, the university wanted to build a power management system utilizing web-based hardware and software. With these goals in mind, they contacted Advantech to help them develop their system and provide them with the means to save energy in the years to come.
Case Study
IoT System for Tunnel Construction
The Zenitaka Corporation ('Zenitaka') has two major business areas: its architectural business focuses on structures such as government buildings, office buildings, and commercial facilities, while its civil engineering business is targeted at structures such as tunnels, bridges and dams. Within these areas, there presented two issues that have always persisted in regard to the construction of mountain tunnels. These issues are 'improving safety" and "reducing energy consumption". Mountain tunnels construction requires a massive amount of electricity. This is because there are many kinds of electrical equipment being used day and night, including construction machinery, construction lighting, and ventilating fan. Despite this, the amount of power consumption is generally not tightly managed. In many cases, the exact amount of power consumption is only ascertained when the bill from the power company becomes available. Sometimes, corporations install demand-monitoring equipment to help curb the maximum power demanded. However, even in these cases, the devices only allow the total volume of power consumption to be ascertained, or they may issue warnings to prevent the contracted volume of power from being exceeded. In order to tackle the issue of reducing power consumption, it was first necessary to obtain an accurate breakdown of how much power was being used in each particular area. In other words, we needed to be able to visualize the amount of power being consumed. Safety, was also not being managed very rigorously. Even now, tunnel construction sites often use a 'name label' system for managing entry into the work site. Specifically, red labels with white reverse sides that bear the workers' names on both sides are displayed at the tunnel work site entrance. The workers themselves then flip the name label to the appropriate side when entering or exiting from the work site to indicate whether or not they are working inside the tunnel at any given time. If a worker forgets to flip his or her name label when entering or exiting from the tunnel, management cannot be performed effectively. In order to tackle the challenges mentioned above, Zenitaka decided to build a system that could improve the safety of tunnel construction as well as reduce the amount of power consumed. In other words, this new system would facilitate a clear picture of which workers were working in each location at the mountain tunnel construction site, as well as which processes were being carried out at those respective locations at any given time. The system would maintain the safety of all workers while also carefully controlling the electrical equipment to reduce unnecessary power consumption. Having decided on the concept, our next concern was whether there existed any kind of robust hardware that would not break down at the construction work site, that could move freely in response to changes in the working environment, and that could accurately detect workers and vehicles using radio frequency identification (RFID). Given that this system would involve many components that were new to Zenitaka, we decided to enlist the cooperation of E.I.Sol Co., Ltd. ('E.I.Sol') as our joint development partner, as they had provided us with a highly practical proposal.
Case Study
Intelligent Building Automation System and Energy Saving Solution
One of the most difficult problems facing the world is conserving energy in buildings. However, it is not easy to have a cost-effective solution to reduce energy usage in a building. One solution for saving energy is to implement an intelligent building automation system (BAS) which can be controlled according to its schedule. In Indonesia a large university with a five floor building and 22 classrooms wanted to save the amount of energy being used.
Case Study
Powering Smart Home Automation solutions with IoT for Energy conservation
Many industry leaders that offer Smart Energy Management products & solutions face challenges including:How to build a scalable platform that can automatically scale-up to on-board ‘n’ number of Smart home devicesData security, solution availability, and reliability are the other critical factors to deal withHow to create a robust common IoT platform that handles any kind of smart devicesHow to enable data management capabilities that would help in intelligent decision-making
Case Study
Splunk Partnership Ties Together Big Data & IoT Services
Splunk was faced with the need to meet emerging customer demands for interfacing IoT projects to its suite of services. The company required an IoT partner that would be able to easily and quickly integrate with its Splunk Enterprise platform, rather than allocating development resources and time to building out an IoT interface and application platform.