Download PDF
Building a Unified Data Platform with ClickHouse: A Case Study on Synq
Technology Category
- Analytics & Modeling - Machine Learning
- Platform as a Service (PaaS) - Application Development Platforms
Applicable Industries
- Buildings
- Cement
Applicable Functions
- Product Research & Development
- Quality Assurance
Use Cases
- Building Automation & Control
- Time Sensitive Networking
Services
- System Integration
- Testing & Certification
The Challenge
Synq, a data observability platform, faced the challenge of managing the complexity, variety, and increasing volumes of data that powered their software system. The company needed to merge operational and analytical needs into a unified data platform. They were dealing with a continuous stream of data from dozens of systems, with frequent bursts of volume when customers ran large batch processing jobs or when new customers were onboarded. The company had set ambitious performance goals for backfilling data and wanted to provide immediate value to customers as they onboarded their product. They also wanted an infrastructure that could serve their first set of defined use cases and provide functionality to support new use cases quickly. Lastly, they aimed to build a single platform that could store their raw log data and act as a serving layer for most data use cases needed by their applications and APIs.
About The Customer
Synq is a data observability platform that analyzes log-level data from complex data ecosystems. It is a large-scale log processing engine that ingests and processes data from dozens of systems. The platform is designed to provide deep integration into ClickHouse clusters with capabilities to detect delayed data loads and uncover hidden data anomalies. It also offers automatically created data lineage and tooling for managing data quality. Synq serves teams at companies such as Typeform, Instabee, and LendInvest, helping them monitor their cloud data stacks.
The Solution
Synq found the solution to their challenges in ClickHouse, a high-performance column-oriented database management system. After a few days of testing, they found that ClickHouse could ingest tens of thousands of rows per second, create query-specific data models, and maintain consistent read query performance under heavy ingest load. To focus their entire development team on the R&D of their platform, they partnered with ClickHouse Cloud. They built a solid ingestion system using the officially maintained Go client clickhouse-go. They also leveraged the ReplacingMergeTree table engine to handle duplicate events. To optimize performance, they created specialized tables that transformed their raw logs data to a format optimized for their queries. They also used the popular data transformation framework dbt to create auditing tables that extract summary statistics about their log data. Finally, they used their ClickHouse cluster as a backbone for many other use cases, including in-app analytics.
Operational Impact
Quantitative Benefit
Related Case Studies.
Case Study
Energy Saving & Power Monitoring System
Recently a university in Taiwan was experiencing dramatic power usage increases due to its growing number of campus buildings and students. Aiming to analyze their power consumption and increase their power efficiency across 52 buildings, the university wanted to build a power management system utilizing web-based hardware and software. With these goals in mind, they contacted Advantech to help them develop their system and provide them with the means to save energy in the years to come.
Case Study
System 800xA at Indian Cement Plants
Chettinad Cement recognized that further efficiencies could be achieved in its cement manufacturing process. It looked to investing in comprehensive operational and control technologies to manage and derive productivity and energy efficiency gains from the assets on Line 2, their second plant in India.
Case Study
Intelligent Building Automation System and Energy Saving Solution
One of the most difficult problems facing the world is conserving energy in buildings. However, it is not easy to have a cost-effective solution to reduce energy usage in a building. One solution for saving energy is to implement an intelligent building automation system (BAS) which can be controlled according to its schedule. In Indonesia a large university with a five floor building and 22 classrooms wanted to save the amount of energy being used.
Case Study
Powering Smart Home Automation solutions with IoT for Energy conservation
Many industry leaders that offer Smart Energy Management products & solutions face challenges including:How to build a scalable platform that can automatically scale-up to on-board ‘n’ number of Smart home devicesData security, solution availability, and reliability are the other critical factors to deal withHow to create a robust common IoT platform that handles any kind of smart devicesHow to enable data management capabilities that would help in intelligent decision-making
Case Study
Commercial Building Automation Boosts Energy Efficiency
One of the challenges to building automation is the multitude of non-interoperable communications protocols that have evolved over the years. Buildings have several islands of automation. Bridging the islands of different automation without losing the considerable investment in each specialized control network is the main focus in this solution.
