下载PDF
Building a Unified Data Platform with ClickHouse: A Case Study on Synq
技术
- 分析与建模 - 机器学习
- 平台即服务 (PaaS) - 应用开发平台
适用行业
- 建筑物
- 水泥
适用功能
- 产品研发
- 质量保证
用例
- 楼宇自动化与控制
- 时间敏感网络
服务
- 系统集成
- 测试与认证
挑战
Synq 是一个数据可观察性平台，面临着管理为其软件系统提供支持的复杂性、多样性和不断增加的数据量的挑战。该公司需要将运营和分析需求合并到一个统一的数据平台中。他们正在处理来自数十个系统的连续数据流，当客户运行大型批处理作业或新客户加入时，数据量会频繁爆发。该公司为回填数据设定了雄心勃勃的性能目标，并希望在客户使用产品时立即为他们提供价值。他们还希望有一个基础设施能够为其第一组定义的用例提供服务，并提供快速支持新用例的功能。最后，他们的目标是构建一个单一平台，可以存储原始日志数据，并充当应用程序和 API 所需的大多数数据用例的服务层。
关于客户
Synq 是一个数据可观测平台，可分析来自复杂数据生态系统的日志级数据。它是一个大型日志处理引擎，可以从数十个系统中摄取和处理数据。该平台旨在提供与 ClickHouse 集群的深度集成，具有检测延迟数据加载和发现隐藏数据异常的功能。它还提供自动创建的数据沿袭和用于管理数据质量的工具。 Synq 为 Typeform、Instabee 和 LendInvest 等公司的团队提供服务，帮助他们监控云数据堆栈。
解决方案
Synq 在 ClickHouse（一种高性能的面向列的数据库管理系统）中找到了应对挑战的解决方案。经过几天的测试，他们发现 ClickHouse 每秒可以摄取数万行，创建特定于查询的数据模型，并在高摄取负载下保持一致的读取查询性能。为了让整个开发团队专注于平台的研发，他们与 ClickHouse Cloud 合作。他们使用官方维护的 Go 客户端 clickhouse-go 构建了一个可靠的摄取系统。他们还利用 ReplacingMergeTree 表引擎来处理重复事件。为了优化性能，他们创建了专门的表，将原始日志数据转换为针对查询优化的格式。他们还使用流行的数据转换框架 dbt 来创建审计表，以提取有关其日志数据的汇总统计信息。最后，他们使用 ClickHouse 集群作为许多其他用例的骨干，包括应用内分析。
运营影响
数量效益
相关案例.
Case Study
Energy Saving & Power Monitoring System
Recently a university in Taiwan was experiencing dramatic power usage increases due to its growing number of campus buildings and students. Aiming to analyze their power consumption and increase their power efficiency across 52 buildings, the university wanted to build a power management system utilizing web-based hardware and software. With these goals in mind, they contacted Advantech to help them develop their system and provide them with the means to save energy in the years to come.
Case Study
System 800xA at Indian Cement Plants
Chettinad Cement recognized that further efficiencies could be achieved in its cement manufacturing process. It looked to investing in comprehensive operational and control technologies to manage and derive productivity and energy efficiency gains from the assets on Line 2, their second plant in India.
Case Study
Intelligent Building Automation System and Energy Saving Solution
One of the most difficult problems facing the world is conserving energy in buildings. However, it is not easy to have a cost-effective solution to reduce energy usage in a building. One solution for saving energy is to implement an intelligent building automation system (BAS) which can be controlled according to its schedule. In Indonesia a large university with a five floor building and 22 classrooms wanted to save the amount of energy being used.
Case Study
Powering Smart Home Automation solutions with IoT for Energy conservation
Many industry leaders that offer Smart Energy Management products & solutions face challenges including:How to build a scalable platform that can automatically scale-up to on-board ‘n’ number of Smart home devicesData security, solution availability, and reliability are the other critical factors to deal withHow to create a robust common IoT platform that handles any kind of smart devicesHow to enable data management capabilities that would help in intelligent decision-making
Case Study
Commercial Building Automation Boosts Energy Efficiency
One of the challenges to building automation is the multitude of non-interoperable communications protocols that have evolved over the years. Buildings have several islands of automation. Bridging the islands of different automation without losing the considerable investment in each specialized control network is the main focus in this solution.
