下载PDF
Instabug's Successful Migration to ClickHouse for Enhanced APM Performance
技术
- 应用基础设施与中间件 - 数据可视化
- 应用基础设施与中间件 - 数据库管理和存储
适用行业
- 水泥
- 建筑与基础设施
适用功能
- 产品研发
- 质量保证
用例
- 基础设施检查
- 时间敏感网络
服务
- 数据科学服务
- 测试与认证
挑战
Instabug 是一个 SDK,它提供了一套用于在整个移动应用程序开发生命周期中监控和调试性能问题的产品,它面临着性能指标的重大挑战。这些指标严重依赖于频繁且大量的事件,这给接收和有效存储这些事件带来了挑战。此外,性能事件的原始格式对用户来说没有用,需要大量的业务逻辑来进行查询和数据可视化。 Instabug 的后端规模庞大,API 平均每分钟约有 200 万个请求,每天有 TB 的数据进出其服务。在构建应用程序性能监控 (APM) 时,他们意识到这将是他们数据规模最大的产品。他们每天以每分钟约 200 万个事件的速度存储约 30 亿个事件。他们还必须提供复杂的数据可视化服务,这些可视化很大程度上依赖于过滤大量数据并快速计算复杂的聚合以获得用户体验。最初,他们像其他产品一样设计 APM,但遇到了 Elasticsearch 的性能问题,尤其是读取方面,而且写入速度也不够快,无法处理其负载。
关于客户
Instabug 是一个 SDK,提供了一套产品,用于在整个移动应用程序开发生命周期中监控、优先级排序和调试性能和稳定性问题。 Instabug SDK 提供崩溃报告和应用程序性能监控 (APM),允许用户监控应用程序性能的各个方面,例如崩溃、处理异常、网络故障、UI 挂起、启动和屏幕加载延迟,以及设置自定义跟踪的能力监视关键代码部分。 Instabug 还通过规则和警报引擎提供工作流程自动化,该引擎与 Jira、Opsgenie、Zendesk、Slack、Trello 等其他项目和事件管理工具集成。他们的后端规模很大,API 平均每分钟约有 200 万个请求,每天有 TB 的数据进出其服务。
解决方案
Instabug 决定尝试不同的数据存储,以寻找 APM 的 Elasticsearch 替代方案,并发现了 ClickHouse。在测试 ClickHouse 并发现其读写性能更好后,他们决定迁移到 ClickHouse。然而,他们无法冻结产品工作以迁移到新的数据存储,并且没有操作 ClickHouse 的经验,因此他们决定使代码和基础设施具有足够的通用性,以允许增量部署和实验。他们重构了代码以抽象处理数据存储,允许它根据一些动态提供的配置读取和写入不同的数据存储。这使他们能够将所有新数据写入 ClickHouse 和 Elasticsearch 以最大程度地减少迁移工作,让特定用户从 ClickHouse 写入/读取数据,而所有其他用户从 Elasticsearch 写入/读取数据,并为 Elasticsearch 和 ClickHouse 添加新功能。迁移到 ClickHouse 大约花了 5 个月的时间,他们在迁移过程中构建的可配置和多功能基础设施仍在使用中,允许他们运行多个集群并根据需要将不同的事件指标数据托管到不同的数据库或集群中。
运营影响
数量效益
相关案例.
Case Study
System 800xA at Indian Cement Plants
Chettinad Cement recognized that further efficiencies could be achieved in its cement manufacturing process. It looked to investing in comprehensive operational and control technologies to manage and derive productivity and energy efficiency gains from the assets on Line 2, their second plant in India.
Case Study
IoT System for Tunnel Construction
The Zenitaka Corporation ('Zenitaka') has two major business areas: its architectural business focuses on structures such as government buildings, office buildings, and commercial facilities, while its civil engineering business is targeted at structures such as tunnels, bridges and dams. Within these areas, there presented two issues that have always persisted in regard to the construction of mountain tunnels. These issues are 'improving safety" and "reducing energy consumption". Mountain tunnels construction requires a massive amount of electricity. This is because there are many kinds of electrical equipment being used day and night, including construction machinery, construction lighting, and ventilating fan. Despite this, the amount of power consumption is generally not tightly managed. In many cases, the exact amount of power consumption is only ascertained when the bill from the power company becomes available. Sometimes, corporations install demand-monitoring equipment to help curb the maximum power demanded. However, even in these cases, the devices only allow the total volume of power consumption to be ascertained, or they may issue warnings to prevent the contracted volume of power from being exceeded. In order to tackle the issue of reducing power consumption, it was first necessary to obtain an accurate breakdown of how much power was being used in each particular area. In other words, we needed to be able to visualize the amount of power being consumed. Safety, was also not being managed very rigorously. Even now, tunnel construction sites often use a 'name label' system for managing entry into the work site. Specifically, red labels with white reverse sides that bear the workers' names on both sides are displayed at the tunnel work site entrance. The workers themselves then flip the name label to the appropriate side when entering or exiting from the work site to indicate whether or not they are working inside the tunnel at any given time. If a worker forgets to flip his or her name label when entering or exiting from the tunnel, management cannot be performed effectively. In order to tackle the challenges mentioned above, Zenitaka decided to build a system that could improve the safety of tunnel construction as well as reduce the amount of power consumed. In other words, this new system would facilitate a clear picture of which workers were working in each location at the mountain tunnel construction site, as well as which processes were being carried out at those respective locations at any given time. The system would maintain the safety of all workers while also carefully controlling the electrical equipment to reduce unnecessary power consumption. Having decided on the concept, our next concern was whether there existed any kind of robust hardware that would not break down at the construction work site, that could move freely in response to changes in the working environment, and that could accurately detect workers and vehicles using radio frequency identification (RFID). Given that this system would involve many components that were new to Zenitaka, we decided to enlist the cooperation of E.I.Sol Co., Ltd. ('E.I.Sol') as our joint development partner, as they had provided us with a highly practical proposal.
Case Study
Splunk Partnership Ties Together Big Data & IoT Services
Splunk was faced with the need to meet emerging customer demands for interfacing IoT projects to its suite of services. The company required an IoT partner that would be able to easily and quickly integrate with its Splunk Enterprise platform, rather than allocating development resources and time to building out an IoT interface and application platform.
Case Study
Bridge monitoring in Hamburg Port
Kattwyk Bridge is used for both rail and road transport, and it has played an important role in the Port of Hamburg since 1973. However, the increasing pressure from traffic requires a monitoring solution. The goal of the project is to assess in real-time the bridge's status and dynamic responses to traffic and lift processes.
Case Study
Bellas Landscaping
Leading landscaping firm serving central Illinois streamlines operations with Samsara’s real-time fleet tracking solution: • 30+ vehicle fleet includes International Terrastar dump trucks and flatbeds, medium- and light-duty pickups from Ford and Chevrolet. Winter fleet includes of snow plows and salters.