下载PDF
High-Speed Content Distribution Analytics for Disney+ with ClickHouse
技术
- 基础设施即服务 (IaaS) - 云计算
- 基础设施即服务 (IaaS) - 云中间件与微服务
适用行业
- 建筑物
- 建筑与基础设施
适用功能
- 物流运输
用例
- 最后一英里交付
- 时间敏感网络
服务
- 系统集成
挑战
Disney+ 的可观察性团队面临着处理和分析其内容分发系统的访问日志的挑战。该团队必须处理 Disney+ 用户生成的大量数据,这需要高度扩展的分布式数据库系统。现有的解决方案,例如Elasticsearch、Hadoop和Flink,无法有效处理海量数据。例如,Elasticsearch 需要大量的重新平衡并使用 Java 虚拟机,从而添加了不必要的虚拟化层。由于数据的大小,该团队正在努力获取所有日志。
关于客户
本案例研究中的客户是 Disney+ 内的 Observability 团队。该团队负责管理支持 Disney+ 内容分发系统的外围系统。他们使用视频数据的访问日志来帮助识别延迟等任何问题。该团队必须处理 Disney+ 用户生成的大量数据,这需要高度扩展的分布式数据库系统。最初,由于数据的大小,该团队在摄取所有日志方面遇到了困难,但在选择 ClickHouse 后,他们已经能够高效地处理和分析日志。
解决方案
Disney+ 的 Observability 团队选择 ClickHouse 作为处理和分析访问日志的数据管道。由于 ClickHouse 的简单性、单二进制设置和轻量级架构,因此选择了 ClickHouse。当前集群由 20 个节点组成,具有 2X 复制功能,提供 160 TB 存储和 2.5 TB RAM。此设置允许团队每秒写入 300 万行 CDN 访问日志,每秒读取 20 亿行 CDN 访问日志。该团队计划升级到具有 32 个节点、每个节点 154 TB、112 个核心的更大集群,这将显着提高他们的数据管理能力。 ClickHouse 的设置就像 HTTP 服务器一样,使其灵活且易于使用。该团队还建议使用 JSON 或键等于值等格式的非结构化日志,以使系统更加灵活。
运营影响
数量效益
相关案例.
Case Study
Energy Saving & Power Monitoring System
Recently a university in Taiwan was experiencing dramatic power usage increases due to its growing number of campus buildings and students. Aiming to analyze their power consumption and increase their power efficiency across 52 buildings, the university wanted to build a power management system utilizing web-based hardware and software. With these goals in mind, they contacted Advantech to help them develop their system and provide them with the means to save energy in the years to come.
Case Study
IoT System for Tunnel Construction
The Zenitaka Corporation ('Zenitaka') has two major business areas: its architectural business focuses on structures such as government buildings, office buildings, and commercial facilities, while its civil engineering business is targeted at structures such as tunnels, bridges and dams. Within these areas, there presented two issues that have always persisted in regard to the construction of mountain tunnels. These issues are 'improving safety" and "reducing energy consumption". Mountain tunnels construction requires a massive amount of electricity. This is because there are many kinds of electrical equipment being used day and night, including construction machinery, construction lighting, and ventilating fan. Despite this, the amount of power consumption is generally not tightly managed. In many cases, the exact amount of power consumption is only ascertained when the bill from the power company becomes available. Sometimes, corporations install demand-monitoring equipment to help curb the maximum power demanded. However, even in these cases, the devices only allow the total volume of power consumption to be ascertained, or they may issue warnings to prevent the contracted volume of power from being exceeded. In order to tackle the issue of reducing power consumption, it was first necessary to obtain an accurate breakdown of how much power was being used in each particular area. In other words, we needed to be able to visualize the amount of power being consumed. Safety, was also not being managed very rigorously. Even now, tunnel construction sites often use a 'name label' system for managing entry into the work site. Specifically, red labels with white reverse sides that bear the workers' names on both sides are displayed at the tunnel work site entrance. The workers themselves then flip the name label to the appropriate side when entering or exiting from the work site to indicate whether or not they are working inside the tunnel at any given time. If a worker forgets to flip his or her name label when entering or exiting from the tunnel, management cannot be performed effectively. In order to tackle the challenges mentioned above, Zenitaka decided to build a system that could improve the safety of tunnel construction as well as reduce the amount of power consumed. In other words, this new system would facilitate a clear picture of which workers were working in each location at the mountain tunnel construction site, as well as which processes were being carried out at those respective locations at any given time. The system would maintain the safety of all workers while also carefully controlling the electrical equipment to reduce unnecessary power consumption. Having decided on the concept, our next concern was whether there existed any kind of robust hardware that would not break down at the construction work site, that could move freely in response to changes in the working environment, and that could accurately detect workers and vehicles using radio frequency identification (RFID). Given that this system would involve many components that were new to Zenitaka, we decided to enlist the cooperation of E.I.Sol Co., Ltd. ('E.I.Sol') as our joint development partner, as they had provided us with a highly practical proposal.
Case Study
Intelligent Building Automation System and Energy Saving Solution
One of the most difficult problems facing the world is conserving energy in buildings. However, it is not easy to have a cost-effective solution to reduce energy usage in a building. One solution for saving energy is to implement an intelligent building automation system (BAS) which can be controlled according to its schedule. In Indonesia a large university with a five floor building and 22 classrooms wanted to save the amount of energy being used.
Case Study
Powering Smart Home Automation solutions with IoT for Energy conservation
Many industry leaders that offer Smart Energy Management products & solutions face challenges including:How to build a scalable platform that can automatically scale-up to on-board ‘n’ number of Smart home devicesData security, solution availability, and reliability are the other critical factors to deal withHow to create a robust common IoT platform that handles any kind of smart devicesHow to enable data management capabilities that would help in intelligent decision-making
Case Study
Splunk Partnership Ties Together Big Data & IoT Services
Splunk was faced with the need to meet emerging customer demands for interfacing IoT projects to its suite of services. The company required an IoT partner that would be able to easily and quickly integrate with its Splunk Enterprise platform, rather than allocating development resources and time to building out an IoT interface and application platform.