Download PDF
Storyblocks' Transformation: Redesigning Data Pipeline with Confluent
Technology Category
- Analytics & Modeling - Machine Learning
- Application Infrastructure & Middleware - Event-Driven Application
Applicable Industries
- Equipment & Machinery
- Oil & Gas
Use Cases
- Inventory Management
- Predictive Maintenance
Services
- Cloud Planning, Design & Implementation Services
- Data Science Services
The Challenge
Storyblocks, a rapidly growing media company, faced challenges with its monolithic application and synchronous REST API calls between services. As the company transitioned from a disruptor to a major industry player, it began to experience issues with the application they had built when the company was started. The issues persisted even after they had split the monolith into microservices. Developers and data engineers were unable to resolve issues quickly or iterate on their search functionality with sufficient agility. The increasing amount of data threatened to slow down productivity and time to market for new features. The initial solution, an AWS Kinesis data pipeline dumping raw data into Amazon S3, began to fail due to lack of scalability and suitability for fully decoupling services.
The Customer
Storyblocks
About The Customer
Storyblocks is a leading media company that provides an unlimited download subscription-based service for stock video and audio. It has over 100,000 customers in the television and video production industry, including NBC and MTV, plus tens of thousands of hobbyists looking to enhance their video projects and productions. Storyblocks’ subscribers can download an unlimited number of clips from a vast and rapidly growing library of stock video, production music, motion backgrounds, sound effects, special effects, and more. As the company transitioned from a disruptor to a major industry player, it began to experience issues with its data pipeline, which led to the partnership with Confluent.
The Solution
To manage these challenges, Storyblocks needed a new solution that could form the backbone of an entirely new data pipeline. They began to use Apache Kafka® and did a proof of concept in conjunction with using a schema registry from Aiven. However, they weren't getting the support for Kafka that they needed from Aiven, so they started to use Confluent Cloud, a fully managed cloud service for Kafka. They also began to spread the use of Kafka as an event bus for more streaming applications and machine learning (ML) features. With this Confluent-backed data pipeline in place, the Storyblocks team could begin to affect a true digital transformation both internally and externally. Instead of implementing a queue for inter-services communication, the team just puts it on the pipeline where it’s stored forever. This sort of infinite storage is powerful for two reasons: Events can be replayed on demand with powerful in-built schema validation and analysts can look at historical data when answering various questions for the business.
Operational Impact
Quantitative Benefit
Related Case Studies.
Case Study
Smart Water Filtration Systems
Before working with Ayla Networks, Ozner was already using cloud connectivity to identify and solve water-filtration system malfunctions as well as to monitor filter cartridges for replacements.But, in June 2015, Ozner executives talked with Ayla about how the company might further improve its water systems with IoT technology. They liked what they heard from Ayla, but the executives needed to be sure that Ayla’s Agile IoT Platform provided the security and reliability Ozner required.
Case Study
IoT enabled Fleet Management with MindSphere
In view of growing competition, Gämmerler had a strong need to remain competitive via process optimization, reliability and gentle handling of printed products, even at highest press speeds. In addition, a digitalization initiative also included developing a key differentiation via data-driven services offers.
Case Study
Taking Oil and Gas Exploration to the Next Level
DownUnder GeoSolutions (DUG) wanted to increase computing performance by 5 to 10 times to improve seismic processing. The solution must build on current architecture software investments without sacrificing existing software and scale computing without scaling IT infrastructure costs.
Case Study
Predictive Maintenance for Industrial Chillers
For global leaders in the industrial chiller manufacturing, reliability of the entire production process is of the utmost importance. Chillers are refrigeration systems that produce ice water to provide cooling for a process or industrial application. One of those leaders sought a way to respond to asset performance issues, even before they occur. The intelligence to guarantee maximum reliability of cooling devices is embedded (pre-alarming). A pre-alarming phase means that the cooling device still works, but symptoms may appear, telling manufacturers that a failure is likely to occur in the near future. Chillers who are not internet connected at that moment, provide little insight in this pre-alarming phase.
Case Study
Premium Appliance Producer Innovates with Internet of Everything
Sub-Zero faced the largest product launch in the company’s history:It wanted to launch 60 new products as scheduled while simultaneously opening a new “greenfield” production facility, yet still adhering to stringent quality requirements and manage issues from new supply-chain partners. A the same time, it wanted to increase staff productivity time and collaboration while reducing travel and costs.
