GoodData, a cloud-based BI tool provider, was looking for a best-of-breed technology to build their CloudConnect platform. They needed a solution that would allow them to access data and get it into their platform efficiently. The company was previously using a REST API-based platform, which worked well for developers familiar with these processes or who had integration technologies in-house. However, this system was not user-friendly for a larger customer base. GoodData needed a solution that would allow anyone to design integrations and do data modeling work visually.

Allant Group, a marketing service provider, was dealing with an increasing amount of diverse data formats due to the surge of international data. They were using a multitude of homegrown applications for data manipulation, which was becoming increasingly difficult to manage with the growing data complexity. The expectation of faster turnaround, higher quality data, and more quality control steps in the process run on data led Allant to search for an ETL to support this. They needed a tool that could integrate with their existing software packages and legacy software, and was easy to use so that the data processing workload could shift from the programmer-side to an operational staff.

The company had been growing through competitor acquisition and had developed an in-house solution to run daily data feeds into Salesforce. However, with tens of millions of records being processed daily, the time needed to load them was getting longer, peaking at 12 hours. This meant the necessary daily updates could no longer be guaranteed. In addition, errors in loading the data into Salesforce, sometimes by as much as 50%, resulted in another day’s (lost) productivity dedicated to fixing the problems. As a result, the financial advisors relying on the data were potentially working with information that was up-to two days old. The company needed a solution that could process data quickly and deliver it, error free, into Salesforce.

The consulting firm was facing challenges with the quality of legacy data during the migration process to Workday. The poor quality of data was slowing down project delivery and taking valuable time away from the firm’s more critical objectives. The firm required a framework that would quickly identify and report data issues. Reconciliation of data in Human Resources was crucial for compliance with legal and regulatory concerns. The firm was looking for a solution that would simplify the data migration process and allow consultants to recoup time they were spending on the manual work.

MyPublisher, a leading online retailer of personalized Photo Applications, was facing a challenge in managing its retail and marketing data. The company wanted to tie this data closely with web traffic and email data to capture customer activity. However, the primary challenge for the data warehouse team was to bring together disparate sources of data in the company. Before the introduction of a data warehouse, MyPublisher was not using a data warehouse at all – ad hoc reports and a number of systems were used instead. This made it difficult for the company to gain a deeper insight into business activity, and do so on a recurring basis.

The financial enterprise uses Oracle Identity Manager (OIM) to track and monitor user access permissions associated with employee transitions and reassignments. However, with more than 6,000 different systems, updates to user permissions result in large volumes of constantly changing data. The disparity in data formats and sources coming from the systems makes feeding OIM a challenge because it’s strict about the data format it ingests. The IT security team had a choice to make: to work with a similar manual approach in-house or utilize specialist data integration software that would automate the process.

The publishing house was facing a significant challenge due to the poor quality of their customer data. The company communicates with its clients by phone, email, and direct mail. However, customer contacts had been stored in different systems and often appeared more than once. Additionally, only 30% of addresses were in the acceptable format. Because of inconsistent and duplicate data records, marketing materials were not being delivered, and some customers could not be reached by phone. Furthermore, multiple marketing packages were sometimes sent to the same address, or the same person received marketing materials at different addresses. On top of the direct costs incurred due to these data errors, the company was also experiencing indirect damages such as loss of credibility and missed opportunities.

The company was facing a challenge with the processing of phone invoices that were sent from providers in three incompatible formats. This made electronic merging and processing impossible, and values had to be manually entered into spreadsheets. The spreadsheets were updated every few months, but this caused inconsistencies in historical data. The company was spending 6 man-days per month handling this burdensome task manually. Additionally, the company’s phone bill—$44,000 monthly—had been growing by 24% per year, but it was difficult to gather adequate insight as to why. The company required a solution that would enable them to automate phone bill processing, develop an efficient calls monitoring system through bill analysis, and monitor private and company phone usage to determine the best possible call rates.

BTC Solutions, a UK-based company, was facing a demand for increased performance of their autoVHC platform due to growing amounts of data and rising customer expectations for a quicker turnaround. Dealers were having difficulties accessing fresh data, leading BTC to search for an ETL tool that could handle fast-changing data. BTC had three main requirements for the data warehouse solution: Data History, Robustness, and Flexibility. The new data warehouse needed to keep track of changes and record all modifications done to the database. It also needed to be able to handle ten times more data than their current database. Lastly, the solution needed to be user-friendly, so that BTC could define and modify the transformation procedures.

The digital insights team at EE, a UK telecommunications company, faced the challenge of integrating their business intelligence from various data sources into one place and visualizing them for a non-technical audience. The data sources included Adobe Analytics, VOC databases, clickstream records, and personalization tools. The team needed to pull data from these sources into Amazon RedShift and then push it all together into Tableau to see the story behind the data. The challenge was to handle this colossal amount of digital data with only a small team and without outsourcing the project to IT services, which would take months to set up and weeks for change requests.

Slovenska sporiteľňa, Slovakia’s biggest bank, required a simple and reliable solution for reconciliation of data from multiple Identity and Access Management systems. They needed an overarching ‘safety net’ that would pick up any violations and omissions. With such a large workforce user access requirements are constantly changing, so the task to keep the user privileges under control was a challenge. The team had given careful consideration to the all the options available to them. One route they pursued was the development of a ‘home-grown’ scripted solution, but it quickly became apparent that this approach had significant flaws. For one, it relied too heavily on one team member creating and managing the script. There was also the potential for the script to become complex and unwieldy over time.

Customology, a division of the GJI group, was facing challenges in managing and utilizing massive amounts of raw transactional and client data in myriad types and formats. As the business grew, it became increasingly difficult to manage the data without a comprehensive data management plan. The data they worked with was not standardized, which led to the development of a single common data model with different presentation layers to have adequate control over the data coming in. Their previous homegrown ETL solution comprised a mixture of off-the-shelf stacks—MSSQL with an ever-growing pool of stored procedures, shell scripts watching hot folders, and the beginnings of microservices written in PHP. While this setup could extract meaningful insights from data, it was slow, cumbersome, and difficult to scale.

Mitsubishi Motors Corporation, a multinational automaker based in Tokyo, Japan, was seeking to improve its supplier relationships on a global scale. The company wanted to offer an easy-to-use solution to interact with its suppliers. In August 2004, Mitsubishi selected Covisint’s platform to help improve the sharing of information and collaborative business processes with its global suppliers. The goal was to provide the supply base with every opportunity to advance performance and understanding of supplier-related requirements of Mitsubishi and its affiliates worldwide.

After its spin-off from Ford Motor Company in 2008, Jaguar Land Rover (JLR) needed a secure solution for global supplier connectivity and collaboration, similar in scope to its legacy company. Updating legacy systems was also necessary, but replacing existing infrastructure was time consuming and cost prohibitive. As a result, JLR faced several technology and flexibility challenges in providing its 3,000 global suppliers with secure access to critical business information. The need to focus on core business processes led JLR to Covisint.

Blue Cross Blue Shield of Michigan (BCBSM) had been relying on Covisint for seven years to manage the access of 6,500 agents to member enrollee package information as well as training and support materials. The challenge was to integrate BCBSM's business applications into a single platform and securely manage access to the applications based on a user’s profile. The agents needed to log into BCBSM’s Agent Secured Services portal, powered by Covisint, to register for training services, access information specific to BCBSM products and manage their overall book of business.

The Wellcome Trust Sanger Institute, a genomic research center, was facing challenges in managing the surge in data volume and computational analysis due to major sequencing technology advancements. The institute's diverse research community, encompassing over 2,000 scientists worldwide, required a robust IT infrastructure with large-scale, high-throughput performance. The unpredictable data growth made it difficult to scale storage sufficiently without overburdening the Institute’s existing 10-GigE network infrastructure or encroaching beyond its one petabyte per floor tile rule in the space-constrained data center. The institute developed a classic “Big Data” problem that was further exacerbated whenever new advances in sequencer technology produced more sequencing data faster than ever before.

Starz, a leading provider of premium subscription video programming, was facing challenges with its legacy HP Enterprise Virtual Arrays (EVAs) 8000s which powered the company’s complete digital media workflow system. The system was lacking scalability, performance, and reliability. The company was experiencing a dramatic increase in both capacity and performance requirements due to the expansion of its programming. The legacy storage infrastructure was causing administrative headaches and jeopardizing the whole file system. The raw performance limitations caused both transcode and encode processes to fail, leading to a lot of manual do-overs and wasted time.

TGen, a leading genomics research institute, was facing challenges with its legacy NAS system which was underpowered and unable to handle concurrent jobs without dragging performance below acceptable levels. Scaling NAS performance was expensive and time-consuming. Moreover, data growth was accelerating, making the existing infrastructure untenable. Genomics, the art of extracting understanding from an organism’s genome, is a complex and data-intensive task. The year-on-year improvements, in volume and accuracy of data being generated by gene sequencing instruments are mind-boggling. As these machines become more productive, the price for gene sequencing, assembly, and analysis drops, enabling new diagnostic methods and disease treatments. However, all this genetic data has resulted in a sea change in how to assemble them into meaningful data, so the analysis can take place.

The Laboratory of Dr. James Knowles at the Zilkha Neurogenetic Institute, Keck School of Medicine at the University of Southern California (USC) was facing a significant challenge. The lab, which is focused on understanding the genetic basis of behavior, cognition, and affect, was struggling with a legacy SAN storage server that was nearing capacity and could not keep up with data access requirements. The storage throughput was hobbled by the network and by the performance limitations of NFS. The storage bottleneck caused by slow uploads was delaying time to discovery. The lab needed a new storage solution that could serve in excess of Gigabyte per second throughput and scale to petabytes in a single name space. The Knowles Lab had a data storage performance problem. They needed to sequence 1,400 full human genomes to support their ongoing studies. This work would generate several terabytes of raw data per day that needed to be transferred, inspected, and aligned to the human genome. Their legacy storage system could only output enough data to the CPU cluster to run a single instance of their Burrows-Wheeler Aligner (BWA) under the Pegasus MPI workflow. Furthermore, they could only upload data to that system at 30-50 MB/second, nowhere near the 100MB/second peak theoretical capacity of the GbE network. This bottleneck was not only an inconvenience, but it was slowing their time to discovery.

The Institute for Computational Biomedicine (ICB) at Weill Cornell Medical College was facing a challenge as they expanded their neuroscience, epigenomics, proteomics imaging facilities and brought online more genetic sequencers. Their legacy methodology of organically adding autonomous storage pools was no longer capable of meeting the computational needs of the researchers. The challenge was transitioning from their legacy method of adding a single dedicated RAID array (at a time), into something that was scalable and could meet their storage needs for years to come. As the data ingest rates continued to raise, the facility needed to look into a more robust, scalable and sustainable storage approach.

The Center for Computational Science (CCS) at the University of Miami is one of the largest centralized, academic, cyber infrastructures in the country. It supports over 2,000 researchers, faculty, staff, and students across multiple disciplines on diverse and interdisciplinary projects requiring high performance computing (HPC) resources. The center's guiding principle is to manage the entire data lifecycle as seamlessly as possible to streamline research workflow. However, the center faced several challenges. The diverse, interdisciplinary research projects required massive compute and storage power as well as integrated data lifecycle movement and management. The explosion of next-generation sequencing had a major impact on compute and storage demands, as it’s now possible to produce more and larger datasets, which often create processing bottlenecks. The heavy I/O required to create four billion reads from one genome in a couple of days only intensifies when the data from the reads needs to be managed and analyzed. The center needed a powerful file system that was flexible enough to handle very large parallel jobs as well as smaller, shorter serial jobs.

The British Antarctic Survey (BAS) was facing a surge in data storage requirements due to its participation in a major global initiative and increased use of scientific modeling. The organization was collecting 10 times the amount of data it gathered just 10 years ago, with the rate of change increasing dramatically. This put pressure on their data collection and storage systems. In addition, BAS became part of a major global initiative, called Super Dual Auroral Radar Network (SuperDARN), which required a major storage expansion. The challenge was finding a solution that could meet the organization’s requirements for high-capacity, high-performance storage within its budget parameters.

The North German Supercomputing Alliance (HLRN) provides scientists across seven North-German states with state-of-the-art storage and compute resources to accelerate scientific breakthroughs in the fields of physics, chemistry, fluid dynamics, engineering and the environment. The scientists, many of whom come from North German universities and other scientific institutions, have combined resources and funding from their respective states and the German federal government to create a powerful, distributed supercomputer system. HLRN’s ability to drive advanced scientific research requires the highest levels of compute power as well as high-bandwidth storage capacity. Given the wide range of data-intensive applications supported by the institute, HLRN sought a Big Data solution that could deliver a significant increase in storage capacity while scaling bandwidth and performance as needed. HLRN also needed to ensure that data could be accessed easily from different geographic locations.

The National Center for Supercomputing Applications (NCSA) was facing dwindling mid-range research funding which drove the need for condo-style campus clusters across a single shared environment. This was extremely complex as it involved accommodating multiple generations of hardware, interconnected technology and storage in one unified system. Ensuring equal access to all types and any number of nodes was complicated, including determining how to handle queuing and configurations. The center sought a blend of IOPS, bandwidth, performance and efficient capacity management in an environment including multiple generations of hardware resources.

Purdue University, a global leader in research, discovery, and innovation, faced the challenge of meeting the needs of up to 1,000 researchers working on several hundred projects. This drove the decision to deploy one large, centralized data repository powered by high-performance storage. The big data variety, velocity, and volume created the need for highly versatile, scalable storage. Diverse workloads required highly flexible storage to accommodate large parallel I/O jobs and many small, random read requests. The university initially looked at the needs of several top research areas, including computational nanotechnologies, aeronautical and astronomical engineering, mechanical engineering, genomics, structural biology, as well as several large projects in the life sciences disciplines. The challenge of managing varied research needs is accommodating both very large parallel I/O jobs and millions of small, random read requests without imposing performance penalties on anyone.

Ringling College of Art and Design faced a challenge of explosive data growth caused by high-resolution, digital file-based workflows. This created a demand for future-proof storage that could scale on demand. The college wanted to use technology to support art as a tool, so that students could be creative without having to manage technology or deal with interruptions to their work. A robust, reliable and transparent storage infrastructure was required to accommodate the college’s desire to give students seamless access to all their digital assets, regardless of platform or location. The college wanted to avoid the management, access difficulty, cost and complexity of siloed storage.

Children’s Mercy Kansas City, a leading children’s hospital, operates the world’s first whole genome sequencing center in a pediatric setting. The hospital’s Center for Pediatric Genomic Medicine collaborates with various professionals to sequence and analyze rare inherited diseases. However, genetic sequencing is very compute- and data-intensive, which puts ever-increasing pressure on the Center’s IT team to deliver ample processing power and highly capable data storage to support testing based on both whole genome sequencing and whole exome sequencing. The Center had previously deployed EMC Isilon storage, which initially met their storage requirement for handling both clinical and research workflows. Over time, however, Children’s Mercy’s traditional scale-out NAS storage lacked the scalability in performance and capacity to address demanding data creation and access needs.

Czech Television, the public television broadcaster in the Czech Republic, was facing a surge in digital video content and high-end video equipment which spurred the need for fast, scalable, robust, and reliable storage. The move to 4K uncompressed DPX workflows created a major spike in storage requirements, as the size of native raw files quadrupled as well as the performance required by the storage. Despite the massive data influx, Czech TV still had to guarantee flawless real-time workflows of concurrent UHD video streams—across ingest, editing, transcoding, distributing, and archiving. They needed to accommodate multiple teams of people working on the same large files all at once to perform necessary color correction, and as well as restore many old films in bad initial condition.

The Institute of Cancer Research (ICR) in London, a world leader in identifying cancer genes, discovering cancer drugs and developing precision radiotherapy, needed a single, central storage infrastructure that would enable users to collect and analyze all types of active research data. The research data service (RDS) had to be broad enough to support eight research divisions and all types of biology, chemistry and physics-related data. It needed to be able to pull in massive amounts of data from different scientific instruments and next-generation sequencers while connecting to various research services from laptops and desktops of every flavor, to high performance supercomputers with CPUs and GPUs.

Kollins Communications, a company that provides all-in-one retail marketing solutions, was faced with an extremely tight deadline to produce over 20 4K videos in four weeks for Samsung Electronics USA. The company needed to reduce the time required to transfer files between editing stations and had a demand to stream three, uncompressed 4K video files simultaneously for a video display. To meet ongoing demands for fast project turnarounds, Kollins employs leading-edge technologies to expedite content creation workflows while keeping pace with an explosion of digital content. The company needed a high-performance storage hub that could expedite the flow of content among a mix of OS X, Windows and Linux workstations. Extremely fast performance was needed to support Autodesk Flame, a high-end 3D visual effects software running on Linux. Additionally, fast and scalable storage was necessary to support Autodesk 3ds Max as well as Adobe After Effects and Premiere Pro software that runs on OS X and Windows.