Download PDF
Ascend.io > Case Studies > Lumiata Case Study: Intelligent Pipeline Orchestration & Automation with Ascend
Ascend.io Logo

Lumiata Case Study: Intelligent Pipeline Orchestration & Automation with Ascend

Technology Category
  • Analytics & Modeling - Big Data Analytics
  • Analytics & Modeling - Data-as-a-Service
  • Platform as a Service (PaaS) - Data Management Platforms
Applicable Industries
  • Healthcare & Hospitals
Applicable Functions
  • Product Research & Development
  • Quality Assurance
Use Cases
  • Predictive Maintenance
Services
  • Data Science Services
  • System Integration
The Challenge
Lumiata, a company focused on making healthcare smarter, was facing challenges with their data pipelines. They were using a mix of Apache Airflow, Apache Spark, Python and over 100,000 lines of custom code to create a Curated Table, which is the basis for their Data Science team to develop the Lumiata Insights. However, with increasing volumes of client data and faster SLA requirements, the process began to strain. Onboarding each new client required bespoke development and the over-extended data engineering team was responsible not only for this development, but also for maintaining and monitoring the pipelines, as well as the health and performance of the underlying Apache Spark jobs. The data science team required a certain amount of experimentation and iteration to develop the Lumiata Insights, but were completely dependent on data engineering to provide necessary adjustments to the Curated Table. This whole process would take six weeks or more, and induced a heavy maintenance burden to keep everything running. As the company looked to scale to take on more clients with their existing team, they needed a new approach.
About The Customer
Lumiata is a company that is focused on making healthcare smarter. They work with major health providers to take data -- ranging from electronic medical records, claims, lab work, physician notes, and more -- and transform it into specific recommendations to improve patient outcomes and reduce delivery costs, known as Lumiata Insights. Their business is built on delivering this advanced data product. Given the foundational importance of data, Lumiata built a world-class team of highly sophisticated data engineers and data scientists. These teams leveraged modern technologies to build the data pipelines that would fuel their Insights product, including Airflow, Scala, and Spark. However, these tools also came with significant engineering overhead and the team’s productivity was being eroded as they battled with long iteration cycles and difficult maintenance.
The Solution
Lumiata opted to give Ascend’s Autonomous Dataflow Service (running on Amazon Web Services) a try. Within three weeks, they were able to migrate all their existing pipelines to the Ascend Service. Creating the Curated Table now involved only 2,000 lines of reusable code, reducing the total codebase to maintain by 98%. Ascend’s declarative programming model ensures the resulting code is focused solely on the data and logic development (not on executing tasks or infrastructure management), so both data engineering and data science teams can collaborate and iterate together clearly and quickly, without risk. Additionally, the automation that the Ascend Service provides fundamentally changed the scale with which they could work with data. When onboarding new datasets, the data engineering team can now ingest and work with all of the source columns with no extra code or complexity. And, since Ascend automatically converts all data to Parquet upon ingestion, they’ve been able to fully eliminate that time-consuming processing stage.
Operational Impact
  • Working from the concise, declarative codebase has made it faster to build new pipelines and relieves much of the maintenance burden for existing pipelines.
  • The data science team now has full visibility into the context of the data and resulting Curated Table. They can clearly trace operations done on data fields, and even pull in new fields or adjust the logic directly.
  • The ability to self-serve updates and experiment rapidly has allowed this team to become Citizen Data Engineers -- giving them more accurate models more quickly and relieving pressure from the central data engineering team.
Quantitative Benefit
  • Reduced the lines of code powering these pipelines by 98%
  • New pipeline creation was 7x faster
  • Process to go from raw data to Lumiata Insight is now 7x faster

Related Case Studies.

Contact us

Let's talk!

* Required
* Required
* Required
* Invalid email address
By submitting this form, you agree that IoT ONE may contact you with insights and marketing messaging.
No thanks, I don't want to receive any marketing emails from IoT ONE.
Submit

Thank you for your message!
We will contact you soon.