Databricks' Transition from Data Silos to a Unified Data Lakehouse

• Analytics & Modeling - Machine Learning
• Analytics & Modeling - Predictive Analytics

• Education
• Equipment & Machinery

• Sales & Marketing
• Warehouse & Inventory Management

• Picking, Sorting & Positioning
• Time Sensitive Networking

• Data Science Services

Chris Klaczynski, a Marketing Analytics Manager at Databricks, was tasked with supporting the primary marketing objectives of driving pipeline generation, growing the database, and improving ROI. However, as Databricks rapidly expanded, the need for centralized and documented data became more and more apparent. Data silos were appearing around the business, including on Chris’ marketing team, where data was stored in its own data warehouse. It was critical that Chris’ newly-built dashboards were supplied with trustworthy, timely data for marketing operations to keep running smoothly. However, without dedicated engineering resources, and in the face of a rapidly expanding marketing team, scaling with demand became next to impossible. Databricks faced a number of challenges with their traditional data warehouse, including issues with their Salesforce and Marketo pipelines, issues appending data natively to existing tables, and schema changes that were always breaking pipelines, resulting in outages and stale, untrustworthy data.

Databricks is a business that prides itself on helping data teams solve the world’s biggest challenges. It was founded in 2013 by the original creators of Apache Spark, Delta Lake and MLflow. Built on a modern Lakehouse architecture in the cloud, Databricks combines the best of data warehouses and data lakes to offer an open and unified platform for data and AI. The company has rapidly expanded, leading to the need for centralized and documented data. However, data silos were appearing around the business, including on the marketing team, where data was stored in its own data warehouse.

Chris began looking for a low-code, turnkey solution that would give his team the reliable pipelines they needed. He decided on a three-point strategy: Get out of the data engineering ‘jungle’ and into insights and predictive analytics, hire analysts and Digital SMEs, not Data Engineers and DevOps, and be self-sufficient with data pipelines and utilize Delta and AutoML. Fivetran immediately stood out as a solution to Chris’ data pipeline needs. In trialing the product, the setup was quick and simple. Chris’s team now uses Fivetran to bring in data from all of their core marketing source systems: Marketo, Salesforce, Facebook Ads and Google Analytics. Once in their lakehouse, Chris’ team joins the data with other sources from Product and Central Teams, transforming it in a meaningful way to run Data Science, machine learning, and generate vital Tableau dashboards for analysis.

• By bringing on Fivetran to handle data ingestion, Chris freed up time to build a full suite of Tableau dashboards that answer the most common questions that his marketing team face, saving his team time and allowing them to focus on adding value with three Data Science and machine learning projects. The downstream impacts of Chris’ decision making are already being felt internally at Databricks. Customer Service and Customer Success have seen how easy the process has been and are asking for more use cases and connectors to be switched on. Access to reliable data has been critical for marketing account managers, who are now more accountable for the forecasts they are making. Increased visibility through sophisticated dashboards has made it easier for teams to track and rapidly act on campaign performance.

• Transitioned from a team of one to a team of five within two years
• Fivetran saves Databricks over 40 hours a month in engineering time – the equivalent of a part-time Data Engineer
• Supports a rapidly growing data team of analysts and leaders who deliver crucial insights to over 90 marketers