Delhivery's Journey: From Data Chaos to Organized Data Catalog with Atlan

• Analytics & Modeling - Data Mining
• Automation & Control - Human Machine Interface (HMI)

• Buildings
• Cement

• Procurement
• Product Research & Development

• Immersive Analytics
• Time Sensitive Networking

• System Integration
• Training

Delhivery, India’s leading fulfillment platform for digital commerce, handles a massive amount of data, over 1.2 TB per day, from its vast network of IoT devices. The company fulfills a million packages a day, 365 days a year, through its extensive network of automated sort centers, fulfillment centers, hubs, direct delivery centers, partner centers, vehicles, and team members. With nearly 60,000 data events and messages per second, data discovery and organization became a significant challenge. The data is organized and processed by hundreds of microservices, which means that ownership over the data is distributed across different teams. As the company grew, the scale and complexity of its data grew even faster. Teams started building their own microservices, motivated by a desire to make data-driven decisions. However, finding and understanding the data became a significant issue. The onboarding process for new team members grew from 1-2 months to 3-4 months due to the growing complexity of the data. By 2019, Delhivery realized it desperately needed a data cataloguing solution.

Delhivery is India’s leading fulfillment platform for digital commerce. The company fulfills a million packages a day, 365 days a year, through its extensive network of automated sort centers, fulfillment centers, hubs, direct delivery centers, partner centers, vehicles, and team members. The company handles over 1.2 TB of data per day from its vast network of IoT devices. The data is organized and processed by hundreds of microservices, which means that ownership over the data is distributed across different teams. As the company grew, the scale and complexity of its data grew even faster. The onboarding process for new team members grew from 1-2 months to 3-4 months due to the growing complexity of the data.

Delhivery started its journey with a data catalog in 2019. The company evaluated traditional enterprise-focused data catalogs like Alation, Collibra, and Waterline, built its own catalog with Atlas and Amundsen, and later adopted the modern SaaS unified data workspace, Atlan. The company started by evaluating commercial products based on features and total cost of ownership. However, none of the products met their requirements. The company then decided to build its own data catalog using Apache Atlas as the backbone of its metadata management framework. However, Atlas was too technical for non-technical team members, so they brought in Amundsen, a data discovery service, as the search interface for their Atlas backend. They also developed new features on top of Atlas and Amundsen to meet their specific needs. However, the user experience was not satisfactory, and the catalog did not integrate well into the daily work of the team members. After seven months of development, the company decided to look for a commercial solution again. They found Atlan, a modern data workspace that excelled at data cataloguing and discovery. Atlan had all the features they were looking for, and the user interface and experience were extremely intuitive. The company decided to go with Atlan, realizing that buying a data cataloguing product would be a lower total cost of ownership than continuing to build their own.

• The implementation of Atlan as a data cataloguing solution has significantly improved the data discovery and organization at Delhivery. The user interface and experience of Atlan are extremely intuitive, making it user-friendly for both technical and non-technical users. Its collaboration-first approach, such as the ability to share data assets as links or its integrations with Slack, has been very promising. The open API backbone of Atlan has given Delhivery the ability to build on top of Atlan and customize the product for their end users. The continuous workshops conducted by Atlan during the Proof of Concept helped onboard new team members, dive deep into advanced features, and maintain traction as team members started documenting data and filling the new metadata stores. The implementation of Atlan has helped Delhivery make their data self-service, discoverable, and clear for their data-driven teams.