Download PDF
Building Karmic’s Data Infrastructure
Technology Category
- Platform as a Service (PaaS) - Data Management Platforms
- Analytics & Modeling - Big Data Analytics
- Application Infrastructure & Middleware - Data Exchange & Integration
Applicable Industries
- Finance & Insurance
- Retail
Applicable Functions
- Business Operation
- Sales & Marketing
Services
- Cloud Planning, Design & Implementation Services
- System Integration
- Software Design & Engineering Services
The Challenge
One of the biggest challenges Yang faced was in choosing and leveraging third-party tools. “How do you weigh vendors when you don’t really know what your needs are, and how those needs will change over time?” For a data warehousing solution, Yang ended up siding with Amazon Redshift, as it met all of his needs for storage, speed, and security. But to get data into Redshift, he needed an ETL solution to match it. Stitch Data was the first provider that caught his eye and that he later implemented - but it wasn’t long before his team outgrew it. “Plug-and-play tools like Stitch work great for straightforward workflows, but we needed more customization and access under the hood to not only comply with our security requirements, but also stay competitive with companies that have more developed data infrastructures” said Yang. “The fact that we didn’t have control over transformations forced us to consider other, more comprehensive options.”
About The Customer
Karmic Labs delivers the future of expense management with a platform for employers, banks, and retailers to manage debit card and fund distribution amongst their customers and members. At a time of momentous growth for Karmic, their ability to build a strong, scalable data infrastructure became increasingly critical. Echoing what Airbnb refers to as “Data Democratization,” Karmic’s Data Science Product Manager, Yang Wang, explains that, “the more accessible data is, the faster we can iterate, and the further we can get in the game.” Yang joined Karmic when that data infrastructure was largely nonexistent, but it soon became one of this team’s highest priorities to fill that gap. “The second you build a software, you want to know what works and what doesn’t. We desperately needed more high-level analysis,” he said.
The Solution
In his research for other options, Yang came across Astronomer’s Managed Apache Airflow module. While he hadn’t heard of Apache Airflow, his research proved that the open-source software had a strong community behind it and was a good fit for the job. “There were no other managed Airflow services out there, and we didn’t have the DevOps resources to run it ourselves” he said. Not long thereafter, he migrated his workflows to our Cloud platform. Karmic now uses Apache Airflow on Astronomer to sync their application database (Postgres) to their data warehouse (Amazon Redshift). Directly on our platform, Yang created a dynamic workflow that both automates that process and complies with Karmic’s security requirements. Due to the sensitive nature of their business, Karmic requires a whitelisted IP and SSH for some database connections. Since Astronomer’s Cloud Airflow service runs in a serverless architecture where each task instance runs in a separate container, there was no immediately obvious place to store the key file needed for an SSH connection (in this case, for Postgres). But by working with Astronomer, Karmic was able to configure a custom Airflow hook that opens an SSH tunnel in each task instance that requires access to that database - and closes that tunnel once the task finishes.
Operational Impact
Related Case Studies.
Case Study
Improving Production Line Efficiency with Ethernet Micro RTU Controller
Moxa was asked to provide a connectivity solution for one of the world's leading cosmetics companies. This multinational corporation, with retail presence in 130 countries, 23 global braches, and over 66,000 employees, sought to improve the efficiency of their production process by migrating from manual monitoring to an automatic productivity monitoring system. The production line was being monitored by ABB Real-TPI, a factory information system that offers data collection and analysis to improve plant efficiency. Due to software limitations, the customer needed an OPC server and a corresponding I/O solution to collect data from additional sensor devices for the Real-TPI system. The goal is to enable the factory information system to more thoroughly collect data from every corner of the production line. This will improve its ability to measure Overall Equipment Effectiveness (OEE) and translate into increased production efficiencies. System Requirements • Instant status updates while still consuming minimal bandwidth to relieve strain on limited factory networks • Interoperable with ABB Real-TPI • Small form factor appropriate for deployment where space is scarce • Remote software management and configuration to simplify operations
Case Study
How Sirqul’s IoT Platform is Crafting Carrefour’s New In-Store Experiences
Carrefour Taiwan’s goal is to be completely digital by end of 2018. Out-dated manual methods for analysis and assumptions limited Carrefour’s ability to change the customer experience and were void of real-time decision-making capabilities. Rather than relying solely on sales data, assumptions, and disparate systems, Carrefour Taiwan’s CEO led an initiative to find a connected IoT solution that could give the team the ability to make real-time changes and more informed decisions. Prior to implementing, Carrefour struggled to address their conversion rates and did not have the proper insights into the customer decision-making process nor how to make an immediate impact without losing customer confidence.
Case Study
Digital Retail Security Solutions
Sennco wanted to help its retail customers increase sales and profits by developing an innovative alarm system as opposed to conventional connected alarms that are permanently tethered to display products. These traditional security systems were cumbersome and intrusive to the customer shopping experience. Additionally, they provided no useful data or analytics.
Case Study
Real-time In-vehicle Monitoring
The telematic solution provides this vital premium-adjusting information. The solution also helps detect and deter vehicle or trailer theft – as soon as a theft occurs, monitoring personnel can alert the appropriate authorities, providing an exact location.“With more and more insurance companies and major fleet operators interested in monitoring driver behaviour on the grounds of road safety, efficient logistics and costs, the market for this type of device and associated e-business services is growing rapidly within Italy and the rest of Europe,” says Franco.“The insurance companies are especially interested in the pay-per-use and pay-as-you-drive applications while other organisations employ the technology for road user charging.”“One million vehicles in Italy currently carry such devices and forecasts indicate that the European market will increase tenfold by 2014.However, for our technology to work effectively, we needed a highly reliable wireless data network to carry the information between the vehicles and monitoring stations.”
Case Study
Ensures Cold Milk in Your Supermarket
As of 2014, AK-Centralen has over 1,500 Danish supermarkets equipped, and utilizes 16 operators, and is open 24 hours a day, 365 days a year. AK-Centralen needed the ability to monitor the cooling alarms from around the country, 24 hours a day, 365 days a year. Each and every time the door to a milk cooler or a freezer does not close properly, an alarm goes off on a computer screen in a control building in southwestern Odense. This type of alarm will go off approximately 140,000 times per year, equating to roughly 400 alarms in a 24-hour period. Should an alarm go off, then there is only a limited amount of time to act before dairy products or frozen pizza must be disposed of, and this type of waste can quickly start to cost a supermarket a great deal of money.
