Improving Test & Learn Process at a Telecom Operator

Technology Category

Analytics & Modeling - Predictive Analytics
Analytics & Modeling - Real Time Analytics
Platform as a Service (PaaS) - Data Management Platforms

Applicable Industries

Retail
Telecommunications

Applicable Functions

Business Operation
Sales & Marketing

Use Cases

Predictive Maintenance

Services

Cloud Planning, Design & Implementation Services
Data Science Services
System Integration

The Challenge

Executives at a telecommunications company were already convinced that test vs. control analytics were the best way to make decisions. However, due to their manual process and highly complicated and biased analyses, they often struggled to accurately evaluate all of their initiatives, and were rarely able to derive the maximum learnings from each one. Further, their test vs. control analyses were limited to initiatives where they conducted a purposeful holdout experiment; for all other initiatives, they relied on regression-based approaches, which could not isolate the true incremental ROI of that action. As one example, the organization struggled to identify the cause-and-effect relationship between the rate increase at the end of a promotion and churn. Without understanding how rate increases caused churn, executives could not confidently change their post-promotion pricing strategy to maximize profitability.

About The Customer

APT, a Mastercard Company, is a leading cloud-based analytics software company that enables organizations to rapidly and precisely measure cause-and-effect relationships between business initiatives and outcomes to generate economic value. Their intuitive and proprietary Test & Learn® software utilizes sophisticated algorithms to analyze large amounts of data, enabling business leaders to conduct experiments and allowing them to make optimal decisions and implement business initiatives at scale. APT also offers products that support decision-making for specific business needs including transaction analysis, space planning, promotion design, category management, and location selection. APT’s client portfolio features some of the world’s best-known brands, including Walmart, Starbucks, Coca-Cola, Victoria’s Secret, American Family, Choice Hotels International, SUBWAY, TD Bank, T-Mobile, and others. APT has offices in Washington, D.C., San Francisco, London, Bentonville, Taipei, Tokyo, Sydney, and Chicago.

The Solution

Test & Learn helped the company make the most rapid, accurate decisions possible by vastly improving the quality, speed, and scope of their analyses. The use cases for experimentation now span the breadth of the business including direct marketing, media, product upgrades, promotion expiration, on-boarding, call center, and more. The client now uses Test & Learn before launching any major program. Not only does it allow them to optimize and target their programs, but it also reduces the risk of trying new ideas. Rapid, in-market experimentation enables executives to confidently invest in initiatives that are empirically proven to maximize customer satisfaction and profits.

Operational Impact

Test & Learn improved the quality, speed, and scope of analyses, enabling rapid and accurate decision-making.
The use cases for experimentation expanded to include direct marketing, media, product upgrades, promotion expiration, on-boarding, call center, and more.
The client now uses Test & Learn before launching any major program, optimizing and targeting their programs while reducing the risk of trying new ideas.

Quantitative Benefit

APT’s software measured an 8% increase in new customer acquisition from direct mail.
A 5-week online display ad heavy-up test showed a 4.2% statistically significant lift in activations.
The cost of acquiring a subscriber was cut by 40% through digital media strategy.