Test and Learn (T&L) is a practice where ideas or hypothesis are tested in a small sample, such as a few branches or a representative portion of a population. This practice mitigates the potential loss if the new concept is not as strong as the existing campaign (the control comparison).
Why is T&L a mentality? Testing costs money. It will raise CPMs and CPAs in the short-term. Fact-based decision making cannot be made without the willingness to invest in a T&L mindset. The benefit is that picking the best creative or offer — which drives incremental response — far outweighs the increase in testing cost. Testing an increase in the offer premium versus a control gives an accurate ROI calculation based on facts—not instinct. This lift can then be used in your strategic planning.
Direct marketing creates an optimal setting for T&L as it is the perfect example of a cause-and-effect relationship. A/B splits, test/control, and current/new are all examples of T&L. Inferences, both good and bad, can be made from a sampling which helps mitigate costs. Rigor must be maintained in keeping sample sizes large enough or tests must be repeated and aggregated over time to achieve the same level of confidence.
The Japanese auto industry evolved from making the cheapest cars to making the most reliable cars by instituting a process called Kanban. Kanban became a mindset of continual process improvement. Anyone on an assembly line could pull a cord to stop the process to eliminate defects. Without this mindset, workers would not have felt empowered to pull the cord. The same
principles can be modified and applied to any industry. “What did we learn?” should be the deliverable!
For the financial industry, digital technology has enabled fast and affordable sampling. Why not test what creative concept is most appealing before spending on a rollout campaign? A $300 offer typically outpulls a $200 offer, but are you attracting offer-driven customers or ones with greater growth potential?
Without testing, an institution will never know for sure, which leads to conjecture—not direct inferences. Optimize your ROMI by coordinating test campaigns and measurement.