Test and learn: a virtuous cycle

Continuous evaluation and improvement of models

« Back to Blog

Turning your predictive analytics practices into a thriving model factory requires closing the loop from model creation, to performance monitoring, to problem and opportunity identification and subsequent model improvement, in a formal and disciplined way. This includes continuous measurement of each model, and creating a virtuous cycle of model improvement through portfolio management of your models.

Continuous measurement

Continuous measurement of model performance is about drawing insights from model performance, and identifying opportunities for model enhancement. You can find these opportunities by stratifying the customer population to isolate pools of customers who behave differently with respect to the predicted outcome; tuning the model for specific pools of risks that contribute to the predictive outcome in different ways; and adding new predictive drivers to the model to help address customer behavior not being explained effectively by the current model.

Corios clients who do a good job of continuous measurement innovate by trying new risk strategies or marketing treatments in order to evaluate unmet customer needs. Through these incremental innovations, they get the opportunity to build pilot test-and-learn programs that yield data useful in building exploratory models, which in turn help them determine whether new models would add further efficacy to their risk and marketing strategies for the future. This is the birthplace of new models, through testing results of pilot programs.

Furthermore, it is helpful to have a comprehensive and standardized measurement schema across the portfolio of all models, so that all models can be compared in terms of efficacy across the board. Seeing the entire model inventory in one place is helpful to bring some perspective on the accomplishments of the business, as well as to evaluate gaps in the model inventory that need to be filled.

Virtuous cycle of model improvements

By tracking model performance and changes in the model inventory, business owners can make smart decisions about investment in staff, skills, model creation, new data and data quality. You cannot improve every aspect of the model factory business all at once; we recommend that carefully-selected targets be selected by the business owner for investment, based on the identification of the best return.

Without a clear identification of which models and decisions based on models need help, and the likely magnitude of the required investment, the business owner’s decision may be off target. This is a significant business concern in new model creation or model retirement; it is our experience that few business owners want to make these decisions to retire models, or sometimes to start to build new models to fill gaps in their portfolio, because the investment feels very substantial and all-consuming. Business owners need to focus on the returns that a good model will contribute to a business decision that derives substantial benefit from the advisory that model scores provide.

The best model factories operated by businesses have a heads-up view of the investments they need to make in their model factory, which new models will provide the best return, and which changes to existing models will yield the best return. Operating this model factory as a portfolio of predictive assets on its contribution to business outcomes truly yields the ideal returns for the firm.

Robin Way

The Founder and President of Corios, Robin’s professional passion lies in democratizing and demystifying the science of applied analytics. An established thought leader fueled with 30 years’ experience in the design, development, execution and improvement of applied analytics models, Robin welcomes every opportunity to move the analytics conversation forward.

Connect with him on LinkedIn , or reach out to Corios to get in touch.