Overstock.com's Analytics Journey To A More Personalized Shopping Experience

A conversation with Craig Kelly, group product manager, Overstock.com

Overstock.com has nearly 5 million products for sale onsite resulting in billions of visits and page views in its historic web logs. Sifting through the massive amounts of sparse data to construct personalization features proved to be an enormous computational challenge, and only a small percentage of visits and page views were converting to purchases.

Craig Kelly, group product manager for Overstock.com, explains how the internet retailer was able to deliver data science at scale and improve data science velocity by over 500 percent resulting in an improved customer experience. Overstock.com also decreased the cost of moving models to production by nearly 50 percent and data scientists can stand up new models five times faster than previously required.

Q: Given the number of products for sale on Overstock.com — as well as billions of site visits and page views — how difficult was sifting through the data generated to construct personalization features

Kelly: Featurizing the data was absolutely the most difficult part of our project and involved multiple attempts at paths that proved to be dead ends. In the end, streaming the vast majority of the raw data needed for features through mParticle, our Customer Data Platform provider, allowed us to have a single source of data and avoided wrangling from several sources. Using Snowflake as our data warehouse allowed us to easily create features out of this consistent raw data set.

Q: What decisions were made to increase conversions and what improvement was seen?

Kelly: We can’t get into too many specifics here due to the proprietary nature of this information, but the biggest decision we made was to avoid assumptions altogether and create as many features as possible. The model diagnostics provided by Databricks allowed us to see which features were most important, and to focus on driving their correlated actions.

Q: What processes/solutions were put in place to keep up with changing market conditions throughout the year?

Kelly: Our models are constantly retuned in order to keep up with market conditions. While our historical data was great for initial model training, the most recent data feeds into retraining and retuning in order to keep up with or even stay ahead of change.

Q: What improvements were seen with regard to the customer experience?

Kelly: We were able to repurpose over 10 percent of our annual advertising budget to drive customer interaction when customers wanted to hear from us, as opposed to when we wanted to talk to customers.

Q: What advice would you give to retailers looking to create an engaging customer experience — one that is personal, fast, easy, and useful — but not quite sure how to create it?

Kelly: Retail marketing comes from the world of catalogs, which is a world of one-to-many. As retailers, we need to shift our mindset from trying to digitize one-to-many catalog experiences and focus on delivering the promise of the true one-to-one experiences that technology enables.

Q: More than half of customers today say they’ve switched companies solely because of poor user experiences. What future do you see for retailers that fail to embrace customer experience as a strategic path to growth?

Kelly: This depends on what your product is, or, other words, what people are buying from you. For niche retailers like Room and Board or Crate and Barrel where the customer is buying the physical product that goes in their homes, that product is the customer experience. For retailers like Overstock, where customers often have their choice of multiple places to buy the exact same physical product, the customer is buying the purchasing experience. This latter-type of retailer will have to invest significantly more in creating a cohesive customer experience across an assortment of millions of products in order to not just stay relevant, but to simply stay in business.

Q: How does data-driven customer experience make a difference?

Kelly: No human being is capable of making sense of the scale of assortment available at massive online retailers. In order to scale a business profitably while delivering an excellent customer experience, these retailers have to use all of the data available.

Q: What type(s) of retail analytics are most important right now? How will that change in the next five years?

Kelly: In retail marketing there’s a still a lot of focus on traditional metrics like Cost Per Click or Revenue on Ad Spend. The most advanced retailers are also pretty good at Lifetime Value, and are deep into the world of Customer Acquisition Cost derived from Multi-Touch Attribution and pathing, although I haven’t seen anyone quite figure it out to this point due to issues with identity management. As those metrics become more standardized and solidified, I think you’ll see the focus expand to include significantly more engagement-based metrics as well as the inclusion of data science-centric model accuracy metrics.