Decoding Metrics in Data Product Management

1. Smoke testing, A/B, or multivariate testing

• • Smoke testing is used at the time of product planning and soft launch. This is done on a very small percentage of your total traffic. Hence, most of your traffic will not see this new feature. You just have to make sure that nothing is broken, the site is working properly, and people can move through it. You are not focussing on end objectives. It is something that you do before you start exposing your site traffic to the new features. 
  • Then A/B testing is a much bold way to test out your product so you could have 20% of your site exposed to the new features and 80% still seeing the old features. You can start from 20 and then slowly start increasing it until you completely open the whole site to the new feature. 
  • In multivariate testing, instead of testing just 2 versions, you may have multiple versions being tested in parallel. You need to make sure that you have the right kind of data you are looking at to ensure that you are getting the right feedback about the product. 

2. Dashboards and signal detection

• • These are used from the soft launch stage to the full-scale stage. The dashboards will give you a variety of metrics that you can keep looking at. But what happens to you as a product manager is that you see the fluctuation in the metric or the movement, and you don’t know if that movement is within random error or is it a real meaningful change in the behavior of the user. 
  • That is where the signal detection comes in. You can apply some statistical techniques to figure out if the movement in the metrics you are seeing is real change or is it something called signal-to-noise ratio. This becomes very important when you are talking to your management. Because if the management has access to the same dashboards and they see some fluctuation they may come running to you.  

3. Multivariate Pre-post analysis of KPIs/other metrics

• • These are used for a post-mortem ROI analysis for the senior executive team. You compare the two across multiple variables- How the customer was behaving before you launched the new features, then how the customer or user is behaving after you launched the new features. 
  • Then you try to understand whether you got an incremental benefit by launching this product or not. This also requires statistical analysis because there are multiple variables involved and you need to make sure that whatever change you are seeing is a real behavior change. This can be used for a postmortem type ROI analysis that senior executives want. 

4. Machine learning/Feedback mining

• • These are used to go beyond numbers, sentiment, or text analysis of customer feedback. What happens in most websites or apps nowadays is that as customers are interacting with the site or the app, they are also making some comments or feedback.
  • You can analyze this data using machine learning or natural language processing and it helps you go beyond pure numbers because decoding data is quantitative. It’s about numbers but then when you do mining of the feedback that your customers are giving you, you can actually go beyond numbers and also measure the sentiments of your customers which tell you a different part of the story about how the site or your product is performing. 

Hence, decoding data is crucial for product managers in their product development journey. They need to understand the challenges posed by different metrics used in product management such as attention, transaction, and productivity metrics, and utilize different tools like machine learning, AB testing, multivariate analysis, etc. to drive their products to success.