- Any Questions?
- 080-4710-6006
- [email protected]
By Dr Chiranjiv Roy – Senior Vice President at SG Analytics
Over the last three decades, the transformation of developing products has changed in leaps and bounds. In this discussion, we will explore the significant shifts that have occurred and examine how emerging product managers can leverage data science across various stages of the product lifecycle. Today, there is a universal drive among businesses to revamp their operations, optimize costs, and boost revenue. Digital readiness is now a prerequisite, with automation catalyzing innovation. The democratization of AI has revolutionized industries, exemplified by the seamless integration of AI in products like Amazon. This shift towards digitalization has become the new standard since the pandemic, where remote work is prompting us to adapt to this “new normal.” In light of these changes, it’s essential to reevaluate our strategies and embrace the challenges and opportunities presented by data science and data-driven products.
The essence of this approach lies in understanding the lifecycle of data products. With the surge in demand for software-as-a-service products, particularly amid the pandemic, there’s a pressing need for higher intelligence in our offerings. This forces product managers and owners to refine product usage, craft customer-centric designs, and fine-tune competitive pricing strategies. Thus, the crux of this approach revolves around leveraging data to its fullest potential. By utilizing data, we can develop and refine data products, creating a cyclical process of optimization that is iterative rather than linear.
The Data Product Journey involves a fusion of people, processes, and technology. It begins with identifying current problems rather than projecting too far ahead, as immediate needs drive revenue considerations. Defining these problems initiates a process to determine whether they stem from customer-facing, business, process, engineering, or other issues, thereby shaping the type of product to develop. People within this process play a pivotal role in defining the product development scheme.
Technology acts as an enabler throughout this journey, aiding in various capacities, either independently or in combination. The journey progresses through stages such as design, production, customer delivery, and experience enhancement. We are starting with data collection, patterns and insights emerge through analytics, leading to scientific analysis to address issues effectively.
Successful implementations are then tested with a select customer base, paving the way for software creation or transformation. This phase is critical, as it integrates knowledge into the product, whether as an enhancement or a completely new offering. This iterative journey highlights the importance of data-driven decision-making and its transformative impact on product development.
The answer lies in a cyclical process that involves design, definition, development, optimization, and redesign. This comprehensive cycle forms the backbone of data product creation.
Designing a data product is a blend of creativity and scientific methodology, guided by established frameworks. The journey begins with defining the product through initial approaches and creating a value pipeline, leveraging data throughout the production phase. Operationalizing data through methodologies like data ops ensures standardization and efficiency.
Software development methodologies such as DevOps facilitate the development phase, with agile frameworks like Scrum adapting to this evolution. Following development, optimization occurs through ML operations, enhancing decision-making with predictive and prescriptive analytics.
Ultimately, product decisions are informed by a combination of instinct, customer feedback, and competitive intelligence, underpinned by rigorous analytics. This shows how complicated data-driven product management can be, as product managers need to think about many things when making or improving products.
Let’s look at a few product examples and check if they are data products or not.
1. Medium- While Medium itself isn’t a data product, data plays a crucial role in tailoring content to readers’ preferences. This feedback loop provides valuable insights to the Medium team.
2. Gmail- Gmail is primarily an email service, but it leverages technologies like natural language processing to enhance user experience. For example, sorting emails using ML operations adds a personalized touch.
3. Instagram- Instagram isn’t a data product per se, but features like tagging, search, and discovery rely on data analytics. User preferences, such as likes and dislikes, are captured and utilized by Instagram’s ML algorithms to optimize content delivery.
4. Google Analytics- Yes, Google Analytics is a data product. It facilitates interaction and provides insights similar to Gmail, Medium, or Instagram, making it a valuable tool for businesses to analyze website traffic and user behavior.
The following are the main varieties of data products:
1. Raw data- This is the initial data collected without any processing. Users can access and manipulate this data according to their needs, with most of the work done on the user’s end.
2. Derived data- Derived data involves some processing done on our end before presenting it to users. For instance, we may add extra attributes to customer data, such as customer segments or likelihood of engagement with specific ads or products.
3. Algorithms- Next we have algorithms or algorithms as a service. We are given some data, we run it through the algorithm- be that machine learning or otherwise- and we return information or insights. A good example would be a Google image: the user uploads a picture and receives a set of images that are the same or similar to the one uploaded. Behind the scenes, the product extracts features classifies the image, and matches it to the stored images, returning the most similar ones.
4. Decision support- In this category, we offer information to users to aid their decision-making process without making decisions for them. Analytics dashboards like Google Analytics fall into this category, providing insights to users for informed decision-making.
5. Automated decision-making- Here, intelligence is outsourced to automate decision-making within a specific domain. Examples include Netflix’s recommendation system or Spotify’s Discover Weekly playlist. Physical manifestations include self-driving cars or automated drones, where decisions are made autonomously based on predefined parameters.
The following are the common ways in which we consume data products:
1. APIs- APIs are primarily for technical users. It’s important to maintain good product practices by ensuring that the API is user-friendly, well-documented, and capable of fulfilling users’ needs efficiently.
2. Visuals- Dashboards and visualizations require some level of statistical literacy. We strive to present information in an easy-to-understand format, while still allowing users to interpret and make decisions based on the data presented.
3. Web Parts- Web elements have become the most common interface for data products in recent years. These interfaces have expanded to include voice, robotics, and augmented reality. While each interface has its unique design, they all focus on presenting decision results to users and explaining the rationale behind AI-driven decisions.
In the journey of building, managing, or enhancing a product, it’s essential to align with the business goals set by the product management team, in coordination with the product owner and manager. This involves striking a balance between user experience and predictive analytics. While not all aspects can be directly experienced, it’s crucial to maintain an equilibrium between experiential knowledge and scientific insights. Given the more methodical nature of science and the tangible nature of experience, a data scientist must also possess strong design skills to convey insights effectively.
Furthermore, advancements in technology have significantly expedited technical delivery. Despite the lengthy development timeline of AI over the past few decades, contemporary approaches to infrastructure and software development, such as software-as-a-service models, allow for rapid deployment. This enables product development to commence swiftly, without the need to create everything from scratch, potentially initiating within a day.
The concept is majorly defined by 3 different personas- design, data engineer, and data scientist. The data engineer handles data sourcing and infrastructure setup, while the data scientist and designer collaborate with the product owner to define problem statements and devise strategies. The collaboration ensures a comprehensive understanding of both the technical and creative aspects.
This collaborative process acknowledges that concept development isn’t a solitary endeavor. Designers must comprehend the data scientist’s perspective, and vice versa, fostering a balanced integration of experience and scientific insights.
Once the concept is refined and tested, it moves to production. The data scientist facilitates the handover to the ML engineer, guiding the integration with the data engineer’s infrastructure. This interplay underscores the importance of ML engineers possessing both data science expertise and practical software development skills.
1. Bridging data science interaction with business stakeholders
Data science product managers shouldn’t be just basically focusing on data, they should be fully tapped into business stakeholders and be able to understand and explain their needs to solve customer problems and figure out product features and industry challenges.
2. Dealing with Data science complexity
Unlike traditional software which does not need to be re-trained, data science products may usually differ from desired performance over time. A PM needs to understand what progress looks like for a product or a feature working with data scientists who extract evaluation metrics that determine the outcome of an experiment. Both PMs and data scientists then must be able to demonstrate their business stakeholders on other teams unquestionably.
3. Balancing agility with data science product development complexity
For accomplishing fast delivery of products, scrum or similar methods are used by software product managers. But for AI/ML, not all stages of the ML cycle work on tight schedules in fixed times. Data science is science, so it is very open-ended and exploratory, some experiments are successes, but some are failures.
Product development is evolving rapidly, with data science playing a central role in ensuring competitiveness in the digital era. Utilizing data to its maximum capacity empowers product managers to foster innovation, streamline processes, and provide unparalleled user experiences. From raw data to sophisticated algorithms, the potential for advancement knows no bounds, offering endless opportunities for business growth and success.
By Dr Chiranjiv Roy – Senior Vice President at SG Analytics
Being a data-driven product manager involves making decisions based on insights derived from data analysis rather than relying solely on intuition or assumptions. Data-driven product managers utilize metrics, analytics, and user feedback to guide decision-making, ultimately aiming to create products that align closely with user needs and business objectives.
Data-driven product strategy involves formulating product plans and initiatives based on insights gleaned from data analysis. A data-driven product strategy prioritizes continuous experimentation, iteration, and validation to ensure that product decisions are grounded in empirical evidence rather than conjecture. By aligning product strategy with data-driven insights, organizations can maximize the effectiveness of their product development efforts and increase their chances of success in the market.
A data-driven approach to product development involves integrating data analysis and insights into every phase of the product development lifecycle. It begins with collecting and analyzing data to identify user needs, preferences, and pain points. Throughout the development process, data is used to inform feature prioritization, design decisions, and product iteration.A data-driven approach to product development enables organizations to build products that are more user-centric, efficient, and successful in the market.
The essentials for a data product manager include bridging data science interaction with business stakeholders, dealing with data science complexity, and balancing agility with data science product development complexity.
