Product Discovery Toolkit
In Product-Led organizations discovery is critical to understanding what users need and determining the right things to build. There are plenty of tools at our disposal to conduct discovery. In this post I want to highlight a few mentioned in Marty Cagan’s Inspired.
What is Product Discovery?
Before diving in to the methods I think it’s important to define what I mean by product discovery. There’s many different types of product discovery that can take place across different organizations and stages of the product development life cycle.
When I talk about product discovery in this post I’m specifically referring to methods and processes that relate to determining what users need, and exploring solutions that we could provide to meet those needs.
MVPs, Prototypes, and Experiments oh my!
Much like product discovery, depending on the organization and team dynamics these words can be used interchangeably. In my experience MVP is one of the most overloaded phrases in tech, it can mean drastically different things to different people, because of this it’s a word I like to avoid.
For my purposes, a prototype is a shallow step into building something in order to provide very specific answers to a very specific set of questions. The methods, depth, and set of questions may differ depending on the prototype being created, but these rules generally apply across the board.
An Experiment on the other hand, is a timeboxed event that is done with real or perspective users. Experiments regularly use a prototype in order to get feedback from users in some form. The distinction is subtle, but i feel it’s important to understand that a prototype is an artifact, where as an experiment is an event or study. The act of simply building the prototype, in many cases, is not enough. We need to put it into the hands of actual users in order to gain useful insights.
Feasibility Prototypes
Starting with this one right out of the gate because I strongly agree with Cagan in that it’s often discounted by Product Managers, or seen as a strong deterrent to going down the path that presents the need for a feasibility prototype.
A Feasibility Prototype is most often created by an engineer when a product team has a solution that treads into unknown territory for the team. Whether it’s a new framework on top of existing architecture, or an entirely new system like Machine Learning or “AI.” A feasibility prototype is designed to validate and assess the performance of the tech itself.
This is likely one of the few prototypes that does not need users. The goal of this prototype is to allow the development teams to gain confidence and reduce the risk of a potential solution. One of the side benefits of a feasibility prototype is that the act of building the solution may present alternative ideas or paths to solve the same problem. This method also helps developers feel invested in the solution, they’re getting to be involved in the process, rather than just coming in at the end to “code monkey” a design across the finish line.
User Prototypes
This is likely the most common understanding of the word “prototype.” A User Prototype is, as Cagan puts it, “a simulation. Smoke and Mirrors. It’s all a facade.” There’s nothing real about a user prototype. While some might consider this description overly ambiguous, it’s about as specific as we can get given the large spectrum of work that falls into this category.
User prototypes can be any level of fidelity, from a barely roughed out interactive wireframe, to a full-featured high-fidelity prototype that appears to function as the “real deal.” The value of a user prototype tends to come in the form of idea validation. It’s great to get feedback on an overall design or to solicit input on the placement of interface elements, as well as communicating how a product may work once developed.
The main drawback to user prototypes is that they rarely have the ability to prove their value. Users may tell you they love an interface design and still end up not using it for one reason or another.
Hybrid Prototypes
The last prototype I want to touch on is the Hybrid Prototype. For me, this is the most interesting of the prototype categories, because it combines different aspects of all the other types in interesting ways.
A common hybrid technique is the Wizard of Oz prototype, as the name implies, the idea behind this technique is to create a “real world” front-end experience that your users interact with, while having an actual human “behind the curtain” completing the tasks that the system would accomplish if built.
One of the major benefits of this type of technique is that it allows you to quickly and relatively easily create a functioning system from the user’s perspective, without having to spend the time or resources to built out the actual tool. This allows product teams to immediately start collecting feedback on “how the system works,” and see if the solution meaningfully impacts a user’s experience with a product.
The counter to the ease and speed of this technique is the abysmal ability to scale this sort of prototype. Because real people are doing the work behind the scenes, any ability to scale this solution requires bringing on more people. This type of prototype is best for small-scale and rapid idea validation and data collection. It’s important to understand these limitations going into this type of prototype and accept that due to the scaling limitations this will need to be converted into a real solution in the long term for success.
Testing with Experiments
If a prototype is the artifact that is created, an experiment is the process of taking that prototype and making it available to real users in order to obtain valuable feedback on the proposed solution.
As with prototypes, there are many types of experiments and tests that can be run. The four main things we often test for are User Value, Usability, Feasibility, and Business Viability. While it’s not required to tackle these concerns in order, it typically makes the most sense. Before doing anything else, it’s important to understand and confirm there is user value in the solution, from there we make sure it’s something they can actually use. Once we have those two problems nailed down it’s time to meet with the development teams and make sure the proposal is actually feasible to build within the timeframe, budget, or other product constraints. Finally, once everyone agrees on those items, we can work with the rest of the organization to understand how the overall business might be impacted by the solution.
Testing for User Value
This might be the most important set of experiments when it comes to product discovery. At the end of the day, if an idea or solution does not provide lasting, meaningful value to our users it doesn’t matter how good everything else is, the wont use or buy it.
Creating and providing user value is at the core or product development, once we know that our users see the value of a solution we know we’re working on the right thing. Three metrics for understanding and determining the user value of an idea are: User Demand, Qualitative Value, and Quantitative Value.
Experiment for User Demand
A common, and relatively low-risk, experiment that teams can run to collect data on and validate the demand for a feature is called the fake door or door to nowhere experiment. The idea behind this experiment is to place a call-to-action in your product and track the people who interact with it. The cta does not take the user to the proposed product or feature when clicked, but instead can direct the user to a message or landing page indicating that your team is collecting data and soliciting feedback on the creation of the feature.
The main reason this type of experiment works is because there should be no indication to the user before-hand that the proposed feature or product does not exist. The act of clicking on the call-to-action is then a true metric of the user wanting to see or use the feature.
If you decide to collect user data and contact information, this type of experiment can also generate a list of users who are ready and willing to engage in conversation with you specific to the proposed feature or product.
Experiment for Determining Qualitative Value
Qualitative Value is the “meat and potatoes” of a product. While demand and quantitative data can let us know if people want, and how they’ll use a product, qualitative value is the why.
A common experiment to determine and understand Qualitative Value for users is the usability test. Sitting down with actual users and having interview style discussions with them where you determine what they’re looking for in a solution, and why they feel like that might meet their needs.
Unlike quantitative value, qualitative value is going to be different for each individual. Collecting meaningful qualitative data is like turning over the pieces of a puzzle, each new piece reveals a little more about the problem and guides you towards the solution.
Experiment for Determining Quantitative Value
Quantitative Value experiments are where product teams can gather large amounts of real evidence on what users are doing, sometimes to the point of statistical significance. Quantitative value is all about collecting data, and using that data to inform product decisions.
The gold standard that most product people are familiar with is A/B Testing. In an A/B test users are shown one of two (or more) different solutions to a user problem. The key is that they are unaware of the test taking place, or which version they are on. In the discovery phase of product development this sort of experiment might look like a relatively small percent of your users (1-3%) receiving the “B” version (typically some form of high-fidelity prototype), while the rest of your users remain on the existing “A” version (your actual product). This type of testing can often provide predictive results of how real-world users would use the new product or feature.
Product Feasibility Experiments
Much like the feasibility prototype, a Feasibility Experiment is working with the engineering team to answer as many potential unknowns as possible. Some of the questions you may look to answer with a feasibility experiment are:
- Do we know how to build this?
- Do we have enough time to build this?
- Do we understand the dependencies involved?
- How will this solution scale?
The end-result of a feasibility experiment is the feasibility prototype mentioned above. The process of running a feasibility experiment is just as important as any other. Bringing in the engineers and giving them the time and space to explore and answer these questions saves time and avoids future roadblocks. It also provides ample space for the team to have discussions about specific needs or requirements, and may even provide alternative routes to the solution. Your engineers want to create something they can be proud of just like the design and product teams, giving them opportunities to be involved helps contribute to the overall health of the product team.
Validating Business Viability
When talking about establishing and validating Business Viability the basic idea is to ensure that all of the different parts of the organization are in alignment with how to build, market, sell, and support the solution. Business viability involves contact with all of the departmental stakeholders that have touchpoints with the product, from speaking with legal to determine things like whether or not you can sell in specific regions, or localization and accessibility requirements, to ensuring sales has a path to market for the product and support has the resources to help users who run into issues.
Closing Thoughts
Product Discovery can often be the most daunting and overwhelming part of the product development lifecycle for someone who is new to the process. Everything listed above and more is all done before we’ve decided to commit to building and shipping any potential solution.
The idea behind this rigorous process however, is that when we do finally decide to build something, we’re pretty darn sure it’s the right thing to build. Successful product teams know what methods, techniques, and processes to use at the right moment to get the right data in order to de-risk ideas and inform decisions.
Sometimes you get lucky and strike gold with your first idea, other times you just keep making the best decision you can in the moment and work towards that success over time with careful planning and deliberate decisions. Keep providing value and your users will thank you.