2.3 Evaluating Problem Evidence

At this point in the chapter, the student who began with a problem noticed in Module 1 has learned how to investigate the experience of the people affected. That is a substantial change. But interviewing is not an automatic truth machine, and a few compelling stories can make us feel more certain than the evidence warrants. Our final academic lesson is about judgment: deciding what the information actually supports, what limitations remain, and which question would be most responsible to investigate next. This is not a lesson about being afraid to act. It is about learning to act with the right kind of confidence—confidence in the quality of your reasoning rather than certainty about an outcome you cannot yet know.

2.3.1 Evidence Strength and Research Limitations

2.3.1.1 Frequency, intensity, and recency tell different stories

Imagine two employees describing the same scheduling problem. One says it occurred once several months ago and was mildly inconvenient. The other says it occurs almost every week and has caused repeated missed appointments with customers. Both accounts are real experiences if reported honestly, but they imply different levels of urgency. In evaluating a problem, three useful dimensions are frequency (how often the problem occurs), intensity (how seriously it affects the person), and recency (how recently it happened). None of these dimensions is sufficient by itself, and a question about one should not quietly become a conclusion about another.

A rare problem may be extraordinarily serious. A safety failure that occurs infrequently can justify significant attention because its potential consequences are severe. A common problem may be irritating without causing much lasting harm. People may also tolerate frequent small inconveniences because existing workarounds are adequate. An investigator who counts mentions but ignores consequences may prioritize the wrong issue. Conversely, a single severe story should not automatically become a claim about the experience of an entire population. The appropriate conclusion is often that the story warrants targeted further investigation.

Recency matters partly because memories change. People may recall a frustrating event differently after time has passed, especially if they have discussed it repeatedly or encountered new information. Questions about a specific recent event can reduce some uncertainty by anchoring an account to circumstances the person remembers, but they do not make the account perfectly objective. An interviewee may misremember a sequence or overlook a detail, just as an investigator might misinterpret it. This is why discovery benefits from multiple forms of evidence where appropriate: accounts, observations, service records, and existing research can illuminate different aspects of the problem.

Suppose Elena hears five stories of appointment difficulties. A careless summary would label the “clinic reminder problem” validated because five people described frustration. A stronger summary would ask which difficulties were related to reminders, how recent the events were, what workarounds occurred, and whether people who had no difficulty experienced different conditions. The investigator might learn that the problems are serious but diverse. That is progress, because a problem statement can now become more specific and a potential intervention can be evaluated against the right mechanism. It is better to understand three distinct causes than to declare one broad problem solved by an attractive but unsupported idea.

2.3.1.2 What a handful of interviews can actually establish

People sometimes ask how many interviews are required before a problem can be called validated. It would be convenient if there were one universal number: five, ten, twenty, or fifty. But research quality does not operate like a loyalty card in which the final stamp unlocks certainty. A small set of interviews can help uncover experiences, clarify language, identify possible patterns, and reveal flaws in a proposed explanation. Whether it can support broader claims depends on how participants were selected, how much their circumstances vary, what was asked, and whether independent information corroborates the accounts.

Qualitative researchers sometimes discuss saturation, a point at which additional interviews seem to reveal few new themes within a particular study. Guest et al. (2006) examined saturation under specific research conditions. Their findings should not be treated as a blanket guarantee that a set number of interviews fully represents a population. A relatively homogeneous group discussing a narrow topic may produce recurring themes quickly, while a diverse population facing a complex institutional process may require much broader investigation. A change in research question can also change what counts as enough evidence.

We therefore need to distinguish exploratory insight from population estimates. If four out of six interviewees describe difficulty finding evening appointments, a researcher may responsibly say that the difficulty recurred in those six accounts. They cannot conclude that two-thirds of all local residents face the difficulty. Estimating prevalence would require a suitable measurement design and sampling approach. Even then, the survey would answer a different question from a detailed interview: one would help estimate how widespread a defined experience is, while the other could help explain what it means and how it occurs.

The most useful habit is to attach a limitation to each finding. “We learned from people who attended evening classes, so we do not know whether daytime students experience the same barriers.” “We interviewed current users, so we may be missing people who stopped using the service entirely.” “Participants described a problem, but we have not observed how the process works in practice.” These statements are not signs of weak thinking. They show that the investigator knows the boundary between evidence and assumption. In professional decision-making, that awareness can prevent a confident but unjustified change from producing unintended harm.

2.3.1.3 Confidence should match the claim, not the excitement

A useful way to evaluate information is to imagine a ladder of evidence. At the bottom is a personal hunch: something may be happening. A little higher is a concrete report from one person. Above that may be similar accounts from several relevant participants, supported by direct observation or records. Higher still could be broader research conducted with methods appropriate to estimating the size or distribution of a problem. The ladder is only an explanatory metaphor; evidence quality is not one-dimensional, and a carefully documented unusual event can be more important than a badly designed survey of thousands of people. The central principle is that stronger or broader claims require evidence suited to those claims.

Think about the statement “Students want longer advising hours.” What does want mean? A few students may say later hours sound nice; some may repeatedly attempt to book appointments but find none outside work; others may have used an evening session successfully. These observations differ in what they support. If several students describe recent missed opportunities, we may have a reason to explore scheduling barriers. We still do not know whether adding evening hours is feasible, whether students would use them consistently, or which alternative changes might be equally effective. Making the claim more precise increases its usefulness.

Camuffo, Cordova, Gambardella, and Spina (2020) examined a scientific approach to entrepreneurial decision-making in a randomized study involving startups. The findings support the value of structured hypotheses and disciplined testing under the conditions studied. But a scientific approach is not a charm that removes every uncertainty, and the experiment does not tell us that a particular student interview is sufficient to approve a business launch. Rather, it reinforces a general learning discipline: make the claim explicit, decide what evidence would be informative, and allow evidence to change the decision.

This is one of the most important transformations in the chapter. At the beginning, a student may have thought confidence meant speaking as though the problem and solution were obvious. By now, confidence should mean the ability to say, “Here is what we observed, here is what it suggests, here is what it does not establish, and here is what we should learn next.” That form of confidence transfers to academic research, workplace recommendations, nonprofit planning, and entrepreneurship. It helps others trust the reasoning even when the final outcome remains uncertain.

What Can You Do?

John Snow, a Water Pump, and the Courage to Question the Obvious

In 1854, physician John Snow investigated a severe cholera outbreak in London's Soho district. At a time when many people attributed cholera to contaminated air, Snow examined where affected residents lived, which water sources they used, and how the pattern of illness related to local pumps. His investigation included mapping cases and questioning people about their water supply. The Centers for Disease Control and Prevention describes his work as an important early example of using evidence to investigate a public-health problem (CDC, 2004). The historical lesson isn't that Snow simply noticed a cluster and instantly proved a theory; evidence was assembled, exceptions mattered, and the wider scientific understanding continued to develop.

The connection to customer discovery may surprise you. Snow was not conducting startup interviews. He was investigating a life-threatening public-health problem, and the methods and ethical stakes were entirely different. Yet the underlying intellectual move is recognizable: rather than accepting the dominant explanation, he sought information that could distinguish among possible causes. He asked where the problem occurred, whom it affected, and which experiences were relevant. Evidence changed what could reasonably be concluded. For a community-college student, that same habit can improve everyday professional judgment even when the stakes are much lower.

Try the reasoning, not the historical field investigation. Think of a routine problem in a setting you understand, such as missed appointments, confusing instructions, slow service, or recurring maintenance delays. Describe the leading explanation people tend to give, then identify two alternative explanations that could produce the same visible result. List the kinds of non-sensitive, appropriately accessible information that would help distinguish them—for example, existing anonymous timing records, a permitted observation of a process, or published research about similar situations. Finally, explain one reason your first explanation might still be wrong even if several people agree with it.

The goal isn't to imitate a scientist working during an epidemic or to conduct unsupervised research on vulnerable people. It is to practice the discipline that made Snow's work consequential: let the evidence challenge the explanation, not merely decorate it.

2.3.2 Problem Validation Versus Solution Validation

2.3.2.1 Real problems do not automatically create successful solutions

Imagine that a local community has a genuine problem with affordable transportation. People describe missed appointments, difficulty reaching jobs, and long waits between connections. These accounts may provide a strong reason for public agencies, employers, and entrepreneurs to investigate improvements. But they do not establish that a privately operated shuttle service will be the best response. The shuttle could run at the wrong times, fail to reach critical destinations, charge more than riders can afford, or require resources beyond what a small operator can provide. A real problem creates the possibility of value; it does not select the correct way to deliver it.

Problem validation asks whether a meaningful difficulty exists for a defined group, under which conditions it occurs, and what evidence supports that understanding. Solution validation asks whether a particular proposed response effectively addresses the difficulty for those people. Business viability asks still more questions: whether the value can be delivered consistently, legally, ethically, and with a workable relationship between costs and resources or revenue. These are related inquiries, but they are not interchangeable. Module 2 concentrates on the first; later modules develop the other two.

An early founder may be especially tempted to blur the categories. Suppose potential customers describe frustration with unreliable home-repair appointments. The founder plans to build an AI voice agent that books appointments automatically. The interviews suggest that reliability matters, but they do not tell us whether customers would prefer to speak with software, whether small businesses can integrate it with existing schedules, or whether the system can handle urgent and sensitive calls responsibly. The problem might be validated while the proposed AI solution remains speculative. The honest next step is to investigate the particular claim about the solution, not to treat general frustration as proof of demand for it.

Blank (2013) places customer learning at the heart of the early venture process, and Shepherd and Gruber (2021) describe validated learning as a central element of the lean startup framework. We should not turn these ideas into a shortcut where one encouraging interview fulfills an entire sequence of learning. Recognizing the distinctions protects entrepreneurs from premature spending and protects customers from solutions that may not address their actual needs. It also helps professionals avoid adopting a trendy tool simply because the underlying problem is important.

2.3.2.2 Compliments, intentions, workarounds, and real choices

Consider four statements a potential customer might make. “That's a wonderful idea.” “I would probably use it.” “I spent two hours last weekend trying to solve that problem.” “Last month, I paid someone to handle the problem because the existing options weren't working.” Each statement contains information, but not the same kind. A compliment reveals a reaction in the conversation. An intention expresses a prediction about future behavior. A workaround describes a reported past action. A payment describes a reported past resource commitment. The last two are often more concrete clues that a problem has consequences, although they do not prove that the customer would adopt the particular offering being considered.

We should resist creating a simplistic hierarchy in which spending money always matters more than anything else. People may experience serious difficulties without having the means to pay for a solution. A public benefit or nonprofit service may address a real need precisely because the affected people lack purchasing power. A person who risks safety or loses essential time may be experiencing a consequential problem even if no comparable commercial product exists. The meaning of evidence depends on the kind of value being created and the decision that must be made. Still, for a proposed business, some later evidence about willingness and ability to pay will eventually be necessary for financial viability.

Workarounds are especially revealing because they show what a person does when an existing system does not fit. A student who sets multiple alarms to register for classes, a technician who keeps a paper backup of a digital system, or a caregiver who rearranges several schedules may be paying a cost in effort or time. But a workaround is not always a sign of dissatisfaction. Some people prefer their informal method because it feels dependable or gives them control. Good discovery asks why the workaround exists and what value it provides before describing it as inefficient. The preferred solution could turn out to remove a feature people find essential.

Think again about Elena. If someone understood the clinic reminder but found it difficult to obtain a required document, a clearer reminder might produce almost no improvement. If the document requirement itself serves a necessary safety or legal purpose, removing it may not be responsible. If a visitor relies on a family member to coordinate preparation, the clinic might need to investigate authorized communication arrangements rather than simply shorten the message. Each possible response requires additional evidence. Understanding the underlying problem doesn't eliminate the need to examine the actual mechanism of a solution.

Entrepreneur-in-Action

Rent the Runway: A Real Test Came Before a National Launch

Rent the Runway's founders, Jenn Hyman and Jenny Fleiss, identified a possibility in the cost and limited use of special-occasion designer clothing. According to the company's published history, the idea began in 2008, and in April 2009 the team organized a campus pop-up where students could try on and rent dresses before the company launched its national website later that year (Rent the Runway, n.d.). The sequence matters. The founders did not rely solely on asking people whether renting a dress sounded appealing; they created an opportunity for potential customers to encounter a version of the service and make a more concrete choice.

The case belongs here because it clarifies the boundary between discovery and testing. Conversations might help a team learn why purchasing expensive clothing for an occasional event is frustrating. A pop-up, by contrast, begins to explore how people respond to a proposed solution under particular conditions. The company's public history describes what the founders did, but it does not provide enough detail for us to calculate how representative the attendees were or to claim that the pop-up alone proved a national market existed. A rigorous reader appreciates the practical lesson without turning the story into a formula that guarantees commercial success.

Now consider an academic counterpart. A college discovers that students have difficulty accessing tutoring. Interviews clarify the circumstances and consequences. A short pilot of different tutoring hours could then investigate whether a proposed change actually improves access. The experiment would require appropriate planning and measurement, but the underlying sequence is the same: understand the problem first, then test a particular response. The story becomes more useful when we see that the reasoning is transferable beyond startup founders and that each stage answers a different question.

See all 24 entrepreneurship case studies →

Venture TrackVT-2C · Optional

Companion to 2.3.2 Problem Validation Versus Solution Validation

Is the Problem Actually Painful?

Consider Lena, who wants to build a service that answers missed calls for independent plumbers. During conversations, several owners say, “That would be great.” She could interpret the compliments as demand and buy expensive software. A better tactic is to look for recent problem evidence. When did a call last go unanswered? How did the owner know it was missed? Did anyone call back? Did the owner lose a confirmed job, or is that still an assumption? Does an answering service, family member, voicemail, or existing phone system already handle the problem?

What Lena learns	What it does—and does not—support
“I love the idea.”	Interest or courtesy; not adoption or payment evidence
“I missed three calls last Friday.”	A reported event; the consequences still need investigation
“I pay an answering service.”	Evidence of an existing resource commitment; not evidence Lena's service is better
“A customer booked another plumber.”	A reported consequence worth understanding; not a population-wide estimate
“I'd pay $200 a month.”	A stated intention, not an actual purchasing decision

Lena can now identify a sharper next question: for which kinds of small businesses are missed calls frequent, consequential, and inadequately addressed by existing options? She still needs to investigate reliability, privacy, integration, pricing, regulations, and actual willingness to pay before promising a service. An early discovery conversation is not permission to make performance claims.

What this teaches

Separate compliments from reported behavior, investigate the cost of a problem, and choose the next test without confusing it with a confirmed sale.

2.3.3 From Discovery to the Next Learning Question

2.3.3.1 Learning ends with a better question, not a declaration of victory

It is natural to want a clean conclusion after collecting and analyzing information. A student who spends time learning about a problem hopes to hear, “You've validated it—now build the solution.” But discovery rarely ends with complete certainty. It usually produces a better description of the problem and a more useful set of questions. A team may learn that the difficulty matters deeply to a narrow group but remains less serious for other people. It may learn that two different causes were combined into one broad complaint. It may discover that an existing service already solves much of the problem for people who can access it, shifting attention toward availability rather than invention.

A valuable end-of-discovery summary therefore includes five elements: who was studied, what was reported or observed, which patterns and exceptions appeared, how strong the evidence is, and what remains unknown. This structure gives readers a chance to examine the reasoning. A manager can decide whether more information is needed before changing policy. A student can identify what additional research might help. A founder can avoid spending money on a preferred product before learning whether it addresses the problem actually observed.

Imagine Elena presenting her findings to an authorized supervisor. Instead of saying, “The reminders are confusing and we need to rewrite them,” she could say, “Our preliminary inquiry suggests that some people have trouble completing preparation requirements, but the reasons appear to vary. Reminder timing and clarity may matter in some situations; document access and scheduling constraints may matter in others. We would need an approved review of the process and appropriate additional evidence before recommending a specific change.” The statement is less dramatic. It is also more useful because it identifies what the organization can investigate responsibly.

The broader lesson is that a well-defined problem is an intellectual achievement. It can prevent an organization from spending resources on the wrong solution, help people with different responsibilities understand each other, and make a later experiment more informative. Entrepreneurial progress is not measured only by how quickly someone produces a product. Sometimes it is measured by how much unnecessary action someone can avoid because they learned to ask a better question.

2.3.3.2 When to continue, widen, narrow, or pause

The next learning step depends on what remains uncertain. If participants describe very different experiences, the investigator may need to narrow the research question to a more coherent situation. If the first interviews all came from one easily reached group, they may need to widen the inquiry to include people whose experiences differ. If a pattern seems strong but its prevalence is unknown, a broader measurement method may be appropriate. If the investigation reveals ethical or legal concerns, the responsible step may be to pause and seek institutional guidance rather than continue collecting information.

A beginner may interpret this flexibility as indecision. In reality, changing the research design when new information appears can be a sign of progress. The purpose of an investigation is not to defend the plan created before the investigation began. It is to reduce uncertainty about a real problem. This is consistent with the logic of hypothesis-driven entrepreneurship and learning discussed by Leatherbee and Katila (2020). Their research supports important elements of structured probing while reminding us that the method's usefulness depends partly on who uses it and how the process unfolds. A technique is not valuable because it sounds entrepreneurial; it is valuable when it produces learning appropriate to the decision.

Not every discovery project should proceed directly toward a solution. A community need may be genuine but require public-policy coordination, funding, specialized expertise, or legal authority beyond one person's control. A customer difficulty may be real but too small to justify a costly intervention. An organization may discover that its own rules create the problem, making internal change more relevant than a new product. These are not disappointing outcomes if they lead to better understanding and more responsible choices. A useful method should be capable of telling us when not to proceed, not merely encouraging us to keep going.

2.3.3.3 The bridge to business models and value propositions

The next module turns toward business model design and strategic thinking. Students who want to launch a venture may be eager to define their product, pricing, and revenue. Students who are interested primarily in professional development may want to learn how organizations arrange people, resources, and activities to create value. Both groups will benefit from the same foundation: an evidence-informed understanding of whom the organization serves and which problem matters. Without that foundation, a business model can become an elegant diagram organized around a problem that has never been understood.

Imagine a service meant to help working students obtain advising. One possible model involves extended office hours, another a partnership with community organizations, and another a digital information tool supported by human advisers. Each approach has different costs, capabilities, constraints, and ways of creating value. The discovery work from this module does not tell us which model will succeed. It gives us the questions that a useful model must answer: whom will it serve, which barrier must it address, what alternatives exist, and what evidence would show that the service is helping?

The same reasoning applies to a home-repair business, an independent tutoring practice, a hospital department, or a nonprofit food program. Understanding a person's problem does not mean handing the person every decision. Professionals still bring expertise, legal obligations, and resource constraints to the process. But without knowing what people actually experience, those decisions risk being technically impressive and practically irrelevant. The next module is where we begin examining how organizations can design coherent ways to deliver value. The responsibility to learn from people does not end when the design begins.

AI Advisor · Coming soon

Customer Discovery Advisor

Coaching on your discovery questions, your interview notes, and how much confidence your evidence deserves.

What to bring

• Before your conversations: the five or so questions you plan to ask, and whom you plan to talk with (2.1.3, 2.2.2).
• Afterward: your notes, with what each person said kept apart from your interpretation (2.2.3.2).
• Your problem statement and evidence note, for example from the Problem Statement Builder (2.2.5).

What the advisor does

• Rewrites leading or hypothetical questions into questions about a specific recent event (2.1.3.2).
• Asks who is missing from your conversations and whether a convenient group is standing in for a wider one (2.1.5).
• Separates compliments and intentions from reported behavior, workarounds, and real choices (2.3.2.2).
• Weighs your evidence by frequency, intensity, and recency, and flags claims stronger than a handful of conversations can support (2.3.1).
• Keeps problem validation apart from solution validation, and ends with a next step: continue, widen, narrow, or pause (2.3.3.2).

What it won't do

• Grade you, or declare a problem or market validated.
• Write interview answers, invent participants, or fill gaps with made-up data.
• Need anyone's name or private details: describe people by role.

Coming soon

About this advisor

A coach, not a grader: the work and the decisions stay yours.

Emerging practices

Staying on Top of Emerging Practices

AI-assisted interviews, automated call summaries, sentiment analysis, and predictive dashboards make it possible to gather and sort customer information rapidly. Yet the central challenges of discovery remain: the system can only analyze the information available to it, and that information may omit important people, contexts, or causes. A model that confidently labels a problem “validated” cannot establish market demand without relevant evidence, and it may misrepresent nuanced or sensitive accounts. As these tools become more common, entrepreneurs and professionals will need to document data sources, protect consent and privacy, review original evidence, and explain how a conclusion was reached. Faster analysis is valuable only when it strengthens the quality of the decision rather than disguising uncertainty.

Module 2 Synthesis: From Having an Idea to Understanding a Person

In Module 1, you learned to see opportunities in familiar situations and to recognize that your own experience can be a starting point for inquiry. Module 2 has taken you beyond the moment of noticing. We have learned that the most obvious explanation for a problem may be incomplete, that people facing similar difficulties may have different goals and constraints, and that a conversation is useful only when it creates room for information we did not already expect. The central skill is not persuading people to agree with us. It is learning to listen in ways that improve our understanding.

You have also encountered the discipline required to turn conversations into evidence. Good inquiry respects people's choices and privacy. It distinguishes past events from hypothetical preferences, recognizes when interview questions encourage flattering answers, and avoids treating a convenient group of participants as representative of everyone. It identifies stakeholders whose roles differ, groups people according to meaningful circumstances rather than stereotypes, and creates problem statements that make the evidence and its limitations visible. These are demanding intellectual habits, but they can make your decisions more useful in almost any career.

Perhaps most importantly, we have separated the problem, the solution, and the business. Evidence that people experience a consequential problem can justify further investigation. It does not prove that your favorite solution will help, that people will adopt or pay for it, or that an organization can deliver it sustainably. Those questions deserve their own careful methods. For now, being able to explain what is known, what remains uncertain, and who still needs to be heard represents significant progress. You are moving from the excitement of discovering an opportunity toward the responsibility of understanding it.

Our continuing fictional student, Elena, began this chapter hoping to fix a confusing reminder. She finishes with a more sophisticated view: the problem may involve several different barriers, the people affected may experience them differently, and the right next question must be chosen with appropriate ethical and organizational safeguards. She is not less entrepreneurial because she has become more cautious. She is better prepared to create value because she is less likely to confuse a plausible answer with a well-understood problem. That is the transformation we wanted this module to make possible.

Key Terms & Definitions

Review Questions

1. How did the purpose of customer discovery differ from the preliminary opportunity recognition you practiced in Module 1? Explain why someone might identify a real difficulty but misunderstand its underlying cause.
2. Compare a leading interview question with a neutral question about a specific past experience. Why might both questions generate enthusiastic responses but very different quality of evidence?
3. Describe a professional setting in which the person who uses a service is different from the person who purchases or authorizes it. Whose perspectives would be necessary to understand the problem responsibly?
4. How can an investigator find recurring themes in several interviews while remaining honest about sampling bias, contradictory accounts, and the limits of what a small group can establish?
5. A person has discovered a significant customer problem and received many compliments about a proposed app. What, if anything, has been validated? What important questions remain unanswered?

Discussion and Reflection

A problem you thought you understood. Think of a situation in which you or someone around you first blamed an individual for a frustrating outcome, then discovered a constraint or process issue that changed the explanation. How might a better question have changed the conversation? Discuss the situation respectfully without revealing another person's private information.

The questions behind the answers. Consider two hypothetical interviews about students' difficulty accessing tutoring. In one, the interviewer repeatedly asks whether people would prefer a mobile app. In the other, the interviewer asks for recent examples of how students sought help. What types of evidence might each conversation produce, and which interpretations would still require further investigation?

Whose voice is missing? Imagine a hospital, library, repair company, or school redesigning a familiar process. Identify the users, purchasers or authorizers, support staff, and other stakeholders. Which group would be easiest to interview? Which group might be hardest to reach? How could that difference affect what the organization believes it knows?

One problem, several segments. Choose an everyday service and imagine two groups that experience it differently. Identify the goal, context, and obstacle for each group, then consider whether one improvement could help both or whether different responses might be necessary. Explain what information you would want before making that judgment.

The responsible conclusion. A fictional investigator hears from five people who describe frustration with an appointment system. Three mention inconvenient hours, two mention unclear instructions, and one mentions transportation. Write two possible summaries: one that overstates the evidence and one that communicates it accurately. Discuss which summary would better support a future decision and why.

References