5.3 Learning-Based Validation Decisions

5.3.1 Evidence and Uncertainty

5.3.1.1 Experiments reduce uncertainty; they do not erase it

When we begin an entrepreneurial project, uncertainty can feel like a flaw that needs to be removed before we move forward. We want to know that the customer will respond, the prototype will work, the organization will be able to deliver, and the costs will fit. A good experiment can help us replace part of that uncertainty with better information. Yet even a carefully designed test cannot reproduce every future circumstance. Customers change, costs shift, competitors respond, and organizations encounter pressures that did not appear during a small trial. Validation should therefore be understood as an evidence-based increase in confidence about a defined claim, not a certificate guaranteeing that the complete venture or improvement will succeed.

This distinction matters for the way we talk about results. If twenty volunteers successfully complete a usability task, we may have stronger confidence that selected participants can use a particular design under similar conditions. We do not know that every future user will succeed, that the service will be affordable, or that there are no accessibility barriers. If a small paid service pilot attracts repeat customers, we have stronger behavioral evidence of value than we had from compliments alone. We still do not know how the business will perform across different seasons or whether labor and marketing costs will remain manageable. The more precise we are about which uncertainty has narrowed, the better prepared we become for the uncertainties that remain.

Shepherd and Gruber (2021) argue that Lean Startup learning is connected to a broader set of business-model and market-opportunity decisions. That connection matters because the result of one experiment feeds into a system of other assumptions. A positive demand signal may justify investigating delivery capacity. A disappointing prototype test may justify revisiting the customer problem. What matters is that our confidence changes for a reason rather than simply rising or falling with our mood. The experiment informs the judgment; it does not replace the person responsible for making the judgment.

5.3.1.2 Confidence should rise and fall in proportion to evidence

Imagine two coworkers discussing a proposed change to the way their department handles requests. One says, “I know the new process will work because everyone in our meeting loved it.” The other says, “We've seen promising feedback from the staff who attended, but we still need to understand whether people can use it during the evening shift.” The second employee is not less committed. They are using a more calibrated form of confidence—confidence that matches what has actually been observed. They can be encouraged by the feedback without making claims that extend beyond it.

A simple evidence ladder can help beginners think about stronger and weaker indications, provided it is not treated as an inflexible ranking. At one end are personal impressions and unsupported claims. More informative evidence might include firsthand descriptions, observations of behavior, successful completion of a task, repeated use, or actual payments. Controlled comparisons may strengthen causal interpretations when carefully designed. Yet each has important limits, and the strongest evidence for one question may be irrelevant to another. An actual purchase may be a strong signal of willingness to pay at a particular price, while an interview may be better for understanding why a service feels disrespectful or confusing.

This is also where we should distinguish evidence of absence from absence of evidence. If an experiment is too small or poorly matched to the question, a null or inconclusive result may not tell us much. If a carefully designed and sufficiently informative test repeatedly fails to reveal the predicted behavior, that can meaningfully reduce confidence in the hypothesis. The same word—“no result”—can conceal very different situations. A responsible decision maker examines the method before deciding how much confidence to change.

5.3.1.3 Decision quality is different from outcome quality

A careful decision can produce a disappointing outcome because of circumstances no one could fully predict. A careless decision can sometimes produce a fortunate outcome. If we judge every decision only by what happened afterward, we may reward luck and punish sound reasoning. Decision quality concerns whether the reasoning, evidence, alternatives, values, and risks were handled responsibly at the time the decision was made. Outcome quality concerns what eventually happened. We need to learn from outcomes, but we should not confuse the two.

Consider a student who chooses to pilot a small service rather than borrow a large sum to launch immediately. The pilot reveals too little demand at a sustainable price, so the student decides not to proceed. Viewed only as an outcome, no business was launched. Viewed as a decision process, the student may have protected scarce resources and gained valuable knowledge. Conversely, someone who makes an unexamined investment and earns money during an unusually favorable season might still have made a poor decision relative to the risks they assumed. Entrepreneurial education should help students recognize that responsible judgment matters even when external outcomes vary.

The research by Camuffo et al. (2020) is useful here because it examines a scientific approach in relation to entrepreneurial decision-making, not simply the number of experiments completed. The important educational question is not whether a student can use the vocabulary of testing. It is whether they can adjust the confidence they place in an idea when evidence provides reasons to do so. That is a professional skill with value long after a particular project ends.

5.3.2 Pivot, Persevere, Pause, or Stop

5.3.2.1 Four different responses to evidence

Imagine a team that has tested a new way to provide community transportation information. Participants understand the instructions, but few use the service at the time expected. The team faces a choice. It could persevere, continuing with the underlying approach while refining details because the evidence supports its central assumptions. It could pivot, changing a major element of the approach—perhaps the audience, channel, or value proposition—while retaining the overall goal of helping people access information. It could pause, delaying a consequential decision while resolving a serious uncertainty, obtaining permission, or addressing a problem with the test itself. Or it could stop, concluding that the particular approach is not justified given the evidence, resource needs, and risks.

These responses are not a simple scale from courage to defeat. They represent different ways of acting responsibly. Perseverance can be wise when evidence supports the core mechanism and remaining difficulties are manageable. A pivot can be wise when the original hypothesis is contradicted but a different opportunity appears plausible. Pausing can be wise when safety or data quality prevents a defensible conclusion. Stopping can be wise when additional investment would expose people or resources to unjustified risk. We should not praise persistence so strongly that students learn to ignore evidence and the limits of their capacity.

In entrepreneurship education, the terms pivot and persevere became prominent through Lean Startup practice, and Shepherd and Gruber (2021) analyze them as part of a wider learning framework. A pivot is not simply changing a logo or making an ordinary revision. It is a consequential change to an important assumption or aspect of the model. A company might change its primary customer segment, pricing logic, delivery method, or core promise. A professional project could similarly change the group it serves or the way the service is delivered. The conceptual distinction helps us name the magnitude of the decision and explain what evidence prompted it.

5.3.2.2 An uncomfortable lesson: a positive test can justify stopping

Suppose an entrepreneur tests a small home-delivery concept and discovers that customers genuinely appreciate it. People order, say the service saved time, and even ask for repeat deliveries. The entrepreneur has gathered meaningful evidence of customer value. But a financial review suggests that each order requires more labor and travel expense than the price can reasonably support. The test of customer value may have succeeded while the underlying business model remains unviable. Continuing unchanged could produce mounting losses even with satisfied customers.

The same pattern appears in public and nonprofit organizations. A program can help participants yet depend on staff hours that cannot be maintained. A pilot can reduce a particular inconvenience while creating an unacceptable burden elsewhere. A digital change can improve completion rates while exposing users to privacy risk. These examples illustrate why our earlier modules matter together. Module 2 taught us to understand people's real problems; Module 3 taught us to see the business or organizational system; Module 4 taught us to develop responsible solutions; Module 5 teaches us to use evidence about those interdependent claims before making a larger commitment.

A positive experimental result therefore does not mean “proceed regardless.” It means “update your understanding of the claim tested.” Perhaps customers value the service but a different delivery method is necessary. Perhaps a project should remain a limited program rather than scale. Perhaps additional funding or partnerships are essential, making a pause appropriate. Perhaps the ethical or financial constraints make stopping the responsible decision. Learning to draw these distinctions is more sophisticated than treating experimentation as a machine that produces green or red lights.

5.3.2.3 An uncomfortable lesson: a negative test can justify continuing differently

Now imagine a prototype that performs poorly during its first usability test. Several participants cannot find an important feature, and the task takes much longer than predicted. A team devoted to confirming its original design may become defensive. A different team recognizes that the test has revealed exactly where the design fails. Instead of abandoning the underlying goal, it returns to the problem statement, studies the confusion, and develops a clearer alternative. In that case, the negative result justifies iteration, a smaller change designed to improve the solution without necessarily replacing its major assumptions.

But there is a crucial boundary. If the test fails because the intended users do not actually value the promised outcome, merely polishing the interface may not solve the problem. If the service requires a resource the organization cannot obtain, a more attractive design will not create that resource. The right response depends on what the experiment challenged. This is why we spent time distinguishing value, demand, feasibility, delivery, and financial assumptions. A diagnosis should guide the remedy. Without that link, a team can remain busy revising details while avoiding the deeper issue.

A good decision memo—whether informal or formal—can state: “We expected X. We observed Y under Z conditions. The result challenges assumption A more than assumption B. Our next action is C because it addresses the largest remaining uncertainty.” The textbook is not asking you to submit such a memo; formal assignments belong in the Instructor Teaching Guide. The purpose of this structure is to make the reasoning visible so colleagues, stakeholders, or future team members can understand why a decision changed.

Entrepreneur-in-Action

Microsoft's Bing: The Small Change Experts Nearly Missed

One of the most revealing stories about experimentation does not begin with a legendary founder. It begins with an employee's relatively modest suggestion inside a large organization. In an account published by Ron Kohavi and Stefan Thomke (2017), a Microsoft employee proposed a change to the presentation of advertising headlines on Bing. The idea was considered a low priority and sat for months. When an engineer eventually ran an online controlled experiment, the results suggested a substantial revenue improvement, prompting scrutiny to rule out a measurement error. The case illustrates how a small proposal can turn out to matter more than experienced decision makers initially expect.

The story is tempting to use as proof that every overlooked idea hides a spectacular opportunity. That would be the wrong lesson. Most tests do not produce dramatic improvements, and the particular result came from a highly instrumented digital service operating at substantial scale. The educational value lies in the separation between authority and evidence. The original judgment that the idea was low priority was understandable in a setting with many competing proposals. The experiment supplied new information that justified revisiting that judgment. The team also checked whether the apparently extraordinary result could have arisen from a technical problem, which is exactly what careful investigators should do.

For students, the story offers a career lesson as well as an entrepreneurial one. You may someday be the junior employee who notices an overlooked possibility. You may also be the supervisor who must decide which ideas deserve attention. Neither role requires pretending that intuition is worthless. Experience can help prioritize questions. But when an important claim can be examined at reasonable cost, evidence offers a way to make the conversation less dependent on rank, confidence, or personal preference. It gives a worthwhile idea a fairer hearing and gives a flawed idea a more constructive exit.

Case source: Kohavi and Thomke (2017), Harvard Business Review, reporting a historical experiment at Microsoft Bing. The reported company-specific performance result is not presented here as typical of experimentation or as a forecast for student ventures.

See all 24 entrepreneurship case studies →

5.3.2.4 Your identity is bigger than your hypothesis

One reason experimentation can feel threatening is that people become attached to ideas they have worked hard to develop. A student may have spent weeks sketching a prototype and feel that unfavorable results are a judgment on their intelligence. A manager may have championed a new workflow and fear that changing it will make them look weak. A founder may have told friends and family about a business and worry that reconsidering it will disappoint them. These feelings are understandable. The solution is not to pretend we have no emotional investment. It is to separate our personal worth and professional integrity from the accuracy of a particular prediction.

A hypothesis is a claim about the world, not a statement about your value as a person. When evidence contradicts it, you have an opportunity to improve your understanding. Sometimes that means making a modest revision; sometimes it means changing a core element of the model; sometimes it means stopping. People often admire entrepreneurial determination, but determination becomes more constructive when it is directed toward understanding and creating value rather than defending every past decision. Commitment to the purpose can coexist with willingness to change the method.

The same principle helps groups learn. A classroom team that treats each test as a competition between members may discourage honest reporting. An organization that rewards only positive results may cause employees to hide problems. A team that values clear reasoning, ethical safeguards, and learning can acknowledge negative findings without humiliation. Building that culture is not easy, but the social environment around an experiment can determine whether the results improve a decision or disappear into a drawer.

Venture TrackVT-5C · Optional

Companion to 5.3.2 Pivot, Persevere, Pause, or Stop

A One-Page Experiment Decision Record

Suppose Sam later conducts an appropriately permitted, insured, small-scale demonstration of his bicycle-repair scheduling process without collecting payments. The fictional scenario shows that participants understand his service description, but several want same-day availability that he could not reliably offer. Sam could ignore their concern and begin advertising widely. A more responsible approach is to record the evidence before deciding whether to proceed.

An example of a founder's decision record

Question	Recorded finding
Assumption	Commuters can use scheduled mobile-repair appointments
Method	Authorized, nonoperational booking-flow walkthrough with selected volunteers
Observation	Most complete the flow; some require faster service than offered
Limitation	Small convenience sample; no real repairs or payments
Decision	Pause broad launch; investigate scheduling expectations and capacity
Next learning question	Which appointment windows are valued and realistically deliverable?

Notice that Sam has not treated criticism as rejection. He has discovered a possible mismatch between the customer's expectations and the business's operational capacity. A responsible next step might be clarifying the service promise, partnering with an established repair provider, or targeting cyclists who can plan maintenance in advance. It might also mean deciding that the venture is not currently workable.

The tactical skill is a disciplined founder habit: record the assumption, method, observed evidence, limits, decision, and next question before building a story about what the test supposedly proved. Learning has value even when the wisest decision is not to launch.

5.3.3 Sequencing Further Inquiry

5.3.3.1 Every answer reveals the next question

Suppose a small community organization tests whether a redesigned online explanation helps volunteers sign up for an event. The early evidence suggests that participants can understand the message and complete the form, but some volunteers still fail to arrive. The next question is no longer simply whether the webpage is readable. It may involve reminders, transportation, competing commitments, or whether the volunteer understood the event schedule. A thoughtful learning process follows the evidence to the next consequential uncertainty. It doesn't repeat the same test merely because the first one was easy to run.

This is sequencing inquiry: ordering questions so that each stage reduces a meaningful uncertainty before the organization makes more costly commitments. In some situations, problem understanding comes first, followed by solution feasibility and then actual demand. In others, a legal or safety constraint must be investigated before any further development would be appropriate. The sequence cannot be identical for every project because the risks and available knowledge differ. The practical skill is to identify what new evidence would most change your next decision and how to obtain it responsibly.

The work of Coali et al. (2024) describes tools that help entrepreneurs connect explanations with evidence-gathering processes. For an introductory learner, the deeper insight is that a good next experiment grows out of the previous interpretation, rather than being chosen from a fashionable list of startup techniques. A landing page, survey, prototype, and pilot are not achievements to collect. They are methods to use when their evidence could answer a question worth asking.

5.3.3.2 A learning roadmap, not a launch calendar

A learning roadmap identifies important unanswered questions, the evidence that could help address them, and the conditions under which a larger commitment would become reasonable. It differs from a simple launch calendar. A calendar says, “We will release the service next month.” A learning roadmap says, “Before expanding, we need reasonable evidence that the service is usable, deliverable, safe, valued, and financially supportable.” Dates can still matter, but they are linked to the quality of learning rather than treated as proof that an idea is ready.

Consider a hypothetical student exploring a bicycle-repair service. The first question might be whether a particular customer group experiences a recurring inconvenience. The second could be whether a limited service appointment solves a problem they value. The third might concern how long safe, quality repairs take under actual conditions. The fourth could involve price, transportation time, and the number of appointments the service can responsibly fulfill. Different kinds of evidence are needed at each stage. One favorable interview cannot substitute for a service-capacity test, and a successful repair demonstration cannot replace an understanding of customer demand.

The roadmap should also recognize stopping points. If a required license or insurance arrangement is unavailable, a live test may be inappropriate. If a safety question remains unresolved, the organization may need professional expertise rather than another informal experiment. If participation would require unfair pressure or intrusive data collection, a simulated or desk-based analysis may be a better educational approach. Entrepreneurship is not a contest to see who can move fastest regardless of consequences. It is the disciplined effort to create value with a responsible understanding of the people and resources involved.

5.3.3.3 Make your next step small enough to learn, important enough to matter

A useful next step has two qualities that may sound contradictory. It is small enough to limit unnecessary cost, commitment, and harm, yet important enough to answer a question that could affect the larger decision. Too small, and the activity becomes a decorative exercise: a team spends time collecting opinions about a minor detail while the core uncertainty remains untouched. Too large, and the team risks significant resources before it has the information needed to act wisely. The art of experimentation lies in finding an appropriate scale of inquiry.

Imagine Elena and the clinic staff considering what to do next. They may have evidence from a carefully supervised usability review suggesting that the revised instructions are easier for certain participants to understand. The next investigation might concern whether the messages reach people reliably and equitably, or whether additional scheduling support is required. But because clinical care, consent, privacy, and patient safety are involved, Elena's responsibility is not to independently redesign the clinic's communications or run unauthorized patient tests. Her responsibility is to raise the question clearly and support qualified staff in choosing an appropriate evaluation method. That is meaningful entrepreneurial contribution within a professional boundary.

By this point in the course, students should see experimentation less as a series of tricks for proving a startup and more as a disciplined way to make decisions under uncertainty. You have learned to recognize a problem, listen to people, understand a system, develop possible solutions, and test consequential claims. The journey now turns toward a different kind of reality: even when evidence suggests that people value an idea, can the resources and finances required to deliver it make sense? That question leads naturally into Module 6, where we will examine financial viability, costs, pricing, forecasting, and the consequences of decisions expressed in money and time.

What Can You Do?

Learn From James Lind's 1747 Comparison

In 1747, Scottish naval surgeon James Lind compared several treatments then used for scurvy among twelve sailors aboard the Salisbury. His report, published in 1753, described groups receiving different treatments under broadly shared conditions. The pair receiving citrus fruits improved notably, though the study was tiny and far from the standards of a modern randomized clinical trial. The historical record is also more complicated than a tidy legend of one experiment instantly convincing the medical establishment. Lind's own interpretations and later naval practice developed over time (Lind, 1753; Tröhler, 2005).

What makes this a useful learning story is the effort to compare explanations rather than rely only on tradition or authority. Lind faced several proposed remedies. His comparison created evidence that could distinguish among them, even though the sample and the scientific knowledge of his time imposed serious limitations. We should not imitate his medical procedures or treat historical experimentation as a model for conducting unapproved tests on people today. Instead, we can translate the reasoning into a safe, nonclinical thought experiment.

Imagine a campus library is trying to reduce confusion over locating reserved books. Three explanations are proposed: the signage is unclear, the notification email is hard to understand, and the pickup area is not visible from the entrance. What would be a fair, authorized way to examine each explanation? Which facts would need to stay comparable? How might you notice if a participant is confused? What would you conclude if improving the sign helped some people but not others? The point is not to select a winner quickly. It is to design a comparison that makes your reasoning more trustworthy without putting participants at risk.

Historical source: Lind (1753), as documented by the James Lind Library and examined by Tröhler (2005). The historical comparison illustrates controlled observation; it is not presented as a contemporary ethical or statistical model.

AI Advisor · Coming soon

Experiment Design Advisor

Coaching on your hypotheses, your experiment plan, and what your results actually justify.

What to bring

• The critical assumption behind your decision, and what kind it is: value, demand, feasibility, delivery, financial, or growth (5.1.2).
• Your hypothesis, method, measure with its denominator, guardrail, and decision rule (5.1.3, 5.2.1).
• After the test: what happened, numbers and observations, including anything unexpected (5.2.3).

What the advisor does

• Turns hopeful statements into testable predictions with a mechanism, a level, and a decision rule set in advance (5.1.3).
• Checks that the method fits the question and names what it cannot establish (5.2.1.3).
• Asks about denominators, guardrails, sample size, and stopping early when the chart looks good (5.2.2).
• Looks for confounds, selection and confirmation bias, and patterns mistaken for causes (5.2.4, 5.2.5).
• Separates decision quality from outcome quality, and ends with persevere, pivot, pause, or stop plus the next test (5.3).

What it won't do

• Grade you, or declare an idea validated.
• Invent data, run statistics on numbers you haven't collected, or declare a result significant.
• Approve tests that put people at risk: it points you to consent, privacy, and the right approvals (5.1.5).

Coming soon

About this advisor

A coach, not a grader: the work and the decisions stay yours.

Emerging practices

Experimentation as a Human Capability

As analytics platforms, automated testing systems, and AI assistants become more accessible, organizations will be able to generate and compare more possible solutions. That growth makes judgment even more important. A machine can suggest a hypothesis, create versions of a message, or summarize a dashboard; it cannot decide independently whether an experiment is fair, whether a person has freely consented, or whether a statistically detectable effect serves the organization's purpose. Some highly automated systems can also encourage continual small optimizations that distract from larger questions of value and trust. The enduring entrepreneurial capability is to connect evidence with responsibility: knowing what matters, what a test can establish, what it cannot, and when the humane decision is to change direction rather than optimize further.

Module 5 Summary: Learning to Be Wrong in Useful Ways

Module 4 invited you to explore solutions without becoming attached to the first idea. Module 5 has taken that willingness one step further. You have learned that a promising design, a positive comment, and a persuasive business model can still contain consequential assumptions. The challenge is to make those assumptions visible, choose evidence that speaks to them, and interpret the results without confusing encouragement with proof.

We began by distinguishing an assumption from a testable hypothesis. We then explored why different questions require different methods, why denominators and participant selection matter, and why a useful metric must connect to a decision. We considered how qualitative accounts and quantitative measures can complement one another, how confounds and biases can distort interpretation, and why even a controlled experiment is not a guarantee about future circumstances. Along the way, we treated privacy, consent, fairness, accessibility, and safety as essential parts of trustworthy learning rather than obstacles to be bypassed in the name of innovation.

Our final insight concerns action. An experiment is not a ritual designed to produce the word validated. It is a disciplined way to decide whether to persevere, revise, pivot, pause, or stop. Sometimes the most valuable result is evidence that helps you avoid an expensive mistake. Sometimes it points toward a better design or a previously overlooked customer need. Either way, the goal is not to win an argument with the world. It is to learn enough about the world to make a better decision within it.

As you begin Module 6, remember: evidence that people want an improvement is meaningful, but it is only part of the question. We must also examine what it costs, how money and capacity move through the system, and whether the promise can be sustained. Financial reasoning is not separate from experimentation; it is another way of testing the claims beneath our plans.

Key Terms & Definitions

A/B Test: A controlled comparison of two alternatives, often used in digital settings, where eligible participants are assigned to conditions and outcomes are compared.

Assumption: A belief being treated as true without sufficient evidence for the decision at hand.

Build–Measure–Learn: An iterative learning cycle in which a team prepares a testable offering or process, gathers observations, and adjusts its understanding and next action.

Causation: A relationship in which one factor produces a change in another factor.

Comparison Group: A group or condition used as a reference when evaluating what happened under a different condition.

Confound: A factor that makes it difficult to distinguish the effect of a proposed change from other influences.

Confirmation Bias: The tendency to favor information that supports existing beliefs while giving less attention to conflicting evidence.

Conversion Rate: The share of a defined group completing a specified action, calculated using a named numerator and denominator.

Critical Assumption: A belief whose failure would significantly undermine a proposed improvement or venture.

Decision Rule: A criterion established before a test to help interpret different possible results and choose a next action.

Denominator: The total number of relevant participants, events, or opportunities underlying a fraction or rate.

Experiment: A planned, ethically appropriate process for gathering evidence about a particular claim.

Falsifiable Hypothesis: A prediction for which conceivable observations could count against its truth.

Guardrail Metric: A measure used to check that improvements in one outcome do not create unacceptable harm or deterioration elsewhere.

Hypothesis: A specific explanation or prediction formulated so that evidence can be collected to evaluate it.

Leading Indicator: An earlier measure that may signal progress toward a later outcome.

Pilot: A limited operational implementation designed to learn about feasibility, performance, and consequences before broader use.

Pivot: A consequential change to a major assumption or element of a venture, service, or strategy in response to learning.

Qualitative Evidence: Descriptive information—such as interviews or observations—that helps explain experience, meaning, and context.

Quantitative Evidence: Numerical information—such as counts, rates, and time measures—used to describe or compare observations.

Random Assignment: Placing eligible participants into experimental conditions by a chance-based process to support fairer comparisons under suitable circumstances.

Sample Size: The number of participants, observations, or events included in a study or test.

Selection Bias: Distortion arising when the people or cases included in evidence differ in relevant ways from those about whom a broader conclusion is made.

Statistical Significance: A statistical judgment about how incompatible observations are with a specified model under a defined testing procedure; not proof of practical importance or truth.

Validation: Evidence-based support for a specified claim under stated conditions, not a universal guarantee that a solution or business will succeed.

Vanity Metric: A measure that appears impressive but provides little information for the decision or outcome that matters.

Review Questions

1. A student team receives enthusiastic comments about a prototype. What exactly has it learned, and what important assumptions may still be untested? Explain how you would distinguish interest from meaningful action.
2. Why should an experiment begin with a decision and a hypothesis rather than a favorite research tool? Describe an example from employment, education, or community life.
3. A website reports 80 signups. What additional information would you need before deciding whether this result supports a claim of meaningful demand? Explain the role of denominators, recruitment, and behavior.
4. How might confirmation bias, selection bias, or an unrecognized confound create a persuasive but misleading success story? Give an example and suggest a safeguard.
5. Describe a situation in which pausing or stopping a project would demonstrate better entrepreneurial judgment than continuing. How might you explain the decision constructively to others?

Discussion and Reflection

Discussion: What would change your mind? Choose an ordinary belief about a service or process, such as “shorter instructions always reduce confusion.” Discuss what observation would challenge the claim and what contextual factors could complicate interpretation.

Thought experiment: Two versions, one outcome. A student group observes 12 completions out of 50 attempts with one message and 18 out of 50 with another. Interpret the percentages and percentage-point difference. What might an appropriate comparison support, and what would still need to be known? Treat the numbers as hypothetical, not as a statistical proof of superiority.

Ethical reflection: When not to experiment. Consider a proposed test that changes a process people rely on for essential services. Identify potential risks, authority questions, privacy concerns, and reasons a simulated exercise or formal review may be more appropriate than a live test.

Professional application: The metric trap. Imagine an organization celebrating more website visits, more meetings, or faster calls. Identify the meaningful human outcome those measures are supposed to represent and at least one guardrail that should be monitored.

Learning reflection: Evidence and identity. Recall a time when information challenged something you believed would work. What made it difficult to reconsider your explanation? What could help you respond with more curiosity and less defensiveness in the future?

References