Math III · S-IC.6

Evaluating Reports Based on Data

Students need to judge whether data-based claims are supported by study design, analysis, uncertainty, and honest interpretation.

Concept Statistics and Probability

Domain Making Inferences and Justifying Conclusions

Read time 5 minutes

What this learning objective is really asking you to learn

This objective asks students to evaluate reports based on data. This is the capstone of the inference sequence. Students are no longer only computing statistics or running simulations. They are judging whether a data-based report is trustworthy.

A report based on data might be a news article, scientific summary, poll, medical study, marketing claim, school report, business dashboard, public-policy analysis, or social-media infographic. The report may include tables, graphs, averages, percentages, margins of error, study descriptions, or causal claims.

Evaluating a report means asking hard questions:

What question was the study trying to answer?
What population is being discussed?
How were data collected?
Was the sample random?
Was there random assignment?
Is the study a survey, experiment, or observational study?
What variables were measured?
Are the graphs honest?
Are margins of error or uncertainty reported?
Are the conclusions stronger than the evidence supports?
Could there be bias or confounding?
Are practical significance and statistical significance distinguished?

This objective is about statistical citizenship. Students should be able to read a claim like “new program improves test scores by 20%” and ask: compared with what? Was there a control group? Were students randomly assigned? How large was the sample? Was the result statistically significant? Could selection bias explain it? What does “20%” mean?

The goal is not cynicism. The goal is disciplined trust. Good data deserve attention. Weak data deserve caution. Misleading claims deserve challenge.

Why students should learn this math

Students should learn to evaluate reports based on data because data claims influence real decisions. People vote based on polls. Patients consider treatments based on studies. Schools adopt programs based on reported results. Businesses change products based on analytics. Governments make policy based on data. Consumers respond to claims about risk, health, price, and performance.

Bad data interpretation can cause real harm. A misleading graph can exaggerate a trend. A biased survey can misrepresent public opinion. An observational study can be reported as causal proof. A tiny effect can be advertised as a breakthrough. A missing denominator can distort risk. A cherry-picked time range can create a false story.

Students need tools to defend themselves against weak claims. They should know that a sample statistic is not automatically a population fact. They should know that association is not causation. They should know that graph scales can mislead. They should know that uncertainty matters. They should know that study design determines the strength of conclusions.

This is one of the most practical objectives in the entire curriculum. It is not only for future statisticians. Every adult lives in a world of claims based on data. The ability to evaluate those claims is part of being educated.

The “why” is that data do not speak for themselves. People interpret data, and those interpretations can be honest, careless, or manipulative. Students need to tell the difference.

The historical machinery: statistics as public evidence

As governments, sciences, businesses, and media began using more data, statistical reports became central to public reasoning. Census data, medical trials, economic indicators, election polls, crime statistics, education studies, and market research all shape decisions. This created a need for statistical literacy among non-specialists.

The history of statistics includes both great successes and major failures. Good data have improved medicine, agriculture, engineering, and public policy. Bad sampling, biased measurement, and overconfident causal claims have also misled people. The lesson is that data require method.

Evaluating reports is the public-facing side of statistics. It asks whether the methods justify the claims. This is the habit students need beyond the classroom.

Where this fits in the big map of mathematics

This objective follows sample inference, margins of error, study design, simulation, and randomized experiments. It asks students to combine all those ideas to critique real reports.

It connects to graph interpretation and data displays from earlier statistics work.

It connects to probability because uncertainty and randomness shape inference.

It connects to modeling because every report is based on choices about variables, measures, and assumptions.

It connects to media literacy, science literacy, and civic reasoning.

The big-map role is evidence evaluation. Students learn to judge the quality of statistical claims.

How to execute the skill technically

Use a report-evaluation checklist:

Identify the claim.
Identify the population.
Identify the data source.
Determine study type: survey, experiment, or observational study.
Check sampling method.
Check whether random assignment was used if causation is claimed.
Check sample size and uncertainty.
Examine graphs for scale, missing context, or distortion.
Look for confounding variables.
Compare conclusion strength to evidence strength.

Example: A report says “Students who eat breakfast score 15% higher, proving breakfast improves test performance.”

Evaluation:

Likely observational unless breakfast was assigned.
Students who eat breakfast may differ in sleep, home support, income, schedule, or health.
The data may show association, but “proving” causation is too strong.
A randomized experiment or stronger design would be needed for causal proof.

Better conclusion: “In this study, breakfast eating was associated with higher test scores, but causation is not established.”

Graph critique example

A graph shows sales increasing from 102 to 106 units, but the vertical axis starts at 100, making the increase look huge. The graph is not necessarily false, but the scale exaggerates the visual effect. A responsible report should make the scale clear and perhaps show a full or context-appropriate axis.

Students should learn that graphs can be technically accurate yet visually misleading.

More report-evaluation examples

Report claim: “A survey shows 80% of students hate the new schedule.” The survey was posted on the student complaint forum.

Evaluation: This is likely a voluntary response sample, not a random sample. Students with strong negative feelings may be more likely to respond. The report may show dissatisfaction among forum respondents, but it should not be generalized to all students without caution.

Report claim: “People who take Supplement X lose more weight, so Supplement X causes weight loss.” The study compares customers who bought Supplement X with people who did not.

Evaluation: This is observational unless researchers assigned supplement use. Supplement buyers may differ in motivation, diet, exercise, income, or health behavior. The data may show association, but causal language is too strong.

Report claim: “A randomized trial found Treatment A had a 12% higher recovery rate than Treatment B, with the difference rarely occurring in randomization simulations.”

Evaluation: This is much stronger evidence, assuming random assignment, appropriate measurement, and no major flaws. The report should still include sample size, uncertainty, and practical importance.

Graph and percentage traps

A report may use relative change to exaggerate. If risk rises from 1 in 10,000 to 2 in 10,000, that is a 100% relative increase but still a very small absolute risk increase. Both facts matter.

A graph may truncate axes or use unequal intervals. Students should check scale, labels, source, and whether a graph shows counts, percentages, or rates.

Funding and incentives

Students should be aware of incentives. Industry-funded research is not automatically false, and advocacy-group reports are not automatically wrong. But source and incentives matter. A careful evaluator asks whether methods are transparent, data are available, and conclusions match evidence.

Problem Library

Problems in the App From This Objective

144 problems across 12 archetypes in the app.

separate claim from evidence.

12 problems Warmup Practice Mixed Review Assessment

Problem 1

Identify claim made in data report A headline says the supplement improves memory based on study results.

Problem 2

Identify claim made in data report ad says 8 out of 10 dentists recommend brand X.

Open in simulator

Problem 3

Identify claim made in data report article says city commute times are rising.

Problem 4

Identify claim made in data report News report states that a new vaccine is 95% effective against a certain virus.

Problem 5

Identify claim made in data report A study found that students who use a tutor improve their grades by an average of 15%.

Problem 6

Identify claim made in data report Company X's quarterly report highlights that their market share grew by 3%.

Problem 7

Identify claim made in data report A political candidate claims that crime rates have decreased by 10% in the last year.

Problem 8

Identify claim made in data report An environmental group states that air pollution levels have dropped significantly over the past decade.

Problem 9

Identify claim made in data report A financial advisor suggests that investing in real estate yields higher returns than stocks.

Problem 10

Identify claim made in data report A health blog asserts that regular exercise reduces the risk of heart disease by half.

Problem 11

Identify claim made in data report A university study concludes that early childhood education leads to better long-term academic success.

Problem 12

Identify claim made in data report A car manufacturer advertises that their new model is 20% more fuel-efficient.

parse study description.

12 problems Warmup Practice Mixed Review Assessment

Problem 13

Identify population, sample, and variables in report survey 1200 adults about weekly exercise.

Problem 14

Identify population, sample, and variables in report experiment assigns 60 students to two apps and measures score gain.

Problem 15

Identify population, sample, and variables in report observational study records diet and cholesterol for 500 patients.

Problem 16

Identify population, sample, and variables in report survey 500 high school students about their preferred college major.

Problem 17

Identify population, sample, and variables in report experiment gives 100 mice different diets and measures weight change.

Problem 18

Identify population, sample, and variables in report observational study tracks 200 birds' migration patterns and nesting success.

Problem 19

Identify population, sample, and variables in report survey 150 local businesses on their hiring plans for the next quarter.

Problem 20

Identify population, sample, and variables in report experiment applies two different pesticides to 80 tomato plants and records yield.

Problem 21

Identify population, sample, and variables in report observational study records daily screen time and academic grades for 300 teenagers.

Problem 22

Identify population, sample, and variables in report survey 250 dog owners about their pet food preferences.

Problem 23

Identify population, sample, and variables in report experiment tests two different website layouts on 120 users and measures time spent on site.

Open in simulator

Problem 24

Identify population, sample, and variables in report observational study analyzes 400 historical weather records and crop failures.

assess randomness, bias, and generalizability.

12 problems Warmup Practice Mixed Review Assessment

Problem 25

Evaluate sampling method in report random digit dialing reaches adults but excludes people without phones.

Problem 26

Evaluate sampling method in report website readers self-select into poll.

Problem 27

Evaluate sampling method in report random sample from full customer list.

Problem 28

Evaluate sampling method in report survey only morning shoppers.

Problem 29

Evaluate sampling method in report every 5th student on the school roster was selected for a survey.

Problem 30

Evaluate sampling method in report randomly selected 50 participants from each of three income brackets.

Problem 31

Evaluate sampling method in report randomly selected 10 neighborhoods and surveyed all households within those neighborhoods.

Problem 32

Evaluate sampling method in report interviewed people leaving a specific movie theater on a Saturday night.

Problem 33

Evaluate sampling method in report participants in a study on rare hobbies were asked to recruit others with the same hobby.

Problem 34

Evaluate sampling method in report online survey distributed via social media, excluding those without internet access.

Open in simulator

Problem 35

Evaluate sampling method in report interviewers were instructed to find 50 men and 50 women, filling quotas by approaching people on the street.

Problem 36

Evaluate sampling method in report survey of all registered voters in the city.

distinguish random assignment and confounding.

12 problems Warmup Practice Mixed Review Assessment

Problem 37

Evaluate study design for causal conclusions in randomized trial assigns treatment and placebo.

Problem 38

Evaluate study design for causal conclusions in observational study finds coffee drinkers sleep less.

Problem 39

Evaluate study design for causal conclusions in before-after study with no control group.

Problem 40

Evaluate study design for causal conclusions in random assignment but high dropout.

Problem 41

Evaluate study design for causal conclusions in a large observational study finds a strong correlation between daily meditation and reduced stress levels.

Problem 42

Evaluate study design for causal conclusions in participants self-select into either a new experimental diet group or a control group.

Problem 43

Evaluate study design for causal conclusions in a quasi-experimental study compares crime rates before and after a new policing strategy in one district, using a similar district as a comparison.

Problem 44

Evaluate study design for causal conclusions in a randomized controlled trial with proper blinding tests the efficacy of a new vaccine.

Problem 45

Evaluate study design for causal conclusions in a cohort study tracks smoking habits and lung health over several decades.

Problem 46

Evaluate study design for causal conclusions in an intervention study provides job training to all unemployed individuals in a city and compares their employment rates to historical data.

Problem 47

Evaluate study design for causal conclusions in patients are assigned to a new therapy based on their physician's recommendation rather than randomly.

Problem 48

Evaluate study design for causal conclusions in a cross-sectional survey finds that people who own pets report higher levels of happiness.

Open in simulator

inspect scale, omitted data, and visualization choices.

12 problems Warmup Practice Mixed Review Assessment

Problem 49

Evaluate graph or summary presentation bar chart y-axis starts at 90 making small difference look huge.

Problem 50

Evaluate graph or summary presentation pie chart percentages sum to 120 percent.

Problem 51

Evaluate graph or summary presentation average reported without spread for skewed data.

Problem 52

Evaluate graph or summary presentation graph omits important comparison group.

Problem 53

Evaluate graph or summary presentation line graph with a y-axis starting above zero.

Problem 54

Evaluate graph or summary presentation time-series graph with non-uniform intervals on the x-axis.

Problem 55

Evaluate graph or summary presentation comparison of groups using raw counts despite different population sizes.

Problem 56

Evaluate graph or summary presentation graph displaying only a limited data range to support a specific narrative.

Problem 57

Evaluate graph or summary presentation 3D bar chart where perspective makes distant bars appear smaller.

Problem 58

Evaluate graph or summary presentation summary implying a causal link from observed correlation.

Problem 59

Evaluate graph or summary presentation generalization about a population based on a very small sample.

Open in simulator

Problem 60

Evaluate graph or summary presentation stacked bar chart with categories that are not mutually exclusive.

interpret uncertainty correctly.

12 problems Warmup Practice Mixed Review Assessment

Problem 61

Evaluate uncertainty use in report candidate leads 51 to 49 with margin of error 4.

Problem 62

Evaluate uncertainty use in report estimate 63 percent reported with no sample size or margin.

Problem 63

Evaluate uncertainty use in report mean 20±2 units.

Problem 64

Evaluate uncertainty use in report report says margin of error accounts for biased wording.

Problem 65

Evaluate uncertainty use in report 95% confidence interval for average weight is 70kg to 75kg.

Problem 66

Evaluate uncertainty use in report Group A mean 50±3, Group B mean 55±3.

Problem 67

Evaluate uncertainty use in report Treatment A improved scores by 10 points (CI 8-12), Treatment B improved scores by 5 points (CI 3-7).

Problem 68

Evaluate uncertainty use in report The exact average income is $65,000 based on our survey.

Problem 69

Evaluate uncertainty use in report The survey found 60% support with a 3% margin of error, so it's very precise.

Problem 70

Evaluate uncertainty use in report Our small pilot study of 20 people showed 80% effectiveness with no margin of error.

Open in simulator

Problem 71

Evaluate uncertainty use in report Since the 99% confidence interval for the effect is 0.5 to 1.5, we are absolutely certain the effect is positive.

Problem 72

Evaluate uncertainty use in report The 95% confidence interval for the treatment effect is -2 to 5.

compare sample to target population.

12 problems Warmup Practice Mixed Review Assessment

Problem 73

Evaluate whether conclusion overgeneralizes in survey of one gym's members claims all adults prefer morning workouts.

Problem 74

Evaluate whether conclusion overgeneralizes in random sample of city residents used to infer city opinion.

Problem 75

Evaluate whether conclusion overgeneralizes in college sample used to claim all teenagers.

Problem 76

Evaluate whether conclusion overgeneralizes in customer feedback from purchasers generalized to noncustomers.

Problem 77

Evaluate whether conclusion overgeneralizes in survey of science museum visitors claims all children enjoy learning about space.

Problem 78

Evaluate whether conclusion overgeneralizes in online news poll about political candidates claims to represent national opinion.

Problem 79

Evaluate whether conclusion overgeneralizes in random sample of students from Northwood High used to infer preferences of Northwood High students.

Problem 80

Evaluate whether conclusion overgeneralizes in study of marathon runners concludes all adults benefit from daily intense exercise.

Problem 81

Evaluate whether conclusion overgeneralizes in feedback from users of a new social media app used to claim all smartphone users prefer its features.

Problem 82

Evaluate whether conclusion overgeneralizes in survey of residents in a wealthy suburb used to make claims about an entire state's economic status.

Problem 83

Evaluate whether conclusion overgeneralizes in random sample of employees from a large corporation used to gauge morale within that corporation.

Open in simulator

Problem 84

Evaluate whether conclusion overgeneralizes in study on college freshmen claims to represent dietary habits of all young adults.

identify correlation-causation errors.

12 problems Warmup Practice Mixed Review Assessment

Problem 85

Evaluate whether conclusion overstates causation in people who eat breakfast score higher, so breakfast causes higher scores.

Problem 86

Evaluate whether conclusion overstates causation in randomized experiment shows treatment group improved more.

Problem 87

Evaluate whether conclusion overstates causation in cities with parks have lower stress, so parks reduce stress.

Problem 88

Evaluate whether conclusion overstates causation in survey finds screen time associated with sleep problems.

Problem 89

Evaluate whether conclusion overstates causation in Students who attend more lectures get better grades, so attending lectures causes higher grades.

Problem 90

Evaluate whether conclusion overstates causation in Areas with more ice cream sales also have more drownings, proving ice cream causes drowning.

Problem 91

Evaluate whether conclusion overstates causation in A survey showed people who meditate regularly experience less stress, so meditation reduces stress.

Problem 92

Evaluate whether conclusion overstates causation in Countries with higher chocolate consumption win more Nobel Prizes, implying chocolate makes people smarter.

Problem 93

Evaluate whether conclusion overstates causation in Workers who use standing desks report fewer back problems, meaning standing desks prevent back pain.

Problem 94

Evaluate whether conclusion overstates causation in A randomized controlled trial found that a new fertilizer increased crop yield significantly.

Problem 95

Evaluate whether conclusion overstates causation in People who own more books tend to earn higher incomes, so owning books leads to wealth.

Problem 96

Evaluate whether conclusion overstates causation in Communities with more streetlights have lower crime rates, indicating streetlights deter crime.

Open in simulator

ask for sample size, selection, design, measurements, uncertainty.

12 problems Warmup Practice Mixed Review Assessment

Problem 97

Identify missing information needed to judge report 70 percent support proposal, no methods given.

Open in simulator

Problem 98

Identify missing information needed to judge report new treatment worked better, no design details.

Problem 99

Identify missing information needed to judge report graph shows increase, axes cropped.

Problem 100

Identify missing information needed to judge report 80% of users are satisfied with the new software update.

Problem 101

Identify missing information needed to judge report New supplement boosts energy levels.

Problem 102

Identify missing information needed to judge report Students using the new curriculum scored higher on standardized tests.

Problem 103

Identify missing information needed to judge report Our new battery lasts twice as long as competitors'.

Problem 104

Identify missing information needed to judge report The local economy grew by 5% last quarter.

Problem 105

Identify missing information needed to judge report Our new filter reduces pollutants by 90%.

Problem 106

Identify missing information needed to judge report Crime rates decreased after the new policy was implemented.

Problem 107

Identify missing information needed to judge report Experiment shows light therapy improves sleep quality.

Problem 108

Identify missing information needed to judge report More people now support the proposed city park.

assess evidence quality and consistency.

12 problems Warmup Practice Mixed Review Assessment

Problem 109

Compare two reports about same issue random sample poll vs website poll with conflicting results.

Problem 110

Compare two reports about same issue large observational study vs small randomized experiment.

Problem 111

Compare two reports about same issue two polls with overlapping margins of error.

Problem 112

Compare two reports about same issue a clinical trial with 50 patients vs a similar trial with 500 patients.

Problem 113

Compare two reports about same issue a survey using snowball sampling vs a survey using stratified random sampling.

Problem 114

Compare two reports about same issue a study without a control group vs a study with a well-matched control group.

Problem 115

Compare two reports about same issue a report based on subjective interviews vs a report based on objective physiological measurements.

Problem 116

Compare two reports about same issue a drug study that was not blinded vs a double-blinded drug study.

Problem 117

Compare two reports about same issue an initial groundbreaking study vs several subsequent replication studies with null findings.

Problem 118

Compare two reports about same issue a 3-month intervention study vs a 5-year longitudinal study on the same health outcome.

Open in simulator

Problem 119

Compare two reports about same issue a research study funded by a pharmaceutical company vs an identical study funded by a government agency.

Problem 120

Compare two reports about same issue a single case study vs a systematic review of randomized controlled trials.

cite strengths, weaknesses, and valid conclusion.

12 problems Warmup Practice Mixed Review Assessment

Problem 121

Write critique of data-based claim ad uses customer survey to claim best product.

Problem 122

Write critique of data-based claim headline claims causation from observational data.

Problem 123

Write critique of data-based claim poll reports estimate with margin of error.

Problem 124

Write critique of data-based claim drug company claims new medication cures disease based on a small, uncontrolled trial.

Problem 125

Write critique of data-based claim school district claims new curriculum improves test scores based on a comparison of current students to previous year's students.

Problem 126

Write critique of data-based claim news report states increasing ice cream sales cause an increase in crime rates.

Open in simulator

Problem 127

Write critique of data-based claim local activist group claims a factory is causing a specific illness based on a few residents living near the factory reporting the illness.

Problem 128

Write critique of data-based claim online poll claims 80% of the population supports a new policy.

Problem 129

Write critique of data-based claim sports commentator claims a player is 'clutch' because they scored the winning point in 3 out of 5 games.

Problem 130

Write critique of data-based claim company claims its new product is preferred by 'most people' based on a taste test conducted only with employees.

Problem 131

Write critique of data-based claim study suggests coffee drinkers live longer based on observing a large group over many years.

Problem 132

Write critique of data-based claim manufacturer claims a new car model is 'safest on the road' based on crash tests at 30 mph.

catch bias, causation, uncertainty, graph, and overclaiming mistakes.

12 problems Warmup Practice Mixed Review Assessment

Problem 133

Correct flawed report interpretation poll lead within margin of error proves winner.

Problem 134

Correct flawed report interpretation observational association proves cause.

Problem 135

Correct flawed report interpretation truncated graph proves huge change.

Problem 136

Correct flawed report interpretation random assignment means sample represents population.

Problem 137

Correct flawed report interpretation correlation between ice cream sales and drownings means ice cream causes drowning.

Problem 138

Correct flawed report interpretation online survey results represent the general public's opinion.

Problem 139

Correct flawed report interpretation a study of 5 people found a cure, so it must be effective.

Problem 140

Correct flawed report interpretation a bar chart with a non-zero y-axis starting point shows massive growth.

Problem 141

Correct flawed report interpretation a trend observed over 5 years will continue indefinitely.

Problem 142

Correct flawed report interpretation a study found people who drink coffee live longer, so coffee extends life.

Open in simulator

Problem 143

Correct flawed report interpretation a survey asking 'Do you agree that our excellent product is the best?' accurately measures customer satisfaction.

Problem 144

Correct flawed report interpretation a statistically significant result means the effect is large and important.