Statistics · AP Statistics
Collecting data
samples, experiments and bias
What data can tell you depends on how they were collected. A good sample lets you speak for a whole population; a good experiment lets you speak about cause and effect.
The figure is a school of 200 students, one ball each. The dark ones have a driver’s licence: 87 of them, a share of 0.435. Nobody doing a survey would know that number. They would ask a sample.
A computer draws 20 students at random (red), and their share with a licence, p̂, drops onto the dot plot. Every sample gives a different answer, but the answers pile up around the truth.
This chapter is about getting data you can trust: how to choose whom to measure, the ways a sample goes wrong, how to design an experiment, and which conclusions each kind of study allows.
do the data we collected tell the truth?
Data vary. Two random samples from the same population give different answers, and statistics is the study of that variation.
The 40 samples of 20 gave values of p̂ from 0.25 to 0.60. None was done wrongly: this is sampling variability, the chance variation of random samples. The values centre on 0.434, close to the true 0.435, and their standard deviation is 0.107.
So one sample’s answer is an estimate, not the truth. Before you trust a number, ask how it was produced:
| question | why it matters |
|---|
| Who was measured, and how were they chosen? | Only a random sample speaks for a population. |
| Was a treatment imposed, and who decided who got it? | Only random assignment shows cause and effect. |
| What was measured, and how was it asked? | A leading question or a bad measurement misleads, however the sample was chosen. |
| Could chance alone explain the result? | Some variation is always there; a pattern must be larger than chance would make. |
The first two are this chapter. The last is the rest of the course: whether a result is larger than random variation would plausibly produce.
population, sample and census
The population is the whole group you want to know about. A sample is the part of it you actually measure.
Name the population exactly: “all 200 students at this school”, not “teenagers”. A conclusion reaches only as far as the population that was sampled.
A census measures every member of the population. For one school it can be done; for a country it is slow and costly, and for some questions impossible (testing every light bulb until it fails leaves none to sell). A well-chosen sample is cheaper and faster, and usually good enough.
A parameter is a number that describes the population; a statistic describes a sample, and is used to estimate the parameter:
Parameters are usually unknown and fixed; statistics are known and vary from sample to sample. Proportions are p and p̂; means are μ and x̄.
Generalizing. A result can be extended from a sample to a population only if the sample was chosen at random from that population. Data from whoever was handy describe only those people.
watching, or doing something
An observational study records what its subjects already do, without changing anything. An experiment deliberately imposes a treatment and measures the response.
Does coffee before a test help? The same 24 students are grouped twice. Dark balls slept 8 hours or more.
Top: observational. The students chose. Tired students reach for coffee, so the coffee group has 3 of 12 well rested and the other group 9 of 12. If the groups score differently, is it the coffee or the sleep? We cannot tell.
Sleep is a confounding variable: it is associated with the explanatory variable (coffee) and it affects the response (the score), so the two effects cannot be separated.
Bottom: experiment. A random shuffle decided, and sleep had no say: 7 and 5 well rested. Random assignment tends to balance every other variable between the groups, the ones you know about and the ones you don’t.
So an observational study can show an association, never cause and effect. A retrospective study looks back at records of the past; a prospective one follows its subjects forward and records as it goes. Sample surveys are observational studies too.
The rule for the whole chapter: generalizing to a population needs random sampling; concluding cause and effect needs random assignment.
the simple random sample
A simple random sample (SRS) of size n is chosen so that every set of n individuals in the population has the same chance of being the sample.
With a table of random digits:
1. Label each student with a different three-digit number, 001 to 200.
2. Read the digits three at a time.
3. Skip 000, numbers above 200, and repeats, until 20 students are chosen.
| 14904 | 88119 | 15775 | 17904 | 80263 | 20028 | 06413 | 17966 | 25544 |
| digits | 149 | 048 | 811 | 915 | 775 | 179 | 048 | 026 | 320 | 028 | 064 | 131 | 796 | 625 | 544 |
|---|
| ✓ | ✓ | – | – | – | ✓ | rep. | ✓ | – | ✓ | ✓ | ✓ | – | – | – |
|---|
The first 7 students chosen are 149, 048, 179, 026, 028, 064, 131. Going on along the table, 115 groups of three give all 20. In this sample 9 of 20 have a licence: p̂ = 0.45.
With a random number generator: generate whole numbers from 1 to 200, ignoring repeats, until there are 20. Or write each name on an identical slip, mix them well in a hat and draw 20.
Samples of people are taken without replacement: once chosen, a student cannot be chosen again. Drawing with replacement puts each one back, so it could be drawn twice.
In an SRS each student’s chance of being chosen is n/N = 20/200 = 0.1. Equal chances for individuals are not enough, though: the methods on the next page give each student the same 0.1, but not every set of 20 is possible.
chance, in steps
Other random methods use chance in stages. Each can be cheaper than an SRS, or more precise.
Stratified random sample. Split the population into groups of similar individuals, the strata; take an SRS from each and combine them. Here the strata are the grades, 5 students from each.
| grade | 9 | 10 | 11 | 12 |
|---|
| with a licence | 2 of 50 | 12 of 50 | 30 of 50 | 43 of 50 |
|---|
Strata should be alike within and differ between, as the grades do here. Then every grade gets its share of the sample, and the estimate varies less from sample to sample than an SRS of the same size. This one gave p̂ = 0.35.
Cluster sample. Split the population into groups, usually by location (rooms, streets, boxes): the clusters. Choose some clusters at random and take everyone in them. Here 3 of the 25 advisory groups, all 24 students in them: p̂ = 0.417. Clusters work best when each is mixed, like a small copy of the population; their gain is cost, one visit per cluster.
Don’t confuse them: a stratified sample takes some from every stratum; a cluster sample takes all from some clusters.
Systematic random sample. Take every kth individual on a list, from a random start among the first k. Here k = 200/20 = 10, and the start is 8: students 8, 18, 28, …, 198, giving p̂ = 0.35. It is quick, and fine unless the list repeats a pattern every k places.
the kinds of bias
A sampling method is biased if it systematically favours some outcomes: over many samples its statistic misses the parameter in the same direction.
The school portal lists only grades 10 to 12; grade 9 is not on it yet. The list a sample is actually drawn from is the sampling frame, and leaving part of the population out of it is undercoverage. On the list 85 of 150 have a licence, 0.567 against 0.435 for the school, so samples from it overestimate the share.
| bias | how it happens |
|---|
| undercoverage | some of the population is less likely (or unable) to be chosen |
| voluntary response | people choose themselves (an online poll, a call-in); those with strong, often negative, opinions answer most |
| convenience | the easiest to reach are chosen: a survey handed out in the student car park finds mostly drivers |
| nonresponse | people who were chosen can’t be reached or won’t answer, and they differ from those who do |
| response bias | answers are untrue: lying about awkward behaviour, wanting to please the interviewer, misremembering |
| question wording | a leading or confusing question steers the answer: “Don’t you agree that…” |
Nonresponse is a problem of who is chosen but doesn’t answer; voluntary response is of who was never chosen at all and answered anyway.
On the exam, name the bias, explain how it arises in this context, and say which way it pushes the estimate and why.
off target, and scattered
Bias and variability are different faults. Bias is a fault of the method; sampling variability comes with every random sample.
Forty samples by each method, the parameter p = 0.435 in blue:
| method | mean of p̂ | SD of p̂ |
|---|
| SRS of 20 from the whole school | 0.430 | 0.104 |
| SRS of 20 from the portal list | 0.597 | 0.111 |
| SRS of 80 from the whole school | 0.432 | 0.042 |
Top: centred on p: unbiased, but the answers scatter. Middle (red): drawn from the portal list, they centre near 0.60. Every sample misses in the same direction; that is bias. Bottom: samples of 80, centred on p with less than half the spread of samples of 20.
A larger random sample reduces variability. It does nothing for bias: 80 students from the portal list would centre near 0.567 just the same, only more tightly. The cure for bias is a better method; the cure for variability is a larger sample.
A common mistake: “the sample was too small, so it was biased”. A small random sample is variable, not biased.
units, treatments, factors and response
An experiment does something to its subjects on purpose and measures how they respond.
Does caffeine speed up reactions? 24 volunteers each take a pill, wait, then hit a key when a light flashes.
| term | meaning | here |
|---|
| experimental units | what the treatments are applied to; people are subjects | the 24 volunteers |
| factor (explanatory variable) | what the experimenter changes | caffeine |
| levels | the values a factor is set to | a caffeine pill, a placebo |
| treatment | the condition a unit gets: one level of each factor | 2 treatments |
| response variable | what is measured on each unit | reaction time (ms) |
With two factors, the treatments are the combinations of their levels: caffeine (yes, no) and sleep (4 h, 8 h) would make 2 × 2 = 4 treatments.
The volunteers were shuffled into two groups of 12: a completely randomized design. The control group took the placebo; it shows what happens without the treatment. Mean reaction times: caffeine 235.1 ms, placebo 260.8 ms.
The volunteers are not a random sample of anyone: experiments almost always use whoever agrees to take part. That limits how far the result reaches (3.7).
comparison, random assignment, control, replication
A well-designed experiment rests on four principles.
1. Comparison. At least two treatments, or a treatment and a control group. Reaction time after caffeine means nothing without reaction time after none.
2. Random assignment. Units are assigned to treatments by chance. It balances the other variables that affect the response (sleep, age, practice) between the groups, so a difference left over is due to the treatment or to chance, not to confounding.
3. Control. Keep everything else the same for every unit: the same time of day, the same screen, the same instructions. That removes sources of variation and of confounding.
4. Replication. Enough units in each group that chance differences between them tend to even out. One volunteer per group could show anything.
Placebo and blinding. People respond to being treated at all: the placebo effect. A placebo is a dummy treatment that looks the same as the real one, so both groups expect the same. In a single-blind experiment the subjects (or those measuring them) don’t know who got which; in a double-blind one, neither do.
Describing random assignment (a frequent exam task): number the volunteers 1 to 24, write the numbers on identical slips, mix them in a hat and draw 12; those get caffeine, the rest the placebo. Say how, with what, and who gets which.
completely randomized and randomized block designs
A completely randomized design assigns all the units to treatments at random. A randomized block design first sorts them into blocks of similar units, then assigns at random within each block.
Regular coffee drinkers may barely respond to one more dose. If, by chance, most of them landed in the caffeine group, the effect would look smaller. So block on coffee habit:
| block | units | caffeine | placebo |
|---|
| coffee drinkers | 8 | 4 | 4 |
| non-drinkers | 16 | 8 | 8 |
Each block is a small completely randomized experiment, and the treatments are compared within each block. The variation caused by coffee habit is taken out of the comparison, so the treatment effect is estimated more precisely.
Block on a variable you expect to affect the response. Blocking is to experiments what stratifying is to samples: group similar units first, then use chance inside each group. The blocks are formed before the random assignment, and a block is not a treatment.
“Control what you can, block what you can’t control, randomize the rest.”
blocks of two
A matched pairs design compares two treatments on pairs of very similar units, or on one unit that gets both treatments.
Here each of 12 volunteers is tested twice, once after each pill, on different days. A coin flip decides the order: 7 took caffeine first, 5 the placebo first. People get faster with practice, so a fixed order would mix up the caffeine effect with the practice effect; randomizing the order balances it.
| volunteer | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
|---|
| caffeine | 241 | 265 | 221 | 207 | 222 | 236 | 233 | 229 | 183 | 231 | 244 | 256 |
|---|
| placebo | 251 | 274 | 215 | 239 | 243 | 244 | 261 | 218 | 213 | 248 | 245 | 251 |
|---|
| difference | 10 | 9 | -6 | 32 | 21 | 8 | 28 | -11 | 30 | 17 | 1 | -5 |
|---|
The analysis uses the differences, placebo minus caffeine, one per volunteer: mean 11.2 ms, standard deviation 14.7 ms. Each volunteer serves as their own control, so the large person-to-person differences in reaction time drop out.
Pairs can also be two units matched closely: twins, two halves of a field, two students of the same score. Then a coin flip decides which of the pair gets which treatment.
is the difference statistically significant?
A difference between treatment groups is statistically significant when it is too large to be plausibly explained by the chance of random assignment alone.
In the caffeine experiment (3.5):
Even if caffeine did nothing, the two groups of 12 would differ a little, depending on who happened to land where. How much? Suppose each volunteer would have had the same reaction time with either pill. Deal the 24 times again at random into two groups of 12, and find the difference in means. Do it 100 times.
Chance alone gave differences from -25.2 to 19.7 ms, centred near 0; none of the 100 was as large as 25.7 ms. So a difference this large is very unlikely to happen by chance alone: it is statistically significant, and it gives convincing evidence that caffeine shortens reaction time for volunteers like these.
Significant means “not plausibly chance”, not “large” or “important”. And a difference that is not significant does not show that the treatment has no effect; the study may just be too small to see it.
what a study’s design allows you to conclude
Random selection lets you generalize to the population the sample came from. Random assignment lets you conclude cause and effect. Each does one job.
| random assignment | no random assignment |
|---|
| random selection | cause and effect, for the population | association only, for the population |
|---|
| no random selection | cause and effect, for units like those studied | association only, for those studied |
|---|
The caffeine experiment (lit): volunteers, randomly assigned. Caffeine caused the faster reactions, but only for people like these volunteers.
The licence survey: an SRS of the school, no treatment. The sample’s share estimates the school’s; nothing about cause.
The coffee study that let students choose (3.2): neither. It shows an association among those students, which sleep may explain.
Common mistakes: claiming a randomized experiment on volunteers applies to everyone; claiming cause from a large random survey. A big sample does not replace random assignment, and random assignment does not replace random selection.
← Exploring two-variable data · Probability and random variables →