The shape of the sampling distribution

POL 211 · Week 6

Juan F. Tellez

University of California, Davis

November 3, 2026

Recap

Last week: the sampling distribution is centered on the truth

\(E[\hat{\Delta}] = \Delta\): the difference in callback rates is unbiased for the gap. Randomizing the names is what makes that gap the effect of the name

Last week: the standard error is the width of the distribution, if the résumés are independent

\[ \text{SD}(\hat{\Delta}) = \sqrt{ \frac{\overbrace{p_{\text{AA}}(1 - p_{\text{AA}})}^{\text{a coin flip's variance}}}{\underbrace{n_{\text{AA}}}_{\text{averaging independent flips}}} \overbrace{+}^{\text{independent groups}} \frac{p_{\text{white}}(1 - p_{\text{white}})}{n_{\text{white}}} } \]

The classic standard error assumes independence. Without it, the variance of a sum picks up a covariance term for every pair of résumés, and this SE is too narrow (more at the end of today)

But that formula needs \(p\), and all we have is \(\hat{p}\)

We never see \(p_{\text{AA}}\) or \(p_{\text{white}}\). So we plug in the callback shares from the one run of data we have:

\[ \widehat{SE}(\hat{\Delta}) = \sqrt{\frac{\hat{p}_{\text{AA}}(1 - \hat{p}_{\text{AA}})}{n_{\text{AA}}} + \frac{\hat{p}_{\text{white}}(1 - \hat{p}_{\text{white}})}{n_{\text{white}}}} = \sqrt{\frac{0.064(1 - 0.064)}{2{,}435} + \frac{0.097(1 - 0.097)}{2{,}435}} = 0.0078 \]

So the standard error in week 1’s table is itself an estimate: a guess, from one run of data, at the width of a distribution we can’t see (Wooldridge 2023, 2)

Is that OK? \(\hat{p}\) isn’t \(p\). Today’s first question: why would plugging in \(\hat{p}\) work?

Every sampling distribution we’ve built looks like a bell

But no single résumé’s outcome looks like a bell. A callback is a 0 or a 1

Today’s second question: why does the sampling distribution look like a bell?

Today: why \(\hat{p}\) works, and why the bell shape

Today

Question Answer
Where does the sampling distribution sit? On the truth (week 5)
How wide is it? \(\sigma / \sqrt{n}\), if the draws are independent (week 5)
Why can we plug in \(\hat{p}\) for \(p\)? Averages settle on their expectation: the law of large numbers
How do we describe a bell? The normal distribution (which needs continuous random variables first)
Why is the sampling distribution a bell? The central limit theorem
What does the bell buy us? About 95% of studies land within 2 standard errors of the truth

The law of large numbers

iid: independent and identically distributed

Last week’s standard error leaned on independence: it wiped out the covariance term for every pair of résumés. Statistics usually states that assumption as iid:

  • Independent: one draw tells you nothing about another (week 5)
  • Identically distributed: every draw comes from the same distribution

The résumé study isn’t identically distributed: each résumé has its own \(p_i\). That’s fine. Week 5 showed the share still centers on the average \(p_i\), and today’s results allow it too, as long as no handful of draws dominates the average

Independence is the more common violation. The end of today shows what happens without it

The law of large numbers: averages settle on the expectation

Take iid draws \(X_1, X_2, \ldots, X_n\), each with mean \(\mu\). As \(n\) grows, their average gets closer and closer to \(\mu\):

\[ \bar{X}_n \rightarrow \mu \quad \text{as } n \rightarrow \infty \]

For coin flips, \(\mu = p\) and the average is the share of 1s, so \(\hat{p} \rightarrow p\)

This holds for any distribution with a mean: a die, a thermometer score, a protest’s size

The precise version says the chance \(\bar{X}_n\) lands more than any tiny distance from \(\mu\) shrinks to zero (“convergence in probability”). Week 7 calls an estimator that does this consistent

Twenty people roll a die: every running average closes in on 3.5

Each line is one person rolling 2,000 times. After every roll, they compute the average of all their rolls so far. Orange: the expectation of one roll, \(E[X] = 3.5\)

The funnel’s width is week 5’s formula: \(\sigma / \sqrt{n}\)

Gold: \(3.5 \pm 2\sigma/\sqrt{n}\), with \(\sigma =\) 1.71 for one roll. Ten times the rolls, a third of the width. Week 5’s formula is the LLN’s speed

Chebyshev’s inequality: a small spread means few values can be far away

Picture a classroom. You know two things: average height and the standard deviation (SD = 3). That means their variance (the average squared distance from the mean) is 9

Chebyshev says that if you know the spread, only so many kids can be really far from the average. The reason is that variance works like a budget. Each kid “spends” from it according to the square of their distance from the average. A kid \(3\delta\) away costs 9 units, so a lot of kids that far out would “use up” more spread than the class has

Formally, for any distribution at all:

\[ P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2} \]

Distance from the mean Chebyshev’s cap, any distribution A normal distribution
2 or more SDs 1/4 = 25% 4.6%
3 or more SDs 1/9 = 11% 0.3%
10 or more SDs 1/100 = 1% about 0%

It assumes nothing about the shape, only a finite variance

Apply Chebyshev to the average, and you get the LLN

Averaging cancels noise, so the spread of the average shrinks: \(\text{Var}(\bar{X}) = \sigma^2 / n\) (the funnel)

Chebyshev, applied to \(\bar{X}\), bounds the chance it misses the truth by \(\varepsilon\) or more:

\[ P(|\bar{X}_n - \mu| \geq \varepsilon) \leq \frac{\sigma^2}{n \, \varepsilon^2} \]

Hold \(\varepsilon\) fixed, however tiny, and let \(n\) grow: the right side goes to 0. That’s the law of large numbers

A die: \(\sigma^2 = 2.92\) (Problem Set 5). Take \(\varepsilon = 0.1\), so \(\varepsilon^2 = 0.01\):

\[ n = 1{,}000: \;\; \frac{2.92}{1{,}000 \times 0.01} = \frac{2.92}{10} \approx 0.29 \qquad\qquad n = 10{,}000: \;\; \frac{2.92}{10{,}000 \times 0.01} = \frac{2.92}{100} \approx 0.03 \]

At most 29% of averages of 1,000 rolls miss 3.5 by 0.1 or more; at most 3% of averages of 10,000

A small spread means the average can’t wander far. More data shrinks the spread toward zero. So with enough data, the average can’t wander far from the truth

Back to the recap: the plug-in works because \(\hat{p}\) is an average

\(\hat{p}_{\text{AA}}\) is the average of 2,435 coin flips, each with expectation \(p_{\text{AA}}\). So the LLN says \(\hat{p}_{\text{AA}} \rightarrow p_{\text{AA}}\)

And if \(\hat{p}\) lands close to \(p\), then \(\hat{p}(1 - \hat{p})\) lands close to \(p(1 - p)\). So the plug-in standard error lands close to the true width:

\[ \widehat{SE}(\hat{\Delta}) \rightarrow \text{SD}(\hat{\Delta}) \quad \text{as } n \rightarrow \infty \]

With a few thousand résumés, the plug-in barely matters. With a few dozen, it can

Check it: the plug-in standard error closes in on the true width as \(n\) grows

Each study plugs its own \(\hat{p}\)’s into the formula; the true width plugs in the true \(p\)’s, which we know because we wrote the DGP. With 200 résumés, 65% of studies land within 10% of the truth. With 4,870, 100% do

The LLN is also why every simulation works

Week 4’s 1,000 reruns of the résumé study: the average of the estimates so far settles on the truth. Every “check it by simulation” slide leaned on this

The canvass DGP from PS5

Step 1, the systematic part: each voter’s expected score

set.seed(211)
canvass = tibble(dem = rbinom(1000, size = 1, prob = 0.5),          # a coin flip: Democrat or not
                 canvassed = rbinom(1000, size = 1, prob = 0.5)) |> # a coin flip: canvassed or not
  mutate(systematic = 40 +              # an uncanvassed non-Democrat
                      5 * canvassed +   # a knock on the door: +5 points (the effect)
                      20 * dem)         # a registered Democrat: +20 points
canvass
# A tibble: 1,000 × 3
    dem canvassed systematic
  <int>     <int>      <dbl>
1     0         0         40
2     1         1         65
3     1         0         60
4     0         1         45
5     0         0         40
# ℹ 995 more rows

The average score for voters with these inputs. It takes just four values: 40, 45, 60, or 65

Step 2, the random part: each voter’s noise

canvass = canvass |>
  mutate(noise = rnorm(1000, mean = 0, sd = 20),   # any number, centered on 0
         therm = systematic + noise)               # the score we'd actually see
canvass
# A tibble: 1,000 × 5
    dem canvassed systematic  noise therm
  <int>     <int>      <dbl>  <dbl> <dbl>
1     0         0         40  27.1   67.1
2     1         1         65 -12.6   52.4
3     1         0         60  17.1   77.1
4     0         1         45   8.90  53.9
5     0         0         40  13.6   53.6
# ℹ 995 more rows

rnorm() draws each voter’s noise. A real researcher sees dem, canvassed, and therm, not the two parts

One run of the study gives one estimate

run_1 = canvass |>
  group_by(canvassed) |>
  summarize(average_therm = mean(therm))
run_1
# A tibble: 2 × 2
  canvassed average_therm
      <int>         <dbl>
1         0          50.1
2         1          53.8

The difference in means: 3.8 points. The true effect we built in: 5

With a coin flip, \(p\) told us everything. The canvass also needs the noise

With the résumés we only ever talked about \(p\). For a coin flip, \(p\) pins down everything: the whole distribution (\(P(Y = 1) = p\)), the center (\(p\)), and the spread (\(p(1 - p)\)). The random part came along for free

The canvass is different. The systematic part (40, 45, 60, or 65) gives the center but says nothing about the spread. The spread lives in the noise

And we need that spread: the random part is what makes estimates vary from study to study (week 4), so its variance is what goes into the standard error.

The noise can be any number: 3.7, −12.04, 26.519… So unlike a Bernoulli, it is a continuous random variable

Continuous random variables

A coin flip has two outcomes. A thermometer score has infinitely many

Discrete: a list of separate values Continuous: any value in a range
Examples A callback (0/1), votes cast, protests in a year A thermometer score, income, a vote share, days until a ceasefire breaks
The distribution A probability for each value A density: a curve
Probability of a single value Can be big: \(P(Y = 1) = 0.08\) Zero
Probability of a range Add up the bars Area under the curve

Make the bars thinner and thinner and they become a curve

The orange curve is the density, \(f(x)\): what the histogram becomes with infinitely many draws and infinitely thin bars

Probability is the area under the curve

# the share of the 100,000 draws that land between 0 and 20
noise |> summarize(share = mean(x > 0 & x < 20))
# A tibble: 1 × 1
  share
  <dbl>
1 0.342

Any one exact value has probability zero

# how many of the 100,000 draws are exactly 3.7?
noise |> summarize(exactly_3.7 = sum(x == 3.7))
# A tibble: 1 × 1
  exactly_3.7
        <int>
1           0

A single point has no width, so it has no area: \(P(\text{noise} = 3.7) = 0\)

So with continuous variables we always ask about ranges: \(P(3.65 < X < 3.75)\), \(P(X > 20)\)

Expectation and variance: the sum becomes an integral

Discrete Continuous
Expectation \(E[X] = \sum_x x \cdot P(X = x)\) \(E[X] = \int x \cdot f(x) \, dx\)
Variance \(\text{Var}(X) = \sum_x (x - E[X])^2 \, P(X = x)\) \(\text{Var}(X) = \int (x - E[X])^2 f(x) \, dx\)

\(\int\) is a sum over infinitely many, infinitely thin slices. Same idea: each value, weighted by how likely it is

And every rule from week 5 still holds: \(E[X + Y] = E[X] + E[Y]\), \(\text{Var}(kX) = k^2 \text{Var}(X)\), the sum rule with its covariance term, \(\sigma^2 / n\)

The normal distribution

The normal is a bell, set by two numbers: its mean and its SD

\(X \sim N(\mu, \sigma^2)\): the mean \(\mu\) moves the bell, the SD \(\sigma\) stretches it. Symmetric, with thin tails

Each unit’s outcome is a random variable, centered on its systematic part

The noise is \(N(0, 20^2)\). Fix a voter’s inputs and the systematic part is just a number. For a non-Democrat nobody canvassed:

\[ \texttt{therm} = \underbrace{40 + 5(0) + 20(0)}_{= \, 40} + \text{noise} \;\sim\; N(40, 20^2) \]

One unit’s outcome Its inputs Systematic part Its distribution
Résumé #1’s callback, \(Y_1\) Its name, experience, age, distance \(p_1 = 0.0545\) \(\text{Bernoulli}(0.0545)\)
A voter’s thermometer score Not canvassed, not a Democrat \(40 + 0 + 0 = 40\) \(N(40, 20^2)\)
Not canvassed, a Democrat \(40 + 0 + 20 = 60\) \(N(60, 20^2)\)
Canvassed, not a Democrat \(40 + 5 + 0 = 45\) \(N(45, 20^2)\)
Canvassed, a Democrat \(40 + 5 + 20 = 65\) \(N(65, 20^2)\)

Each unit’s outcome is a random variable, and its systematic part sets the center. For a coin flip, the center also fixes the spread, \(p(1 - p)\). The canvass noise has its own spread: SD 20 for every voter

The 68–95–99.7 rule

For any normal: 68% of draws within one SD of the mean, 95% within two, 99.7% within three

Week 5’s rules turn every normal into the standard normal

Take any \(X\) with mean \(\mu\) and SD \(\sigma\), and standardize it:

\[ Z = \frac{X - \mu}{\sigma} \]

\[ E[Z] = \frac{E[X] - \mu}{\sigma} = 0 \qquad \text{Var}(Z) = \frac{\text{Var}(X)}{\sigma^2} = \frac{\sigma^2}{\sigma^2} = 1 \]

The shift rule moves the center to 0; the \(k^2\) rule (with \(k = 1/\sigma\)) shrinks the variance to 1

If \(X\) is normal, \(Z \sim N(0, 1)\). \(Z\) counts SDs from the mean: the z-score

A z-score puts a voter on the standard normal’s scale

An uncanvassed non-Democrat: \(\texttt{therm} = 40 + \text{noise} \sim N(40, 20^2)\). One of them scores 80

\[ z = \frac{80 - 40}{20} = 2 \]

Two SDs above the mean: warmer than about 97.7% of voters like them. A canvassed Democrat who scores 80 has \(z = (80 - 65)/20 = 0.75\): warmer than about 77%

Let’s take a 10-minute break

R has four functions for every distribution

Function What it gives Normal You’ve used
r*() Random draws rnorm(n, mean, sd) rnorm(), rbinom() since Problem Set 1
d*() The density’s height (a probability, for discrete) dnorm(x, mean, sd) the curves on these slides
p*() The area to the left of \(x\): \(P(X \leq x)\) pnorm(x, mean, sd) new
q*() The \(x\) with a given area to its left qnorm(p, mean, sd) new

Swap the letters after the first one for other distributions: rbinom(), dbinom(), pbinom(), qbinom()

pnorm() gives the area to the left

# share scoring 60 or below
pnorm(60, mean = 40, sd = 20)
[1] 0.8413447
# share scoring between 20 and 60
pnorm(60, 40, 20) - pnorm(20, 40, 20)
[1] 0.6826895

qnorm() runs pnorm() backward

Which score marks the top 2.5%? Ask for the point with 97.5% of the area to its left:

qnorm(0.975, mean = 40, sd = 20)
[1] 79.19928
qnorm(0.975)   # the standard normal: in SDs from the mean
[1] 1.959964

1.96: the middle 95% of any normal runs from 1.96 SDs below the mean to 1.96 above. “About 2” in the 68–95–99.7 rule is 1.96. You will see this number for the rest of your career

Your turn: the canvass, with pnorm() and qnorm()

In the canvass DGP:

  • Uncanvassed non-Democrats score \(N(40, 20^2)\)
  • Canvassed Democrats score \(N(65, 20^2)\)
  1. What share of uncanvassed non-Democrats score above 50?
  2. Problem Set 5 warned that a few scores land below 0. What share of uncanvassed non-Democrats?
  3. What score marks the warmest 10% of uncanvassed non-Democrats?
  4. What share of canvassed Democrats score above 50?

Check one of your answers with rnorm() draws

The canvass with pnorm() and qnorm(): answers

1 - pnorm(50, mean = 40, sd = 20)   # 1. above 50
[1] 0.3085375
pnorm(0, mean = 40, sd = 20)        # 2. below 0
[1] 0.02275013
qnorm(0.90, mean = 40, sd = 20)     # 3. the warmest 10% start here
[1] 65.63103
1 - pnorm(50, mean = 65, sd = 20)   # 4. canvassed Democrats above 50
[1] 0.7733726

The central limit theorem

Week 5 gave the center and the width. The CLT gives the shape

The average of \(n\) independent draws, \(\bar{X}\) Where it comes from
Center \(E[\bar{X}] = \mu\) Week 5: the expectation of a sum
Width \(\text{SD}(\bar{X}) = \sigma / \sqrt{n}\) Week 5: the variance of a sum of independent draws
Shape About normal, when \(n\) is large Today: the central limit theorem

Average more draws, and a bell appears, whatever you start from

First column, one draw of each: a callback (two bars), a die (flat), a protest’s size (a long right tail). Orange: the normal curve with week 5’s center \(\mu\) and width \(\sigma / \sqrt{n}\)

The central limit theorem

Take iid draws \(X_1, \ldots, X_n\), each with mean \(\mu\) and variance \(\sigma^2\). When \(n\) is large,

\[ \bar{X} \;\overset{\text{approx.}}{\sim}\; N\!\left(\mu, \; \frac{\sigma^2}{n}\right) \]

Whatever the distribution of one draw. Equivalently, the standardized average is about standard normal:

\[ \frac{\bar{X} - \mu}{\sigma / \sqrt{n}} \;\overset{\text{approx.}}{\sim}\; N(0, 1) \]

“Large” depends on the start: a few draws for a die, hundreds for a rare event or a skewed one

Why a bell? Extremes need everything to go the same way

Two dice sum to 12 one way (6 + 6) and to 7 six ways. With ten dice, a sum of 60 needs all ten to show 6; middling sums can happen millions of ways

The LLN and the CLT answer different questions

Law of large numbers Central limit theorem
The question Where does the average end up? What shape is its scatter around that spot?
The answer On the expectation: \(\bar{X} \rightarrow \mu\) A bell: \(\bar{X} \overset{\text{approx.}}{\sim} N(\mu, \sigma^2 / n)\)
As \(n\) grows The scatter shrinks toward zero The scatter stays a bell, just narrower
What it buys us \(\hat{p} \rightarrow p\), so the plug-in works; simulations work About 95% of studies land within 1.96 SEs: confidence intervals and tests (weeks 8 and 9)
Today’s picture The funnel of running averages The bells in the grid

One way to keep them straight: the LLN is about one number homing in. The CLT is about the whole distribution of that number on its way there

Zoom out and the average collapses onto 3.5. Zoom in and it’s a bell

The résumé gap is a difference of averages, so it’s normal too

Week 4’s 1,000 reruns, with the normal curve centered at −3.2 points with SD 0.78 points. Each callback share is an average of thousands of coin flips; their difference is normal too

Your turn: protest sizes

Week 5’s protest: 100 people (probability 0.6), 1,000 (0.3), or 10,000 (0.1)

Mean 1,360, SD about 2,908

  1. Simulate the average size of 5 protests, 1,000 times, and plot a histogram. A bell?
  2. Do the same for 30 protests and for 200 protests. Where does it start to look like a bell?
  3. Compare the SD of your 200-protest averages to what week 5’s formula says it should be

Protest sizes: answers

Five protests: still skewed. Thirty: close. Two hundred: a bell, with SD 203 against \(2{,}908 / \sqrt{200} \approx\) 206

When the draws aren’t independent

Linked draws: the bell is wider than the formula says

What if résumés covary? Résumés at one employer share its mood (a 2% chance if the job is filled, 14% if it’s urgent). Orange: the bell week 5’s formula predicts

Orange: the bell week 5’s formula predicts

The more résumés share an employer, the wider the real spread. And one decision for all 1,000 (week 5’s scenario B)? The share is 0% or 100%. No bell at all

Why independence is key: each linked draw adds less new information

1,000 reruns of each design, 1,000 résumés per study. Made-up data.
Design Real SD of the share (points) Formula’s SD (points) Real ÷ formula Worth this many independent résumés
Independent 0.85 0.86 0.99 1,015
4 per employer 0.89 0.86 1.04 930
50 per employer 1.59 0.86 1.86 289
One decision for all 27.60 0.86 32.17 1

Week 5: the variance of a sum has a covariance term for every pair. Independence wipes them all out, which is where \(\sigma^2 / n\) comes from. Linked draws bring them back, so the real spread is bigger than the formula says

Bertrand and Mullainathan sent four résumés to each ad. For the callback share, their 4,870 résumés act like somewhat fewer. Week 10: when to cluster

The payoff

Center, width, and shape of the sampling distribution from one run of data

Centered on the truth (week 5), with a standard error of 0.78 points from one run of data (week 5), and normal (today). So 95% of studies land within 1.96 standard errors of the truth

Check it: 95% of week 4’s reruns land within 1.96 SEs

gaps |>
  summarize(share_within = mean(abs(gap - (-0.032)) <= 1.96 * sd(gap)))
# A tibble: 1 × 1
  share_within
         <dbl>
1        0.952

The simulation and the normal curve agree. With the normal, we get the number from pnorm() without simulating:

pnorm(1.96) - pnorm(-1.96)
[1] 0.9500042

The payoff: −3.2 points, give or take 0.78

Week 1’s table said -0.032 (0.0078). Now we can read every piece of it:

  • -3.2 points: our estimate of the gap
  • 0.78 points: its standard error, estimated from one run of data (the LLN says plugging in \(\hat{p}\) is fine)
  • A bell: the CLT, so 95% of studies land within \(1.96 \times 0.78 \approx 1.5\) points of the truth

95% of studies like this one land within 1.5 points of the true gap

So unless ours is one of the unlucky 5%, the true gap is between -4.7 and -1.7 points. Zero is far outside that range. Weeks 8 and 9 turn this into confidence intervals and tests

Next week

Exam 2 in the first half of class: weeks 4–6 (probability, random variables, expectation and variance, the normal, the LLN and the CLT). The practice set is up today

Then estimators: if unbiasedness isn’t everything, what else makes an estimator good? Four estimators of the same number, and the trade-off between bias and variance

References

Wooldridge, Jeffrey M. 2023. “What Is a Standard Error? (And How Should We Compute It?).” Journal of Econometrics 237: 105517. doi:10.1016/j.jeconom.2023.105517.