POL 211 · Week 4
University of California, Davis
October 20, 2026
Write your answers in your blue book
| Callback (1) | |
|---|---|
| African American-sounding name | -0.032 |
| (0.0078) | |
| Intercept | 0.097 |
| (0.0055) | |
| Résumés | 4,870 |
Linear regression. Outcome: callback = 1 if the employer called back, 0 if not. Standard errors in parentheses.
The estimated treatment effect, -3.2pp: how much less likely AA-sounding names to receive callback than White-sounding names
The standard error, 0.0078, is “how much the estimate would vary if we ran the study again.”
Today: how to think about why the estimate varies
The résumé study found AA-sounding names got about 3 points fewer callbacks
Send the same résumés out again, to a fresh batch of employers, and we’d get a different number. Maybe 2.5 points. Maybe 4
This is because the Data-Generating Process (DGP) has a random component
Probability is the language for the random part: how much would the answer change if we ran the world again?
Let’s generate data from a DGP we control completely
The systematic part: each résumé’s chance of a call, \(p_i\)
The random part: each employer then flips a coin weighted by \(p_i\)
set.seed(211)
resumes = tibble(name = sample(c("white", "black"), size = 4870, replace = TRUE), # a coin flip picks each name
exper = sample(0:10, size = 4870, replace = TRUE), # years of experience
age = sample(22:55, size = 4870, replace = TRUE),
distance = sample(0:30, size = 4870, replace = TRUE)) |> # miles from the job
mutate(p = 0.099 - 0.032 * (name == "black") + # Black-sounding name: -3.2 points
0.004 * exper - # each year of experience: +0.4 points
0.0005 * (age - 22) - # each 10 years older: -0.5 points
0.001 * distance) # each 10 miles away: -1 point
resumes# A tibble: 4,870 × 5
name exper age distance p
<chr> <int> <int> <int> <dbl>
1 black 6 49 23 0.0545
2 white 4 23 23 0.0915
3 white 6 48 0 0.11
4 white 4 49 10 0.0915
5 black 0 41 0 0.0575
# ℹ 4,865 more rows
We wrote \(p\) down ourselves: we know the systematic part exactly
We want to turn those probabilities into call (0/1), where the probability that call = 1 depends on \(p\)
# A tibble: 4,870 × 6
name exper age distance p call
<chr> <int> <int> <int> <dbl> <int>
1 black 6 49 23 0.0545 0
2 white 4 23 23 0.0915 0
3 white 6 48 0 0.11 0
4 white 4 49 10 0.0915 0
5 black 0 41 0 0.0575 0
# ℹ 4,865 more rows
rbinom() flips one coin per résumé, each weighted by its own \(p\)
# A tibble: 2 × 2
name callback_rate
<chr> <dbl>
1 black 0.0680
2 white 0.0882
The difference in means: -2.0 points. The true gap we built in: −3.2
# A tibble: 2 × 2
name callback_rate
<chr> <dbl>
1 black 0.0615
2 white 0.101
This time: -3.9 points. Even when we know exactly how the world works, and by how much, the estimate changes

The name shifts the chance of a call. It doesn’t decide any one résumé: most résumés with either name get no call, and some Black-named résumés get calls that white-named ones don’t
The callback rates we computed in runs 1 and 2 were \(\hat{p}\): shares in the data. The chances we wrote in Step 1 were \(p\). With real data, all we ever get is \(\hat{p}\):
\(p\) is a setting on the DGP: we never see it. \(\hat{p}\) (read “p-hat”) is the share in the one run we got: our estimate of \(p\)


| Source | What it means | In the résumé study |
|---|---|---|
| Sampling | We see only part of the whole | These employers, in these two cities, that year |
| Theoretical | Our theory leaves things out | The hiring manager’s mood that morning |
| Fundamental | People are partly random, even with a complete theory | Rerun the same morning, and the same manager might decide differently |
The three sources come from Gailmard (2014, 73–78)
Which source is most at work: sampling, theoretical, or fundamental?
We just watched the estimate change from run to run. The next few weeks build a mathematical language for that variation, so we can:
An experiment is one run of a random process: send one résumé to one employer
The sample space, \(S\), lists every outcome that could happen:
\[ S = \{\text{callback}, \; \text{no callback}\} \]
(A die works the same way: roll it once, and \(S = \{1, 2, 3, 4, 5, 6\}\))
Sorry
Statistics, probability, and math are full of terribly overlapping concepts. This creates a lot of confusion
The outcomes “callback” and “no callback” aren’t numbers. To do math with them, we translate:
\[ Y(\text{callback}) = 1, \qquad Y(\text{no callback}) = 0 \]
A random variable is that translation: a rule that turns each outcome in \(S\) into a number. We name random variables with capital letters
One hiring process, many possible random variables:
| Random variable | What it records |
|---|---|
| \(Y_i\) | Whether résumé \(i\) gets a callback |
| \(Z_i\) | How many days until the employer responds |
| \(W_i\) | Whether the employer offers an interview |
With random variables in hand, we can deploy concepts from statistics/probability to quantify uncertainty
Before an employer opens résumé #1, we don’t know whether they’ll call. Call it \(Y_1\): it will turn out to be 1 (callback) or 0 (no callback)
Once they decide, we see data: \(y_1 = 0\)
| Written | What it is | |
|---|---|---|
| Random variable | \(Y\) | The number before the machine produces it: it could be 0 or 1 |
| Data | \(y\) | The number the machine produced this time |
This lets us make probability statements about \(Y_1\), like \(P(Y_1 = 1) = 0.05\)
but not about \(y_1\): once we’ve seen it, it’s just 0 or 1
Definition: describes the probability of every possible outcome for a random variable
Résumé #1 from Step 1 (Black-sounding name, 6 years of experience, age 49, 23 miles away) got \(p_1\) = 0.0545:
\[ P(Y_1 = 1) = 0.0545, \qquad P(Y_1 = 0) = 0.9455 \]
Together, these are a probability distribution: every value \(Y_1\) can take, and how likely each one is
These are the true probabilities: settings on the DGP, known only because we built it in Step 1. With real data we never see them. We see outcomes: the empirical distribution, the real-life “footprint” of the true probability

\[ Y_i \sim \text{Bernoulli}(p_i) \]
Read it as: “\(Y_i\) is drawn from a Bernoulli with probability \(p_i\).” It takes the value 1 with probability \(p_i\) and 0 with probability \(1 - p_i\)
The formula:
\[ P(Y_i = y) = p_i^{\,y} \, (1 - p_i)^{1 - y}, \qquad y = 0 \text{ or } 1 \]
Plug in \(y = 1\): you get \(p_i\). Plug in \(y = 0\): you get \(1 - p_i\). One formula, both cases
One parameter: once we know \(p_i\), we know the whole distribution
rbinom()
This formula is what rbinom() draws from in Step 2
| Type of outcome | Example | A distribution often used |
|---|---|---|
| Binary (0/1) | A callback | Bernoulli |
| Count | Number of protests in a week | Poisson |
| Continuous | Vote share; income | Normal (week 6); lognormal |
| Unordered categories | Democrat, Republican, independent | Categorical |
| Ordered categories | Strongly disagree … strongly agree | Ordered categorical |
Each family describes how probability is spread across the values an outcome can take, and each has parameters that set its shape. For anything beyond 0/1, picking the family is part of building the model
\[ Y_i \sim \text{Bernoulli}(p_i), \qquad p_i = \text{baseline} - \text{Black name} + \text{experience} - \text{age} - \text{distance} \]
| Term | In symbols | In the résumé study |
|---|---|---|
| Estimand | \(\Delta = p_{\text{Black}} - p_{\text{white}}\) | The true gap in callback chances: a setting on the machine |
| Estimator | \(\hat{p}_{\text{Black}} - \hat{p}_{\text{white}}\) | The rule: subtract the two callback shares |
| Estimate | \(\hat{\Delta}\) = -3.2 points | What the rule gave in the one run we saw (6.45% – 9.65%); the same number as week 1’s regression |
Before the data exist, the estimator’s answer is a random variable. The estimate is its realization: the \(y\) to its \(Y\)
Before Bertrand and Mullainathan ran their study, \(\hat{\Delta}\) was a random variable: built from 4,870 Bernoulli draws, it could have come out −1 point, −3 points, −5 points…
The −3.2 points they reported is one draw
And like any random variable, \(\hat{\Delta}\) has a distribution: every value the gap could take, and how often

This is what the DGP would generate across many runs of the study. We never see it: Bertrand and Mullainathan ran once. It’s hypothetical
# the DGP from Steps 1 and 2
resumes = tibble(
name = sample(c("white", "black"), size = 4870, replace = TRUE),
exper = sample(0:10, size = 4870, replace = TRUE),
age = sample(22:55, size = 4870, replace = TRUE),
distance = sample(0:30, size = 4870, replace = TRUE)) |>
mutate(
# systematic: the same rule every run
p = 0.099 - 0.032 * (name == "black") + 0.004 * exper -
0.0005 * (age - 22) - 0.001 * distance,
# random: new coin flips every run
call = rbinom(4870, size = 1, prob = p))All we have to do is run this code thousands of times, compute the gap each time, and pile up the gaps
A function takes inputs (here, a variable and a guess) and returns an output (here, a number)
Today: no inputs, and the output is a whole made-up dataset. The function is the data-generating process
make_resumes = function() {
tibble(name = sample(c("white", "black"), size = 4870, replace = TRUE),
exper = sample(0:10, size = 4870, replace = TRUE),
age = sample(22:55, size = 4870, replace = TRUE),
distance = sample(0:30, size = 4870, replace = TRUE)) |>
mutate(p = 0.099 - 0.032 * (name == "black") + 0.004 * exper -
0.0005 * (age - 22) - 0.001 * distance, # step 1: systematic
call = rbinom(4870, size = 1, prob = p)) # step 2: random
}
run_resumes = function() {
make_resumes() |>
summarize(gap = mean(call[name == "black"]) - mean(call[name == "white"])) |>
pull(gap) # just the estimate
}# A tibble: 1,000 × 2
run gap
<int> <dbl>
1 1 -0.0202
2 2 -0.0354
3 3 -0.0168
4 4 -0.0305
5 5 -0.0321
# ℹ 995 more rows
One row per study. Each gap is a \(\hat{\Delta}\) the study could have produced

Each dot is one whole study. The orange line is the true gap we built in (−3.2 points)


The hypothetical curve from before, made visible. What stands out? And how could we quantify how much the estimate bounces?
# A tibble: 1 × 2
average_gap spread
<dbl> <dbl>
1 -0.0318 0.00776
Center: the average of the 1,000 estimates is the −3.2 points we built in. The estimator is unbiased: it lands on the truth on average, though any one run can miss
Spread: the standard deviation of the 1,000 estimates says how far a typical run lands from the truth
Both should yield essentially the same result. Use one as a check on the other (Xu 2026)
Today we took the simulation path. Weeks 5–8 take the analytic path
Bertrand and Mullainathan ran their study once. We can’t rerun reality
But how much to trust their −3.2 points depends on how much it would bounce. If the pile is narrow, −3.2 means a real gap; if the pile is wide, −3.2 could be a fluke of which résumés got which names
Every tool for the rest of the course (standard errors, confidence intervals, p-values) is a statement about that pile. The hard part, and the next few weeks’ job, is learning about it from a single run
Today we simulated the sampling distribution of estimates. Next week we start deriving it
Expected value and variance of a random variable, and how they behave when we add or average
The payoff: why the sampling distribution is centered on the truth, why bigger studies make it narrower, and the formula behind week 1’s 0.0078
POL 211 · Fall 2026