library(tidyverse)
library(juanr)Problem Set 2: Bias, Simulation, and Measurement
Due: Monday, October 12 by 11:59 PM. Email both your .qmd and your rendered .html to Max Welch (mwelch@ucdavis.edu).
This set leans on the R you learned at Quant Camp. When you get stuck on filtering, grouping, or plotting, go back to the Quant Camp materials. They’re the fastest way to get unstuck.
Use AI the way people used Google, R’s help pages, and Stack Overflow before AI: ask precise, limited questions to get unstuck on part of a question. Passing off AI-generated work wholesale as your own is not allowed. See the syllabus and the Department’s policy on generative AI.
Format: Write your answers in one Quarto document, saved in the sets/ folder of your course project. Start it with this header, so the HTML renders as one self-contained file:
---
title: "Problem Set 2"
author: "Your name"
format:
html:
embed-resources: true
---Your document should read as a short report: brief prose explaining your reasoning, not just code and output.
Libraries you will need:
Parts marked Methods portfolio are required if you plan to choose methods as a subfield. They’re optional, and recommended, for everyone else.
Question 1: Separate the effect from the bias
A state gives violence-prevention grants to some of its cities. Imagine we could see every potential outcome:
| Homicides per 100,000 residents | |
|---|---|
| Cities with a grant, as they are | 12 |
| Cities with a grant, if they had not gotten one | 15 |
| Cities without a grant, as they are | 6 |
What is the treatment, and what is the outcome? State the estimand in words.
What is the difference in means we’d see in real data and what inference would that imply about the grants?
Split the difference in means into the effect of the grant on the cities that got one, plus the baseline difference, as we did in class. Show your work.
Is there evidence in the table that the grants were not handed out at random? And: why does this evidence fall short of proof?
- Work backward. Suppose you only had the two rows anyone can see (12 and 6). How high would the hidden row have to be for the grant to have no effect at all? For the grant to have cut homicides in half? What would the baseline difference be in each case? Which of these do you find believable, and why? This is the logic of a sensitivity analysis: ask how big the bias would have to be to produce the estimate we see (Bueno de Mesquita and Fowler 2021, 179).
Question 2: Which way does the bias go?
For each finding below, name the most likely confounder (if any), say which way it pushes the treatment and the outcome, and sign the bias. Does the bias make the finding look stronger or weaker than it really is?
Candidates for Congress who raise more money win by bigger margins.
Governments that spend more on their armies face more rebellions.
College students who live on campus get higher grades than students who live off campus. Consider two confounders: family income, and being a first-year student. Sign the bias from each one. Can you sign the total?
Size the bias. Suppose the treatment and a confounder are both 0/1, and the confounder has the same effect on everyone’s outcome. Then the bias in the difference in means turns out to be:
\[ \text{Bias} = \gamma \times (p_T - p_U) \]
where \(\gamma\) is the confounder’s effect on the outcome, \(p_T\) is the share of treated units that have the confounder, and \(p_U\) is the share of untreated units that have it.
Check it on the made-up world from class. Half of all countries are rich. Rich countries are democracies 75% of the time; poor countries, 25% of the time. Being rich lowers the civil war rate by 2 points.
- What share of democracies are rich? What share of non-democracies? What does the formula say the bias is? Compare it with the baseline difference on the “Hypothetical” slide.
- Back to case (c). Suppose high-income students are 60% of on-campus students and 40% of off-campus students, and high income adds 0.3 to a student’s GPA. First-years are 70% of on-campus students and 10% of off-campus students, and being a first-year lowers GPA by 0.2. When each confounder adds its own fixed amount to the outcome, the biases add up. What is the total bias? Which confounder wins?
Question 3: Weighted coin flips with rbinom()
In class, we used sample() to pick among elements in x with equal probability. Here we will learn to use rbinom() to give different probabilities to different units: to make rich countries more likely to be democracies than poor ones, say.
rbinom() in plain words
Each draw is one flip of a weighted coin: it comes up 1 with chance prob, and 0 otherwise. size = 1 means one flip per draw. In these examples, you always want one flip per draw.
Run
rbinom(100, size = 1, prob = 0.9)a few times. Then changeprob. Report the percent of the draws equal 1 for each value ofprobyou selected.probcan also be a list of chances, one per draw. Run this a few times:rbinom(6, size = 1, prob = c(0.1, 0.1, 0.1, 0.9, 0.9, 0.9))The first three observations have a 10% chance of being a 1, whereas the latter three have a 90% chance.
Generate a town of 1,000 people, classifying them as “younger” or “older” with equal probability. (Note: make sure to set a seed with
set.seed(211).)Now, create a
votedvariable that gives “older” people a 70% probability of voting and “younger” people a 40% probability of voting. What share of older and younger people voted? Why isn’t it equal to the probabilities you assigned? (Hint: you’ll want togroup_by()to compare age groups.)
Question 4: Simulate confounding
Your example. Pick a yes/no treatment, a numeric outcome from your field, and the effect size of the treatment on the outcome.
Simulate data. Simulate data for your example with 10,000 units (describe your unit of analysis). Explain/justify your choice of parameters for
rnorm(),rbinom(), etc.Compute the difference in means. How close is your estimate to your estimand? Why isn’t it exact?
A confounder. Name and justify a confounder in your example. Add a simulated confounder
zto your data. Compute the difference in means. How far is it from your estimand?Hold the confounder fixed. Compute the difference in means separately for units with and without the confounder. (Hint: review the lecture activity with income, democracy, and war.) How close is each one to your estimand?
1,000 reruns. Rerun your simulation 1,000 times with 50 units each, and again with 5,000 units each.
expand_grid()makes one row for each unit in each run, so every column gets new random draws in every run:reruns = expand_grid(run = 1:1000, unit = 1:50) |> # your z, x, and y lines from (d), with n() as the number of draws mutate(...) |> # the difference in means within each run group_by(run) |> summarize(gap = mean(y[x == 1]) - mean(y[x == 0]))(
y[x == 1]keeps the values ofywherexis 1.) Plot each set of 1,000 gaps in a histogram, with a vertical line at your estimand. Describe what you see.
Question 5: Noise in the outcome vs. noise in the treatment
In class, we said that noise in the measurement of the outcome did not bias estimates, while noise in the measurement of the treatment biases it towards zero. Here you’ll work through the intuition, but with a yes/no treatment.
A yes/no treatment can’t be off by a little: each label is either right or wrong. So for a yes/no treatment, noise means mislabeling: at random, some treated units get recorded as untreated, and some untreated units as treated.
Why mislabel? For a hypothetical 0/1 treatment of your choosing, explain how such measurement error (“mislabeling”) could happen.
One world. Simulate 10,000 observations, each is treated (
x) with a probability of 0.5. The outcome (y) is normally distributed around a mean of 50, +/- 10. Make the effect of the treatment on the outcome equal to 5. Compute the difference in means. How close is it to the estimand?Noise in the outcome. Suppose we measure
ybadly. Addy_noisy = y + rnorm(n(), mean = 0, sd = 10)to your data. Compute the difference in means ofy_noisybetween treated and untreated units. What happened to the estimate? What happened to the spread of the outcome? (Hint: compare the standard deviation ofyandy_noisy.)Noise in the treatment. Now suppose we measure
ywell but measurexbadly. We’ll use this code to mislabel each unit with a 20% chance:your_data = your_data |> # rbinom() picks who gets mislabeled; 1 - x turns a 1 into 0 and a 0 into 1 mutate(x_noisy = if_else(rbinom(n(), size = 1, prob = 0.2) == 1, 1 - x, x))What share of units were mislabeled? (Hint: the average of all rows where
x_noisydoes not matchxis what you want.)Noisy treatment effect. Compute the difference in means of
yacross your noisyxvariable. What happened to the estimate?
- How much does mislabeling shrink the estimate? Rerun (d) with mislabeling chances of 0, 0.1, 0.2, 0.3, and 0.5. Plot the average estimate against the mislabeling chance. What happens at 0.5, and why might that be?