library(tidyverse)
library(juanr)
library(qss)Problem Set 8: Confidence Intervals
Due: Monday, November 23 by 11:59 PM. Email both your .qmd and your rendered .html to Max Welch (mwelch@ucdavis.edu).
This set leans on the R you learned at Quant Camp. When you get stuck on filtering, grouping, or plotting, go back to the Quant Camp materials. They’re the fastest way to get unstuck.
Use AI the way people used Google, R’s help pages, and Stack Overflow before AI: ask precise, limited questions to get unstuck on part of a question. Passing off AI-generated work wholesale as your own is not allowed. See the syllabus and the Department’s policy on generative AI.
Format: Write your answers in one Quarto document, saved in the sets/ folder of your course project. Start it with this header, so the HTML renders as one self-contained file:
---
title: "Problem Set 8"
author: "Your name"
format:
html:
embed-resources: true
---Your document should read as a short report: brief prose explaining your reasoning, not just code and output.
Libraries you will need:
Parts marked Methods portfolio are required if you plan to choose methods as a subfield. They’re optional, and recommended, for everyone else.
Question 1: How wide should the interval be?
In class we reran the résumé study 100 times and drew each rerun’s 95% interval. Here you’ll make that plot yourself, then change the confidence level.
To keep the focus on the intervals, skip the résumés: draw each rerun’s estimate straight from the sampling distribution. In the résumé study, that’s a normal centered on the true gap of −3.2 points, with a standard error of 0.78 points:
true_gap = -3.2 # percentage points
set.seed(211)
studies = tibble(run = 1:1000,
gap = rnorm(1000, mean = true_gap, sd = 0.78), # each rerun's estimate
se = 0.78) # its standard errorTake the first 20 reruns. For each one, build a 95% interval and record whether it contains the true gap. Plot the 20 intervals, with a vertical line at the truth, colored by whether each one catches it. How many catch the truth? How many did you expect?
Now use all 1,000 reruns, and for each one build 50%, 70%, 95%, and 99% intervals. For each level, report the coverage (the share of intervals that contain the truth) and the average width of the intervals. Plot the first 20 intervals at each level side by side.
Discuss the tradeoff. What do you gain and lose as you move from a 50% interval to a 99% one? Why not always report a 99% interval?
Question 2: Reading an interval
A poll of 800 likely voters finds that 54% support a ballot measure. The pollster reports a 95% confidence interval of 50.5% to 57.5%.
Check the interval by hand: compute the standard error of a share, \(\sqrt{\hat{p}(1 - \hat{p}) / n}\), and build the interval.
Which of these statements are right? Explain each answer in a sentence or two.
- “There’s a 95% chance that true support for the measure is between 50.5% and 57.5%.”
- “The interval 50.5% to 57.5% came from a method that catches true support in 95% of polls.”
- “95% of likely voters have a level of support between 50.5% and 57.5%.”
- “If another pollster ran the same poll, there’s a 95% chance its estimate would land between 50.5% and 57.5%.”
- “Any level of true support between 50.5% and 57.5% is compatible with this poll; levels outside it are not.”
- “The poll suggests the measure will pass.”
- Statement iv is wrong. Find the right number: if both polls are independent and each has the standard error you computed in (a), what’s the chance the second poll’s estimate lands inside the first poll’s 95% interval? Check your answer with a simulation.
Question 3: An 81% interval
A researcher runs a small pilot survey before fielding a big one. Simulate that pilot by drawing 15 respondents from the therm data in juanr (the July 2017 VOTER Survey), and look at ft_police, how warmly each respondent feels toward the police (0 to 100):
set.seed(211)
pilot = therm |>
filter(!is.na(ft_police)) |>
slice_sample(n = 15)The researcher wants an 81% confidence interval for the average.
Build the 81% interval for the average feeling toward the police, using the \(t\) distribution: which multiplier does
qt()give you, and why?Compare your multiplier with
qnorm()’s multiplier for an 81% interval. Explain the difference.Treat the whole survey as the population: compute the average of
ft_policeacross all respondents. Did your interval catch it? How often should an 81% interval miss?
Question 4: Why qt() works at any sample size
Draw samples from a normal population with mean 50 and SD 25. For each sample, estimate the average, estimate the SD with sd(), and build two 95% intervals: one with qnorm() as the multiplier, one with qt(). Do this 10,000 times for each sample size \(n\) = 3, 5, 10, 30, and 100.
For each sample size, compute the coverage of each kind of interval. Show your results in a table.
Describe the pattern. When does the choice of multiplier matter?
Explain why the
qnorm()interval undercovers in small samples, and why the problem goes away as \(n\) grows.
- The \(t\) distribution of \(T = (\bar{Y} - \mu)/(S/\sqrt{n})\) doesn’t depend on \(\mu\) or \(\sigma\). Show this algebraically: write each draw as \(Y_i = \mu + \sigma Z_i\), where \(Z_i \sim N(0, 1)\), and show that \(T\) can be written using only the \(Z_i\)’s. Then confirm it by rerunning your simulation for \(n\) = 3 and 5 with a very different mean and SD.
Question 5: Reading both ends of a published interval
Gerber, Green, and Larimer (2008) ran a famous field experiment before Michigan’s 2006 primary. Households were randomly assigned to get a mailing or not. The “Neighbors” mailing listed whether each person, and each of their neighbors, had voted in past elections, and promised to send an updated list after the election. The data are social in the qss package (load them with data(social)).
Compared with the control group (no mailing), the Neighbors mailing raised turnout by 8.1 percentage points, with a 95% confidence interval of 7.6 to 8.7 points. Turnout in the control group was 29.7%.
What inferences can we draw from this particular interval about directionality?
What inferences can we draw from this particular interval about the magnitude of compatible effects? Express your answer as a share of the control group’s turnout, too.
Question 6: An effect in one group, not in the other
Use the therm data again, this time looking at ft_dem, how warmly each respondent feels toward the Democratic Party (0 to 100). The “effect” here is a gender gap: the difference in average feeling between women and men (sex), within each racial group (race).
Among Black respondents, estimate the gender gap (women’s average minus men’s) and its 95% interval. (Hint: the standard error of a difference in means is \(\sqrt{s_1^2/n_1 + s_2^2/n_2}\).) What can you say about the gap?
Do the same among Asian respondents. What can you say?
Is the gender gap among Black respondents different from the gender gap among Asian respondents? Estimate the difference between the two gaps and its 95% interval.
A classmate says: “That’s a contradiction. There’s a gender gap among Black respondents but not among Asian respondents, so the gaps must be different. How can we not be able to tell them apart?” Explain why this isn’t a contradiction.