Bias, simulation, and measurement

POL 211 · Week 2

Juan F. Tellez

University of California, Davis

October 6, 2026

Last week: the data-generating process + causality

Every dataset is one run of a DGP

Systematic part: the patterns our theories are about, like how much a name changes an employer’s chance of calling back

Random part: everything else, like which manager opened the résumé and their mood that morning

The fundamental problem of causal inference

A causal effect compares two worlds: the same résumé, sent once with each name

We only ever see one, so we borrow a group to stand in for the missing world

Today

  1. Bias: when the groups we compare can’t stand in for each other
  2. Simulation: make up a world where we know the truth, and check which way the bias goes
  3. Measurement: when the measure does not capture the concept

Plus: a 10-minute break

Bias

Why bother with fake résumés (Bertrand and Mullainathan 2004)?

What if they had compared real Black and white job seekers, and counted who got called back?

But real applicants differ in more than their names: schooling, work history, neighborhood, the jobs they apply for

The fake résumés differ only in the name

Three terms

What it is In the résumé study
Estimand The true quantity we want to learn about The average effect of an African American-sounding name on callbacks
Estimator The recipe we apply to the data Difference-in-means: subtract one group’s callback rate from the other’s
Estimate The number the recipe gives us 3.2 points

“The estimate is the number we get as a result of our analysis. The estimand is the true quantity of interest in the population that we are trying to learn about.”

— Bueno de Mesquita and Fowler (2021, 95)

An estimate is wrong for two reasons

\[ \text{Estimate} = \text{Estimand} + \text{Bias} + \text{Noise} \] (Bueno de Mesquita and Fowler 2021, xix)

Bias: “errors that occur for systematic reasons”

Noise: “idiosyncratic errors that occur because of chance” (Bueno de Mesquita and Fowler 2021, 95)

Bias shifts the whole cloud of estimates; noise spreads it out

Week 1: our dataset is one run of the process. Imagine 15 runs of the résumé study, with new employers each time. Each blue dot is one run’s estimate

The orange dot is the estimand, the true effect. Each blue dot is the estimate from one run of the study. Simulated data

Randomization removes bias; more data only shrinks noise

\[ \underbrace{3.2 \text{ points}}_{\text{estimate}} = \underbrace{\text{true effect}}_{\text{estimand}} + \underbrace{0}_{\text{bias}} + \underbrace{\text{noise}}_{\text{luck of the draw}} \]

Randomization makes the bias zero, on average. The two piles of résumés are alike on average, so the comparison isn’t tilted

Noise is luck, and luck cancels out. With 10 employers, one odd employer moves the average a lot. With 5,000, it barely moves it. Last week’s standard error measures it

Bias doesn’t cancel. If the comparison itself is off, every new data point repeats the same mistake.

Today: two big sources of bias, confounding and measurement

Confounding

Does democracy make civil war less likely?

6 of the 6,343 country-years. Data: Fearon and Laitin (2003), via the juanr package
Country Year Polity score Democracy? GDP per capita ($1,000s) New civil war
India 1989 8 Yes 1.2 1
Peru 1981 7 Yes 3.1 1
Rwanda 1990 -7 No 0.8 1
China 1998 -7 No 2.5 0
Switzerland 1998 10 Yes 16.1 0
United States 1998 10 Yes 20.6 0

The unit of analysis is the country-year. civil_war is 1 in a year a civil war began

Polity score: experts rate how democratic a country’s institutions are, from –10 (full autocracy) to 10 (full democracy). We call a score above 5 a democracy

Civil wars broke out less than half as often in democracies

Country-years, 1945–1999. A country-year counts as a democracy if its Polity score is above 5 (on a scale from –10 to 10). Data: Fearon and Laitin (2003), via the juanr package

Does democracy prevent civil war?

In this question
Estimand How much being a democracy changes a country’s chance of a civil war starting
Estimator Difference in means: the war rate in democracies minus the rate in non-democracies
Estimate -1.3 points

What might bias this estimate?

A skeptic: democracies are also much richer

Democracies are richer

Average GDP per capita: $6.2k in democracies vs. $2.4k in non-democracies

Richer countries have fewer civil wars

A new civil war began in 2.6% of the poorest third of country-years, and 0.5% of the richest third

The gap between democracies and non-democracies could be income, not democracy

A confounder shapes both the treatment and the outcome

A confounder is a variable that affects both the treatment (\(X\)) and the outcome (\(Y\))

Confounding is systematic

Confounding is systematic part of DGP: the data is being generated such that democracies tend to be wealthier and have more/less conflict

More data won’t fix it: 6 million country-years would still mix democracy with income

The gap we see = the effect + a baseline difference

\[ \underbrace{\text{Difference in means}}_{\text{estimate}} = \underbrace{\text{Effect of democracy on democracies}}_{\text{estimand}} + \underbrace{\text{Baseline difference}}_{\text{bias}} + \text{Noise} \]

Baseline difference: how much less civil war the democracies would have had even without democracy, compared to the non-democracies (Bueno de Mesquita and Fowler 2021, 166)

Democracies are richer, so they’d have fewer civil wars anyway. The baseline difference is negative, and it makes democracy look more peaceful than it is

Hypothetical: separate effect from bias

Imagine if we could see the counterfactual:

Civil war rates in a made-up example. Made-up numbers
Civil war rate
Democracies, as democracies 1.0%
Democracies, if they had not been democracies 1.5% (we never see this)
Non-democracies, as non-democracies 2.5%

\[ \underbrace{1.0 - 2.5}_{\text{what we see: } -1.5} = \underbrace{(1.0 - 1.5)}_{\text{effect on democracies: } -0.5} + \underbrace{(1.5 - 2.5)}_{\text{baseline difference: } -1.0} \]

Two-thirds of the gap we see turns out to be bias

Randomization sets the baseline difference to zero

If a coin flip had picked the democracies, they’d be no richer than the rest, on average. Let’s simulate this:

set.seed(211)

coin_flip = wars |>
  # flip a coin for each country-year: heads, it's a "democracy"
  mutate(coin = sample(x = c("Democracy", "Not a democracy"), size = n(), replace = TRUE))

coin_flip |> select(country, year, civil_war, democracy, coin)
# A tibble: 6,343 × 5
  country  year civil_war democracy coin           
  <chr>   <dbl>     <dbl> <chr>     <chr>          
1 USA      1945         0 Democracy Not a democracy
2 USA      1946         0 Democracy Democracy      
3 USA      1947         0 Democracy Democracy      
4 USA      1948         0 Democracy Democracy      
5 USA      1949         0 Democracy Not a democracy
# ℹ 6,338 more rows

Sampling

sample() picks among the elements in x, n times, each time returning its pick to the x (“replace = TRUE”)

When a coin picks the democracies, they’re no richer than the rest

coin_gdp = coin_flip |>
  group_by(democracy) |>
  summarize(gdp = mean(gdp_cap))   # average GDP per capita, in $1,000s

coin_gdp
# A tibble: 2 × 2
  democracy         gdp
  <chr>           <dbl>
1 Democracy        6.19
2 Not a democracy  2.37

Real democracies vs. the rest: $6.2k vs. $2.4k.

coin_gdp = coin_flip |>
  group_by(coin) |>
  summarize(gdp = mean(gdp_cap))   # average GDP per capita, in $1,000s

coin_gdp
# A tibble: 2 × 2
  coin              gdp
  <chr>           <dbl>
1 Democracy        3.68
2 Not a democracy  3.69

Coin-flip “democracies” vs. the rest: $3.68k vs. $3.69k

Randomizing democracy makes the income gap zero, on average

Real GDP per capita for 6,343 country-years; the democracy labels come from simulated coin flips. Data: Fearon and Laitin (2003), via the juanr package

Every coin-flip gap falls within $0.32k of zero. Randomization turns a systematic difference (income) into a random one (the luck of the flip), and random differences average out

Which way will the bias go?

Two questions about a confounder:

  1. does it push the treatment up or down?
  2. Does it push the outcome up or down?

Waffle House is a Southern institution

Waffle House stays open so reliably that FEMA uses it to judge disaster damage

States with Waffle Houses have more divorce

Average divorce rate in the 25 states with at least one Waffle House and the 25 without. Data: McElreath (2020)

Do cheap, delicious waffles put marriages at risk?

Every Southern state has a Waffle House

Number of US states, by region and whether they have a Waffle House. Data: McElreath (2020)
Region Has a Waffle House No Waffle House
Not South 11 25
South 14 0

The South has more divorce and more Waffle Houses. It’s a confounder

And no Southern state lacks a Waffle House, so we can’t even compare within the South. No variation, no comparison

Same-direction confounders give the treated group a head start

Ask the two questions about the South:

  1. Does it raise the chance of a Waffle House? Yes (+). 14 of the 25 states with a Waffle House are Southern; none of the 25 without one are
  1. Does it raise divorce? Yes (+). The South has 10.7 divorces per 1,000 adults; the rest of the country has 9.3

So the Waffle House states would have more divorce even with no waffles. They start ahead, so the baseline difference is positive

The gap we see is the effect of waffles plus that head start

Positive bias: waffles look worse for marriage than they are

Simulation

How can we know which way the bias will go?

Real data never tell us the true effect. A simulation can

In a simulation, we make up the data-generating process ourselves, so we know the truth

  1. Write the systematic part. Pick the true effect (for waffles: zero), then add the confounder’s effects. That’s where the bias comes from
  2. Add the random part. Each state gets its own random nudge. That’s the noise
  3. Test our intuition. Does the estimate land where we thought it would?

Step 1: 50 states, and nothing affects anything

set.seed(1181)   # makes the random draws the same every time we run this

fake_states = tibble(
  south = sample(c(0, 1), size = 50, replace = TRUE, prob = c(0.7, 0.3)),
  waffle = rnorm(50, mean = 20, sd = 4),     # Waffle Houses per million residents
  divorce = rnorm(50, mean = 9, sd = 1.5)    # divorces per 1,000 adults
)

In plain words

sample() makes about 3 in 10 states Southern

rnorm() gives each state a waffle rate and a divorce rate: a center plus its own random nudge

No confounding; the line is flat

Step 2: the South adds Waffle Houses, and the line stays flat

set.seed(1181)

fake_states = tibble(
  south = sample(c(0, 1), size = 50, replace = TRUE, prob = c(0.7, 0.3)),
  waffle = rnorm(50, mean = 20, sd = 4) + 10 * south,   # the South adds 10
  divorce = rnorm(50, mean = 9, sd = 1.5)
)

The waffle line

Systematic part: 20, plus 10 in the South. That + 10 * south is the arrow from the South to waffles

Random part: each state’s nudge (sd = 4)

The Southern states slide right. Divorce still has nothing to do with anything

Step 3: the South adds divorce, and the line tilts up

set.seed(1181)

fake_states = tibble(
  south = sample(c(0, 1), size = 50, replace = TRUE, prob = c(0.7, 0.3)),
  waffle = rnorm(50, mean = 20, sd = 4) + 10 * south,
  divorce = rnorm(50, mean = 9, sd = 1.5) + 2 * south   # the South adds 2
)

The divorce line

Systematic part: 9, plus 2 in the South

Random part: each state’s nudge (sd = 1.5)

waffle appears in neither

The Southern states move up. Now the states with the most waffles have the most divorce

Each line of code draws one arrow

The arrows are the systematic part. waffle line: an arrow from the South to waffles. divorce line: an arrow from the South to divorce

waffle appears in no systematic part, so no arrow runs from waffles to divorce. The true effect is zero. We know the estimand

Waffles look like they raise divorce, even though they do nothing

The slope of the line from step 3: a state with 10 more Waffle Houses per million residents has about 1.1 more divorces per 1,000 adults

\[ \underbrace{0.11}_{\text{estimate}} = \underbrace{0}_{\text{estimand}} + \text{bias} + \text{noise} \]

The South raises waffles (+) and raises divorce (+). We guessed positive bias, and that’s what we got

Let’s take a 10-minute break

Rerun the world 1,000 times: the bias stays, the noise shrinks

Orange line: the true effect (zero). Dashed line: the average estimate. Simulated data

Each rerun keeps the systematic part and redraws the random part. The confounding is in the systematic part, so the bias comes back every run: the slopes center on about 0.11, not zero. More states only tighten the spread

Compare within the South and outside it, and the line goes flat

fake_states |>
  mutate(region = if_else(south == 1, "South", "Not South")) |>
  ggplot(aes(x = waffle, y = divorce, color = region)) +   # color splits the data by region
  geom_point() +
  geom_smooth(method = "lm", se = FALSE) +                # so we get one line per region
  labs(x = "Waffle Houses per million residents", y = "Divorces per 1,000", color = NULL)

Within each region, the line is flat. The slope of 0.11 through all the states was the South, not waffles. Holding the South fixed removes the bias

Your turn: Change the bias

set.seed(1181)
fake_states = tibble(
  south = sample(c(0, 1), size = 50, replace = TRUE, prob = c(0.7, 0.3)),
  waffle = rnorm(50, mean = 20, sd = 4) + 10 * south,
  divorce = rnorm(50, mean = 9, sd = 1.5) + 2 * south
)

ggplot(fake_states, aes(x = waffle, y = divorce)) +
  geom_point(aes(color = factor(south))) +
  geom_smooth(method = "lm", se = FALSE)
  1. Run it. Then move color = factor(south) into ggplot(aes()). Does the tilt go away?
  2. What if the South lowered divorce? Change the effect and look again.
  3. Give waffles a real effect: add + 0.1 * waffle to the end of the divorce line. Is the line through all the states steeper than the lines within each region?

We can often guess which way the bias goes

Ask two questions about a confounder: does it push the treatment up or down? Does it push the outcome up or down?

Confounder raises the outcome Confounder lowers the outcome
Confounder raises the chance of treatment Positive bias: estimate too high Negative bias: estimate too low
Confounder lowers the chance of treatment Negative bias: estimate too low Positive bias: estimate too high

Same direction: positive bias. Opposite directions: negative bias (Bueno de Mesquita and Fowler 2021, 177–78)

Income should make democracy look more peaceful than it is

What income does Sign
Effect on the treatment (democracy) Richer countries are more often democracies +
Effect on the outcome (civil war) Richer countries have fewer civil wars –
Bias –

Democracies are mostly rich, and rich countries have fewer civil wars anyway. The democracies start ahead on peace, so the baseline difference is negative

Negative bias: our estimate (-1.3 points) is lower than the true effect. The true effect is closer to zero, or even above it

Prediction: hold income fixed, and the gap should shrink. Let’s check

Your turn: does the gap survive within income groups?

library(juanr)

wars = fearon |>
  filter(!is.na(polity2), !is.na(gdp_cap)) |>      # drop rows missing either one
  mutate(democracy = if_else(polity2 > 5, "Democracy", "Not a democracy"),
         civil_war = if_else(onset > 0, 1, 0),
         income = ntile(gdp_cap, 3))                # 3 equal-sized groups: 1 = poorest

wars |>
  group_by(democracy) |>
  summarize(war_rate = mean(civil_war))
  1. Run it. Do you get the same gap?
  2. Are democracies richer? Check with summarize()
  3. Add income to group_by(). Does the gap survive within each income group?

Within every income group, the gap shrinks

Income groups are thirds of all country-years by GDP per capita, 1945–1999. Data: Fearon and Laitin (2003), via the juanr package

All countries: a gap of -1.29 points. Within each income group: -0.01, -0.44, and -0.74

Comparing within income groups is what “controlling for income” means

Week 1: adding a control for respondent sex barely moved the résumé estimate, even though sex could affect \(Pr(callBack)\)

Why? because the names were randomized

Here, holding income (roughly) fixed moves the estimate a lot, in the direction we predicted. That’s the sign of a confounder

POL 212 does this with regression. The logic is the same: compare democracies and non-democracies that are alike on income

Your turn: which way does the bias go?

Name a likely confounder, sign the bias, and say what it does to the estimate

  1. People a campaign phoned voted more often than people it didn’t
  2. Countries that got UN peacekeepers had more violence afterward than countries that didn’t
  3. Students who got tutoring score lower than students who didn’t
  4. Back to our data: civil wars can make a country less democratic. Which way does that bias the democracy gap?

Signing the bias: answers

Comparison Likely culprit Effect on the treatment Effect on the outcome Bias
Phone calls and turnout Voted in past elections + campaigns call regular voters + Up: calls look more effective than they are
Peacekeepers and violence How bad the war was + peacekeepers go to the hardest cases + Up: peacekeepers look harmful
Tutoring and test scores Struggling in school + struggling students get tutored – Down: tutoring looks harmful
Democracy and civil war Civil war itself (reverse causality) – war erodes democracy + Down: democracy looks more peaceful

We can’t always sign the bias in advance

Confounders can pull opposite ways

Income pushes the bias down. Population pushes it up: democracies are a bit bigger, and bigger countries have more civil wars

Which push wins? Hard to guess. For confounders no one measured, impossible to check

The sign gives the direction, not the size

“Negative” says the estimate is too low, not by how much

A big bias can even flip the sign. Police go where crime is high, so places with more police can have more crime, even if police reduce it (Bueno de Mesquita and Fowler 2021, 176–77)

Signing the bias is useful, but not a fix. Take POL 285 to learn more about how to address these issues

Your turn: make up your own confounded world

Start from the waffle code:

  1. Pick a confounder, a treatment, and an outcome from your own field or everyday life. Write the code so the treatment does nothing
  2. Before you run it, sign the bias: will the line through all the data tilt up or down?
  3. Run it and plot it. Compare that line with the lines within each level of the confounder. Were you right?
  4. Bonus: give the treatment a real effect, and make the bias big enough to flip the sign of the estimate

Measurement

Measurement is the second half of a data-generating process

Week 1’s definition had two halves:

“A data-generating process (DGP) is a rule or set of rules governing the social or political events that an analyst wishes to study and the rules by which observations of its results come to be represented in a dataset.”

— Gailmard (2014, 70)

So far today: the first half, the events themselves. Now the second half: how a civil war, a regime, or a person’s race turns into a number in our data

That step has its own systematic part and its own random part

Every variable is a measure of some concept

Concept What we measured
Race (week 1) The name on a résumé
Democracy A Polity score above 5
Wealth GDP per capita
Civil war Fighting between a state and rebels that killed at least 1,000 people

Before we compare anything, ask: does the number capture the concept?

A variable’s type limits the math we can do with it

Type What the numbers mean Example
Nominal Just labels Country, region
Ordinal Order, but the gaps may not be equal Polity score, “agree/disagree” survey scales
Interval Equal gaps, but no true zero Year
Ratio Equal gaps and a true zero GDP per capita, population

These four types come from Stevens (1946)

Measuring democracy as ordinal or interval has consequences

We chose “Polity above 5.” Another cutoff gives another answer

New civil wars by regime, under three definitions of democracy. Country-years, 1945–1999. Data: Fearon and Laitin (2003), via the juanr package
Democracy if Polity is above… Share counted as democracies War rate, democracies War rate, the rest Gap (points)
0 43% 1.29% 1.96% -0.67
5 34% 0.82% 2.12% -1.29
9 19% 0.33% 1.98% -1.65

A cutoff turns a scale into a yes/no. It’s a choice, and here it moves the gap by more than a point

Three questions to ask of any measure

  1. Face validity: on its face, does it look like the concept?
  1. Content validity: does it cover every part of the concept?
  1. Construct validity: does it track the concept, and only the concept? Does it behave the way the concept should?

For the fuller version, see Adcock and Collier (2001)

Face validity: Do Polity scores appear sensible?

Polity scores for 10 countries in 1998. Data: Fearon and Laitin (2003), via the juanr package
Country Polity score Democracy? (above 5)
United States 10 Yes
United Kingdom 10 Yes
South Africa 9 Yes
India 9 Yes
Brazil 8 Yes
Mexico 6 Yes
Russia 4 No
Iran 3 No
Indonesia -5 No
China -7 No

The United States and the UK at the top, China at the bottom. It passes the sniff test, a low bar

Content validity: Polity leaves out who gets to vote

Polity scores in years when many adults couldn’t vote. Data: Fearon and Laitin (2003), via the juanr package
Country Years Polity score Who couldn’t vote
Switzerland 1950–1970 10 Women, in federal elections, until 1971
United States 1945–1964 10 Most Black Southerners, until the Voting Rights Act of 1965

Polity scores competition and limits on the executive, not how many people can vote. It misses part of what most of us mean by democracy (Paxton 2000)

Does this matter? It depends on your theory; what aspects of democracy does your theory depend on? How exactly does democracy affect your outcome?

Construct validity: does the measure track the concept, and only the concept?

Week 1: the name on a résumé measured race. But a name also signals class (Fryer and Levitt 2004)

A measure that picks up a second concept fails this test: we can’t tell which concept moved the outcome

A second check: does the measure relate to other things the way the concept should? A good democracy measure should agree with other democracy measures, like V-Dem from week 1

A measure can miss in the same two ways as an estimate

\[ \text{Measured value} = \text{True value} + \text{Bias} + \text{Noise} \]

Invalid: systematic error

Misses in the same direction every time

We are interested in broad participation, but Polity gives Switzerland a 10 before women could vote

Unreliable: random error

Misses at random

Two coders read the same country differently; a survey respondent answers differently on another day

A stable result: the middle of the Polity scale is most dangerous

Country-years, 1945–1999, by Polity score. Data: Fearon and Laitin (2003), via the juanr package

A well-known finding: regimes in the middle (“anocracies”) have the most violence (Hegre et al. 2001). The story: dictators can crush resistance, and democracies give people a peaceful way to seek reform. Regimes in the middle can do neither

Part of Polity’s middle range is defined by political violence

Two of Polity’s parts have a “factional” category, for politics that is “intense, hostile, and frequently violent” (Polity’s codebook, quoted in Vreeland 2008)

So a country sliding into violence gets scored toward the middle. The treatment is measured partly with the outcome: the bias in the measure depends on the violence

Drop those parts, and the link between the middle and civil war disappears (Vreeland 2008)

Is there more violence in the middle?

A more recent study, using machine learning, finds the pattern is still there (Jones and Lupu 2018)

Violence gets counted where someone can see it

Much conflict data comes from news reports. Violence far from reporters is more likely to be missed (Weidmann 2016)

Areas with cell phone coverage seem to have more violence (Pierskalla and Hollenbach 2013). Part of that “effect” is that violence in covered areas gets reported more (Weidmann 2016)

A biased measure of the outcome can create an effect that isn’t there

Noise in the outcome blurs the line. Noise in the treatment flattens it

Noise in the outcome and noise in the treatment can produce different patterns:

Orange dashed line: the true effect (a slope of 1). Black line: the line through what we measured. Simulated data

Noise in \(Y\) leaves the slope near 1. Noise in \(X\) cuts it about in half

Why: noise in the treatment mixes up who got what

Noise in the outcome

Pushes each point up or down at random

At any value of X, the ups and downs cancel. The line stays put; we just see it less clearly. More noise, no bias

Noise in the treatment

Pushes each point left or right at random

The points at the far right are partly there because of a big random push. Their Y is ordinary, so the right end of the line sags. The line flattens toward zero: bias

Wrapping up

Before you trust an effect estimate, ask three questions

  1. Who stands in for the missing world? If the groups differ in other ways, the estimate is biased (confounding)
  1. Which way would the bias push it? Sign it with the two questions, and test your reasoning with a simulation
  1. Does the number capture the concept? Measures can be biased or noisy (measurement)

References

Adcock, Robert, and David Collier. 2001. “Measurement Validity: A Shared Standard for Qualitative and Quantitative Research.” American Political Science Review 95(3): 529–46.
Bertrand, Marianne, and Sendhil Mullainathan. 2004. “Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” American Economic Review 94(4): 991–1013. doi:10.1257/0002828042002561.
Bueno de Mesquita, Ethan, and Anthony Fowler. 2021. Thinking Clearly with Data: A Guide to Quantitative Reasoning and Analysis. Princeton, NJ: Princeton University Press.
Fearon, James D., and David D. Laitin. 2003. “Ethnicity, Insurgency, and Civil War.” American Political Science Review 97(1): 75–90.
Fryer, Roland G., and Steven D. Levitt. 2004. “The Causes and Consequences of Distinctively Black Names.” Quarterly Journal of Economics 119(3): 767–805. doi:10.1162/0033553041502180.
Gailmard, Sean. 2014. Statistical Modeling and Inference for Social Science. New York: Cambridge University Press.
Hegre, Håvard, Tanja Ellingsen, Scott Gates, and Nils Petter Gleditsch. 2001. “Toward a Democratic Civil Peace? Democracy, Political Change, and Civil War, 1816–1992.” American Political Science Review 95(1): 33–48.
Jones, Zachary M, and Yonatan Lupu. 2018. “Is There More Violence in the Middle?” American Journal of Political Science 62(3): 652–67.
McElreath, Richard. 2020. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. 2nd ed. Boca Raton, FL: CRC Press.
Paxton, Pamela. 2000. “Women’s Suffrage in the Measurement of Democracy: Problems of Operationalization.” Studies in Comparative International Development 35(3): 92–111.
Pierskalla, Jan H., and Florian M. Hollenbach. 2013. “Technology and Collective Action: The Effect of Cell Phone Coverage on Political Violence in Africa.” American Political Science Review 107(02): 207–24.
Stevens, S. S. 1946. “On the Theory of Scales of Measurement.” Science 103(2684): 677–80.
Vreeland, James Raymond. 2008. “The Effect of Political Regime on Civil War: Unpacking Anocracy.” Journal of Conflict Resolution 52(3): 401–25.
Weidmann, Nils B. 2016. “A Closer Look at Reporting Bias in Conflict Event Data.” American Journal of Political Science 60(1): 206–18. doi:10.1111/ajps.12196.