The Experimental Approach
Module 4
History of Randomised Experiments
The methodology of randomized experiments serves as a critical foundation for causal inference in economics and other sciences. The application of randomized control trials in social sciences drew significant inspiration from clinical trials in medicine. Over centuries, experimental design evolved from small-scale testing to explicit statistical randomization.
| Experimenter & Year | Context & Design | Key Contribution |
|---|---|---|
| James Lind (1774) | Recruited eight sailors with scurvy. Divided them into two groups of four (lemons vs. control). | One of the earliest recorded uses of randomized trials in medicine to test the efficacy of vitamin C against scurvy. |
| Ronald Fisher (1930s-40s) | Authored Statistical Method for Research Workers and Design of Experiments. | Proposed the explicit use of randomization in experimental design to achieve clean causal inference. |
| Austin B. Hill et al. (1948) | Tested if streptomycin cures tuberculosis. Used an RCT because the drug was in limited supply. | Led to the widespread adoption of RCTs in the medical community to evaluate treatment efficacy. |
| Jonas Salk (1954) | Designed a double-blind randomized control trial to test the polio vaccine. | Used Fisher's exact test to measure treatment effects, proving the vaccine reduced the risk of polio. |
Social and Lab Experiments
Randomized experiments eventually transitioned from medicine to social sciences and laboratory settings. Initial social experiments aimed to test policy interventions, while early lab experiments focused on individual decision-making and social interactions.
| Researchers & Year | Experiment Focus | Key Findings & Context |
|---|---|---|
| RAND Corporation (1971-1982) | Causal effect of healthcare insurance and copays on healthcare usage. | Cost-sharing treatments led to fewer physician visits, fewer hospitalizations, and less overall spending without adverse health effects. |
| Burtless and Hausman (1977) | Tested if a negative income tax (cash transfers) reduced willingness to work. | Found no effect on individual labor supply, but the experiment was underpowered and had too many treatment arms. |
| Thurnstone (1931) | Tested indifference between pairs of goods (hats, coats, shoes). | First to use lab experiments to deduce individual indifference curves. |
| Von Neumann & Morgenstern (1944) | Proposed four reasons for lab experiments in game theory. | Suggested experiments can construct utility curves, predict future behavior, and test behavior toward complex risks using real money. |
| Dresher & Flood (1950) | Implemented a version of the Prisoner's Dilemma over 100 repetitions. | Found that human participants rarely play the socially efficient strategy, but also depart from the rational Nash equilibrium prediction. |
| Guth et al. (1982) | Bargaining between a proposer and a receiver in the Ultimatum Game. | Showed that proposers generally offer 40-50% of the pie, and offers below 20% are routinely rejected because people dislike unfair options. |
An Overview of Experiments
An experiment is a method that develops identical environments and changes one variable at a time to cleanly establish causal inference. The primary advantages of experimental methods include precise control over the data collection process and the ability to capture psychological variables of interest that are typically unobservable in standard observational data.
| Discipline | Definition & Role of Experiments |
|---|---|
| Behavioral Economics | A sub-discipline of economics focused on psychological factors in decision-making. |
| Experimental Economics | A methodological approach using experiments as a tool to study behavioral economics, standard economics, or psychology. |
Lab Experiments
Lab experiments place human subjects into highly controlled environments to test specific theories, price discovery mechanisms, and psychological biases. These environments are strictly monitored to isolate the exact mechanism driving behavior. Participants are financially incentivized based on their choices to ensure decisions reflect true preferences.
| Type of Auction | Bidding Mechanism | Payment Rule |
|---|---|---|
| First Price Sealed Bid (FPA) | Bidders submit sealed bids. Highest bidder wins. | Winner pays their exact bid amount. |
| Second Price Sealed Bid (SPA) | Bidders submit sealed bids. Highest bidder wins. | Winner pays the amount of the second-highest bid. |
| Third Price Sealed Bid (TPA) | Novel experimental format. Highest bidder wins. | Winner pays the amount of the third-highest bid. |
| Aspect | Characteristics of Lab Experiments |
|---|---|
| Advantages | Total researcher control over data collection. Clean identification of mechanisms. Precise causal inference. Allows for multiple specific treatment variations. |
| Disadvantages | Low external validity. Generalizability is limited because the subject pool primarily consists of university students responding to small financial incentives. |
Lab-in-the-field Experiments
To counter the reliance on standard student populations, researchers developed lab-in-the-field experiments (also known as artefactual field experiments). This methodology brings the controlled laboratory setup directly into natural settings, such as setting up computers in a rural village. Standard lab participants are often referred to using the acronym WEIRD (Western, Educated, Industrialized, Rich, and Democratic). Lab-in-the-field experiments specifically target non-WEIRD populations to answer context-specific questions.
| Aspect | Characteristics of Lab-in-the-field Experiments |
|---|---|
| Advantages | Greater generalizability than traditional lab experiments. Maintains a high degree of control over the experimental environment. Reaches targeted, relevant populations (e.g., farmers, specific caste categories). |
| Disadvantages | Less generalizable than large-scale natural field experiments. Expensive and logistically challenging to transport laboratory paraphernalia to remote areas. |
Online and Survey Experiments
Online platforms (such as Amazon MTurk, Prolific, and Qualtrics) allow researchers to deploy experiments to massive subject pools across different geographies. A specific subset is the survey experiment, where treatment conditions (such as varied framing of economic policies) are embedded directly within a survey to measure attitudes, expectations, and narratives.
| Aspect | Characteristics of Online & Survey Experiments |
|---|---|
| Advantages | Highly cost-effective. High generalizability due to large-scale data collection from diverse locations. Allows for quota-based representative sampling. |
| Disadvantages | High participant dropout rates. Researchers cannot control the environment (participants may be distracted, watching TV, or multitasking). Survey experiments are often unincentivized. |
Field Experiments and Randomized Control Trials
Field experiments and Randomized Control Trials (RCTs) observe outcomes in naturally occurring environments (schools, hospitals, factories) without subjects explicitly knowing they are part of a systematic study. This prevents observation biases such as the Hawthorne effect or social desirability bias.
| Concept | Distinction |
|---|---|
| Field Experiment | Tests precise theoretical predictions and mechanisms in a natural setting. |
| Randomized Control Trial (RCT) | Tests hypotheses directly linked to large-scale policy evaluations and implementations. |
| Case Study | Researchers | Finding & Context |
|---|---|---|
| Charitable Giving | John List et al. | Door-to-door field experiment. Found that people often donate due to the social pressure of disliking saying no, rather than pure altruism. |
| Labor Market Discrimination | Bertrand & Mullainathan | Field experiment sending identical resumes with distinct racially identifiable names. Proved severe racial discrimination in interview callback rates. |
| Malaria & Bed Nets | Michael Kremer & Ted Miguel | RCT evaluating treated bed nets on child mortality and income. Contributed to a 2019 Nobel Prize. |
| Educational Interventions | Banerjee, Cole, Duflo, Linden | RCT testing the Balsakhi program (remedial tutors) and CAL program (computer-assisted learning) in India to increase student learning levels. |
| Aspect | Characteristics of Field Experiments & RCTs |
|---|---|
| Advantages | Highly generalizable. Considerable external validity. Captures authentic actions free from laboratory-induced biases. |
| Disadvantages | Extremely expensive. Logistically demanding. Difficult to isolate precise underlying mechanisms without complementary survey measures. |
Non-Randomised Experiments
While randomization is the gold standard for causal inference, experiments can still be conducted without strict randomization to carefully measure outcomes, capture social norms, and understand societal mechanisms that are otherwise unmeasurable.
| Case Study Focus | Researchers | Methodology & Context |
|---|---|---|
| Nation Building & Trust | Blouin and Mukand | Compared Rwandan villages exposed to a government radio program against unexposed villages using a Trust Game. |
| Poverty & Cognitive Bandwidth | Anandi Mani et al. | Tested farmers' cognitive functions pre-harvest (when poor) and post-harvest (when rich) to show financial scarcity decreases cognitive bandwidth. |
| Gender Differences in Competition | Gneezy, List, et al. | Compared behavior in a patriarchal society (Maasai) versus a matrilineal society (Khasi) to prove competitive traits are driven by nurture, not nature. |
| Low Promotability Tasks | Linda Babcock et al. | Measured the likelihood of men and women volunteering for tasks that do not lead to promotion. Found women receive and accept these requests more often. |
| Social Integration | Gautam Rao | Used dictator games to compare wealthy Delhi students exposed to poor peers (due to a court mandate) against unexposed students. |
Ethics in Research and Historical Context
Because economic experiments involve human subjects, strict ethical scientific conduct is mandatory. Modern ethical frameworks were developed in direct response to severe historical violations of basic human rights.
| Historical Violation | Context & Impact |
|---|---|
| Nazi Twin Experiments | Conducted by Joseph Mengele (1943-1945) in Auschwitz involving amputations and disease infection under the pretext of genetic research. |
| Tuskegee Syphilis Study | Conducted in rural Alabama (1932-1972). African American men with syphilis were deliberately denied penicillin to observe the disease progression. |
| Stanford Prison Experiment | Conducted by Zimbardo (1971). Students acted as guards and prisoners. Guards became sadistic, resulting in severe psychological trauma. |
| Milgram Obedience Experiment | Conducted at Yale (1961-1962). Participants were pressured by authority to administer supposedly lethal electric shocks to learners, causing severe distress. |
Following these violations, the Belmont Report (1979) established the foundational pillars of modern research ethics.
| Belmont Report Pillar | Definition & Application |
|---|---|
| Respect for Persons | Treating participants as autonomous agents. Requires full informed consent, right to withdraw at any time, and special protections for vulnerable populations (minors, prisoners). |
| Beneficence | The obligation to maximize benefits for the scientific community while minimizing risk and harm to the participants. |
| Justice | Ensuring fair distribution of benefits and burdens. Prevents the exploitation of vulnerable populations for the sake of scientific advancement. |
To enforce these pillars, modern protocols mandate ethics training (such as the CITI program) and require full experimental designs to be approved by an Institutional Review Board (IRB). In behavioral economics, deception (providing objectively false information) is strictly prohibited. The IRB categorizes research based on the level of risk to human subjects.
| IRB Review Category | Description |
|---|---|
| Exempt Review | Granted for studies with minimal risk (e.g., standard educational settings or basic surveys). |
| Expedited Review | Granted when risk is no more than minimal. The IRB chair quickly approves with minor modifications. |
| Full Board Review | Required when the assessed risk is greater than minimal risk. The entire board evaluates the protocol. |
Project
An experimental project applies behavioral economics principles to a specific context. The methodology requires clearly identifying a research question, laying down predictions, and designing a randomized experiment with at least one treatment condition and one control condition. Researchers use tools like Qualtrics (which has an inbuilt randomizer) for data collection.
A crucial component of result analysis is the balance table, which ensures that the randomized assignment successfully created statistically similar groups. The balance table compares the means of covariates (age, gender, income) between the treatment and control groups. Different statistical tests are utilized depending on the data type of the covariate or outcome.
| Statistical Test | Application Rule |
|---|---|
| T-test | Used when testing differences between continuous variables (e.g., age, income). |
| Chi-Square Test | Used when testing differences between binary or categorical variables (e.g., female: yes/no). |
The final analysis requires comparing the main outcome variable between the treatment and control columns. If the treatment was successful, a statistically significant difference should emerge, allowing the researcher to establish causality rather than mere correlation.
Ultra-Quick Revision (Exam Essentials)
Key Concepts & Distinctions
- Behavioral vs. Experimental Economics: Behavioral economics studies the psychological factors of decision-making, while experimental economics is the methodological tool used to test those factors.
- Types of Experiments Hierarchy:
- Lab Experiment: Ultimate control, low generalizability, WEIRD subjects.
- Lab-in-the-field: Brings lab control to targeted non-standard field populations.
- Online/Survey: High scale, low cost, minimal control.
- Field/RCT: High external validity, subjects act naturally, highest cost.
- Field Experiment vs. RCT: Field experiments test precise theoretical predictions in nature. RCTs test hypotheses specifically linked to policy interventions.
- Randomization: The required mechanism to isolate variables and establish pure causality rather than correlation.
- Balance Table Purpose: Proves that the treatment and control groups are statistically identical regarding baseline characteristics prior to the intervention.
Must-Know Terms
- WEIRD: An acronym describing the typical university subject pool (Western, Educated, Industrialized, Rich, Democratic).
- Hawthorne Effect: The alteration of subject behavior purely because they know they are being observed by researchers.
- Deception: Providing false information to subjects. Strictly banned in experimental economics.
- Institutional Review Board (IRB): The committee that reviews research protocols to protect human subjects from harm.
- Belmont Report: The 1979 ethical foundation document establishing Respect for Persons, Beneficence, and Justice.
- Ultimatum Game: A strategic lab game proving humans reject unfair financial distributions, violating rational Nash equilibrium predictions.
- T-test vs. Chi-Square: Statistical tools to compare means. T-test is for continuous data, Chi-Square is for binary data.