Comparing Treatments Using Randomized Experiments - Statistics
Card 1 of 30
Decide significance: If a randomization test gives $p=0.03$ at $\alpha=0.05$, what is the conclusion?
Decide significance: If a randomization test gives $p=0.03$ at $\alpha=0.05$, what is the conclusion?
Tap to reveal answer
Reject $H_0$; the difference is statistically significant. $p < \alpha$ means reject null hypothesis.
Reject $H_0$; the difference is statistically significant. $p < \alpha$ means reject null hypothesis.
← Didn't Know|Knew It →
Decide significance: If a randomization test gives $p=0.42$ at $\alpha=0.05$, what is the conclusion?
Decide significance: If a randomization test gives $p=0.42$ at $\alpha=0.05$, what is the conclusion?
Tap to reveal answer
Fail to reject $H_0$; the difference is not statistically significant. $p > \alpha$ means insufficient evidence to reject null.
Fail to reject $H_0$; the difference is not statistically significant. $p > \alpha$ means insufficient evidence to reject null.
← Didn't Know|Knew It →
Identify the correct simulation step: What must be kept fixed when shuffling labels in a randomization test?
Identify the correct simulation step: What must be kept fixed when shuffling labels in a randomization test?
Tap to reveal answer
The observed responses and group sizes; only treatment labels are shuffled. Simulates null hypothesis by reassigning treatments randomly.
The observed responses and group sizes; only treatment labels are shuffled. Simulates null hypothesis by reassigning treatments randomly.
← Didn't Know|Knew It →
What does it mean if the randomization-test $p$-value is small (for example, $<0.05$)?
What does it mean if the randomization-test $p$-value is small (for example, $<0.05$)?
Tap to reveal answer
The observed difference is unlikely by chance; evidence of a treatment effect. Small p-values suggest the difference isn't due to chance.
The observed difference is unlikely by chance; evidence of a treatment effect. Small p-values suggest the difference isn't due to chance.
← Didn't Know|Knew It →
Which decision rule uses significance level $\alpha$ with a randomization-test $p$-value?
Which decision rule uses significance level $\alpha$ with a randomization-test $p$-value?
Tap to reveal answer
Reject $H_0$ if $p\text{-value} \le \alpha$. Standard hypothesis testing decision rule.
Reject $H_0$ if $p\text{-value} \le \alpha$. Standard hypothesis testing decision rule.
← Didn't Know|Knew It →
What is the definition of a $p$-value in a randomization test for two treatments?
What is the definition of a $p$-value in a randomization test for two treatments?
Tap to reveal answer
Proportion of simulated stats $c^6$ as or more extreme than observed. Measures how extreme the observed difference is.
Proportion of simulated stats $c^6$ as or more extreme than observed. Measures how extreme the observed difference is.
← Didn't Know|Knew It →
What is the standard randomization-test assumption about the treatment effect under $H_0$?
What is the standard randomization-test assumption about the treatment effect under $H_0$?
Tap to reveal answer
Under $H_0$, treatment labels are exchangeable (no real effect). If no effect exists, any assignment gives similar results.
Under $H_0$, treatment labels are exchangeable (no real effect). If no effect exists, any assignment gives similar results.
← Didn't Know|Knew It →
What is the null hypothesis for a randomization test comparing two treatments on a proportion?
What is the null hypothesis for a randomization test comparing two treatments on a proportion?
Tap to reveal answer
$H_0:\ p_1 - p_2 = 0$ (no treatment effect). Tests if treatment proportions are equal (no effect).
$H_0:\ p_1 - p_2 = 0$ (no treatment effect). Tests if treatment proportions are equal (no effect).
← Didn't Know|Knew It →
What is the null hypothesis for a randomization test comparing two treatments on a mean?
What is the null hypothesis for a randomization test comparing two treatments on a mean?
Tap to reveal answer
$H_0:\ \mu_1 - \mu_2 = 0$ (no treatment effect). Tests if treatment means are equal (no effect).
$H_0:\ \mu_1 - \mu_2 = 0$ (no treatment effect). Tests if treatment means are equal (no effect).
← Didn't Know|Knew It →
What statistic estimates $bc_1 - bc_2$ from sample data in a two-treatment experiment?
What statistic estimates $bc_1 - bc_2$ from sample data in a two-treatment experiment?
Tap to reveal answer
The difference in sample means, $\bar{x}_1 - \bar{x}_2$. Sample means estimate population means.
The difference in sample means, $\bar{x}_1 - \bar{x}_2$. Sample means estimate population means.
← Didn't Know|Knew It →
What statistic estimates $p_1 - p_2$ from sample data in a two-treatment experiment?
What statistic estimates $p_1 - p_2$ from sample data in a two-treatment experiment?
Tap to reveal answer
The difference in sample proportions, $c^6p_1 - c^6p_2$. Sample proportions estimate population proportions.
The difference in sample proportions, $c^6p_1 - c^6p_2$. Sample proportions estimate population proportions.
← Didn't Know|Knew It →
Find the observed statistic: $\bar{x}_1=18.2$, $\bar{x}_2=16.9$; what is $\bar{x}_1-\bar{x}_2$?
Find the observed statistic: $\bar{x}_1=18.2$, $\bar{x}_2=16.9$; what is $\bar{x}_1-\bar{x}_2$?
Tap to reveal answer
$1.3$. Direct subtraction: $18.2 - 16.9 = 1.3$
$1.3$. Direct subtraction: $18.2 - 16.9 = 1.3$
← Didn't Know|Knew It →
What parameter is compared when the response is quantitative for two treatments?
What parameter is compared when the response is quantitative for two treatments?
Tap to reveal answer
The difference in population means, $bc_1 - bc_2$. Compares average outcomes between two treatment groups.
The difference in population means, $bc_1 - bc_2$. Compares average outcomes between two treatment groups.
← Didn't Know|Knew It →
What parameter is compared when the response is categorical (success/failure) for two treatments?
What parameter is compared when the response is categorical (success/failure) for two treatments?
Tap to reveal answer
The difference in population proportions, $p_1 - p_2$. Compares success rates between two treatment groups.
The difference in population proportions, $p_1 - p_2$. Compares success rates between two treatment groups.
← Didn't Know|Knew It →
What is the response variable in a randomized experiment comparing two treatments?
What is the response variable in a randomized experiment comparing two treatments?
Tap to reveal answer
The outcome measured on each experimental unit. The variable we measure to compare treatment effects.
The outcome measured on each experimental unit. The variable we measure to compare treatment effects.
← Didn't Know|Knew It →
What is the key purpose of random assignment in a randomized experiment comparing two treatments?
What is the key purpose of random assignment in a randomized experiment comparing two treatments?
Tap to reveal answer
To reduce confounding and support a cause-and-effect conclusion. Random assignment ensures groups are comparable except for treatment.
To reduce confounding and support a cause-and-effect conclusion. Random assignment ensures groups are comparable except for treatment.
← Didn't Know|Knew It →
What is the typical center of the randomization distribution for $\bar{x}_1-\bar{x}_2$ under $H_0$?
What is the typical center of the randomization distribution for $\bar{x}_1-\bar{x}_2$ under $H_0$?
Tap to reveal answer
Approximately $0$. Under $H_0$, no treatment effect means difference centers at 0.
Approximately $0$. Under $H_0$, no treatment effect means difference centers at 0.
← Didn't Know|Knew It →
Which tail(s) should be used for a $p$-value when the alternative is $H_a:\ \mu_1-\mu_2>0$?
Which tail(s) should be used for a $p$-value when the alternative is $H_a:\ \mu_1-\mu_2>0$?
Tap to reveal answer
Right-tail probability (simulated differences $\ge$ observed). One-sided test looks only at differences in predicted direction.
Right-tail probability (simulated differences $\ge$ observed). One-sided test looks only at differences in predicted direction.
← Didn't Know|Knew It →
Which tail(s) should be used for a $p$-value when the alternative is $H_a:\ \mu_1-\mu_2\ne^0$?
Which tail(s) should be used for a $p$-value when the alternative is $H_a:\ \mu_1-\mu_2\ne^0$?
Tap to reveal answer
Two tails: simulated differences with $|\text{stat}| \ge |\text{observed}|$. Two-sided test considers extreme differences in either direction.
Two tails: simulated differences with $|\text{stat}| \ge |\text{observed}|$. Two-sided test considers extreme differences in either direction.
← Didn't Know|Knew It →
Find the observed statistic: Treatment A has $12/40$ successes and B has $6/40$; what is $c^6p_A-c^6p_B$?
Find the observed statistic: Treatment A has $12/40$ successes and B has $6/40$; what is $c^6p_A-c^6p_B$?
Tap to reveal answer
$0.15$. $\frac{12}{40} - \frac{6}{40} = 0.30 - 0.15 = 0.15$
$0.15$. $\frac{12}{40} - \frac{6}{40} = 0.30 - 0.15 = 0.15$
← Didn't Know|Knew It →
If the observed difference is near the center of the randomization distribution, what is the conclusion?
If the observed difference is near the center of the randomization distribution, what is the conclusion?
Tap to reveal answer
Not statistically significant; insufficient evidence of a treatment effect. Common results suggest random variation, not treatment.
Not statistically significant; insufficient evidence of a treatment effect. Common results suggest random variation, not treatment.
← Didn't Know|Knew It →
If the observed difference is in the extreme tail of the randomization distribution, what is the conclusion?
If the observed difference is in the extreme tail of the randomization distribution, what is the conclusion?
Tap to reveal answer
The difference is statistically significant; evidence of a treatment effect. Extreme results suggest treatment causes the difference.
The difference is statistically significant; evidence of a treatment effect. Extreme results suggest treatment causes the difference.
← Didn't Know|Knew It →
What is the $p$-value in a randomization test comparing two treatments?
What is the $p$-value in a randomization test comparing two treatments?
Tap to reveal answer
Proportion of simulated statistics at least as extreme as observed. Counts how often random chance produces results this extreme.
Proportion of simulated statistics at least as extreme as observed. Counts how often random chance produces results this extreme.
← Didn't Know|Knew It →
What is the correct interpretation of a small $p$-value in a randomized experiment?
What is the correct interpretation of a small $p$-value in a randomized experiment?
Tap to reveal answer
The observed result is unlikely if $H_0$ is true. Low probability under null suggests treatment effect.
The observed result is unlikely if $H_0$ is true. Low probability under null suggests treatment effect.
← Didn't Know|Knew It →
What is the purpose of random assignment in a randomized experiment comparing two treatments?
What is the purpose of random assignment in a randomized experiment comparing two treatments?
Tap to reveal answer
To reduce confounding and support a cause-and-effect conclusion. Random assignment ensures groups are comparable except for treatment.
To reduce confounding and support a cause-and-effect conclusion. Random assignment ensures groups are comparable except for treatment.
← Didn't Know|Knew It →
What is a Type I error in the context of comparing two treatments with $H_0$ true?
What is a Type I error in the context of comparing two treatments with $H_0$ true?
Tap to reveal answer
Rejecting $H_0$ and claiming a treatment effect when none exists. False positive: concluding effect when there's none.
Rejecting $H_0$ and claiming a treatment effect when none exists. False positive: concluding effect when there's none.
← Didn't Know|Knew It →
What parameter is compared for two treatments when the response variable is categorical (success/failure)?
What parameter is compared for two treatments when the response variable is categorical (success/failure)?
Tap to reveal answer
The difference in population proportions, $p_1 - p_2$. Compares success rates between two groups.
The difference in population proportions, $p_1 - p_2$. Compares success rates between two groups.
← Didn't Know|Knew It →
Identify the correct scope: In a randomized experiment, what supports a cause-and-effect conclusion?
Identify the correct scope: In a randomized experiment, what supports a cause-and-effect conclusion?
Tap to reveal answer
Random assignment of experimental units to treatments. Random assignment eliminates confounding variables.
Random assignment of experimental units to treatments. Random assignment eliminates confounding variables.
← Didn't Know|Knew It →
Identify the correct scope: In a randomized experiment, to what population can results be generalized?
Identify the correct scope: In a randomized experiment, to what population can results be generalized?
Tap to reveal answer
Only to the population represented by the random sample (if one exists). Without random sampling, can't generalize beyond participants.
Only to the population represented by the random sample (if one exists). Without random sampling, can't generalize beyond participants.
← Didn't Know|Knew It →
What is a Type II error in the context of comparing two treatments with a real effect present?
What is a Type II error in the context of comparing two treatments with a real effect present?
Tap to reveal answer
Failing to reject $H_0$ when a treatment effect actually exists. False negative: missing a real treatment effect.
Failing to reject $H_0$ when a treatment effect actually exists. False negative: missing a real treatment effect.
← Didn't Know|Knew It →