Testing for Differences Between Two Groups: t test

Reading: SPSS Base 9.0 User's Guide: Chapter 18, T Tests
Homework: t test
Download: ttest.sav        (Download Tips)

1. Overview

Suppose you have two groups, an experimental group and a control group. You may wish to test the null hypotheses that there are no differences between your two groups as measured by your dependent variable. Your task is to choose the appropriate statistical test for your data.

In this set of notes we will look at three statistical tests that are designed for interval data:
a) the independent t test,
b) the dependent t test, and
c) Welch's t test;
and two tests that are designed for ordinal data:
d) the Mann-Whitney U; and
e) the Wilcoxon test.

Each of those statistical tests make certain assumptions about your data. Choosing the appropriate test is a matter of determining which assumptions have been met by your data. In general, tests designed for interval data will be more powerful than tests designed for ordinal or nominal data. Because of this our strategy is to first check if the t-test assumptions have been met. If the t-test assumptions are met we will use it, if the t-test assumptions are not met then we will use one of the tests designed for ordinal data.

There are four basic assumptions:

The question about whether the two groups are independent or related is really a question about whether the means in the two groups are independent or not. If participants were randomly assigned to the two conditions, then the groups are considered to be independent. If the two means are based on the same people, then the means are considered to be dependent. For example, if group 1 was a pretest and group 2 was a posttest and the same people responded to the pretest and posttest, then the means are considered to be dependent. Another way of saying this is as follows: means that are "between subject" means are usually independent. Means that are "within subject" are dependent. Means that are "repeated measures" are dependent means.

The scale of measurement question is answered by considering the nature of the dependent variable. Does the dependent variable meet all the rules of an interval scale? Does it only meet the rules of an ordinal scale? If the dependent variable is nominal then none of these tests may be used.

The final two questions about the distributions (shape and homogeneity) of the means in the two groups will be answered by looking at the output generated by the SPSS procedure explore.

Several examples will be given. For each example we will first determine which statistic to use by asking the four questions about the statistical assumptions. Then we will run the appropriate statistical test of the null hypothesis. Finally we will interpret the output.

top

2. The Decision Tree

The decisions that must be made can be summarized in terms of the following decision tree.

Notes on the score distribution assumptions:

   (a) Kurtosis is not viewed as being a major threat to the
       t test.  If the two populations are symmetric, and if the
       variances are equal, then the t test may be used.  

   (b) If the two populations are symmetric, and the variances
       are not equal, then use Welch's t test.

   (c) Skewness is not a problem if the skewness is in 
       the same direction.  If the variances are equal then
       use a t test.

   (d) If skewness is in the same direction and the variances
       are unequal, then if the sample sizes are equal use 
       Welch's t test.

   (e) In most instances in social science combined sample sizes
       of 40 or more would be considered "moderately large."

   (f) See Myers and Well (1991, p. 69) for additional discussion
       of these points.

top

3. Independent Samples: The Data

The first data set we are going to look at is from a psychophysiological study of terror management. Briefly, participants were asked to write an essay either in support of or opposed to their own attitude towards the gulf war. They were then shown eight statements about death (e.g., Thinking about being buried after I die makes me anxious). Skin conductance responses were measured while the participants viewed each of the death statements. The skin conductance responses were scored as the number of skin conductance responses (SCR) made in response to viewing each statement.

The data are stored in the file ttest.sav. The variables in terror.sav are shown in Table 3.

The hypothesis was that writing an essay that was opposed to their own opinion would make the participants hypersensitive to the death related statements. Whereas writing an essay that was in support of their own attitude would not lead participants to be hypersensitive to the death related statements. Hypersensitivity to the death related statements was measured by the psychophysiological measure of skin conductance. Specifically it was predicted that participants who wrote an essay opposed to their own attitude would show a greater number of skin conductance responses to the death item than would participants who write an essay in support of their own attitudes. The hypothesis was tested using the sum of the number of skin conductance responses to the first four items. The first four items were selected because previous research has found habituation effects, the psychophysiological responses to repeated stimuli tend to get smaller across presentations. Consequently the effect was expected to be greatest for the first four items. The syntax commands that computed the sum of the number of skin conductance responses across the first four items and across the last four items are

top

4. Independent Samples: Testing the Assumptions

Be begin by testing the assumptions of the t test. If the assumptions are met, then the t test will provide the most powerful test of the hypothesis. If the assumptions of the t test are not met, then we should consider using other statistical tests.

Assumption 1 (independence). The observations in the two groups are independent because participants were randomly assigned to the two essay conditions.

Assumption 2 (scale of measurement). The scale of measurement for the total number of skin conductance response measure is ratio.

Assumption 3 (normality). This assumption can be tested by running several normality tests.

The Explore procedure can be used to provide information related to the normality assumption.  We have already looked at several of the statistics and graphics displays available in the Explore procedure including: histograms, stem and leaf plots, skewness and kurtosis statistics, normal probability plots, and the Kolmogorov-Smirnov and Shapiro-Wilk tests of normality (see Explore: I).  So far we know how to produce those statistics for a variable. The assumptions of the t test demand that we look at those assumptions within two different groups of people. In this case, we need to look at the skin conductance responses for those people randomly assigned to the bolster condition and for those people randomly assigned the refute condition. This is accomplished by moving the independent variable (bolster) to the factor list in the Explore window.

Here is the explore syntax that will create the appropriate statistics and graphs...

The syntax includes a "BY" keyword on the VARIABLES= subcommand. The independent variable (bolster) was added after the BY.  Adding the independent variable will create the requested statistics and graphs for each level of the independent variable. 

Statistical tests of the homogeneity assumption are provided by the spread-level tests within the explore procedure. However, the t test procedure automatically provides the Levene test of homogeneity. We will use that test instead of running spread-level tests in explore.

The stem and leaf plots for the two conditions are shown in Table 5. The sample sizes are small so it is difficult to see any shape to these curves.

The skewness and kurtosis statistics for the two conditions are shown in Table 6. The 95% confidence interval around the skewness and kurtosis scores for both cells include zero indicating that both scores are normally distributed.

The Kolmogorov-Smirnov (KS) and Shapiro-Wilk (SW) tests of normality are presented in Table 7. The only potential problem is the SW test for the "Refutes own attitude" cell, Shapiro-Wilk(7) = .772, p = .029. The KS is nonsignificant and the skewness and kurtosis tests are nonsignificant for that cell so we will assume that this data does meet the normality assumption.

4. Assumption 4 (homogeneity). Boxplots, as provided by the explore procedure, give a graphic representation of homogeneity. The boxplots are presented in Figure 1. The boxes are about the same height, indicating that the variances are homogeneous. The data also suggest that the t test will be significant. The median for the "Refutes own attitude" cell is higher than the median for the "Bolsters own attitude" cell and the boxes themselves do not overlap.

top

5. Independent Samples: Running the t Test

The t test procedure for independent groups is selected by clicking

Analyze
   Compare Means
          Independent-Samples T Test...  

The independent samples t test dialog box presents the variables in the active file in the box on the left. The dependent variable, nscr1_4 in this example, should be moved to the Test Variable(s): box. The Grouping Variable: is the independent variable, bolster in this example. Once the independent variable has been moved to the Grouping Variable: box the Define Groups... button will become available. Press that button. Our grouping variable, bolster has two levels (1 = Bolsters own attitude and 2 = Refutes own attitude. Enter "1" for Group1: and "2" for Group2:. This specifies that the two groups for the t test will be composed of the individuals in those two conditions. If there were any other conditions (or values) for bolster, those conditions would be excluded from the analysis.

It is possible to specify groups by using a Cut point. For example, suppose you wanted the use the median age to divide the participants in to the two groups. You would enter the median age as the cut point. One group would be composed of those individuals at or above the median age, the other group would be composed of those individuals below the median age.

There are three Options. The first option allows you to set the confidence interval for the mean (the default is the 95% confidence interval). The second and third options are used when you have specified more than one test variable. The exclude cases listwise option will delete any cases if that case has a missing value on any of the test variables or on the grouping variable. The exclude cases analysis by analysis will only exclude a case for the analysis in which it has a missing value.

Output: Group Statistics

The output from the t test includes a table for group statistics, see Table 8, and a table for the statistical tests, see Table 8. Table 7 shows the number of cases in each cell, the mean, the standard deviation, and the standard error of the mean.

The formula for the standard error of the mean is the standard deviation divided by the square root of the number of cases -

For "Bolsters own attitude" the standard error of the mean is

SE_M = 3.44 / Ö6
        = 3.44 / 2.449
        = 1.40

For the "Refutes own attitude" the standard error of the mean is

SE_M = 2.76 / Ö7
        = 2.76 / 2.6457
        = 1.04

Note: there is always the possibility of rounding error in these computations because we are using a rounded value for the standard deviations. It would be preferable to have at least three decimal places available when rounding to 2 decimal places.

Output: T-Test Statistics

The table that reports the t-test statistics includes Levene's test of homogeneity of variance.  The null hypothesis for the Levene test is that the variances are homogeneous.  For this set of data the Levene's test  is not significant, F(1, 11) = 1.209, p = .295, indicating the the null hypothesis cannot be rejected, that is, the variances are homogeneous (see Table 9). The homogeneity assumption has been met.

There are two rows of statistics displayed. The statistics in the row labeled "Equal variances assumed" should be used whenever Levene's test is not significant, that is, when the variances are homogeneous. The statistics in the row labeled "Equal variances not assumed" should be used whenever Levene's test is significant, that is, when the variances are not homogeneous. The t statistic in the "Equal variances not assumed" row is also called Welch's t test.

The T Test

The general formula for a t test is:

t test = mean difference / standard error of the difference

The t value (equal variances assumed) in this example is -3.351. 

Mean Difference

The mean difference is found by subtracting the mean of group 2 from the mean of group 1. We assigned the values for the two groups when we were using the dialog box to set up the independent t test. In this example the "1 = bolsters own attitude" condition was assigned as the value of Group 1:. We could have assigned the "2 = refutes own attitude" as the value for Group 1:. If we had done the latter the mean difference would have been positive rather than negative. For that reason the t test value is normally reported as the absolute value of the statistic rather than as the signed value of the statistic. That is, we normally drop the sign of the t test when it is reported. 

Note that if the two means were exactly the same, the mean difference would be zero. It would be rare to find the means from two treatment to be exactly the same.  The statistical test tells us how likely it is that the obtained mean difference is due to chance variation.  We want to know if the obtained mean difference is significantly different from zero.

Degrees of Freedom

When the variances are homogeneous the degrees of freedom value is found as the sum of the degrees of freedom in each cell. The degrees of freedom in a cell is the cell n - 1

df = (n₁ - 1) + (n₂ - 1)
    = (6 -1) + (7 - 1)
    = 5 + 6
    = 11

When the variances are not homogeneous the formula for the degrees of freedom is more complicated. It normally results in a fractional value for degrees of freedom, as in the example above. The formula is -

Standard Error of the Difference

When the variances are homogeneous, the standard error of the difference is computed by summing the standard deviations and dividing by the square root of the sum of the ns. This is also called the standard error of the difference with pooled variance estimates.

          = 6.20 / Ö13
          = 6.20 / 3.6056
          = 1.72

When the variances are not homogeneous, the standard error of the difference is found using separate variance estimates. The formula is -

           = 1.75

95% Confidence Interval of the Difference

If the 95% confidence interval of the mean difference includes zero, then the mean difference is not different from zero.  In that case the two means are not significantly different from each other.  The 95% confidence interval for the mean difference found as:

CI₉₅ = M_diff ± (cv)(SE_diff)

where the critical value (cv) is the value needed for the t-test statistic at the .05 level of significance with n df. With 11 degrees of freedom the critical value of the t test is 2.201 (from a table of the critical values of the t distribution).

CI₉₅ = M_diff ± (cv)(SE_diff)
        = -5.76 ± (2.201)(1.72)
        = -5.76 ± 3.79

So the 95 % confidence interval ranges from -9.55 to -1.97.  It does not include zero, so the mean difference is significantly different from zero.  That is, the two means are significantly different from each other.

Reporting the Results

The t test statistic is reported as follows

t(df) = t value, p = significance.

In our example the values would be:

t(11) = 3.35, p = .006.

In addition to reporting the value of the t test you should also report the means and standard deviations for the two groups. The APA Publication Manual (1994) also recommends reporting effect sizes. 

When reporting the results of any statistic the focus of description should be on the relative effects of the treatment for the participants in each group rather than on the statistic itself. This is hard to because you have been focusing on the statistics themselves, searching for the elusive significant effect that may allow your study to be published. The value of the statistic is used to justify your conclusion about the findings, it is rarely the focus of the interpretation. The following is a poorly constructed description of these results.

This description tells us that something happened, but does not describe how the independent variable or treatment affected the two groups in question. 

Write a description of the results shown in Tables 8 and 9. When you have finished press the Submit button.  

You can then press the View Descriptions button to see what you and your peers have (anonymously) written. Press the reload or refresh button your web viewer to update the descriptions as others submit them.

top

6. Effect Size for a t Test

SPSS does not report any effect size statistics within the output for a t test. The most commonly used effect size estimates are Cohen's d (Cohen, 1988) and the effect size correlation. Cohen's d is found by dividing the mean difference by the pooled standard deviation. The effect size correlation for a t test is computed as the Pearson's product moment correlation between the independent variable with two groups and the dependent variable. By convention, effect size measures are positive if the mean difference supports the hypothesis, and negative if the mean difference is opposite to that predicted by the hypothesis. 

A description of these effect size measures and the formula's for computing effect sizes can be found at:

Effect Size (ES).

Web-based calculators for computing Cohen's d and the correlation ratio from the means and standard deviations, and from the t test value can be found at:

Effect Size Calculators

Cohen's d for this data was 1.84, a large effect size.

top

7. Related Samples: The Data

Earlier we said that this type of social-psychophysiological data tends to show habituation. We could test the hypothesis that habituation was shown in this data by testing whether the sum of the skin conductance responses over the first four items, nscr1_4, is higher than the sum of the skin conductance responses over the last four items, nscr5_8.

top

8. Related Samples: Testing the Assumptions

Each participant has contributed two scores, the sum of the skin conductance responses to the first four items, nscr1_4, and the sum of the skin conductance responses to the last four items, nscr5_8. A dependent groups t test is a test of whether the differences between those two scores is different from zero. The assumptions of the dependent groups t test are assumptions about the difference scores. In order to test the assumptions we will need to compute the difference score. The following syntax command will create a difference score, scrdiff, by subtracting the sum of the SCRs to last four items from the sum of the SCRs to the first four items:

The difference score, scrdiff, will be positive if there is a habituation effect. It will be negative if the sum of the SCRs for the last four items are greater than the sum of the SCRs for the first four items. It will be zero if there is no differences in the number of SCRs to the first- and last-four items.

Assumption 1 (independence). The observations in the two groups are related because the same participants contributed to both scores. This is a repeated measures design.

Assumption 2 (scale of measurement). The scale of measurement for the difference score is ratio, there is a rational zero point.

Assumption 3 (normality). The shape assumption is that the difference scores are normally distributed. We can use the explore procedure to look at the stem-and-leaf plots, the skewness and kurtosis statistics, and the normality tests.

The stem-and-leaf plots for the difference score is shown in Table 11. The plot looks like it should be normal, although there is one extreme value at positive end of the scale.

The skewness and kurtosis statistics for the difference score is shown in Table 12. The 95% confidence interval around the skewness and kurtosis scores for both cells include zero indicating that the difference score is normally distributed.

The Kolmogorov-Smirnov (KS) and Shapiro-Wilk (SW) tests of normality are presented in Table 13. None of the statistics are significant indicating that the difference score is normally distributed.

All of the assumptions of the dependent t test have been met.

top

9. Related Samples: Running the t Test

The t test procedure for dependent groups is selected by clicking

Analyze
   Compare Means
          Paired-Samples T Test...  

The paired-samples t test dialog box presents the variables in the active file in the box on the left. The dependent variable pairs, nscr1_4 and nscr5_8 in this example, should be moved to the Paired Variable(s): box. Note that the t test is run on the original variables, nscr1_4 and nscr5_8, and not on the variable we created to test the assumptions, scrdiff.

There are three Options. The first option allows you to set the confidence interval for the mean (the default is the 95% confidence interval). The second and third options are used when you have specified more than one test variable. The exclude cases listwise option will delete any cases if that case has a missing value on any of the test variables or on the grouping variable. The exclude cases analysis by analysis will only exclude a case for the analysis in which it has a missing value.

The output from the t test includes a table for group statistics, see Table 14, a table showing the correlation between the two variables, see Table 15, and a table for the statistical tests, see Table 16.

Table 14 shows, for each variable, the number of cases, the mean, the standard deviation, and the standard error of the mean. The formula for the standard error of the mean was given above in the dependent t section.

Table 15 shows the correlation between the two variables. The two variables are positively correlated, r(N = 15) = .57, p = .027.

Table 16 shows the t statistics for the paired differences. The hypothesis that these data would show a habituation effect was supported. The mean decrease in the number of SCRs from the first four items to the last four items was 2.0, t(14) = 2.185, p = .046.

The formula for the 95% confidence interval was described in the independent t section above.

Reporting the Results

Here is one way that this data could be reported...

The description includes a statement of the hypothesis, how the hypothesis was tested, the relative effect of the treatment (reading the eight statements), the means and standard deviations for each condition, the effect size reported as Cohen's d, and finally the supporting evidence, the t-test statistics. Notice that the t-test statistics are not in parentheses.  APA style dictates that you cannot have back to back parentheses.

A Note on Effect Size Measures for Related Samples T Tests 

The means and pooled standard deviations rather than value of the t test itself should be used when computing Cohen's d or the effect size correlation for related samples (see Dunlop, Cortina, Vaslow, & Burke (1996).  The effect size will be biased upwards, that is the effect size value will be too large, if the value of the dependent t is used to calculate effect sizes.

top

10. References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.

Dunlop, W. P., Corina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170-177.

Hinkle, D. E., Wiersma, W., & Jurs, S. G. (1994). Applied statistics for the behavioral sciences (3rd ed.). Boston: Houghton Mifflin.

Norusis, M. J. (1990). SPSS Introductory Statistics Student Guide. Chicago, IL: SPSS Inc.

Misanan, J. R., & Hinderliter, C. F. (1991). Fundamentals of Statistics for Psychology Students. New York: HarperCollins.

Myers, J. L., & Well, A. D. (1991). Research Design and Statistical Analysis. New York: HarperCollins.

©Lee A. Becker, 1997-1999 -revised 11/03/99