Missing data

In longitudinal studies in psychology, there may often be substantial attrition, resulting in non-ignorable missingness in datasets. There's a whole area of research devoted to what to do with missing data, which I am not summarizing here. I am summarizing some helpful findings for future reference.

First, a very practical piece of advice taken from Graham (2009): "... if the loss of cases due to missing data is small (e.g., less than about 5%), biases and loss of power are both likely to be inconsequential." (p. 554).

If >5% of values are missing: remember that MCAR is better than MAR, which is better than MNAR.

MCAR: missing completely at random. The missingness is completely random. This is great, but very unlikely. "If data are MCAR, then consistent results with missing data can be obtained by performing the analyses we would have used had their been no missing data, although there will generally be some loss of information. In practice this means that, under MCAR, the analysis of only those units with complete data gives valid inferences." (source)

MAR: missing at random. Missingness depends only on observed data. We can use methods like EM (expextation maximization) and FIML (ful-information maximum likelihood) to estimate the model.

MNAR: missing not at random. Missingness depends on onobserved data. There is no perfect way to deal with this. Now, we have a problem! Check Graham (2009) for a starter on how to deal with this. 

We can never be sure if data is MAR of MCAR. Maybe we haven't observed enough data to find that data was not missing at random. With all the data in the world, we may be able to turn MNAR into MCAR. But if we had all the data in the world, we wouldn't have missing data to begin with.

Collins, Schafer and Kam (2001) compared restrictive versus inclusive strategies for taking into account missing data. Restrictive is when no or minimal additional variables are used in missing data procedures, and inclusive is when additional variables are used in missing data procedures. Inclusive is always better. Of course; when the amount of missing data is minimal, or data is in fact MAR or MCAR, it never hurts to include additional variables, because it they won't really influence imputed values or parameter estimates. This is in line with advice of, for example, Graham (2009), who states 'Use auxiliary variables' as the first strategy to reduce the biasing effects of attrition. Inclusive strategies can be performedboth with maximum likelihood estimation (ML; where missingness is taken into account in parameter estimation) and with multiple imputation (MI; where several datasets are created where missing values are imputed). However, Collins et al. (2001) note that ML approaches as implemented in most software tend to encourage inclusive restrictive strategies, whereas MI approaches tend to encourage inclusive strategies.

Practical pieces of advice from Collins et al. (2001) for real-world data analysis: "Generally, we found that when missingness does not exceed 25% and the correlation between the cause of missingness and the variable subject to missingness was .4, omitting the cause of missingness from the analysis had a negligible effect. When missingness exceeds 25% or the correlation between the cause of missingness and the variable subject to missingness is as large as .9, our results indicate that there are often substantial problems with bias, efficiency, and coverage." (p.347).

References

Collins, L. M., Schafer, J. L., & Kam, C. M. (2001). A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330.

Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual review of psychology, 60, 549-576.

Small sample df's with multiple imputation

When Multiple Imputation (MI) is used in SPSS, output of subsequent analyses of datasets may show huge df-values.

Barnard and Rubin (1999) suggested an adjustment of df-values. Abstract:

"An appealing feature of multiple imputation is the simplicity of the rules for combining the multiple complete-data inferences into a final inference, the repeated-imputation inference (Rubin, 1987). This inference is based on a t distribution and is derived from a Bayesian paradigm under the assumption that the complete-data degrees of freedom, \nu_{com}, are infinite, but the number of imputations, m, is finite. When \nu_{com} is small and there is only a modest proportion of missing data, the calculated repeated-imputation degrees of freedom, \nu_{m}, for the t reference distribution can be much larger than \nu_{com}, which is clearly inappropriate. Following the Bayesian paradigm, we derive an adjusted degrees of freedom, \tilde{\nu_{m}} with the following three properties: for fixed m and estimated fraction of missing information, \tilde{\nu_{m}} monotonically increases in \nu_{com}; \tilde{\nu_{m}} is always less than or equal to \nu_{com}; and\tilde{\nu_{m}} equals \nu_{m} when \nu_{com} is infinite. A small simulation study demonstrates the superior frequentist performance when using \tilde{\nu_{m}} , rather than \nu_{m}."

Formulae

\nu_{com} is complete data df's

m is number of imputations

\nu_{m} is the repeated imputation df's

\tilde{\nu_{m}} is always less than or equal to \nu_{com}

\tilde\nu_{m} = \nu_{m} * ( 1 + \nu_{m}) / (\hat{\nu_{obs}} )^(-1)

in which:

\hat{\nu_{obs}} = \lambda (\nu_{com}) * \nu{com} * (1- \hat{\gamma_{m}} )

\lambda (\nu) = (\nu+1) / (nu+3)

\hat{\gamma_{m}} is approximately the Bayesian fraction of missing information for the unknown quantity of interest. Hard to calculate by hand. An SPSS macro can be found here.

Barnard, J. and Rubin, D.B. (1999). Small-Sample Degrees of Freedom with Multiple Imputation. Biometrika, 86, 4, 948-955.

See also: Van Ginkel, J. R., & Van der Ark, L. A. (2005). SPSS syntax for missing value imputation in test and questionnaire data. Applied Psychological Measurement, 29, 152-153.