Teaching to the test

Teaching to the test is a threat to internal validity. It can be seen as a combination of maturation and testing effects (Cook & Campbell, 1979). Teaching to the test occurs when familiarity with a test enhances the performance of test takers. It is a well-known concept in educational settings, where it implies that students obtain higher grades on a test, (partly) because they are more familiar with the testing material or items, and not (exclusively) because they have developed greater ability or skill.

When a test is used for measuring something other than ability or skill, for example in a questionnaire about mental health, 'obtaining a higher grade' loses it's relevance. After all, obtaining a high grade is probably of importance for any test taker, smart or stupid, in a high stakes educational situation, in order to gain access to higher education, for example. However, obtaining a high grade on a questionnaire measuring depression in order to gain access to treatment is probably only of importance to test takers suffering from substantial depressive symptoms. At the same time, a test taker would need no (or very little) training in order to fake being depressed on a questionnaire, whereas faking arithmetic skill probably takes a lot more effort.

So, enhanced performance on mental health would in this case mean an enhanced ability to assess depressive symptomatology. This would not result in artificially higher (or lower) scores. What kind of artifact in the data this may cause, I'm not sure, but it seems imaginable such an increased ability would result in decreased error variances.


Literature

Cook, Thomas T.D. & Campbell, D.T. (1979). Quasi-Experimentation: Design & Analysis Issues for Field Settings. Rand McNally College Publishing Company, Chicago, IL.

Lazear, E.P. (2006). Speeding, Terrorism, and Teaching to the Test. The Quarterly Journal of Economics, 121 (3), 1029-1061.

(The last citation is not very relevant to the above, but nevertheless very interesting, as it demonstrates that, and in which cases, teaching to the test may be beneficial to a student's learning process. An interesting analogy is made to traffic patrols, tax fraud and fighting terrorism.)