When "reliability" fails, must a measure be discarded? The case of kinesthetic aftereffect.

Résumé

Critics of kinesthetic aftereffect (KAE) recommend abandoning it as a personality measure largely because of poor test-retest reliability. Although no test can be valid if lacking true reliability, to discard a measure because of poor retest reliability is an oversimplification of validation procedures. This pitfall is exemplified in the present paper by a reexamination of KAE. KAE scores involve measures before (pretest) and after (test) aftereffect induction. Internal analysis of a KAE study with 65 undergraduates showed the following: Differential bias is present; its locus is the 2nd-session pretest; its form makes 2nd-session pretest scores functionally more similar to 1st- and 2nd-session test sores and functionally more dissimilar to 1st-session pretest score. Given this 2nd-session bias, the retest correlation tells us nothing about the true reliability of a 1-session KAE score. However, if a measure possesses external validity, it must to some degree show true reliability. Based on a literature review of 1-session KAE validity studies, it is concluded that 1-session KAE scores are valid and hence show true reliability. KAE remains a promising personality measure. (40 ref) ( (c) 2002 APA, all rights reserved)