National add-on study on mode effects
in PISA 2018
Lead: DIPF
Joint Research with: IPN, TUM
PISA 2015 included an important change for the test students: the switch from paper- to computer-based testing. It took advantage of the far greater possibilities for designing tests and exercises unlocked by computers. However, this also raised many questions on the comparability of the two test forms, which will be scrutinized more closely as part of a national add-on study for PISA 2018.
On the one hand, the project will test whether response behavior differs between processing exercises on paper and using a computer. In some cases, the test exercises for the computer differ substantially in design from the paper-based versions. This prompts questions of whether this places different demands on the participants and thus changes with regard to the competence measured in each case (mode effect on construct interpretation), whether the tasks can distinguish in the same way between more or less competent pupils (mode effect on item discrimination) and, furthermore, whether the difficulty of the tasks changes to the same extent for all persons or groups (moderated mode effect on difficulty). It is also important to investigate whether mode effects differ between schools and which school characteristics (e.g. computer equipment or use of computers in class) can explain them.
On the other hand, it is about the comparability of the study results of all previous PISA studies. Besides the description of the results of the respective current PISA study, the national PISA collective report always also analyses and interprets development (trend). The switch to computer-based testing means that a straightforward comparison is no longer possible. Mode effects must always also be considered in analyses and comparisons. This add-on study therefore investigates the extent to which such mode effects are pronounced and the extent to which they affect comparability in order to be able to make reliable statements about national trends in the future.
Information from the DIPF on the national supplementary study (mode effects in PISA 2018) can be found in
Goldhammer, F., Harrison, S., Bürger, S., Kroehne, U., Lüdtke, O., Robitzsch, A., Köller, O., Heine, J.-H & Mang, M. (2019). In-depth analyses of the change of mode from paper to computer. In K. Reiss, M. Weis, E. Klieme & O. Köller (ed.), PISA 2018 Basic education in international comparison (pp. 163-186). Münster: Waxmann. (only available in German)