Understanding evaluation and comprehension processes by reconstructing the actions of individuals in underlying activity spaces

Dr. Carolin Hahnel, Research associate DIPF

Technology-based assessments provide many opportunities to improve the way an assessment can be carried out or to enhance the interpretability of its data. A contributing factor here includes the investigation of behavioral data that can be collected over the course of a test session. For example, by recording log data (mouse clicks, keyboard strokes, etc.), it is supposed to be possible to represent construction-relevant processes in order to obtain information about successful solution strategies or problems that someone has experienced while working on a task. However, the use of process indicators from log data also presents difficulties. Some process indicators may allow ambiguous interpretations under certain circumstances (e.g., Does a person have a short processing time because he or she was very efficient or unmotivated?). Accordingly, conclusions about cognitive or motivational processes that cannot be directly observed should be well grounded.

The habilitation project addresses questions of the contextualization of log data and the interpretability of the process indicators derived from them. It aims at investigating the extent to which recorded actions of individuals can be related to theoretical assumptions about the skills under investigation. In this way, ambiguities in the interpretation of process indicators should be reduced and processes of information processing by persons could be studied in greater depth. In particular, different reading situations in which persons have to deal with multiple documents or Internet websites will be examined. In such situations, persons may encounter repetitive, complementary or even contradictory information, as different authors place particular emphasis on certain points in their texts or represent their own perspectives. The processes involved in evaluating, integrating and understanding the information read will be represented and examined in detail using log and process data.

References:

Hahnel, C., Kroehne, U., Goldhammer, F., Schoor, C., Mahlow, N., & Artelt, C. (2019). Validating process variables of sourcing in an assessment of multiple document comprehension. British Journal of Educational Psychology, 89, 524–537. https://doi.org/10.1111/bjep.12278➚

Schoor, C., Hahnel, C., Artelt, C., Reimann, D., Kröhne, U., & Goldhammer, F. (2020). Entwicklung und Skalierung eines Tests zur Erfassung des Verständnisses multipler Dokumente von Studierenden. Diagnostica, 66(2), 123–135. https://doi.org/10.1026/0012-1924/a000231➚

 

 

Mentor: Prof. Dr. Frank Goldhammer