Detection of item pre-knowledge and memory effects and testing of simulation-based items in computer-based assessment under high-stakes conditions – Development of an analysis method for process data from the MYSKILLS test procedure
Lothar Persic-Beck, ZIB-associated Researcher DIPF
The dissertation project deals with methodological challenges within the MYSKILLS test procedure. MYSKILLS➚ aims at improving the assessment of non-formally acquired occupational competences of fugitives and low-skilled workers with the aid of computer-based assessments.
The first study examines whether construct irrelevant factors can be identified using a finite state model (Kroehne & Goldhammer, 2018) to conceptualize log data from the MYSKILLS test procedure. It will be evaluated if meaningful features can be derived which describe the response process and indicate behavioral patterns shown by the participants. Furthermore, log-events will be used to identify item responses which do not contribute to the measurement of the construct covered by the test.
The second study examines the sensitivity and specificity of existing data forensic methods to detect item pre-knowledge and compares them to a newly developed method with data from the MYSKILLS test procedure. For this purpose, an experiment will be conducted. Four items that were not selected for the final test booklet will be included in the test procedure. Participants will be divided into two groups. In each group the participants will receive the correct answers to two of the four additional items, while the others will serve as control items. For the log data, it is expected that memory effects comparable to memory effects of item pre-knowledge will be found for the presented items. With this data set it is possible to determine the false positive and false negative detection rates of analysis methods for item exposure and thus their applicability for the MYSKILLS testing procedure.
The third study analyzes whether simulation based items are more robust to construct-irrelevant characteristics such as item pre-knowledge. For this study a new set of simulation based items will be developed for the specialist in information technology test from the MYSKILLS testing procedure. The items will consist of small tasks that required participants to write and execute code using a script pad or use googleTM’s blockly to fulfill a programing task. The items can be approached in different ways and the evaluation of log data makes it possible to keep track of the solution behavior. It can therefore be expected that solutions are harder to memorize and exposed solutions can be identified more easily. To test this hypothesis, a part of the solutions will be leaked in an experimental setting.
Literature
Kroehne, U., & Goldhammer, F. (2018). How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items. Behaviormetrika. https://doi.org/10.1007/s41237-018-0063-y➚
Betreuer: Prof. Dr. Frank Goldhammer