An atomized approach to assessing energy problem solving in physics using multidimensional item response theory

Introduction: Problem solving is a central competence in STEM education, yet many secondary school students struggle to coordinate the multiple skills required for successful problem solving. Early assessment of problem-solving skills can support individual feedback during this pivotal phase of schooling. However, existing assessment approaches focus mainly on complete problem solutions, which are resource-intensive and cannot adequately capture skills of students who fail in early phases of the problem-solving process.

Methods: To address this gap, the atomized problem-solving test (APST) was developed as a digital instrument that independently assesses four problem solving subprocesses: Representation, Planning, Execution, and Evaluation. The APST was evaluated in two consecutive studies with a total of 800 German secondary school students within a web-based learning environment on energy conservation. Multidimensional item response theory (MIRT) was used to examine item quality and dimensional structure, complemented by supplemental assessments of conceptual knowledge, school grades, and rubric based analyses of written problem solutions.

Results: The analyses supported a four-dimensional structure aligned with the theoretical design of the APST. The items showed acceptable model fit and reliable measurement of the intended subprocesses. All APST dimensions were moderately associated with conceptual knowledge of energy and with school grades in physics and mathematics, while no meaningful correlations were found with gender or native language. Evaluation emerged as a distinctive subprocess, showing strong associations with other subprocesses–particularly Execution–alongside evaluation-specific skills.

Discussion: The results indicate that the APST enables valid and reliable assessment of problem-solving subprocess skills in secondary physics education. At the same time, the findings underscore limitations of atomized assessments for measuring general problem-solving competence, as independent decision making is not assessed. The prominent role of Evaluation highlights its integrative function within the problem-solving process and points to important implications for both assessment design and future research.