Variation in assembling assessments using automated test assembly methodologies: item-pool constraints and response-time targets

The rise in popularity of computer based testing and continuous testing windows has increased the need to rapidly assemble forms but doing so is time intensive and costly. This study aims to examine variation in observed-score distributions, computational time, and accuracy of automatically assembled forms due to three factors; item-pool size, item-property divergence, and the use of three different response time constraints to meet a target estimated response time. Automated test assembly was implemented using the OPTMODEL in base SAS® version 9.4, SAS® version 15.2, with an optimization function that assembles forms that are observed-score pre-equated to a reference form.

Simulated data were used to automatically assemble five forms in a 3x2x2 fully crossed design. Results indicate that only the response time constraints used had a significant impact on the number of iterations the system required to find an acceptable solution. The lack of significant findings supports the claim by Shao, Liu, Yang, and Tsai (2019) that this method of ATA can successfully assemble multiple forms. This study adds addition evidence to the flexibility and reliability of this method of ATA in successfully assembling forms with less-than-optimal item pools.