Getting Rasch about assessment: Using a psychometric approach to select and confirm items in the new Victorian licence test

Selecting assessment tasks and items for the new Victorian licence test relied in part on reducing a large number of potential assessment items to a smaller set of items with a high level of internal consistency. This made use of a psychometric method commonly used in the development of ability tests ? Rasch modelling ? that does not appear to have been applied to on-road tests. The Rasch approach allows test developers to assess the performance and calibration of items and test takers against a common underlying ability dimension.

The Rasch approach was used to select potential items for the first, screening stage of the test from a pool of items derived from the current Victorian test, the Californian DPE, and the first stage of the New Zealand full licence test. A trial involving 352 learner drivers provided data for the analysis. The analysis identified a small set of internally-consistent assessment items that were used as the basis for the screening stage of the test.

The same approach was used to select potential items for the main part of the new test from a large pool of assessment items originally developed with a focus on construct validity (see McDonald and Harrison?s paper at this conference). Data for the analysis were collected in a trial involving over 400 learner drivers. The Rasch analysis identified a subset of internally-consistent items that were then considered for inclusion in the new test.

A third trial with over 500 experienced learner drivers was used to assess performance of the selected test items. A Rasch analysis confirmed that the items in Stages 1 and 2 have a high level of internal consistency and are appropriately calibrated against the ability level of experienced learner drivers.

This paper summarises the results of the trials and their implications for the Victorian licence test, and discusses the potential value of the Rasch approach for licence test development in future.