Reading Fluency                                                                                           2-23-02

 

About June of last year someone drew our attention to conflicting instructions for scoring Reading Fluency in the Test Book and the ACH Examiner’s Manual. Reading Fluency is a true-false test with a 3-minute time limit, therefore, a “correction for guessing” must be applied. A subject who is guessing has a 50-50 chance of selecting the correct answer for an item. One guesser might attempt 10 items in the 3 minutes and would get 5 correct by chance. Another guesser might attempt 50 items in the same 3 minutes and would get 25 correct by chance. Both subjects guessed but one ends up with a score of 5 and the other with a score of 25. After subtracting the number of errors from the number correct, each of these subjects would have a score of zero.

            The clerical staff correcting these tests were instructed erroneously to count skipped items as incorrect. If a “correction for guessing” is applied, this procedure unfairly penalizes subjects who prefer not to guess. “Unfair” because all skipped items are counted as incorrect but only half of them would have been incorrect if the subject had guessed rather than skipped.

Below is a recent post in response to the question about the possible impact of changed scoring upon Reading Fluency norms:

 

From: "Barbara Wendling" <bjwendling@worldnet.att.net>

To: <IAPCHC@yahoogroups.com>

Sent: Tuesday, February 12, 2002 3:17 PM

Subject: Re: [IAPCHC] WJ III Reading Fluency

 

 

> Here is information about the norming of Reading Fluency: (source of info. -> Dr. Woodcock)

>

> In the norming, skipped items were counted as errors, but this happened infrequently. The scarcity of skips occurring results in an imperceptible impact on the norms. However, the effect on the occasional individual who skips several items could be significant in a clinical evaluation. We don't want to unduly penalize individuals for not guessing.

>

> After reviewing a sampling of 800 norm subjects across a representative span of ages, only 16 skipped any items. Seven of those 16 only skipped one item. Nine of those 16 were ages 6 or 7, suggesting that skipped items occur more at the younger ages.

>

 

Let me draw attention to and provide a little more detail about two important statements in the post:

 

First, the impact on norms or other group statistics would be imperceptible. Upon realizing that the norming data had been scored with skipped items counted as incorrect we undertook a review of the Reading Fluency norming test data to evaluate the differential effect of the two scoring procedures. We pulled protocols for a sample of 800 norming subjects. Many of the 800 subjects had data for both Forms A and B of the test, thus, a total of 1072 tests were checked for instances of skipped items. Sixteen tests with skipped items were identified and then re-scored to determine the increase in W score that would have occurred if the skipped items had not been subtracted from the number correct. The increase in W ranged from 0 to 16. The average increase for the 16 tests was about 5 W units. The sum of the gains was about 68. The impact on group statistics (e.g., descriptive statistics, norms, and factor analyses) was estimated by dividing the sum of the gains by the number of tests reviewed (68/1072 = 0.06 W). The means, on average across the sample, would have been higher by 6/100^th of a W unit if skipped items had not been counted as errors. Another estimate was obtained by determining the impact upon standard deviation (SD). An inspection of the SDs reported for Reading Fluency in Appendix A of the WJ III Technical Manual indicates a median SD of about 50 W units across all ages. Dividing the median SD by the average gain (50/.06 = 833) shows an effect size of about 1/800^th of a SD. These two estimates provide evidence of the “imperceptible” impact upon group statistics.

Second, even though it happened infrequently in the norming sample, it is important that clinicians do not count skipped items as errors. For example, one subject in the sample of tests reviewed received a new score that was 16 W points higher. The scoring instructions in the Test Book are correct. The problematic statement appeared in an early printing of the ACH Examiner’s Manual and was corrected in later printings. If you have an older manual on hand, you may wish to turn to page 48 and delete the second sentence under Scoring (“Score as incorrect any items skipped prior to the last item the subject completed”). We all apologize for the confusion this may have caused some of you.