September 19, 2024 by Doug McCurry from BooBook Education
Key questions about assessing the most basic educational skill
What does a ‘reading test’ test?
Is it a good reading test?
A reading test about winning the Olympic medal count
Reading is the most basic and important skill taught in the education system. How reading is assessed in the most basic question about the assessment of educational achievement. Testing reading is fundamental to NAPLAN, PISA, GAT and VCAA On Demand Testing. Teaching and assessing reading is fundamental to the work of teachers, and the strengths and weaknesses of reading tests is an important issue for teachers.
In the following reading passage and test items I will offer a kind of answer to the question in the sub-headings about what the reading test is testing and whether the test is a good test of reading.
The following text about different ways of determining a winner of the overall Olympic Games is the basis of fifteen multiple choice questions (MCQ) and 1 short answer question.
A link is given at the end which will allow students do the test electronically and have it scored by Google Forms. There is another link that will allow teachers to download the test and use it with a class or classes. The student responses to a downloaded version of the test will be automatically scored and the results aggregated for a teacher's group. Such a testing procedure could be a very cost-effective reading assessment.
You may have some suspicions about the value of formal reading tests. I certainly did at one time, but working as a test writer showed me that there are strong and weak reading tests. A good reading test is a demonstrably powerful assessment. If I was now a classroom teacher I would have a schedule of reading tests using Google Forms which I would use for different groups in different years. This repeated testing would allow me to assess the reading ability of this or that group and construct local norms for my school. Local norms would allow me to assess changes in my students and the effectiveness of my teaching over time.
My confidence in the test results would depend on my confidence in the quality of the tests I was using. I will use the following test items as a way of suggesting some key issues about quality in reading tests.
In the interests of full disclosure, I point out that I wrote the following stimulus (based on a draft by ChatGTP) and the questions in two hours. I think it offers a good basis for testing reading and reasoning. It is the basis for testing reading and reasoning rather than merely testing reading comprehension.
There are four major ways of determining the winner of the Olympic medal count. 1. Gold Medal Count This method emphasizes the highest achievement in each event. It might be preferred because winning a gold medal is the ultimate goal in the Olympics. 2. Total Medal Count This method counts the highest total number of medals (gold, silver, and bronze) and rewards overall consistency and excellence. It recognizes the breadth of a country's athletic talent and competitiveness, rewarding countries that perform well across a wide range of sports. 3. Weighted Medal Count This method assigns the values of 3 points for gold, 2 for silver, 1 for bronze. It provides a balanced view, recognizing the higher value of gold while also accounting for other podium finishes. It rewards both gold medals and overall performance. 4. Per Capita Medal Count For this method either the gold medals, the total number of medals or the weighted medal counts are divided by each country's population. This method acknowledges the relative success of countries, based on population size. The per capita method emphasizes the efficiency of Olympic success and often highlights the achievements of smaller nations. The following nations typically perform highly on the per capita medal count method. · Jamaica is dominant in track and field, particularly sprinting events. · New Zealand excels in rowing, rugby, and sailing. · Slovenia often performs well in winter sports, such as skiing and snowboarding. · Fiji is famous for its rugby sevens team. Each method has its advantages and disadvantages, and the choice of method can reflect values such as excellence, consistency, fairness, or narrative impact. Different countries or organizations may prefer one method over another based on their own priorities and perspectives. (324 words) |
Notice that the text has been organised to give four alternatives (the usual number and the number assumed by ChatGTP for MCQ) which means that ten of the fifteen MCQ are written to one of two sets of options. ChatGTP also used the same pair of options for four of its ten items.
The big quality issue in reading MCQ is the extent to which the items are literal comprehension (that was stated or implied in the text) and the extent to which the items are inferential reasoning (that can be deduced or extrapolated from the text).
Here are the questions in the electronic test and a comment on each.
1. The phrase 'per capita' is best described as meaning
most common.
most important.
for a population. *
as a result of affluence.
The first is a vocab item, but it is not easy even with a dictionary.
Google defines the term 'per capita' as follows.
for each person; in relation to people taken individually.
'the state had fewer banks per capita than elsewhere'
The meaning in the context of the passage of 'for a population', or 'per head count', can be inferred from the statements 'relative success', 'based on population size' and 'divided by each country's population'. lnferring the meaning of the phrase in context from the Google definition is sophisticated thinking because the Google definition directly deal with these issues.
2. The third method can be described as weighted because
it is unbiased.
it is one sided.
all medals have an impact.
different medals have different impact. *
The key of D for this item is an extrapolation from the passage and to get the right answer students have to see that 'impact' in the item is a kind of synonym for the word 'value' in the passage.
Items 3 to 9 ask students to link a statement about a method to one the titles. The text has been constructed to give four options and this makes posing a question in a stem and using the same options a cost-effective way of constructing MCQ.
3. Which method would have the best claim to being the fairest?
Gold Medal Count
Total Medal Count
Weighted Medal Count
Per Capita Medal Count *
Item 3 is a best answer rather than the only possible answer to the question. One would want to monitor this question to see whether it performs satisfactorily. If students get D as the answer it is worth keeping, if they are all over the options you would get rid of it.
4. Which method does not distinguish between the value of medals?
Gold Medal Count
Total Medal Count *
Weighted Medal Count
Per Capita Medal Count
The answer to this item depends on recognising the relationship between the word 'rewards' in the passage and the word 'value' in the stem of the item.
5. Which method adjusts for the comparative situation of different countries?
Gold Medal Count
Total Medal Count
Weighted Medal Count
Per Capita Medal Count *
This item involves a recognition of the relationship between the word 'relative' in the text and the word 'comparative' in the item.
6. Which method of determining the winning country takes type of medals won into account?
Gold Medal Count
Total Medal Count
Weighted Medal Count *
Per Capita Medal Count
In this item students have to construe 'type of medals' in the item as gold, silver and bronze in the text.
7. Which method gives more emphasis to excellence?
Gold Medal Count *
Total Medal Count
Weighted Medal Count
Per Capita Medal Count
In this item students have to recognise that 'highest achievement' in the passage is related to 'excellence' in the item.
8. Which method is best described as a compromise?
Gold Medal Count
Total Medal Count
Weighted Medal Count *
Per Capita Medal Count
In this item students have to see that the phrase 'balanced view' in the passage is related to the idea of a 'compromise' between different methods.
9. Which method would most likely advantage the more affluent countries?
Gold Medal Count
Total Medal Count *
Weighted Medal Count
Per Capita Medal Count
This issue is not dealt with in the passage, but we can extrapolate to the conclusion that wealthier countries (with larger populations) would be able to support a wider range of sports (and gain more medals) than poorer and smaller countries. Hence, as is actually the case, some wealthier countries (such as the United States) show a preference for the total medal count result.
10. What is the biggest disadvantage of the gold medal count method?
It is biased to big countries.
It is biased to small countries.
It is neither one thing nor the other.
It does not recognise levels of achievement. *
The answer to this item is based on the inference that the gold medals alone method does not recognise other levels of performance.
11. What is the biggest disadvantage of the total medal count method?
It is neither one method nor the other.
It is biased to bigger and richer countries. *
It is biased to small and poorer countries.
It does not take general excellence into account.
This answer can be inferred from the idea that big countries participating in more sports are advantaged in total medal count.
12. What is the biggest disadvantage of the weighted medal count method?
It is biased to big countries.
It is biased to small countries.
It is neither one thing nor the other. *
It does not take general excellence into account.
This answer can be inferred because weighted medal count does not have the clear focus on either excellence or broad achievement.
13. The medal performance of which of the smaller nations is most shaped by its geography?
Jamaica
New Zealand
Slovenia *
Fiji
This answer can be inferred about an area like Slovenia with mountains and winter snow. Unlike Jamaica and Fiji, New Zealand has snow, but it is successful in sports other than those of winter.
This item might be objectionable because you do have to know something about the geography of each of the countries to answer it.
14. The medal performance of which of the smaller nations is shaped most by genetics and the race of its population?
Jamaica *
New Zealand
Slovenia
Fiji
This answer can be inferred because most of the Jamaican population had ancestors from Africa, and Africa has produced great athletes, particularly sprinters.
Again, this item might be objectionable because you do have to know something about the racial makeup of Jamaica and the dominance of runners with African heritage. And some Fijians are physically big which is an advantage in Rugby.
15. The medal performance of which of the smaller nations is shaped by its broad sporting culture?
Jamaica
New Zealand *
Slovenia
Fiji
New Zealand is the answer to this question because it excels in a range of different sports.
16. Short answer question:
Which method of determining a winning country is best and why?
___________________________________________________________________
___________________________________________________________________
This is an interestingly open question that requires considerable insight. To answer it well students have to understand the likely outcomes of each method, consider what the term 'best' might mean (easiest, fairest, most likely to encourage excellence etc.), and offer reasons for preferring some criteria to others.
A conclusion
Apart from a few specialist concepts like per capita and some background knowledge, the passage is quite explicit rather than opaque and allusive, and the language of the passage is quite clear and straight forward.
The answers are not literal comprehension that can be answered by parroting the language of the text. Most of the items involve a process of relating terms in the text with synonyms or related terms in the question. This means that students have to understand the meaning of the words in context and the meaning of the terms in the item.
The questions about advantages require students to understand each method. The questions about disadvantages require students to see the reverse of the descriptions of the advantages of a method.
These items are inferential rather than literal comprehension. I would claim that these items test some of the most fundamental aspects of reading and are a valid test of reading.
The following is the link for students to do the test and have it scored.
The following is the link for teachers to download a copy of the test, set it up for themselves in their Google Drive, and have the results of their students automatically recorded.
https://docs.google.com/forms/d/1xicrnMI7fTXsjWep3mghe8nQRl49I8xpVE6UmsXJjwA/copy
