November 28, 2024 by Doug McCurry from BooBook Education

Flirting with ChatGPT to Score Writing

I have been totally opposed to computer scoring of writing tests for a range of reasons.

I have researched such scoring of writing and have published results that challenge the validity and reliability of computer scoring. I am pleased that I had a part in opposing and stopping the replacement of human markers with computer scoring in NAPLAN. I have also argued in print that is would be worthwhile to support the current single marking of NAPLAN with a second computer score. This use of computer scoring of writing technology to support human scoring is of no interest to ACARA because they want to replace human scorers to save money.

I have been very impressed recently with the output of ChatGPT when I have used it for various tasks. The processes of the large language models used in CGPT are unlike those used in the so-called 'natural language processing' that underpins current commercial computer scoring of writing (CSW) programs. So, I thought I would see what CGPT could do in terms of scoring writing scripts, and particularly what qualitative feedback the process could offer from the scoring of writing scripts. The old computer scoring processes promised qualitive feedback from marking, but I never saw any feedback from such scoring other than the most generalised kind that was more or less useless.

I did a study of the computer scoring of writing in 2005, the results of which were published in Assessing Writing in 2009.^[1] In that study some hundreds of Y12 writing scripts were used to ‘train’ two commercial computer scoring programs. The 200 electronic scripts used in the study had each been scored by four trained markers in a system level testing program, so the human scores (used either as a total of four scores out of 10 or an average of the four scores) were a robust human assessment of the writing test scripts.

The two commercial computer scoring algorithms were ‘trained’ on the human scores of 120 of the scripts, and the algorithms derived from these ‘training scripts’ were then used to blind score the other 80 scripts. The writing task scored was complex, the students were in Y12, and the scripts were of a quite consistently high standard. It was my hypothesis that the computer algorithms would not be able to deal with such complex and high level performances as well as human scorers could. And the trial showed that this was the case.

I thought I would use the trial scripts from that study on which I had four human and two computer scores to see how CGPT would score such scripts, and in particular what kind of qualitative feedback it could produce. I presumed it would be less good at predicting human scores that the commercial programs, but I wanted to see what qualitative feedback it could produce.

The following table show correlations between and average of four human scores (HumAv), a CGPT score, the two computer scoring algorithm scores (CSW 1 and 2), and a count of the number of words in each script. Correlations with word count are used in studies of CSW because it is widely recognised that there is a higher correlation between word count and the computer scoring of writing scripts than for human markers. It is claimed by critics of CSW that while word count might correlate with quality in writing tests (so it is important in CSW algorithms), word count is a trivial characteristic of a writing sample, and can be easily gamed by test candidates. (‘Write as much as you can in the time kid!’)

Table 1 Correlation between four scores and word count

Table 2 Correlations between four separate human scores, CGPT and word count

The correlation between the human average and CGPT is fairly high at 0.61.
The correlation of computer score 1 is higher at 0.73.
The correlation between the three computer scores vary between 0.32 and 0.62.

The correlations to Words are quite different for the computer scores. The correlations of the CSW1 is fairly high at 0.57, and the correlations to Words are low at 0.21 for CGPT and 0.33 for CSW2.
This suggests the higher correlation to HumAv for CSW1 is related to a higher relationship to word count than HumAv or CSW2. The relationship to Word count is under operator control for CSW1 and CSW2.

The correlations between separate human markers are the highest in the tables. Human markers will agree with each other more than any of the computed scores will agree with them or with another computer. This was the hypothesis tested in the study 2005 and so it proved to be the case in this study.

This data shows that CGPT is a fairly strong predictor of human scores on this data, and that the relationship of CGPT to HumAv is not influenced much by the number of words in a writing sample, as is shown in the result for the commercial CSW1 algorithm.

These positive results for CGPT could interest to teachers because (unlike the commercial CSW programs) CGTP is readily accessible (free or at a modest charge), can be used on a normal computer, and is very easy to use.

It is very easy to get access to CGPT, and when one is registered, one only has to use a couple of simple commands to score scripts with CGPT from a Word document.

What makes CGPT interesting for teachers and students is that it can produce a range of qualitative feedback from the processing of the writing scripts.

No more than the following command is needed to successfully score batches of 10 scripts in a Word document with CGPT.

On the basis of the levels described in the BBE Described Levels document, score between 1 and 10 the scripts in the attached document.
Using the six digit number at the beginning of each script as identification, put all scripts ids and the scores into a table.

The BBE Levels of Performance used in this command are shown below. This command produces a table of results that can be readily copied and manipulated. But there is much more to the CGPT scoring of writing scripts than the production of scores.

I repeated the scoring exercise with the addition of the following command requesting a commentary on the scripts.

On the basis of the score between 1 and 10 for the scripts found after the six digit identification number at the beginning of each piece in the attached document, give a commentary on each script describing

• The main contention and supporting arguments
• Any evidence or reasoning supporting the view
• The substance and plausibility of the claims made
• The clarity, precision and impact of the language used
• The accuracy and correctness of the language used
• How the piece might be improved and why changes or additions would be an improvement.

This command did not specify discussion in terms of the BBE Levels document so as not to limit the comments. The command produced commentary on the scripts of the kind discussed below. These examples show the following characteristics of the CGPT scoring and its commentary on the scripts.

Unlike CSW1 and CSW2, CGPT produced the kind of score produced by human scoring with a slightly higher mean score but a bigger standard deviation than HumAv.

Mean and standard deviation for HumAv and CGPT

CGPT was consistently able to accurately gloss the view presented in a script in a sentence. It did not identify supporting arguments, but then it is usually not possible to identify supporting arguments in weaker scripts. As can be seen in my comments on the scripts below, the higher the level of the script the more it is possible to tease out supporting arguments.

The comments on reasoning and evidence are also limited by the approach of the writers of the scripts.

The language issues to be dealt with in the dot points usually got a quite definite response from CGPT, although the comments were not very precise and references to spelling were common.

The comments on improving the piece were fairly informative, and tended to reveal an assumption that a balanced approach considering both sides of an issue is superior to direct advocacy of a position.
Including counter arguments was commonly recommended.
Specific examples, evidence and clearer explanation were seen as valuable additions and improvements.
Attention to precision of language to improve clarity, readability and impact was commonly recommended.
Consideration of ‘ethical concerns’ was also commonly recommended.

One of the most interesting (and powerful) characteristics of CGPT is the way it can reflect on its own processes and outputs.

Some sample scripts and the CGPT commentary on them

Below are six more of less typical examples (in so far as any scripts are typical) of below, middle and above middle scripts used in this study followed by the qualitative feedback from CGTP in a table using the headings of the CGPT output. The scripts were written in response to a collection of material about the human treatment of animals. The scripts were scored by four human markers and no discrepancy procedure had yet been applied to the scores.

The responses to this task are hard to assess as it has no set topic, and students can choose how they approached it and what they will offer as the substance of their response. This openness makes it harder for humans to assess and much harder for the parameters of natural language processing computer scoring to converge upon a score.

The first two examples are given low scores by humans and are typically given slightly higher scores by CGPT. The bot typically gives a narrower range of marks with fewer low and high marks that humans.

Note that in all the examples below the description of the view presented in the first column in one sentence is accurate.

The BBE Levels of Performance in Writing at Year 12

Sample scripts

170549

Human Scores 6 4 4 4

Equality for all species

Humans and animals are currently not equal. The way in which animals are treated is not fair, this problem should be rectified resulting in equality for all species. Humans live in and make a society rife in speciesist attitudes, some are more obvious than others and some are here to stay due to our dependancy on these resources.

Agriculture is here to stay due to its enourmous impact on a global level if it was to be eliminated. Science testing and research is slowly decreasing, with an increasing amount of products not being tested on animals. Recreation and entertainment is basically taking advantage of animals and as such it should be eliminated purely due to the inhumane treatment of animals. Wildlife has good origins with subsistence hunting which is needed for survival of every species. The other branches of hunting is what is inhumane and just for the fun of the hunt.

Agriculture, very prominent and also big bussiness in todays society. Today you can go to the supermarket and buy a vast array of animals which have been parceled up into serving sizes. These animals do not always live as free as possible due to the dollars being earnt a very important factor in the business. It is usually a very traumatic time for the animals heading to the abattoir and waiting at there to be slaughtered.

This is a very speciesist attitude but not a lot can be done to slow this very wide industry, due to its enourmous size and dependancy humans have on the industry.

Science is very prominent, while a lot of testing is done on animals for medicle and cosmetic reasons the cosmetic factor is decreasing with a lot of products on the market having the symble representing it was not tested on animals. This is heading in a good direction as speciesism is very obvious in this area due to the animals not benifiting at all after they have been tested upon.

Recreation is an area which should be eliminated altogether. For example circuses, people go to watch animals perform tricks but what they don’t care for is what goes on behind the curtains all the taunting and beatings they go through to be able to perform the tricks to a high standard. The A.C.T government banning circuses was a good idea due to the trauma the animals go through even though they now just head out to Queanbeyan. Rodeos are also inhumane to animals getting them all worked up and aggressive just for the fun of riding or roping them. This is just taking advantage of animals and not rearly treating them equally.

Wildlife is a mixed bag with its origins as a survival mechanism which comes out in every species. An example of this is a food web where one species will eat another species hunting does not just occur in the human species. It is the other types of hunting which should be stemmed due to these types of hunting being basically for the fun of the sport and the trophies which are put on the wall. Although it is not as inhumane due to the animal living in open spaces and seems to be free that animal could be a mate or parent of another animal which is very dependant. This is a good case of where humans think they are above animals.

Through research already done shows animals do have feelings which should be acknowledged by humans because we know what it is like to lose someone we love. We should be able to live in harmony with respect for other species making the world a better place for all species to live in.

The markers gave this first example 3 scores of 4 and one on 6.

This is a good example of a script that is poor without being obviously inadequate. It is not brief and insubstantial. It is typical of lower scores in being garbled and awkward at sentence level and the whole seeming to wander around the topic. The thoughts seem conflated, random or fractured and there is poor continuity between the thoughts. The student can’t get ideas or language clear. The piece seems unplanned and unorganised.

The script is a clear 4 and could perhaps be lower?

The CGPT score of 5 is too high. The comment is too bland and does not identify the fundamental lack of well-formed ideas and sentences in the script.
The recommended improvement of adding specific examples and clear explanations does not recognise the fundamental weakness of the piece.
The comment doesn’t address the fundamental problem with clarity and coherence of thought and language.

577979

Human scores 3 4 3 4

Hunting Animals: is it good or bad

In this time we live in, Animal Liberation has raised issues about Hunting animals and how it can be cruel and not nice.  This can be true for same reasons but in other cases hunting animals is need and is not cruel.  Like when the population of an animal is to high for an area and when farms need to protect their crops which animals would destroy if left alone.  Also a place where hunting animals is needed is to supply food for people.  The only area which can be seen as cruel and inhumane is people hunting for sport and recreation.

An area where hunting animal is need is when the population of an animal species is to high for an area and they are causing an impact to the enviroment.  The place where this is need the most is on farms.  If a farmer leaves the animals to over-populate then the damage on his land and crops would be so large that he would loss a lot of money.  So the simple solution is to hunt occasional or to hunt a lot once every few years when the numbers are high.  Likewise in national parts, rangers have to hunt animals so numbers do not get to high.  A good example of this is the killing of kangaroos.  When the numbers are high because of the lack of predators and diseases, which useally kills many kangaroos.  If the roo’s are not hunted then other animals and plants are harmed, so this sort of hunting is need.

The other reason where hunting can be considered a good and right thing to do is hunting for food.  Many animals in the world hunts for their food and we humans fit into this catorgie.  Most of our meats are domestic animals which are killed in abattoirs.  But now we can also get kangaroo meat and other types which have to be huntered and shoot in the wild.  These huntered meats can be seen as the same as a domestic animal, like a sheep or cow.  Just another animal which is killed for our food. Some people believe and see that the killing of wild animals is more humane than that of the slaughters in abattoris.  They believe that the domestic animals will know whats going to happen to them by instinct and scent.  It is completely different in the wild.  The animal does not know the end is caming and the way which proffessional hunters canshoot the end is quick.  So Animal liberation groups would think hunting animals for food is bad, but they have no argument against it.

What Animal Liberation groups can strongly argue against is the hunting of animals for sport and recreation.  The going out and killing an animal for no real reason at all.  Only so they could have a skin or a head on the wall.  This can be seen as cruel because the animal is being used as someones fun and entertainment and the end result is thos kill it.  As well many of the people shooting for fun are amateurs an go alot of there kills are not straight away.  For so animals the dieing proscess could be long and painful and the animal suffers a lot.

The hunting of animal has being going on for a very long time and only in the last few years have people thought if it is cruel and bad or good.  In same cases, it is cruel and bad but in same case it is needed for a number of reasons.

The humans give 3 and 4 for this second piece, and again CGPT goes for the higher score of 5.

As is common for this level, the piece seems wandering and unorganized. The student only deals with the issue of hunting, and attempts to give a balanced consideration of the issue, but the reactions seem simple minded. The piece has a degree of system, but the student does not follow through with thoughts.

While the language is weak it is better than the first example?

The CGPT score is the same as the previous example, but the comment seems a little more positive?

Does CGPT approve of the attempt to be ‘balanced’?

566772

Human scores 3 6 6 3

Survival Of The Fittest

Most Australians are unwilling to accept the fact that animal research, animal liberation and kangaroo culling are actually are extremely benefical and productive towards our society. It produces a good quantity of dedication & production from our society. With the result of natural selection humans were chosen to be on top of the food chain it shows the results of natural selection survival of the fittest.

Animal Research plays a crucial role in our society and modern medicine, we achieve with the use of animals crucial discoveries in medicine with developing all kinds of new research, treatments and awesome break-throughs all most everyday of our lives; we are finding exciting and new ailments with animal research. The results are amazing for the benefit of humankind. It is amazing to find the results in which benefits humankind; with the development of antibiotics, development of vaccines, open-heart surgery, diabetes research, hypertension treatment anti-ulcer drug development, and organ transplants including heart values (Animal Research: Jack H. Botting and Adrian R. Morrison.) It amazes me that people find it so bad to use animal in research to help medicine, but they feel it is too cruel for the animals but they want human guinea pigs instead.

Animal Liberation is the Australian Animal Rights Organisation is designed to help end all the human involvement with animals in which harm is involved also and like the RSPCA, they are designed to stop the cruelty towards animals but there is not cruelty that I have heard of, maybe it is the people of those organisations that actually are the cruel ones, as in the natural world the survival of the fittest,  as we are all animals, humans are on the top of the food chain we have the right to eat & kill what we want. What is the difference from a turkey from the butcher and we killing from nature.

As kangaroo culling is known by most Australians, people tend to think that it is the slaughter/killing of innocent little baby kangaroos, but to find out that they are huge destroying animals really comes to a shock. The demands on Australian agriculture and on animal welfare. Due to public ignorance about culling; it is essential to keep the grazing & crops safe and not attacked by the kangaroos. Australia has found a humane way of dealing with kangaroos and keeping down the numbers and the damage of the kangaroos that they do every year on the farms and the land, the meat trade of the kangaroos is a great way of helping the economy and keeping down the numbers.

The interests of the animal depands on us, the humans, as we are the top dog of the food chain. I do see how they are being exploited, as the saying goes this is simply survival of the fittest, and that is the trueth, if you are not fit you are weak, you will not live to see another day.

This script clearly splits the human markers with 2 scores of 3 and 2 scores of 6. This script would go to discrepancy adjudication.

The language is not well-controlled and the proofing is poor.

It presents a clear point of view (so could be scored 6?) but the comments are rather simple and crude (so could be scored 4?), and the language is awkward. The idea that some animal rights advocates propose using humans as ‘guinea pigs’ is simplistic, and the whole is a rather crudely Darwinian view.

The CGPT comment on the substance and plausibility of the argument is good, but the comment about the language being informal, imprecise and repetitive is not particularly informative. The comment about the need for greater nuance is vague and the suggestion about addressing the ‘ethical concerns’ would not change the crudeness of the argument.

As an adjudicator, I would want to split the difference here and give this script 5.

222482

HUMAN SCORES 6 5 6 6

Do you feel you could live without it?

How does one value a life? Do you value another more than your own? Wheather it be the life of a human or non-human animal, most of us would put our life before theirs. It is only natural to value your existance more than anothers, the same as it is natural to have the urge to hunt and do other so called barbaric things. The human species have made our way to the top of the food chain and most abuse that fact. Most humans live with meat being one of the top foods in their diet, they have come to depend on it. To have people try and take it away would create a virtual war. The so called meat-eating humans tend not to care where this food has come from, if they really gave it thought would they care? Who are these non-meat eating humans otherwise known as vegetarian to say what is right and what is wrong. As humans most of us see ourselves supperior to these animals, thats why we have these animals as our slaves. Wheather it be horses, dogs, hens, sheep or cattle we see them all there to further our existance. The comfort and consideration of these animals is not considered they are merely a money makeing tool.

Due to the evolution of man we have become superior to all beasts. If we are not stronger than them we are smarter and more advanced. We as humans rule the food chain. We have animals to do labor for us, make money for us and also for us to eat. Of course there are those who do not abuse the rights of animals, these people the vegetarians who do non of the above. These people see the slaughter of animals inhumane yet it is a necessity for others to survive. Isn’t it amazing that even though we have reached the top of the food chain the human race is the only species where there are meat and non-meat eaters.

The farming of animals for a specific purpose, for example battery hens is one of the main forms of creulty shown to animals in Australia today. Animals seen to be given no chance at a free life is seen to many as a disrespectful act. These farmers only appear to care for the net profit, in the case of battery hens it is extremely high. Where as free range has are a lot more exspensive to the farmer and not as widely used. They are more exspensive to buy as well and we as consumers mainly see this as a way to save money. All that most seem to care about is the all mighty dollar as oppose to the animals rights. Once again if people took sometimes to consider where the eggs come from they might reconsider buying the cheaper eggs. However most Australians put their financial situation before the well being of a hen.

It is our biological make up to have the urge to hunt. We have evolved into this species as hunting being one of our prime sources of food. However as we need to conserve our endangered species. This may result in the killing of endangered species enemies and threats in the wild such as rabbits. Australia as a whole is against the killing of endangered species which is why we are against the killing of such animals as the whale as it is done in such a in-humane way and to such a magnificent creature. There is some need for the hunting and killing of different species mainly due to control reasons, for example the kangaroo population, in Australia some say it is wrong to kill a national icon. However it is of an absolute necessity to control these populations. The way it is carried out though is also a topic of much discussion with animal rights being such a highly strung topic, the way these animal species are to be controlled cannot appear barbaric and unsensitive to the animal involved.

We live in a society where anyone is free to speak about what ever you wish to. However everyone is free to do what they wish to do as long as it is within the law. The meat-eaters of society don’t try and control what the vegetarians do with their eating habits, so why do the vegetarians feel they can control what the meat eaters eat? As long as it is within the law of this country you can eat what ever you want to eat.

Three of four markers gave this script a score of 6, and CGPT gave it a score of 5.

The opening reflections of the piece seem buoyantly pessimistic, and the introduction seems to wander. It does offer a kind of reasoning about the superiority of humans to animals. Financial issues are seen as most important in the second paragraph. Hunting is justified as a biological ‘urge’ in the third paragraph, and the importance of endangered species and the kangaroo as a national icon are mentioned. In conclusion meat eating is seen as a civil liberties issue and acceptable because within the law.

The CGPT comment generously described the piece as ‘philosophical musings’ and finds the reasoning ‘basic but sound’. The comment sees the piece as touching on ‘the ethics of animal treatment’ but does not suggest the understanding is as limited as it is.

There is no clear line of development that works up to conclusion in the piece. The student does not understand the opposing case, and does not accommodate or refute it. It is suggested in the CGPT commentary that ‘specific examples and evidence’ be incorporated to improve the piece, but this does not adequately identify or address the limitations of it. Vegetarianism is not a civil liberties issue.

Even with the limitations of the piece, the human score of 6 seems appropriate.

170615

Human scores 8 8 7 8

Activists for Animal Rights are Hiding Behind a Wall of Weak Arguments

Animals are not always treated in a humane way, but to cut out the use of animals by humans is unacceptable. Activists argue that killing animals is unnecessary and that it is a trait only common in the human species. Activists argue that if one person can be a vegetarian, anyone can. Activists argue that using animals for medical research only benefits humans. These may seem like good reasons for preserving animal lives but these arguments are weak. The use of one species by another is natural and benefits everyone, humans and animals.

The main argument formed when people refer to animal rights is the killing of animals. Those who say there is no place for a human to take the life of an animal are blinded by their hearts and cannot see the benefit to the animal population, humans included. Ecosystems are very balanced societies and can be effected greatly by the increased number of a particular species. There is a balance which allows all living things, flora and fauna, to live in harmony. If the population of kangaroos, for example, is allowed to increase, this balance will be disturbed and it is for this reason farmers Australia-wide have resorted to culling. When culling is performed it sets the balance back to normal, taking out the weaker of the species leaving the stronger of that species to continue balanced.

Some people say we as humans view ourselves as being of a “superior” species and that culling is a human trait. It is here the argument, given by those with this view, fails. When you are watching ‘The National Geographic Channel’ or one of many wildlife documentaries, animals ranging from lions to insects are found to be preying on smaller or weaker animals/species for food. This process of natural culling is Mother Nature’s way of keeping the balance and at the same time providing carnivorous species with food.

A major argument for animal rights is that if everyone became a vegetarian we would have no need to kill animals for food. It is here I again introduce documentaries and wonder whether Animal Liberation activists would watch these programs and decide that it would be beneficial for lions, bears, frogs and other carnivorous species to become vegetarian. There are many animals in the natural kingdom, including ourselves, that rely on the meat of other animals both for nourishment, and to obtain vitamins and minerals not easily obtainable in other foods. Unlike humans though, animals do not have the technology, science or medical knowledge to replace meat with supliments such as iron tablets.

Medical research plays an important part in the way human society is and there are not many people who would begrudge themselves, or someone close to them, another chance at life because the cure was discovered through animal research. If by experimenting on, for example, ten rabbits, medical scientists discovered a vaccine that prevented the death of millions of people, then there is a good chance that the sacrifice of ten animals is well worth the advantages it brings to millions more. At this point the idea that animal research only advantages humans needs to be addressed. For those in agreement with this idea, they have not taken into account the pluses of medical research for species other than our own. Without the discovery of antibiotics and other medicines used by vetinary surgeons worldwide, the lives of many animals each day would not be improved, or in some cases saved. Medical research benefits all living things and as long as it is carried out in a humane way, there are more benefits than downfalls.

Animal Rights is an emotionally charged issue that many people form their opinion on without looking at counter-arguments. Although the slaughter of animals is an unattractive thought, culling is natures invention and is carried out to ensure the balance of ecosystems. The usage of animal species for food also occurs in nature and through medical experimentation on animals the standard of living for all species, including ours, is improved. Activists can develop as many arguments as they like, but before they take into consideration counter-arguments, their arguments will always be weak. 

Three of four markers gave this script an 8 whereas CGTP gave the script a 6.

The approach of the piece is clearly polemical, it is focused on argument and offers a deliberate and distinctive argument of its own.

The piece begins by outlining the opposing arguments, and the first paragraph ends with the challenging claim that the ‘use of one species by another is natural and benefits everyone, humans and animals.’ The second paragraph addresses the issue of animal rights and argues that ecological balance requires that the weaker species be subject to the stronger. The third paragraph argues that the stronger preying on weaker is ‘Mother Nature’s way’ of keeping ecological balance. The fourth paragraph about medical research argues that while animal experiments should be carried out in a humane way, they are of value to both animals and humans. The last paragraph deals with the ‘emotionally charged’ issue of Animal Rights and concludes that animal rights activists have to counter the argument that nature ensures a natural balance through animals using other animals for food.

The piece is not without limitations, but it is a clear 8 because the language is clear and controlled and the argument is definite, organised and consistent.

The CGPT comment describes the argument as ‘plausible but limited in depth’, but it is not accurate to describe the piece as needing to explore ‘counterpoints’. This is what it actually does. The language is described as clear but ‘some sentences could be more concise’.

The CGPT score of 6 clearly underestimates the strength of this piece.

578020            

Human scores 7 6 5 6   

Animal Research the Saviour of Humankind

Animal research has played a significant and enormously beneficial role in the development of modern medical treatments. It has been these developments in medical treatments that has substantially improved the lives of all humankind.

Since the use of animal research began, medical scientists have been able to dramatically improve existing medical treatments. An example of this would be clearly demonstrated by patients with diabetes or cancer. Through the use of animal research, medical scientists have discovered new and exciting treatments, which reduce the pain and required dosage for these patients. There are a number of other examples, which have proven that animal research has been very beneficial to the improvement of existing medical treatments.

At present, the world is polluted by many thousands of diseases that threaten the life and even the existence of humankind. Unfortunately, the emergence of new and dangerous diseases do occur, as they have in the past. These particular diseases have the potential to destroy the lives of thousands. To prevent such events from occurring, animal research has proved the conclusive results which have led to the development of vaccines. It has been these vaccines which have prevented humankind from being infected by these dangerous diseases. This treatment may well of saved the lives of millions of people throughout the world. This is why animal research is so beneficial to humankind, because it prevents the emergence of new and dangerous diseases.

Every year many thousands of patients wait in frustration for an organ donor. Unfortunately, some patients are forced to wait too long and it results in their death. However, this deeply heart felt experience may be a thing of the past. Recently, medical researchers have begun experiments using animals as possible organ donors for humans. The results have been simply amazing, and more importantly it means patients would no longer have to wait in horror for an organ donor. This would mean in future animal organs would become readily available due to efforts of animal research. This is another enormous benefit that animal research has once again produced for humankind.

Animal research does result in the death of many thousand of innocent animals every year. However, if animal research was to be restricted in any way, we would ultimately preventing new and exciting medical treatments that would benefit all of humankind. Obviously, the death of an animal is very unfortunate, but it is because of their death that medical scientists are able to improve and save the lives of millions.

There is no doubt that animal research has been a definite saviour for mankind. Firstly, it has improved existing medical treatments for a number of diseases like diabetes and cancer. Secondly, it has prevented the emergence of new and dangerous diseases through the development of vaccines. Thirdly, it has opened the door for and possibly saved the lives of thousand of organ waiting patients. Finally, animal research may take the life of many animals every year, but through animal research the benefits to humankind are endless.

This is an unusual case in that the markers gave a range of 5 to 7 and CGPT gave a 7 for all criteria other than Structure and Organisation.
Scores of 5 to 7 would not send the script to discrepancy adjudication, and would average out at 6.

The piece is focused on the issue of medical research and the benefits it has brought to all humankind. It is suggested in the second paragraph that the use of animals in research has been part of a dramatic improvement in medical treatments. The third paragraph mentions the continued threat of dangerous diseases which means that the results of medical experiments on animals are necessary. The fourth paragraph mentions the possibility of organ transplantation from animals which would unfortunately involve the death of animals but could improve the lives of many humans. The piece concludes by reiterating the view that animal research may take the lives of many animals, the benefits to humankind are ‘endless’.

CGPT gives this piece 7 (which most of the markers do not) because it is ‘well-substantiated and plausible, though slightly one-sided’. The language is described as ‘clear and precise’, and the piece is described as considering ‘ethical concerns’ and including counterarguments. But, according to CGPT, the piece lacks specific examples, and analogies would improve the argument.

These claims are accurate enough, but the piece does not deal with whether there is unnecessary cruelty to animals in medical experiments, and whether there is enough effort to limit mistreatment of animals in research. It is questionable whether the piece is a firm 7 score.

As in all the CGPT comments, it accurately identifies the argument, and in this case is quite accurate about the substance of the argument and the characteristics of the language and presentation.

As with three of the four markers, I doubt that this piece deserves a score of 7.

The following is a kind of black box stress test for computer scoring. I wrote the script (rather quickly in 2005) to be brief and simple in language but as offering sophisticated thoughts about the topic. The aim was to see whether computer scoring could recognise substantial thought in concise and simple language.

325649            

Intended score 9

The life of a lamb is less precious than that of a human being.

It has been argued that animals have the same rights as humans.  This view seems to be based on the idea that animals feel and can suffer as we humans do, and so they can be said to have the same rights as we humans do.  This line of reasoning is not strong. 

Human rights are lovable, human fictions.  We are prepared to argue that we should not treat other humans in a way that we would not wish to be treated ourselves, and we imagine this argument into what we call human rights.  I can sympathise with and accept this line of reasoning.  But I cannot agree with the extension of this reasoning to other species.  It is not clear that a cow or a pig feels and suffers as you and me do.

Those who claim animals have the same rights as humans project their own feelings onto those of animals.  One might reasonably ask the animal rights advocates whether all living creatures have rights?  Do flies and mosquitoes have the same rights as humans?

Although I believe the claim that animals have the same rights as humans is absurd, I think that treating animals humanely is very important.  It degrades humans to treat animals cruelly.  We become more human when we treat each other and animals with humanity.  But this is not to blur or collapse the distinction between humans and animals.

I am opposed to cruelty to animals not because animals have the same rights as humans or that they suffer as we do.  I am opposed to cruelty to animals because they do suffer and because we humans have to constantly exercise our humanity by causing each other and animals no undue or pointless suffering.

Animal rights advocates often justly identify acts of human cruelty to animals.  I have my doubts whether animal experimentation can take place humanely.  But I do not think we have to draw the conclusion that farming animals, using animals in medical research or hunting are necessarily unacceptably cruel.  To recognise that animals suffer and that cruelty to animals is unacceptable is not to conclude that vegetarianism is a necessity, or that zoos and circuses are necessarily cruel, or that hunting (and even hunting for sport) is necessarily unacceptable.

If animals have rights, they have the right to be treated carefully, humanely and to be caused no unnecessary or avoidable suffering.  But this does not give animals the same right as humans or mean that what should not be done to humans cannot be done to animals. 

The commercial computer algorithms clearly failed this test. One gave the piece a score of 4 and the other gave a score of 5. We cannot know what markers would have scored this script (and I am not an impartial observer), but in my view it should be given at least a score of 8. Interestingly, CGPT did better at recognising the strength of this piece than the ‘natural language’ computer algorithms by giving it the highest score of 7 in the set to this piece.

The CGPT comment gives an accurate one sentence summary of the argument and recognises that it involves ‘a well-reasoned argument that challenges the idea of equal rights’. The comment that the language strengthens the script's persuasiveness by careful phrasing seems appropriate. The phrasing is simple, but it is also careful. The suggestion that the piece would be improved by the addition of examples or analogies seems nugatory.

The CGPT score for this script is significantly more appropriate than those of other methods of computer scoring. The comments on the piece are quite subtle, while the other the computer scoring methods do not offer comments to explain or justify a score.

These examples show in my view that CGPT offers feedback on scripts that is worth considering. I find some of the comments of CGPT of value, and would be inclined to take a second look at any script in which my score differs in more than 1 mark from CGPT.

Appendix The Reflexive Use of CGPT

The following is CGPT feedback about this research process and the outcomes of it. The comments are responses to the following questions. The response is a good overview of the spirit of the BBE Described Levels. This seems to me useful feedback to teachers and students.

What the key issues in the BBE described levels?
What are the characteristics of high, medium and low level scripts?

The key issues in the BBE described levels relate to the development of thought, expression, and engagement with the prompt. Here's a summary of the characteristics for high, medium, and low-level scripts.

Low-Level Scripts (1- 4)

- Lack of focus and substance: These scripts often fail to grasp the core issue or prompt.
- Simplicity and crudeness: The handling of the issues tends to be reductive and lacks nuance, with clumsy or crude expression of ideas.
- Unclear and inaccurate expression: The language is often unclear or inappropriate for the context.
- Basic understanding: There is some comprehension of the prompt but ideas are simple, predictable, and poorly developed.
- Disorganized: The structure may be either disorganized or rigidly mechanical, without clear argument progression.

Medium-Level Scripts (5 - 6)

- Some grasp of the issue: These scripts show some understanding and an ability to define the topic or comment on it.
- Purposeful but predictable: There is a sense of direction and organization, but often the approach is standard, obvious, and lacks deeper insight.
- Competent but bland: The writing is straightforward, functional, and competent but not particularly engaging or insightful.

High-Level Scripts (7 - 10)

- Substantial thought and engagement: These scripts demonstrate significant thought about the issue, going beyond the obvious and recognizing complexities.
- Ability to formulate and support a point of view: Writers at this level are able to clearly articulate and support a well-developed perspective, showing critical thinking and engagement with the prompt.
- Lucid and precise writing: The writing is fluent, clear, and vivid, often showing sensitivity, involvement, and responsiveness to the issue.
- Nuanced understanding: High-level scripts reflect an ability to analyze the complexities of the issues, with an insightful and critical approach.
- Distinctive voice: There is a developed, distinctive tone or attitude, and the writing is generally characterized by originality and vivid expression.

In summary, high-level scripts demonstrate depth, clarity, and sophistication in both thought and language, while low-level scripts struggle with basic comprehension and clarity, offering predictable or simplistic responses. Medium-level scripts are competent but tend to lack depth or originality.

The following is CGPT feedback from the following question about its own feedback. Again it is quite accurate and informative. 

Summarise the comments on the scripts offered in the attached document CGPT Commentary.
What does the commentary criticise and value?
What are the characteristics of high, medium and low level scripts?

The commentary in the document offers insights into the evaluation of various scripts, identifying strengths and weaknesses across several dimensions: argument depth, organization, language use, and the engagement with ethical concerns related to animal rights and human exploitation of animals.

What the commentary criticizes:

1. Lack of Depth: Many scripts are criticized for not delving deeply enough into ethical concerns, alternative viewpoints, or the complexities of the issues they discuss.
2. Simplistic Arguments: Several essays provide straightforward defenses or critiques but are considered too simplistic or lacking in nuance.
3. Repetition and Unclear Language: Some essays are described as repetitive or lacking clarity, with language that does not engage the reader or fully develop the argument.
4. Weak Structure: Disorganization or scattered ideas are a common critique, as well as essays that fail to maintain a clear focus or thesis.

What the commentary values:

1. Balanced and Nuanced Arguments: High-scoring essays tend to present a balanced discussion, critically engaging with both sides of an issue.
2. Ethical Engagement: Essays that explore the ethical dimensions of animal rights and human exploitation in a thoughtful way are often praised.
3. Clear Structure: Well-organized essays with a logical flow of ideas and coherent structure are valued.
4. Engaging Language: Essays that use vivid, precise, and engaging language receive higher praise, especially when they manage to balance emotional appeal with objective reasoning.

Characteristics of High, Medium, and Low-Level Scripts:

• High-Level Scripts (Scores 8-10):
  These scripts are well-structured, with a clear thesis and thoughtful engagement with both sides of an argument. They provide nuanced ethical analysis, engage deeply with the topic, and use precise, fluent, and persuasive language. The arguments are well-developed and supported by detailed examples.
• Medium-Level Scripts (Scores 5-7):
  These essays are generally clear and organized but may lack depth in exploring complex ethical issues or alternative viewpoints. They often present a solid argument but could benefit from more detailed analysis or engagement with counterarguments. The language is typically clear but may lack sophistication or variation.
• Low-Level Scripts (Scores 2-4):
  These essays often suffer from poor structure, unclear language, and disjointed arguments. They may touch on relevant issues but fail to develop their ideas fully. Repetition, grammatical errors, and lack of clarity are common, and they often oversimplify the ethical concerns, offering limited insight into the topic.

The overall evaluation emphasizes the importance of depth, clarity, engagement with ethical concerns, and sophisticated use of language for higher scoring scripts.

^[1] McCurry, D. M. (2009). Can Machines Score a Broad and Open Writing Test as Well as Human Readers?, Assessing Writing 15 pp. 118-129

McCurry, D. M., (2010), The machine scoring of writing, English in Australia Volume 45, Number 1, 2010 pages 47-52.

McCurry, D. M.(2012), Computer scoring and quality of thought in assessing writing in Measuring writing: Recent insights into theory, methodology and practices, Studies in Writing, Vol 23