Format

BMJ links


 

BMJ: 'Was message of sudden infant death study misleading?' by Jonathan Gornall


A significant omission: the case of the BMJ, Jonathan Gornall and the missing word.


Webster’s ‘sophistry’: Jonathan Gornall replies


'The presumption of murder': BMJ Rapid Response from Richard Webster

The man who has the answers: Professor Ray Hill (scroll down to bottom of page to download articles)



 

Roy Meadow, the statistics of cot deaths and other fragments

RICHARD WEBSTER

7 February 2007

………………………………………………

Towards the end of last year the appeal court judgment in the case of Meadow v GMC prompted me to look again at Professor Sir Roy Meadow's use of statistics and at other aspects of the cot death debate. The essay which I embarked on then remains unfinished but the publication by the BMJ of an article on the same subject by the freelance journalist Jonathan Gornall has prompted me to reproduce here selected fragments of that essay. These fragments will be added to over the next few days and I will also try to add a small library of relevant links. . . .

……………………………………………… 

1 Conviction by mathematical error?

LAST MONTH, on Thursday 26 October 2006, three judges in the Court of Appeal overturned a high court judge’s ruling giving immunity from disciplinary action to expert witnesses such as Professor Sir Roy Meadow, who gave misleading statistical evidence to the jury which convicted Sally Clark of murdering her two children.

The Court of Appeal ruled that no such immunity should be granted to experts. As was widely reported at the time, however, it upheld a finding by the high court judge, Mr Justice Collins, that the General Medical Council was wrong to find Professor Meadow guilty of serious professional misconduct over his grossly inaccurate evidence that the chances of two deaths which could be classified as ‘SIDS’ (Sudden Infant Death Syndrome) in a family like Mrs Clark’s were one in 73 million.

This finding was understandably welcomed by Professor Meadow who said: ‘ I am glad that the Court of Appeal has agreed with the previous High Court judgment that my evidence in the trial of Mrs Clark nearly seven years ago was not an example of serious professional misconduct. The GMC was wrong in its judgment of me.’

What was not prominently reported, however, was that all three judges on the panel took the view that Meadow had given misleading evidence and their eventual finding on this issue was split by a majority of 2-1. Lord Justice Auld said Professor Meadow was ‘undoubtedly guilty of some misconduct.’ He added: ‘I could not contemplate erasure [from the medical register] as an appropriate penalty for Professor Meadow’s uncharacteristic honest errors in this difficult case.’

Lord Justice Thorpe, the other judge who dismissed the appeal over serious professional misconduct, said: ‘Professor Meadow’s evidence, flawed though it was, fell far short of serious professional misconduct.’

However the senior judge on the panel, Master of the Rolls Sir Anthony Clarke, dissented from the view of his colleagues. He accepted Professor Meadow’s claim that he had given his evidence in good faith but he still concluded that the manner in which he had done so added up to a case of serious professional misconduct and gave his reasons for arriving at this conclusion.

The fact that such a senior appeal court judge should so emphatically disagree with his fellow judges on this issue is, I believe, highly significant. Moreover, the rationale on which Sir Anthony based his conclusion raises, or ought to raise, a series of grave questions about the conduct of medical experts in cases such as Sally Clark’s. That being so it is perhaps time to revisit not simply the specific issue of Professor Meadow’s use of statistics but the whole question of what is sometimes referred to as the ‘diagnosis’ of Munchausen’s Syndrome by Proxy.

Let us start by considering Professor Meadow’s use of statistics. When he told the jury that the odds against two of Sally Clark’s children dying of unidentified natural causes (or Sudden Infant Death Syndrome) were 73 million to one, he was leaving the area of his expertise – paediatric medicine – and entering a field in whose basic principles he was completely unschooled – statistical probability. He did so without alerting the jury or the court to the fact he was no longer addressing them as an expert and thus invested a statistical fallacy with all the authority his medical status endued him with.

In giving this part of his evidence he relied upon the draft version of the latest CESDI report ( Confidential Enquiry into Stillbirths and Deaths in Infancy) . This report, whose principal author was Professor Fleming, Professor of Infant Health and Development Physiology at the Institute of Child Health at Bristol University, was commissioned by the Department of Health. It was a study of factors contributing to sudden and unexpected deaths in infancy. In relaying some of the statistics from this report to the jury, Professor Meadow misconstrued them, committing three elementary statistical errors simultaneously. So serious were these errors that they moved Dr Stephen Watkins, a trained epidemiologist with an expertise in statistics, to compose an editorial which was published in the British Medical Journal under the title ‘Conviction by Mathematical Error’. In the editorial he did not argue that Meadow’s abuse of statistics had led to the conviction but rather that the prominence of misleading statistics in Meadow’s evidence meant that these could have overshadowed the other evidence. He suggested that this necessarily rendered the conviction unsafe.

The draft version of the paper which Meadow used as his source indicated that there was one SIDS death for every 1,300 deaths. It also identified three important factors which, if present in a particular family, were capable of increasing the risk of such a death, namely the presence of a smoker, the absence of any wage-earner and the fact that the mother was below the age of 26. In families in which none of these factors applied it was said that the odds against one unexplained infant death occurring by natural causes were much higher than indicated by the general figure; instead of 1,300 to 1 they were 8543 to 1. In families in which all three risk factors were present, they were significantly lower, namely 214 to 1.

The authors of the paper made clear, however, that although the three factors they had singled out were perhaps the most important, there were also other factors which would affect the probability of a SIDS death occurring.

Meadow’s first statistical error was that he ignored this caveat. Although he was now moving from abstract generalisation to concrete particular, where specific risk factors might have been identified, he assumed without inquiry that the Clark family should automatically be assigned to the lowest category of risk. Since the risk factor was subject to a large variation – from 214 to 1 to 8543 to 1, this was potentially significant.

Whatever error Meadow had introduced in this manner though, was now massively compounded by his next step. Here, in fairness to Meadow, it must be said that the paper he was relying on was itself presented in such a way that the unwary and uninformed reader might well misconstrue it.

The CESDI report contained the following passage:

For a family with none of these three factors, the risk of two infants dying as SIDS by chance alone will thus be 1 in (8,543 x 8,543) i.e. approximately 1 in 73m. For a family with all three factors the risk will be 1 in (214 x 214) i.e. approximately 1 in 46,000. Thus, for families with several known risk factors for SIDS, a second SIDS death, whilst uncommon, is 1,600 times more likely than for families with no such factors. Where additional adverse factors are present, the recurrence risk would correspondingly be greater still.

… When a second SIDS death occurs in the same family, in addition to careful search for inherited disorder there must always be a very thorough investigation of the circumstances – though it would be inappropriate to assume maltreatment was always the cause [italics added] (quoted in General Medical Council v Meadow, 2006, paragraph 137) .

The crucial words here are ‘by chance alone’. What these words were intended to convey was that the calculation which arrived at the figure of 1 in 73 million was an abstract mathematical exercise which could not properly be applied to any instance of recurrent infant death in a particular family. For this method of squaring the original odds could only be used legitimately if it could be shown that the causes of the two deaths were entirely independent of one another – a negative which in practise could never be established with complete certainty. Since the two children whose deaths had given rise to the allegations against Sally Clark were siblings who shared the same genetic inheritance and the same environment, there was no reason to make such an assumption. Indeed a particular set of genetic or environmental circumstances might actually mean that the death of a first child was predictive of a second death.

However, although a table containing the 1 in 73 million figure was put before the jury, the text quoted above, with its specific caution that ‘it would be inappropriate to assume maltreatment was always the cause’ was never presented to the jury and Meadow himself made no reference to it in his evidence.

The errors Meadow made by assuming that the Clark family fell into the lowest category of risk, and by accepting the squaring method, were both serious. Yet these two errors were relatively minor in comparison to the third statistical fallacy on which Meadow’s evidence rested. This was what Watkins has called ‘the lottery effect’ (also known as ‘the prosecutor’s fallacy’) which consists in calculating the ‘chance of a specific individual, identified in advance, having a particular experience and [using] that as the measure for the chance of some individual, out of all those at risk, suffering it.’

This fallacy is perhaps best illustrated by considering a hypothetical weekly lottery in which every member of the UK population of 60 million purchases a ticket bearing a different combination of numbers. When one particular person – let us call her Mrs Smith – wins this lottery, the odds against her doing so in that particular week were, before the event, 60 million to one. But once Mrs Smith has produced her ticket to show that she has won, the authorities do not invoke these odds in order to suggest that she has committed an act of fraud. Because in reality, although the odds against any one specified person winning this prize are very high, the odds against an unspecified person winning are not high at all. Indeed, in our hypothetical lottery it is clear that one person wins the lottery every week.

Although it is less common for two babies in the same family to die of natural causes than it is for somebody to win the lottery, a recurrent death of this kind is by no means an unheard of event. According to figures which Watkins cites, it occurs to somebody once every twelve to eighteen months. As he points out, when the judges in Sally Clark’s first appeal suggested that Meadow should have replaced his figure of 1 in 73 million by the claim that a double infant death in the same family was ‘very rare’, they clearly had not grasped this point.

As the Royal Society of Statistics indicated, in a press release issued after the first trial of Sally Clark, the question which Meadow should have addressed was not the one concerned solely with the chances of two natural deaths occurring in the same family:

‘The jury needs to weigh up two competing explanations for the babies’ deaths: Sudden Infant Death Syndrome [SIDS] or murder. Two deaths by SIDS or two murders are each quite unlikely, but one has apparently happened in this case. What matters is the relative likelihood of the deaths under each explanation, not just how unlikely they are under one explanation.’

In other words if, for the sake of argument, the chances of two unexplained natural infant deaths in a family were 1 in 500,000 and the chances of a double murder were also 1 in 500,000, then, if all other evidence was equal or equivocal, there would be a 1 in 2 chance that Sally Clark’s children died of natural causes. There is a big difference between this and the 1 in 73 million chance which Professor Meadow Meadow, in his guise as an expert, had confidently put before the jury, comparing these odds to the chance of successfully backing four different 80-1 outsiders to win the Grand National in four successive years.

Although it is far from clear that this point has been fully assimilated by the courts, it was made eloquently by Mr Justice Jack during his summing up in the trial of Trupti Patel in June 2003 where Mrs Patel was eventually acquitted by the jury of murdering three of her children:

I said that I would have a direction for you on the subject of there being three deaths. You have heard from some of the Prosecution witnesses the idea that the fact of three deaths makes it more likely that the cause was unnatural. Certainly with three deaths one must be suspicious and look the more carefully, for it is potentially a very serious situation. But I am going to ask you to put out of your minds the idea that because there are three that makes it more likely that the causes are unnatural: that is asphyxiation by Trupti Patel. I think that would be a dangerous approach in this case for two reasons.

The first is this: suppose that something happens and there is only one possible event as the cause for it. However rare or common that event may be, it must be the cause: straightforward. If it is rare the unexpected has happened. Suppose, though, that there are two possible events as the cause. One is a common event and one rare. It can then be said that the common event is the more likely cause. Suppose, however, that the two events are both rare; perhaps very rare. They are nonetheless equally likely as the cause even though they are rare, because they are competing with each other to be the cause .

So it is not enough to say that an event is rare so it is unlikely to be the cause of something. One has to look at the likelihood of the other possible cause, or other possible causes. That is the danger with what may be happening here in saying that three SIDS deaths in a family would be very unusual, therefore the deaths are unnatural. How rare would three asphyxiations be particularly where, as is the case here, the mother loved her children and was immediately distraught and regretful? We simply do not know. We have not had any evidence about that. It is hardly common is it? That is obvious. That is the competing cause of the deaths and nobody has evaluated its likelihood.

This direction, which may well have been directly influenced by the Press Release issued by the Royal Society of Statistics some two years previously, puts one key part of the statistical argument extremely well.

The jury in the Sally Clark trial had been given no such clear guidance by the judge and it was Meadow’s evidence which overshadowed the entire proceedings.

What rendered his evidence even more likely to mislead was that he gave it on the assumption that the Defence would be arguing, on Sally Clark’s behalf, that the deaths of her two children should be regarded as SIDS deaths. In fact this was not the position they took; they argued simply that the two deaths had arisen from natural causes. This is important for the simple reason that the chances against two children’s deaths being classified as SIDS (in other words natural causes which cannot be explained or identified by current medical knowledge) are not the same as the chances against two children dying from any kind of natural cause. This is because the latter category would necessarily include not only deaths which could not be explained, but deaths which could – such as those occasioned by identifiable genetic disorders or infectious diseases. The figure of 1 in 73 million was irrelevant to this broader category and Professor Meadow’s evidence should have made this clear.

It is a matter of some interest that, although the question of Professor Meadow’s statistics has now been examined by at least eight judges, and by many more journalists, it is far from clear that any of these examinations has succeeded in conveying the sheer magnitude of the statistical errors which he perpetrated in his evidence.

The most recent judicial appraisal, in the case of GMC v Meadow in the Court of Appeal (October 2006) is no exception to this. It is therefore a matter of some interest that, even though the gravity of Professor Meadow’s errors are not acknowledged adequately in any part of the judgment, one of the three judges, Sir Anthony Clarke, still came to the conclusion that the GMC was right to find Professor Meadow guilty of serious professional misconduct.

In reaching this conclusion he based his views on the statement of the duties of expert witnesses set out by Mr Justice Cresswell in the case which has become known as The Ikarian Reefer (a case involving a Panamanian vessel alleged by its insurers to have been the subject of arson):

1. Expert evidence presented to the court should be, and should be seen to be, the independent product of the expert uninfluenced as to form or content by the exigencies of litigation.

2. An expert witness should provide independent assistance to the court by way of objective unbiased opinion in relation to matters within his expertise. An expert witness in the High Court should never assume the role of an advocate.

3. An expert witness should state the facts or assumptions upon which his opinion is based. He should not omit to consider material facts which could detract from his concluded opinion.

4. An expert witness should make it clear when a particular question or issue falls outside his expertise … (General Medical Council v Meadow, 2006, paragraph 70)

Sir Anthony went on to say that in his opinion it was ‘of the utmost importance that an expert should only give evidence of opinion which is within his particular expertise and that, where a statement, whether made in writing or orally, is outside his expertise, he should expressly say so.’ It was in his view essential that ‘All reasonable attempts should be made to check the validity of an opinion which is not within the expert’s expertise.’ The care which any expert should be expected to take when giving evidence was especially necessary in cases which involved grave allegations:

These are simple precautions which should be taken by experts because of the risk that the opinion might be wrong, with what may be very serious consequences. This seems to me to be of particular importance in a serious criminal matter such as the trial of a defendant for murder (paragraph 71).

In developing his argument Sir Anthony quotes the view of Lord Justice Kay as expressed in the judgment in Sally Clark’s second appeal:

Putting the evidence of 1 in 73 million before the jury with its related statistic that it was the equivalent of a single occurrence of two such deaths in the same family once in a century was tantamount to saying that without consideration of the rest of the evidence one could be just about sure that this was a case of murder.

Sir Anthony said that he entirely agreed with this view: ‘Professor Meadow must surely have appreciated that that was the case or, if he did not, he was to my mind grossly negligent in not doing so.’ He went on to set out his conclusion in the following terms:

The essential features of his evidence which have persuaded me that Professor Meadow’s shortcomings amount to serious professional misconduct and not simply to professional misconduct are that he did not simply state that the statistics were relevant only to SIDS deaths … and that they were not relevant to and did not help to decide whether the deaths of the children were caused by natural causes. On the contrary, the way in which he gave the evidence quoted in paragraphs 79 and 80 above in my opinion suggested that his opinion was that the 73 million to 1 statistic, the one in a hundred years’ chance and the Grand National analogy all applied to the chances of these deaths being caused by natural causes. The second court of appeal expressed that opinion in … their judgment … albeit without hearing full argument on the point, and I agree. To put it at its lowest, there was to my mind a serious risk that the jury would so understand the evidence and accept it …

It is true that Professor Meadow did not intend to mislead the jury and that no-one challenged what he did but, as Kay LJ put it in paragraphs 102 and 175 quoted above, that was the picture that would be uppermost in the jury’s minds and was tantamount to saying that without consideration of the rest of the evidence one could be just about sure that this was a case of murder. In my opinion Professor Meadow should have appreciated that there was (to put it no higher) a serious risk that that would be the effect on the jury and should not have made the unqualified statements that he did. In short this is one of those rare cases in which the FPP [the Fitness to Practise Panel of the General Medical Council] was correct to hold that the expert was guilty of serious professional misconduct without acting in bad faith (paragraphs 92, 95).

Even though Sir Anthony does not begin to convey the full gravity of Professor Meadow’s statistical errors, he has no hesitation in reaching this conclusion. What he is saying in effect (although it should be made clear that he does not use these terms himself) is that any expert giving evidence to a court which is dealing with a count of murder must be absolutely scrupulous. In failing to conduct the necessary checks on the evidence that he gave, so that he actually presented evidence which was grossly misleading, Professor Meadow was behaving unscrupulously. Because his evidence might in itself have provided the jury with a basis for convicting Sally Clark, the matter was very grave indeed and justified a finding of serious professional misconduct even though his evidence was apparently given in good faith . . . .

Conviction by mathematical error

………………………………………………

© Richard Webster, 2006/7

www.richardwebster.net

 

TOP

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

......................

Richard Webster, 2004