“No one comes up here without a damn good reason.”
* * *
Regular readers will recall that I have previously written about the UK’s new healthcare guidelines for ME/CFS, as published by the National Institute for Health and Care Excellence (NICE) in 2021. Whereas the old guidance had proposed that patients be given a combination of graded exercise therapy (GET) and CBT, the newly updated guideline completely rejected these two treatments. Having completed the most extensive research review ever conducted on the matter, NICE soberly concluded that there was insufficient empirical evidence to support their continued use.
The episode is a case study in using updated science to address medical error in the face of professional territoriality. Like many other commentators, I immediately predicted that those (few) practitioners who previously championed the GET/CBT approach would be especially resistant to any new treatment standard. After all, they had staked their careers and reputations on the credibility of the old therapies. But I pointed out that even though such resistance would be inevitable, ultimately it would prove futile. When it comes to scientific revolutions, history shows that science always wins in the end.
I guess we are still in the midst of the adjustment phase with this one.
Martin Rücker, a investigative journalist in Germany, has revealed the details of an as-yet-unpublished opinion piece that has been written by a cadre of these individuals. He outlined their overall approach and some information about the content of the piece in an extensive Twitter thread. You can see it all here (note: the Tweets are in German):
It seems that a key objective of the paper is to highlight “eight major errors” perpetrated by the NICE committee “in the guideline process and outcome” of the ME/CFS review. According to Martin’s information, these “major errors” are as follows:
David Tuller (with whom I have co-written pieces in the past) and Adam Lowe (a member of the NICE ME/CFS committee) have dissected the various “errors” here. Others have done so extensively on social media, and on patient-led message boards, such as Science 4 ME.
Suffice to say, these so-called “errors” are not really errors at all.
* * *
What strikes me most about the list of claimed errors is not their lack of substance, but their lack of logic.
These days, we hear a lot of catastrophising about the post-truth society. In my view, modernity’s crisis is more complicated than that. The issue isn’t the absence of veracity, it is the absence of validity. The problem is not that society has moved beyond a norm of expecting claims to be true, it is that, by and large, we no longer check to see if they are even logical.
The problem of living in a post-logic society is that it returns us to an age of deference. We revert to an existence where (self-styled) authority figures are enabled, if not encouraged, to say whatever comes into their minds, knowing that they will go unchallenged even if they end up spouting nonsense.
This “eight-so-called-major-errors” paper is an excellent example of this problem.
Let us count the ways…
* * *
So-called ‘Error’ #1:
“[NICE] created a new definition of CFS/ME, which automatically downgraded the certainty of trial evidence”
Problem
This claim is erroneous on its face. In framing its guideline, NICE did not “create a new definition” of ME/CFS. It worked with the definition formulated by the Institute of Medicine (now the US National Academy of Medicine) which dates back to 2015. This is the definition that the US Centers for Disease Control uses when providing information to healthcare providers. In other words, the definition here is not “new”. It is well established and widely employed.
With regard to logic, using the IOM definition did not “automatically downgrade the certainty of trial evidence.” This is because the “certainty of trial evidence” was already of an inherently low grade, having been reliant on self-report data gathered using non-blinded methods. In other words, the trial evidence suffered because so many researchers conducted sloppy research. Focusing on subjective measures of so-called “recovery” — and overlooking objective ones — represents extremely poor research practice.
Fallacy
The IOM definition has no bearing on whether researchers should use subjective or objective measures, so claiming it had a bearing on the certainty of trial evidence is spurious.
This is an example of a red herring fallacy: the authors present a misleading premise, from which they then make an irrelevant inference.
* * *
So-called ‘Error’ #2:
“[NICE] omitted data from standard trial end points used to assess efficacy”
Problem
This claim notes that NICE prioritised some types of outcome data over other types. Moreover, it implies that NICE did this in a fashion that was arbitrary, in order to distort the research evidence in favour of its own argument. In reality, NICE applied a consistent standard of prioritising long-term outcomes over short-term ones. This makes sense, because ME/CFS is a long-term illness and so long-term outcomes are more important.
Specifically, a therapy that produces short-term benefit but not long-term benefit must be seen as less useful than a therapy that does the opposite (i.e., one whose effects are lasting rather than brief).
Fallacy
By accusing NICE of being partisan in cherry-picking which outcome data to prioritise, the authors are employing an appeal to motive. They are implying that NICE deliberately skewed the evidence to serve its own ends (whatever those ends might be). This is fallacious because, as outlined above, NICE’s strategy was logical, clearly stated, and systematically applied, and therefore clearly not arbitrary.
The accusation that NICE is engaging in unfair practices here is itself an example of poisoning the well, a form of fallacious argumentation where supposedly adverse information is used in an effort to discredit another party.
* * *
So-called ‘Error’ #3:
“[NICE] discounted trial data when assessing treatment harm in favour of lower quality surveys and selective comments from qualitative studies”
Problem
NICE found that the available research did not record patient harms particularly well, with most studies simply ignoring the possibility that patients could ever suffer a negative consequence from therapy. This reflects a common standard of practice in ME/CFS clinics, at least in the UK. According to freedom of information data, there appears to be no evidence of any treatment-related patient harms ever having been formally recorded by any British ME/CFS clinic. (Disclosure: I was one of the authors of the study that found this.) A significant subset of patients drop out of treatment. It is unfortunate that UK clinics generally do not operate a policy of asking them why they do so.
Nonetheless, several surveys have found that ME/CFS patients often suffer adverse reactions to treatments, especially GET. As NICE could find no evidence that GET was actually therapeuetic, its new guideline advises that GET should therefore not be offered to patients. The guideline document explains that this is because reports of patient harms are not outweighed by empirical evidence of therapeutic benefit.
Fallacy
In dismissing, out of hand, the findings of patient surveys, the authors are engaging in a form of fallacious argument occasionally referred to as ‘Bulverism.’ This is where speakers simply assume that their opponents are wrong and then pursue their argument from that premise. In reality, they have not made the case as to why — or even if — their opponents’ claims are flawed.
This is a form of ad hominem argument, classified as ‘circumstantial’ ad hominem. In other words, it is being asserted that the circumstances dictate that survey respondents must be biased. This is fallacious because, even though survey evidence is less robust than evidence from, say, controlled scientific research, this does not in and of itself make survey findings automatically wrong.
While better data may be gathered in the future, the various patient harm surveys that have been conducted to date remain the best source of current evidence that we have on this issue.
* * *
So-called ‘Error’ #4:
“[NICE] downplayed the importance of fatigue as an outcome, even though this is the primary symptom of CFS/ME”
Problem
It is misleading to suggest that NICE “downplayed” the importance of fatigue as an outcome of ME/CFS, or in other respects. For example, three of the four cardinal symptoms specified by NICE as essential to a diagnosis of ME/CFS are fatigue-based (i.e., debilitating fatigue, post-extertional malaise, and unrefreshing sleep). It follows that if a patient is to recover from ME/CFS — that is, if they are no longer to have such a diagnosis — then they should no longer be exhibiting these fatigue-related symptoms.
In his interview with David Tuller, Adam Lowe describes how the concept of “fatigue” can be seen as generic. For example, while patients might complain of generalised fatigue to begin with, as their condition develops they start to appreciate their different fatigue-related symptoms in a far more specified manner (for example, they start to differentiate between orthostatic intolerance and, say, brain fog). Therefore, it makes sense for clinicians to consider outcomes in a similarly detailed way.
In terms of what “core outcomes” should be focused on the most, the NICE guideline explicitly recommends that identifying the best outcomes to use should be an urgent priority for new research. As such, NICE was meticulous in not downplaying the importance of fatigue per se, or of any other symptom. Instead it concluded that such declarations of “importance” should be rigorously evidence-based.
Finally, in terms of its 356-page review of the evidence for psychobehavioural interventions (including GET and CBT), NICE did not “downplay the importance of fatigue as an outcome.” In fact, it emphasised the importance of fatigue as an outcome. In a section entitled “The outcomes that matter most”, NICE stated the following:
Fatigue/fatigability, unrefreshing sleep and physical and cognitive dysfunction are recognised as key symptoms of ME/CFS. The worsening or improvement of these symptoms reflect the impact of an intervention or strategy.
So, rather than downplaying fatigue’s importance as an outcome, NICE highlighted it as a key symptom of ME/CFS. Indeed, according to NICE, fatigue is one of the “outcomes that matter most.”
Fallacy
The nearest I can find to any “downplaying” of emphasis is where NICE point out that subjective self-reports of fatigue are inherently less precise than objective measures of, say, physical endurance, and that, as such, blinding is especially important when conducting research that uses fatigue as an outcome. So NICE does not “downplay” fatigue, it merely points out that fatigue is relatively hard to measure when you compare it to (many) other symptoms.
The tactic of misrepresenting an opponent’s argument in order to make it easier to refute is an example of a straw man fallacy. Claiming that NICE “downplayed the importance” of something — when it actually identified that something as one of “the things that matter most” — helps the authors to convey an impression that they have successfully proved NICE wrong.
Straw man arguments are an interesting fallacy because they help to show how poor reasoning is often intertwined with the generation, and spread, of misinformation.
* * *
So-called ‘Error’ #5:
“[NICE] failed to synthesise and GRADE trial evidence adequately”
Problem
This claim relates to the standardised procedure, known as the GRADE system, that NICE uses to evaluate the quality of research trials. The GRADE system is widely endorsed by research organisations as a way to promote evidence-based medicine. As per standard practice, NICE used the GRADE system to evaluate hundreds of study findings from ME/CFS therapy research, publishing its assessment of CBT and GET in a detailed 356-page document on the NICE website.
It is important to note that the grading was done by an independent NICE technical team, who do this job for all the NICE reviews. In other words, it was not conducted by the ME/CFS review group. This reflects a standard approach applied by NICE to the evaluation of evidence bases for medical treatments. While all such systems require human judgement to some extent, on the face of things, it is difficult to see why NICE should be accused of misapplying GRADE in this particular guideline review as opposed to in any other.
In relation to ME/CFS, some critics of the new guideline have tried to argue that behavioural therapies should be exempt from systems like GRADE. They claim that behavioural interventions should not be judged by the same standards as are applied to other medical interventions. For example, they argue that it is unfair to expect behavioural therapy research to use blinding. However, this is problematic for two reasons.
Firstly, the very same voices have previously claimed that blinding is in fact essential for such research. They have argued that blinded studies provide “the best and only reliable evidence” for the effectiveness of “any intervention” in “any condition.” That they would now adopt the opposite position out of allegiance to their own preferred therapeutic approach is, to say the least, indicative of a double standard.
Secondly, the main purpose of GRADE is to minimise the problems that arise when clinicians are asked to subjectivity evaluate their own preferred treatment approaches. Such biases are particularly common in behavioural research. (Disclosure: I have written two books highlighting these problems.) Therefore, far from being unsuitable, it could be argued that GRADE is, in fact, especially appropriate to the evaluation of behavioural therapies.
Fallacy
The idea that behavioural therapies are somehow distinct and so should be evaluated differently from everything else exemplifies the fallacy of special pleading. This is where a speaker identifies their own cause as an exception to a general principle without justifying why. It implies that a speaker is aware of a problem with their position, but wants that problem to be ignored and their position to be privileged, regardless, over the positions of others.
It also hints at a common reasoning error in which an observer assumes that their own insight unassailably represents the true facts of a situation. In other words, special pleading is justified on the basis that the speaker “just knows” they are right and that everyone else must therefore be wrong.
Interestingly, this assumption — that one’s own standpoint is objective and definitive — is known in philosophy as the psychologist’s fallacy.
I kid you not.
* * *
So-called ‘Error’ #6:
“[NICE] misrepresented graded exercise therapy as mandating fixed increments of change when the major trials defined it as collaborative, negotiated, and symptom dependent”
Problem
This claim implies that “mandating fixed increments” and “collaborative, negotiated, and symptom dependent” are mutually exclusive concepts. They are not. GET can involve both at the same time. That said, what is meant by “collaboration” or “negotiation” can sometimes appear quite arbitrary.
The GET manual from PACE, one of the aforementioned major trials, is usually identified as being definitive. This manual instructs practitioners to “negotiate meaningful goals” and to “negotiate and add baseline of exercise at low intensity” with patients at the beginning of GET (see p. 38). But then, as its next step, the manual specifies a fixed increment approach as the way to proceed (e.g., “Add 20% duration, up to 30 minutes” ; again, see p. 38).
The NICE guideline actually defines GET as follows:
In this guideline, graded exercise therapy is defined as first establishing an individual’s baseline of achievable exercise or physical activity, then making fixed incremental increases in the time spent being physically active.
That seems to capture the PACE Trial guidance quite accurately.
We know that GET produces symptom exacerbation in many ME/CFS patients. As such, it is worrying that the PACE manual explicitly directs practitioners to get patients to persist with GET even if they experience symptom “setbacks” or “crashes.” The manual states that the purpose of GET during a crash is to “maintain exercise as much as possible.” Practitioners are directed to warn patients about “the many negative consequences of rest” (p. 51).
“Some participants may be resistant to this approach,” the manual warns. If a patient is especially uncooperative, the practitioner is advised to try telling them that “hurt does not equal harm.”
So much for “collaborative” and “symptom dependent” then.
In summary, NICE did not misrepresent GET when it referred to “fixed incremental increases.” By contrast, the authors are being somewhat loose with their terms when they describe GET as “collaborative, negotiated, and symptom dependent.”
Fallacy
It seems the authors are trying to defend GET by claiming that it is something other than what they claimed it was before. Attempting to change the meaning of a widely accepted term after an objection is raised is a logical fallacy known as definitional retreat. It is a form of moving the goalposts, a class of equivocation that seeks to deflect legitimate criticism by unilaterally imposing new criteria on an ongoing discussion.
* * *
So-called ‘Error’ #7:
“[NICE] deviated from NICE recommendations of rehabilitation for related conditions, such as chronic primary pain”
Problem
It is true that the NICE recommendations for ME/CFS are not the same as those for chronic primary pain. However, it is not true that they should be expected to be the same.
Whatever one might say about the degree of relatedness that exists between two conditions, it is self-evident that separate conditions — by virtue of being separate — will typically require distinctive treatment recommendations.
What works for condition A will not necessarily work for condition B. This, after all, is why conditions have different names, different diagnostic criteria, and different NICE guidelines.
Fallacy
Arguing that two separate constructs must be treated the same way because they share one or more common property is known as an association fallacy. It involves a breach of formal syllogistic logic because it suggests that when A and B are the same in some important respects, it means they are the same in all important respects. However, some is not the same as all.
* * *
So-called ‘Error’ #8:
“[NICE] recommended an energy management approach in the absence of supportive research evidence”
Problem
There is considerable irony in the authors’ claim that NICE have recommended a practice for which there is poor (or no) research evidence.
Part of the irony is the authors’ own persistence in referring to both GET and CBT as “evidence-based” treatments for ME/CFS, even though that 356-page NICE research review showed they are not evidence-based at all.
The NICE guideline describes energy management as follows:
A self-management strategy that involves a person with ME/CFS managing their activities to stay within their energy limit, with support from a healthcare professional.
The “approach” that NICE recommends is based on the precautionary principle, and is intended to ensure that patients are not encouraged to continue with activity if their symptoms worsen (contrary to, say, the above-mentioned PACE guidance on GET). It is by no means presented as a “treatment” for ME/CFS, for which “evidence” might be required. In fact, the NICE guideline obliges practitioners to explain to patients that energy management “is not curative” (p. 30).
On a wider point, it is misleading to characterise NICE as being somehow ambivalent about the need for evidence bases. The NICE guideline explicitly points out that there is “a lack of effective evidence” on energy management approaches (p. 77), and highlights the need for research on self-monitoring management strategies to be prioritised (p. 62).
So yes, NICE recommended an energy management approach in the absence of research evidence for that approach. Such a statement might be true, but it is also banal. This is because: (a) NICE recommended this form of energy management out of an abundance of caution; (b) it pointed out that research evidence was lacking; and (c) it did not claim in any way that energy management was an evidence-based cure for ME/CFS (in fact, it stated the opposite).
This hardly seems like much of an “error” by NICE.
Fallacy
It would appear that this claimed “error” is intended to reveal NICE’s hypocricy in rejecting GET and CBT for lacking evidence. However, such reasoning smacks of a familiar fallacy known as “Two wrongs don’t make a right.” In philosophy, this is classed as a fallacy of relevance. Anyone who has ever tried to break up a fight among four-year-olds will understand why this reasoning is askew.
As well as being fallacious, such reasoning poses the following problem for the arguer: if the arguer wants the target of their criticism to be condemned for their wrongdoing, then this suggests the arguer must also be aware that their own behaviour is similarly wrong and so open to similar condemnation.
As such, if we are to condemn NICE for promoting unevidenced interventions, then surely we must also condemn those who promote GET or CBT (seeing as GET and CBT are also unevidenced).
But of course the authors will simply respond that GET and CBT are evidence-based after all, despite the fact that the research shows otherwise. This insistence on relentlessly repeating a point regardless of contradition or refutation is yet another logical fallacy, known as proof by repeated assertion. It is commonly employed in political rhetoric: for example, it is a key feature of Trumpism.
It is also an essential component of brainwashing.
And of gaslighting.
Who knew, right?
* * *
As you can see, the “eight-so-called-major-errors” paper is not quite the exposé that it intends to be. Rather than revealing something rotten in the heart of NICE and its ME/CFS guideline, it instead serves to highlight the utter weakness in the position of those denialists who are being left behind by regulatory change.
The authors of the piece are listed in this Tweet. There are 48 of them. Seeing as this is intended to be a formal academic review paper, and not a scientific study for which all the named people gathered primary data, we must assume that every single one of these esteemed individuals contributed intellectually to the content of the manuscript.
For example, we must assume that they did not just agree to have their name listed on the byline as if they were signing a petition or an open letter. After all, the journal in which they plan to publish their paper “adheres to the highest standards concerning its editorial policies on publication ethics,” which would make any use of ‘honorary’ (i.e., unearned) authorship a complete no-no.
Also, I wonder how many of these authors have provided CBT or GET services to ME/CFS patients in the course of their (paid) work, or who regularly earn royalties, consultancy fees, or other income relating to their promotion of these treatments? There is nothing wrong with people in such a situation authoring papers, of course. I just suspect the journal will need to set aside at least a couple of pages for the hefty conflicts-of-interest declaration that will be required in this case.
But then again, given the logical incoherence of the arguments mounted by these authors, and the consequent vapidity of their claim that NICE made eight (or even any) “major errors”, by rights this paper should never be accepted for publication in any journal that values its academic reputation.
Based on even the snippets we have seen thus far, the peer review process should see this paper rejected by the journal in double-quick time.
Of course, I say “should” see this paper rejected.
Because, then, that’s another problem, isn’t it?
The post Eight (or more) logical fallacies in that paper bemoaning the new NICE guideline for ME/CFS appeared first on The Science Bit.
