Those Who Dismiss Healthcare (and Healthcare IT) Adverse Events Reports as Mere "Anecdotes" Are Losers - Supreme Court-Style

At my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture" I wrote:

... It's the EMR "anecdotalists" (as opposed to the "Markopolists") who say that "anecdotes" of HIT-related injury are meaningless. They deem reports of safety issues and HIT-related misadventures and risk as simply "anecdotal", and that "anecdotes don't make evidence" (or "anecdotes don't make data").

For "anecdotes" of patient harm due to medical devices even from the most reliable of sources to be counted as "evidence" of device risk, apparently, the stories need to be blessed with Statistical Holy Water. The Holy Water must also be of a brand approved by the academic pundits.

For me, this is no longer merely a professional debate. My elderly mother became one of those "anecdotes" in May last year.

I address the casual, Dogbert-style, waving-of-the-hand "Bah!" dismissal of health IT harm "anecdotes" at numerous other posts as well, such as "EHR Problems? No, They're Merely Anecdotal" and "Health IT: On Anecdotalism and Totalitarianism".

Bah! Your Health IT adverse events reports are anecdotes, and anecdotes don't make data!

In those posts I also mention how Australian informatics professor Dr. Jon Patrick had essentially hit the flaws of this argument out of the Southern hemisphere with a short editorial in the journal "Applied Clinical Informatics" entitled "The Validity of Personal Experiences in Evaluating HIT." That essay is free at the link and is worth reading.

Interestingly and thankfully, the "anecdotes are meaningless" crowd have now lost, and lost big - Supreme Court style. In fact, the U.S. Supreme Court has shown far more common sense than many esteemed academics and industry pundits.

As noted in this post at Derek Lowe's pharmaceutical industry "In the Pipeline" blog, the company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.

The Supreme Court would have none of that argument:

"Matrixx�s [Zicam's manufacturer - ed.] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.

The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:

... We conclude that the materiality of adverse event reports cannot be reduced to a bright-line rule ... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.

This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.

In essence, it is saying that adverse events reports, especially repeated ones, from trustworthy sources are not to be lightly dismissed, but should serve at the very least as red flags that there may be a systemic problem requiring further investigation.

One wonders how and if public healthcare IT vendors will begin disclosing "anecdotal" reports of their products causing patient harm to their own stockholders.

One also wonders if the academic anecdotalists (up to the level of the chair of the Office of the National Coordinator for Health Information Technology at HHS) will cease their unfettered dismissal of health IT AE reports as mere "anecdotes" and therefore let's roll out this 100% beneficent technology nationwide ASAP, e.g., as at my Feb. 2011 post "A Somewhat Harsh Farewell to David Blumenthal of ONC, From a Patient Injured by Health IT - My Mother":

"Nothing [ONC has] found would give them any pause that a policy of introducing EMR's could impede patient safety." - David Blumenthal

That sounds a bit like the refrain of the makers of Zicam.

One might also wonder if the anecdotalists merely lack common sense, or are using this form of epistemological dementia to obscure conflict of interest.

On a final note, my favorite comment at the aforementioned "In the Pipeline" blog story is this by anonymous commenter "Still Scared of Dinosaurs":

One of the most important ideas real statisticians must get into their heads is "Thou shalt not worship the 0.05 threshold". The whole concept of "statistical significance" for AEs is idiotic and the fact that Matrixx based any part of their defense on it indicates that their stupidity did not end when they named the company.

Perhaps this Dilbert cartoon is apropos to the Supreme Court decision:


-SS

Addendum:

I thought it appropriate to share these thoughts with the leadership of the Joint Commission, the organization that accredits healthcare organizations in the United States:

From: Scot Silverstein
Sent: Sunday, March 27, 2011 10:41 AM
To: MGiuntoli, Anita; Chassin, Mark; Schyve, Paul; Legaspi, Shirley
Cc: Ross Koppel; 'David Kreda'
Subject: Re: MATRIXX INITIATIVES, INC., ET AL. v. SIRACUSANO ET AL.

Not a complaint this time [about health IT failure - ed.], but an observation.

The JC has noted health IT risks in the Sentinel Events Alert "Safely implementing health information and converging technologies" of 2008.

The company that made "Zicam", a zinc-based over-the-counter cold remedy, tried to defend shareholder suits that the company withheld case reports of Zicam causing permanent loss of smell via arguing that such reports "did not reach a level of statistical significance", i.e., were "anecdotal." The case went to the U.S. Supreme Court.

The Supreme Court would have none of that argument:

"Matrixx�s [Zicam's manufacturer] premise that statistical significance is the only reliable indication of causation is flawed. Both medical experts and the Food and Drug Administration rely on evidence other than statistically significant data to establish an inference of causation. It thus stands to reason that reasonable investors would act on such evidence.

The full court decision is at this link: http://www.supremecourt.gov/opinions/10pdf/09-1156.pdf (PDF file), but a passage I consider key to this issue is as follows:

... Because adverse reports can take many forms, assessing their materiality is a fact-specific inquiry, requiring consideration of their source, content, and context.

This is common sense incarnate. It applies not just to drugs, but to medical devices, to health IT, and to other domains as well.

I believe JC should start to pay serious attention to "anecdotal reports" of health IT-caused patient injury, and consider reliable reporting of these events as an Accreditation standard.

As I noted in my July 2009 JAMA letter to the editor "Health Care Information Technology, Hospital Responsibilities, and Joint Commission Standards" in response to Koppel and Kreda's JAMA article on HIT industry practices, "hold harmless" and "gag" clauses must go, and be replaced with proactive reporting of healthcare IT-related "events."

Scot Silverstein

-- SS

Read More

Australian ED EHR Study: Putting the Lie to the Line "Your Evidence Is Anecdotal, Thus Worthless" Used by Eggheads, Fools and Gonifs

At my Sept. 2010 post "The Dangers of Critical Thinking in A Politicized, Irrational Culture" I lamented that while my early mentor in biomedicine Victor P. Satinsky MD taught the wise credo "critical thinking always, or your patient's dead" in the 1970's, our culture had become so perverse that this credo had been largely supplanted with:

"Critical thinking anytime, and your career's dead."

That post was in reaction to continued heckling on a professional mailing list, the American Medical Informatics Association (AMIA) Clinical Information Systems Working Group (cis-wg). On this list, Oregon Health Sciences University professor William Hersh expounded on how the evidence of health IT dangers was largely "anecdotal" therefore to be discounted, and how I, specifically, "didn't know the literature on health IT."

In that post I put the lie to the latter figment. On the former prevarication Dr. Jon Patrick, author of the recent thorough dissection of problems with the ED EHR system being rolled out in public hospitals in New South Wales, Australia (see my Mar. 5, 2011 post "On an EMR Forensic Evaluation by Professor Jon Patrick from Down Under: More Thoughts"), hit the ball out of the park:

Prof. Patrick to AMIA cis-wg:

I think such defences are particularly unuseful especially with respect to the dismissal of personal stories and experiences as "anecdotes", hence committing them to the realm of folklore. I offer these notions as a counterpoint.

Discounting Anecdotes:

1. Is a perfidious and specious act.

2. It denies early warning signs of problems.

3. It denies a voice and disempowers the working clinical community who have to operationalise decisions made by others.

4. It denies a route to process improvement within an institution - which is most important for EBM and incremental review of local processes.

5. It defends software manufacturers from fault rectification - cuts off even a need to deliberate on it. Critics of the value of anecdotes are squarely on the side of the faulty and deficient manufacturer.

6. A rule of project management is that projects consist of 3 components, cost, quality and time and if their needs to be a compromise it has to be on quality. Anecdotes are early warning signs of such a compromise.

Prof. Patrick had to once again put the lie to this refrain at a comment on the HISTalk blog yesterday, where the anonymous proprietor had written in a review of Patrick's EHR study:

... On the other hand, I wouldn�t say it�s [Prof. Patrick's Cerner FirstNet study] necessarily unbiased, it focuses on implementation of a single department application that didn�t go well for a variety of reasons (despite many successful FirstNet implementations elsewhere), it uses the unchallenged anecdotal comments of unhappy users who make it clear they liked their previous EDIS better, and it nitpicks (I wasn�t moved to find a pitchfork when I learned that the primary keys in the Millennium database aren�t named consistently).

[Considering the complexity and changeability of healthcare and the corresponding software lifecycle, I duly note that that latter attitude about 'failing to find pitchforks' regarding breaches of sound software engineering practices seems to be a symptom of the larger health IT disease that Prof. Patrick writes about - ed.]

Dr. Patrick then knocked the ball out of the Southern Hemisphere with a comment of his own about anecdotalism and a link to an expansion of the aforementioned ideas he'd shared on "discounting anecdotes":

Prof. Patrick to HISTalk owner:

Your Comment �it uses the unchallenged anecdotal comments of unhappy users � is not only unfair but unreasonably inaccurate. The comments made by the users are the Directors of 7 EDs and so they have a right to carry authority by virtue of the experience but also the number of 6 out 7 presenting a view of Firstnet as unfit for purpose has numeric validity, which they justify with about 20+ pages of their comments � see Part 2 Appendix 2.

I would also point readers to my editorial about the role of personal experiences being the most useful information to understand the nature of socio-technical failures. http://aci.schattauer.de/en/contents/archive/issue/1124/manuscript/15463/show.html

The essay at that link, "The Validity of Personal Experiences in Evaluating HIT", is an editorial in Johns Hopkins informaticist Chris Lehmann's brilliant new journal "Applied Clinical Informatics."

The editorial is available free, and is a must-read for anyone in a decision-making or managerial role in mission critical domains, including our elected representatives.

In the editorial Dr. Patrick concludes. similarly to his earlier AMIA mailing list opinion:

... the denial of recounted personal experiences in discussion and analysis of HIT is biased and specious and has the effect of:

1. Denying early warning signs of problems.

2. Denying a voice for the working clinical community who have to operationalise decisions made by others and thus disempowers them.

3. Denying process improvement within an institution � which is most important for Evidence Based Medicine and incremental review of local processes.

4. Discourages staff from engaging in any form of process improvement hence worsening the sense of disenchantment.

Every legitimate personal experience of a HIT deserves to be considered on its merits lest we wish to retreat from process and product improvement. Mechanisms of censorship both implicit due to contrived processes of disinformation and disempowerment or explicit due to contractual specifications will lead to more waste, lost productivity, contempt for the providers, and distress among frontline staff rather than increased productivity and improved patient health and safety as we all desire.

In my view, the drivers or motivators for the "anecdotalist" accusation are these, singly or in combination:

• Too much "education" to see the nose on one's face, as in, to think zebras and unicorns instead of horses when hearing hoofbeats outside one's midwest U.S. abode (eggheads);
  
• Too little common sense (fools), as in Scott Adams' example: "IGNORING ALL ANECDOTAL EVIDENCE - Example: I always get hives immediately after eating strawberries. But without a scientifically controlled experiment, it�s not reliable data. So I continue to eat strawberries every day, since I can�t tell if they cause hives";
  
• Too much concern for the possible interruption of flow of money or power in one's direction (gonifs).
  

In conclusion, the anecdotalist refrain of "your evidence is anecdotal" [therefore of little or no value] when used repetitively against competent observers is the refrain of eggheads, fools and gonifs.

In healthcare, the end result is "your patient's dead."

As at my Mar. 2011 post "Hospital: "While We're the Bee's Knees in IT, We Aren't Perfect And We Are Always Willing To Look In The [Smashed Up, Rear-View] Mirror", my "anecdotal mother" is sadly an example.

As for myself, I am a Markopolist (see my Sept. 2010 post "Health IT: On Anecdotalism and Totalitarianism").

-- SS

Read More

EHR Problems? No, They're Merely Anecdotal; the Truth Must Be That I Attract Bad Electrons and Stale Bits

My mother, who suffered an iatrogenic cerebral hemorrhage in May contributed to by an EMR's interference with clinicians, fell the other day in the bathroom.

Her fall was hard; she struck her back and knocked out one of the mounting posts for the bathroom tissue - completely out of the wall, the wallboard now with a large gaping hole in it.

In an elderly person, falls can result in injuries such as this one, a painful hematoma on her back.

She went to a local hospital where x-rays were done; aside from a large bruise and hematoma (collection of blood under the skin) over her back at the point of impact, miraculously nothing was broken.

At triage I went over her medication list in great detail, ensuring both the data input and the triage printout record of her meds were complete and precise. She went home.

The very next day, in mid afternoon she had sudden onset of speech difficulty (expressive aphasia) and right sided weakness, symptoms of loss of blood flow to the left side of the brain, while sitting in a chair talking on the phone. (The aphasia was the same symptom that led to her May 2010 presentation and travails, resulting in left carotid stenting, but with accidental cessation of a critical medication and severe complications, domino-style.)

So into the hospital she went via ambulance again. The ambulance crew copied her meds off a list I keep on my mother's refrigerator onto a scrap of paper. In the hospital the ED nurse reviewed the meds with me from the scrap, but I informed the ED nurse that doing so was not necessary since I'd just carefully checked the ED EMR med list at triage less than 24 hours earlier at the triage station.

The ED nurse then replied - "we're not using the ED EHR med lists right now, the system's been 'glitchy' today."

Me - " 'Glitchy?' What does that mean?"

ED nurse - "Sometimes the EHR pulls up the patient's meds, sometimes it doesn't." (direct quote).

Needless to say, these were not exactly words I enjoyed hearing.

My mother was having a repeat of the ischemia to the brain or "TIA" (transient ischemic attack, i.e., threatening to have a stroke), only this time the ED EHR itself was also having a TIA.

In this progressive "paperless" setting, I was the sole conduit of accurate information about her meds. However, not every elderly patient has an advocate with my background...

My mother's TIA symptoms improved somewhat and she went to ICU, and was set up for a slew of tests to see what should be done, but these "every time I enter a hospital" EHR problems are getting a bit beyond what I consider as mere personal bad luck.

She was then transferred to a tertiary care hospital's critical care floor for neurological problems. Before transfer, I asked to see the results of her neck and brain scans.

A doctor brought them up on the computer screen, but rapidly scrolled down to the impression section. The doctor hoped I didn't see what was at the top of the radiological report. But I did. I saw a statement like this:

"A duplicate medical record number, previously unknown, was discovered for this patient."

Out of exasperation, I did not raise a commotion, but I can only wonder what data might have gone into that "previously unknown" silo.

After transfer to the tertiary hospital, the commercial EHR on a cart on wheels ("COW") outside my mother's room was displaying the EHR main screen, with a "patient worklist" window also open in the screen's center.

(The GUI appeared, by the way, to be that of obsolete Windows 2000 or NT 4.0, although possibly it could have been XP set to display the older GUI appearance, but the icon appearance suggested the former possibility).

Superimposed over the central patient worklist window, though, was a dreaded Microsoft crash window, exactly like this one from the Web:

An error window like this was superimposed on the EMR screens being used to manage my mother's care. Click to enlarge.

In asking the RN about this, I was told this window popped up a lot, and was simply dismissed by users with one of the two buttons. The IT dept. had told clinical staff the problem was due to users "loading illegal software on the hospital computers." (This COW, incidentally, lacked any portals for thumb drives, floppies, etc.) It sounded like IT would fix it when they managed to get around to it. The nurses went about their business, ignoring this screen when it popped up unpredictably but regularly.

Somehow, this did not inspire within me great confidence in the integrity of that EHR and its data.

As an aside, I remember hearing a story like this over ten years ago in a past life as CMIO of a large hospital, in the Cath Lab as I detailed here:

... The informaticist [a.k.a. me - ed.] first asked to see what had been installed in the cath lab by MIS. The informaticist found workstations running the application under Windows 3.1, an unreliable platform especially unsuited for critical care environments, because "Windows NT and other OS's such as UNIX were not supported by MIS." When shown a short demo of data entry by a nurse after a cardiac cath case, the workstation crashed, displayed a "general protection fault" error and hexadecimal debugging data. It had to be rebooted, with resultant time and data loss.

The informaticist asked the nurse about the crash and was told it happened frequently, up to several times per day per workstation. When the informaticist asked if MIS had requested a detailed log be kept of the crashes and error messages to help resolve the problem, the answer was no. MIS felt diagnosis and repair was the vendor's responsibility. When the informaticist asked the nurse exactly what had been explained to clinicians about the crashes, the nurse replied that cath lab staff had been told by MIS "don't worry about it, you can't understand it, we'll make it better."

The informaticist remembered, from medical school and residency, being told never to say such a thing to patients as it was considered inappropriate and too paternalistic in the modern age of medicine, especially with the elderly. This was an ironic and somewhat perverse scenario for a critical care area, the informaticist thought.

I find the repeat of a story like this simply stunning.

As probably 2/3 of my healthcare-worker students have related stories of EHR-induced clinical problems in their organizations in the past several courses I've taught [typical examples of student stories are at this link], and other mentees with worse tales in their CMIO roles, and now with my own experiences getting more and more theatre-of-the-absurdish, I offer this thought:

Perhaps my experiences are merely anecdotal, due to some bad karma that causes me to attract bad electrons and stale bits, disrupting EMR operations.

Yes, that must be it.

-- SS

Read More