Make your live is better

Make your live is better.

Your Fammily is Your live

Your Fammily is Your live.

Care your future

Be healty .

This is default featured post 4 title

Go to Blogger edit html and find these sentences.Now replace these sentences with your own descriptions.

This is default featured post 5 title

Go to Blogger edit html and find these sentences.Now replace these sentences with your own descriptions.

Showing posts with label Healthcare IT. Show all posts
Showing posts with label Healthcare IT. Show all posts

Tuesday, March 15, 2011

How Lousy Health IT (And Other Healthcare Refuse) Gets Pushed on Na�ve and Malleable Physicians, Hospital Executives and Governments

Over at "Hooked: Ethics, Medicine, and Pharma", University of Texas Medical Branch at Galveston (UTMB) Institute for the Medical Humanities professor Howard Brody MD, PhD offers a fascinating piece "How Does the Drug Industry Exert Power? An Anthropological Perspective."

The piece is based on writings of of Kalman Applbaum, of the Department of Anthropology, University of Wisconsin-Milwaukee, and summarizes how the drug industry achieves its aims in today's society.

In Prof. Brody's post:

... Applbaum begins by teaching us marketing neophytes about "channels." Marketing drugs is highly complicated because there's no direct pipeline from the manufacturer to the end-user. Lots of folks control and manipulate the pathway that delivers the drug eventually to the mouth or body of the patient--most notably the physician who has to write the prescription, but also insurance companies, government regulators, managers of hospital formularies, and numerous other players.

The key to selling drugs successfully today is first, to be able to control the channel:

"Pharmaceutical manufacturers, like other marketing-driven enterprises, have realized that it is less in the product, the brand or even the patent where their fortunes lie, but in the stream, the marketing channel.

Once you control the channel, you can insert any product you like into it, no matter how useless or dangerous." [Such as grossly unfit for purpose, clinically disruptive, mission hostile healthcare IT - ed.]

But, if Rule One is control the channel, Rule Two is not to appear obviously to do so.

Read the entire post at the above link.

The observation that "once you control the channel, you can insert any product you like into it, no matter how useless or dangerous" explains many observations about the healthcare IT industry as well, such as how reports from the U.S. National Research Council and many others (e.g., here, here, and here) can simply be ignored.

In the health IT industry, that "channel" is controlled, right up to the highest levels of numerous governments.

Observations such at my April 2010 post "Healthcare IT Corporate Ethics 101: A Strategy for Cerner Corporation to Address the HIT Stimulus Plan";

at my post just a few days later on the "disappearance" of materials cited in my above expos�, "Cerner - Fuqua School of Business 'Corporate Ethics 101' Paper and Website Disappear";

and at my Jan. 2010 post "More on Perversity in the Healthcare IT World: Is Meditech Employing Sockpuppets?" (where I exposed a likely industry shill through IP forensics), among others, can be placed better into context as components of "controlling the channel" for health IT, and not appearing obvious in doing it.

Physicians especially need to re-learn the mantra "Critical thinking always, or your patient's dead" of my late mentor, cardiothoracic surgery pioneer Victor Satinsky MD.

Medical schools also should be developing and teaching courses in how the healthcare industries attempt to "control the channel."

These issues go far beyond the mere "tension management" issues of forbidding students, residents and attending physicians from accepting coffee cups, lunches, stethoscopes or other gifts of nominal value from the industry.

-- SS

Monday, March 14, 2011

AMA, Deluded on Health IT, Begs For Doctor Penalty Extension - Not Penalty Termination

The American Medical Association (AMA) has become worse than useless:

AMA urges issuance of EMR penalties be delayed

Congressional Quarterly's CQ Weekly (3/14, Zeller) reports that "Congress strongly believes that electronic medical records will bring down the cost of healthcare, so much so that it has passed two laws providing incentives for doctors who upgrade their systems and levying penalties on those who don't."

But, the American Medical Association is urging that the penalties be delayed. Notably, the association "says a Government Accountability Office study released in February bolsters its point that the two laws create competing incentives that contradict each other." [Just our sub-15% approval rating Congress at work - ed.]

The article adds that the Department of Health and Human Services "announced that doctors will face penalties of 1 percent of their Medicare fees if they don't start issuing electronic prescriptions this year," which the AMA says would be a waste because physicians would buy e-prescribing software that they would discard when they install records systems in 2012.

[Lest they face other Medicare penalties for non-"meaningful use" of EMR's - ed.]

Aside from the Congressional mayhem created by conflicts between e-Prescribing and EMR coercion laws, if the AMA were not completely useless, and truly represented physicians (and patients), they'd be pushing not for a delay in penalties, but for their abolition.

This would help return HIT to a sensible voluntary path from the path of irrational exuberance it's now on.

-- SS

Friday, March 11, 2011

Is Microsoft slowly edging towards an "exit stage left" in health IT?

Interesting item seen here:

From Dabney: �Re: former Sentillion exec departures from Microsoft. Microsoft transferred their 800 Health Solutions Group people into the small-to-medium commercial sector group (Microsoft Business Solutions) last Monday. Peter Neupert and his whole organization have been pushed out of the incubation group in Microsoft Research with the guys who sell Microsoft Axapta ERP and CRM for small commercial customers. That will mark the end of acquisitions and spending of Microsoft on health because they haven�t had any significant sales of Amalga UIS in the past year after already withdrawing Amalga HIS and Amalga RIS/PACS from the market. Microsoft is slowly edging towards an exit stage left in health IT.

Why would this not surprise me if true?

Because I predicted it.

In July 2006, nearly 5 years ago, in my July 2006 post "Bill, Have You Lost Your Mind?"

Nobody was listening, just as nobody seems to be listening to my current dire predictions for the National Program for IT in the HHS� .

Wait until 2016...

-- SS

Wednesday, March 9, 2011

ONC: "The Benefits Of Health Information Technology: A Review Of The Recent Literature Shows Predominantly Positive Results"

The Office of the National Coordinator for Health IT in the US (ONC) has just published an article in "Health Affairs" entitled "The Benefits Of Health Information Technology: A Review Of The Recent Literature Shows Predominantly Positive Results", Buntin, Burke, Hoaglin and Blumenthal, Health Affairs, 30, no.3 (2011):464-471, doi: 10.1377/hlthaff.2011.0178.

It is available at the hyperlink above, but may not be publicly accessible.

The authors all are, or were, ONC officials:

Melinda Beeuwkes Buntin (Melinda.buntin@hhs.gov) is director of the Office of Economic Analysis, Evaluation, and Modeling, Office of the National Coordinator for Health Information Technology (ONC), Department of Health and Human Services, in Washington, D.C. Matthew F. Burke is a policy analyst at the ONC. Michael C. Hoaglin is a former policy analyst at the ONC. David Blumenthal is the national coordinator for health information technology.

The abstract is as follows:

ABSTRACT
An unprecedented [indeed- ed.] federal effort is under way to boost [coerce? - ed.] the adoption of electronic health records and spur innovation in health care delivery.We reviewed the recent literature on health information technology to determine its effect on outcomes, including quality, efficiency, and provider satisfaction.We found that 92 percent of the recent articles on health information technology reached conclusions that were positive overall. We also found that the benefits of the technology are beginning to emerge in smaller practices and organizations, as well as in large organizations that were early adopters. However, dissatisfaction with electronic health records among some providers remains a problem and a barrier to achieving the potential of health information technology. [Some? That sounds like an understatement - ed.] These realities highlight the need for studies that document the challenging aspects of implementing health information technology more specifically and how these challenges might be addressed.

I have long stated, at least since 1999, that:

Healthcare information technology (HIT) holds great promise towards improving healthcare quality, safety and costs.

The new ONC review article is certainly pointing in this direction. Perhaps Health Affairs will release it to general circulation.

However, I also wrote:

As we enter the second decade of the 21st century, however, this potential has been largely unrealized. Significant factors impeding HIT achievement have been false assumptions concerning the challenges presented by this still-experimental technology, and underestimations of the expertise essential to achieve the potential benefits of HIT. This often results in clinician-unfriendly HIT design, and HIT leaders and stakeholders operating outside (often far outside) the boundaries of their professional competencies. Until these issues are acknowledged and corrected, HIT efforts will unnecessarily over-utilize precious healthcare resources, will be unlikely to achieve claimed benefits for many years to come, and may actually cause harm.

Whether the new ONC article demonstrates that these issues are starting to approach resolution, or is just another opinion paper not fully supported by facts, is not certain.

Two charts are presented that summarize the findings (click to enlarge):


(Click to enlarge.) Evaluations Of Outcome Measures Of Health Information Technology, By Type And Rating



(Click to enlarge.) Health Information Technology: Study Design And Scope Factors Associated With Positive Overall Conclusions

There are several caveats. The first has to do with possible selection bias that can be present in any review article.

On article selection for the review:

... we decided that to be included in this review, an article had to address a relevant aspect of health IT, as listed in the Appendix [7]; examine the use of health information technology in clinical practice; and measure qualitative or quantitative outcomes. Analyses that forecast the effects of a health IT component were included only if they were based on effects experienced during actual use. ... Using this framework, the review team removed 2,692 articles based on their titles. An additional 1,270 articles were determined to be outside the study�s scope after the team examined the article abstracts. For example, 269 abstracts focused solely on health IT adoption.

[What, exactly, does that mean? Were the problems with HIT adoption potentially significant towards causation of lack of benefit, and/or presence of harm, for instance? - ed]

By the third review stage, the review team had 231 articles. An additional forty-three were excluded after further review because they did not meet the criteria, and thirty-four review articles were dropped from the analyses because they did not present new work.

[What does that mean? Did they drop highly comprehensive articles showing uncertainty in the literature about HIT, such as Greenhalgh's "Tensions and Paradoxes in Electronic Patient Record Research: A Systematic Literature Review Using the Meta-narrative Method" from University College London? That article appeared in the Dec. 2009 Milbank Quarterly. I wrote about it at this link.]

This left 154 studies that met our inclusion criteria, 100 of which were conducted in the United States. This is comparable to the 182 studies found over a slightly longer time period that were evaluated by Goldzweig and colleagues.

I should also note that no mention is made of independent reviewers of the article corpus and elimination process. It appears the entire effort was conducted within ONC itself, where a bias towards finding positive results is likely present (and understandable).

Another caveat is the the Health Affairs/ONC article appears to bypass a body of literature, both peer-reviewed and non-peer reviewed, that sheds doubt on health IT in its present form from a number of angles such as I recently aggregated at "An updated reading list on HIT" and at "2009 a pivotal year in healthcare IT." Bypassing literature such as this is a possible major weakness.

Further, in the current political environment, it is not hard to imagine that articles highly critical of health IT, or revealing major mishaps and possibly exposing organizations to litigation, are scarce.

I addressed that in a paper "Remediating an Unintended Consequence of Healthcare IT: A Dearth of Data on Unintended Consequences of Healthcare IT." The paper itself was initially found needing revisions, largely in format, by a small group of blinded reviewers in the Medical Informatics community (with the exception of one faux-newshound who commented that "material like this could be read in any major newspaper", a rather perplexing comment considering the topic). Rather than revise, and not being on the tenure treadmill, I chose instead to publish at the Scribd link above.

The ONC paper does acknowledge this:

A recent study found that for clinical trials, studies with positive results are roughly four times more likely to be published than those without positive findings. Because the articles were limited to health IT adopters, we anticipated that authors more often approached studies looking for benefits rather than adverse effects.

They do, however, then issue a value judgment:

It is important to note that although publication bias may lead to an underestimation of the trade-offs associated with health IT, the benefits found in the published articles are real.

I"m not sure my mother would find that value judgment heartening. She suffered a crippling brain hemorrhage that would not have occurred if paper had been used in the ED rather than an EHR. (I say this from personal experience; I once worked in that ED, pre-EHR.)

I note that if a pharmaceutical company were to issue such a value judgment about a drug as justification for national marketing, they'd likely be nailed to the cross...

The ONC paper also ignores accounts of "near misses" and actual patient injury from impeccable sources, such as in "Health informatics � Guidance on the management of clinical risk relating to the deployment and use of health software (PDF). UK National Health Service, DSCN18 (2009), formerly ISO/TR 29322:2008(E)":

Annex A

Examples of potential harm presented by health software

GP prescribing decision support
In 2004 the four most commonly used primary care systems were subjected to eighteen, potentially serious, realistic scenarios including an aspirin prescription for an eight year old, penicillin for a patient with penicillin allergy and a combined oral contraceptive for a patient with a history of deep vein thrombosis. Using dummy records, all eighteen scenarios failed to produce appropriate alerts by all of the systems, most of the time. The best score was a system that flagged up seven appropriate alerts. The health organization clearly has, in such a system deployment, a key responsibility to ensure that knowledge bases used within a design are correctly populated and aligned with clinical practice within their organization.

Inadvertent accidental prescribing of dangerous drugs (such as methotrexate)
This incident occurred when a user of a primary care system attempted to issue two repeat items. The items were highlighted and instead of the issue selected repeats button, the prescribe acute issues from the formulary button was pressed. This brought up the formulary dialogue which contained the high risk items. Either the issue button was then pressed or the particular items were double clicked. When the warning messages came up, they were all ignored and proceed and issue selected. The user chose the first item presented on the formulary list, which just so happened to be a methotrexate injection. In this particular case, it was determined that patient risk was minimal as the treatment was rarely used in primary care and would, in practice, be rejected by the pharmacist. To preclude any recurrence of the problem, access to the high risk formulary was removed from the formulary part of the acute drug issue dialogue. This example again demonstrates the need to align clinical practice and authority levels with the knowledge and rule bases within the system. Wherever possible, design and implementation of health software systems should be undertaken to improve control and accuracy, note introduce new exposures. Furthermore the hazard and risk assessment of this situation may well not apply in other settings, e.g. prescription issue by nursing staff on a ward versus a pharmacist in a retail store.

Incorrect patient details retrieved from radiology information system
This incident arose from the fact that medical reference numbers (MRNs) are usually prefixed by an alpha code. Some hospitals however do not use these prefixes and identical MRNs can be generated. This gave rise to the creation of shared MRNs and subsequent confusion of records in the central datastore when retrieval key is the Medical Record Number. Four specific instances were found where a patient number had been entered in the radiology information system and incorrect patient details had been retrieved. The manufacturer could have built in an appropriate format check during development. Alternatively, the problem could have been spotted by the health organization if a structured risk assessment had been undertaken.

Drug mapping error
Sodium valproate 200 mg slow release was incorrectly mapped to sodium valproate 200 mg in a formulary encoded into a health software system. These are anti-epilepsy drugs and thus the implications for patient safety could be significant. This particular incident is just one of many that have been reported in relation to drug mapping.

An initial investigation indicated that 35 prescriptions had been generated using the incorrect map. Corrective action included contacting the relevant primary care practices to check upon patient health and the supplier to correct the mapping process to ensure no further incorrect prescriptions were generated. As before, this was a design/coding error by the manufacturer but was compounded by the health organization not checking the mappings and failing either to build in appropriate prescribing controls, or map the controls to health organization individuals with the appropriate experience and authority.

Pre-natal screening
The ages of women who had undergone pre-natal screening were wrongly computed by a health software system. As a result 150 women were wrongly notified that they were at no risk. Of these, four gave birth to Down's syndrome babies and two others made belated decisions to have abortions.

Patient identification
A student died of meningitis because of a misspelling of her name and inadequacy in computer use. The student was admitted and a blood test proved negative for meningitis. The following day another blood test was taken and filed on a new computer entry but the letter ?p was missed in the spelling of the name. When a doctor looked up results they were presented with only the first negative tests result because of the misspelling. If the second result had been seen it would have triggered further investigations and probable diagnosis of meningitis. The investigating panel concluded that problems with the health software system had been greater than first thought and in this case there was a combination of a misspelled name and the doctor not being able to use the computer system property. The health software system could have been designed to use unique numbers either instead of the name or in addition to it.

They also missed consideration of serious IT defects reports as in the FDA's Maude database that I wrote about at my Jan. 2011 post "MAUDE and HIT Risks."

Another confounding factor is the issue of possible unreliability of the medical literature itself as expressed in a recent post on the IBM Watson supercomputer exuberance:

Consider the issue of the medical literature suffering from numerous conflict of interest and dishonesty-related phenomema making it increasingly untrustworthy, as pointed out by Roy Poses in a Dec. 2010 post "The Lancet Emphasizes the Threats to the Academic Medical Mission", at my Aug. 2009 post "Has Ghostwriting Infected The "Experts" With Tainted Knowledge, Creating Vectors for Further Spread and Mutation of the Scientific Knowledge Base?" and elsewhere on this blog.

They do state what I have been writing about for over a decade now:

... In fact, the stronger finding may be that the �human element� is critical to health IT implementation. The association between the assessment of provider satisfaction and negative findings is a strong one. This highlights the importance of strong leadership and staff �buy-in� [which will only occur if the systems are not miserable examples of poor engineering, not due to P.R. or the irrational exuberance of others - ed.] if systems are to successfully manage and see benefit from health information technology. The negative findings also highlight the need for studies that document the challenging aspects of implementing health IT more specifically and how these challenges might be addressed.

Dear ONC: see this site, and the many posts since 2004 on this blog.

I was surprised to see this:

... Taking a cue from the literature on continuous quality improvement, every negative finding can be a treasure if it yields information on how to improve implementation strategies and design better health information technologies.

As in my post at this link, does this mean that 'anecdotal' accounts of HIT problems will no longer be summarily dismissed? I wonder.

I have also written in the past that in order to truly understand a domain, one must look at both evidence of the upside, and evidence of the downside. My concern is that the latter was not well addressed in this paper. A beneficial technology with a significant downside, especially in medicine, is not ready for national rollout (cf. Vioxx).

In summary, the new ONC paper may present a glimmer of hope that health IT is starting to produce real results. On the other hand, its possible deficiencies and biases might also make it more a political statement than anything else. It will certainly be used as such from the high government perch of HHS regardless. This has already started:

... President Obama and Congress envisioned that the HITECH Act would provide benefits in the form of lower costs, better quality of care, and improved patient outcomes. This review of the recent literature on the effects of health information technology is reassuring: It indicates that the expansion of health IT in the health care system is worthwhile.

[Note the use of "is worthwhile" as opposed to "may be worthwhile", a continuation of the "it's proven, nothing else to say" style I noted
at my July 2010 post "Science or Politics? The New England Journal and "The 'Meaningful Use' Regulation for Electronic Health Records." Such statements of absolute certainty are of concern; they remind me of the global warming debate - ed.]

...Thus, with HITECH, providers have an unparalleled opportunity to accelerate their adoption of health information technology and realize benefits for their practices, institutions, patients, and the broader system. [Ditto - ed.]

Does the article truly show a breakthrough, or is it a flawed review by a governmental agency that will be used for political purposes? I simply do not know which.

I am certain, however, that there will be active debate and dissection of this paper and its source articles in the months to come by those with more time, resources, and expertise than I have at my disposal.

-- SS

March 10, 2011 addendum:

Trisha Greenhalgh at Barts and The London School of Medicine and Dentistry and the author of the aforementioned comprehensive review article "Tensions and Paradoxes in Electronic Patient Record Research: A Systematic Literature Review Using the Meta-narrative Method" (link to my essay), had this observation about the following passage in the ONC paper:

�Our findings must be qualified by two important limitations: the question of publication bias, and the fact that we implicitly gave equal weight to all studies regardless of study design or sample size.�

Prof. Greenhalgh relates: "Given these very fundamental acknowledged biases, I�m very surprised anyone published this paper in its present form.�

-- SS

March 14, 2011 addendum:

Dr. Roy Poses had this to say:

Dr Silverstein commented on a new review of research of health care information technology whose results were mainly optimistic. The authors were from the US government agency that promotes health care IT, so its optimism is not surprising. However, its credibility was unclear, since it did not appear to be systematic. [In general, one at least assesses and takes into account the methodologic quality of studies used in a systematic review, and does not "give equal weight to all studies regardless of study design or sample size" - ed.]

Furthermore, the authors had no methodologic standards whatsoever for article inclusion. [They did, sort of, as in my paragraphs "On article selection for the review:", but one might term the standards "loose" - ed.] The review included qualitative studies that were probably not meant to be evaluative, and observational studies subject to severe methodologic bias.

The publication of this review demonstrated how the conventional wisdom is continually reinforced based on the strength of the influence of its proponents, rather than the strength of the supporting evidence. Adaptation of new drugs and devices should be based on evidence that their benefits outweigh their harms, rather than the enthusiasm and financial interests of their proponents.

-- SS

Wednesday, March 2, 2011

Vladimir Putin and Common Sense on IT

Vladimir Putin may be known for showing off his pectorals riding horses bare chested, but he also seems to have a substantial amount of gray matter between the ears.

Our country, including the healthcare IT sector, could probably learn something from him:

A Walled Wide Web for Nervous Autocrats

Wall Street Journal
Jan. 8, 2011

By EVGENY MOROZOV

At the end of 2010, the "open-source" software movement, whose activists tend to be fringe academics and ponytailed computer geeks, found an unusual ally: the Russian government. Vladimir Putin signed a 20-page executive order requiring all public institutions in Russia to replace proprietary software, developed by companies like Microsoft and Adobe, with free open-source alternatives by 2015.

The move will save billions of dollars in licensing fees, but Mr. Putin's motives are not strictly economic. In all likelihood, his real fear is that Russia's growing dependence on proprietary software, especially programs sold by foreign vendors, has immense implications for the country's national security. Free open-source software, by its nature, is unlikely to feature secret back doors that lead directly to Langley, Va.

It's also less likely to feature a backdoor to a hacker's basement, or a lot of bugs ("glitches") that some health IT vendors and customer organizations allow to accumulate into the thousands before lifting a finger.

This brings to mind the adage that "if you want something done right, do it yourself."

... For ordinary Internet users, there is one silver lining: The embrace of open-source technology by governments may result in more intuitive software applications, written by a more diverse set of developers.

[Read the whole article at the link above - ed.]

More intuitive IT applications could solve a lot of the current health IT problems, such as the mission hostile user experience of many closed-source products from major vendors.

Afrer major IT debacles including the �13 billion abyss of the National Programme for IT in the NHS (NPfIT), the UK also seems to be learning:

New approach urged for government IT
E-Health Insider
2 March 2011
Lyn Whitfield

A new report into government IT failures has warned that previous inquiries may have embedded problems by focusing on inappropriate �best practice� instead of looking for alternative approaches.

The report, from the Institute for Government, says �existing �best practice� project models do not deal with the fundamental issues at the heart of government IT.�

[As I've stated here, one has to consider the "worst practices" as well, the "thou shall not's." Typical Milquetoast, touchy-feely "best practices" models and their "sanitized accounts of successful projects" (cf. Greenhalgh et al.) avoid such inconvenient realities like the plague - ed.]

It also argues that: �By implementing these same, flawed project techniques in an increasingly rigid fashion, these traditional solutions can act to exacerbate the problems further.�

[This sounds familiar, typical of the inflexible, dogmatic business IT culture, descended from the punched-card programmable tabulator culture of the 1920's - see this link - ed.]

Instead, it calls for a new that focuses on using government�s huge buying power to get better deals for what it calls �platform' technologies � such as server capacity and PCs � while encouraging departments to adopt �agile� methodologies to deliver systems to meet the particular challenges they face.

... Andrew Adonis, Director of the Institute for Government said: �The billions spent on cancelled IT projects, such as ID cards and National Programme for IT in the NHS, demonstrate precisely why we need a much more flexible approach to government IT.

... The report cites the national programme as one �symptom of failure� in government IT; the tendency for projects to run late and for departments to struggle to keep suppliers on board or hold them to their original delivery requirements as a result.


Agile software development methodologies have traditionally been anathema to the culture of IS departments involved in mercantile, manufacturing and management varieties of business computing. As I've written many times, however, those methodologies are essential in healthcare IT, such as here and here. At the latter link I observed:

... In fact, in my observations IT personnel are the true Luddites [as opposed to clinicians who are often accused of that reactionary characteristic- ed.], clinging to inappropriate, rigid business-IT views on the healthcare IT development and implementation process (vs. more appropriate and modern agile methodologies), holding unshakable, stereotypical views about physicians, and remaining unreasonably obstinate on clinician complaints about "clunky" health IT user experiences.

Perhaps Mr. Putin and Mr. Adonis deserve a copy of the book "Medical Informatics 20/20: Quality And Electronic Health Records Through Collaboration, Open Solutions, And Innovation", Jan. 2007, by the VistA (open-source EMR) pioneers.

The "Open Solutions" part of that title says it all.

-- SS

Friday, February 25, 2011

MedInformaticsMD in WSJ again: No Patient Will Ever Say, 'Quick, Watson, the Needle!'

Following along the lines of my Jan. 2011 post IBM's Watson, Jeopardy, and "Revolutionizing Medicine" , the Wall Street Journal published this Letter to the Editor by me today:

Wall Street Journal

LETTERS
FEBRUARY 25, 2011

No Patient Will Ever Say, 'Quick, Watson, the Needle!'

Regarding Ray Kurzweil's "When Computers Beat Humans on Jeopardy" (op-ed, Feb. 17): Librarians often receive requests for information as cryptic as "Jeopardy!" clues, from people who are sometimes not even sure what they're seeking. Watson is, in essence, a librarian that retrieves facts.

Regarding natural language processing and fact-retrieval systems like IBM's Watson, medicine is about cognition. It's about human judgment born of experience in dealing with ambiguity, not just of language but also, and this is critical, of observations, findings, lab data, image interpretation, etc. It is about human intuition, assemblage and the integration of a huge amount of disparate information in ways not well understood even by its practitioners. The end result is not just the recall of a piece of information, obscure as the information might be.

I consider predictions of Watson spearheading cybernetic miracles in medicine to imply just as grandiose a valuation to the technology as the statements I heard two decades ago about the health information technology of the day, or even today, "revolutionizing medicine." That has not happened.

A cybernetic librarian is no physician. As for me, for now I'll stick to people in my own medical care. Let Mr. Kurzweil see the machine for his ailments.

Scot Silverstein, M.D.

Philadelphia


I am pleased that my counter-marketing-hype opinion on the meme of health IT "revolutionizing medicine" was deemed fit to print by the Journal.

Those in senior leadership positions, as are included in the WSJ's readership, need to see that meme challenged.

-- SS

Friday, February 18, 2011

Massive Nationwide Medicare Bust: 111 Charged For Scams Worth $225 Million

The article "Massive Nationwide Medicare Bust: 111 Charged For Scams Worth $225 Million" in the HuffPo reports that:

MIAMI � Federal authorities charged more than 100 doctors, nurses and physical therapists in nine cities with Medicare fraud Thursday, part of a massive nationwide bust that snared more suspects than any other in history.

More than 700 law enforcement agents fanned out to arrest dozens of people accused of illegally billing Medicare more than $225 million. The arrests are the latest in a string of major busts in the past two years as authorities have struggled to pare the fraud that's believed to cost the government between $60 billion and $90 billion each year. Stopping Medicare's budget from hemorrhaging that money will be key to paying for President Barack Obama's health care overhaul ...


Read the entire story at the link above.

As I wrote at my Feb. 15, 2010 post "Does Possible EMR-Facilitated Upcoding (Also Known as "Fraud") Need Investigation by CMS, And Could it Explain HIT Irrational Exuberance?", those "Federal authorities" need to look into how EHR's facilitate upcoding, which is a form of fraud that is likely worth far, far more than a mere $225 million:

(Click to enlarge. $225 million might be mere chicken feed. See here.)


Worse, without careful scrutiny, EHR-mediated fraud might not be as easily detected as the "conventional" Medicare fraud mills, where, for example, it can get highly obvious something's amiss:

... A Brooklyn, N.Y., proctologist was charged with billing $6.5 million for hemorrhoid removals, most of which he never performed.

Finally, one might wonder if any of those caught in this recent sting operation were using EHR's...

-- SS

Tuesday, February 15, 2011

Does EHR-Incited Upcoding (Also Known as "Fraud") Need Investigation by CMS, And Could it Explain HIT Irrational Exuberance?

It has long puzzled me why an experimental technology that costs tens or hundreds of billions of dollars to roll out organizationally and nationally, that individual medical centers are spending upwards of $100 million or more upon, and that has a track record as poor as suggested in posts such as "An Updated Reading List on Health IT" and "MAUDE and HIT Risks: Mother Mary, What in God's Name is Going on Here?" is so popular.

At a Feb. 2010 post "A Lawsuit Over Healthcare IT Whistleblowing and Wrongful Discharge" I summed up other critical HIT issues succinctly, with abundant hyperlinks to other materials as below:

... I have written about health IT problems extensively on this blog and at my academic website on HIT failure. These include but are not limited to: medical informatics specialists ignored by nonclinician IT personnel [the link is to another case involving critical care]; inverted and paradoxical organizational structures where IT facilitators become HIT project leaders and clinical leaders become HIT project facilitators; ill-conceived and poorly implemented mission hostile health IT; perverse and in fact clinically cavalier IT "politics"; failure to obtain patient informed consent as if health IT is an elite world not subject to the same ethical obligations as medicine; probable violations of Joint Commission safety standards and hospital executive fiduciary obligations, and numerous others.

At a Jan. 2011 post "Healthcare IT Delirium" wrote that:

... On top of an irrational exuberance (see this blog query) largely unsupported by the literature (e.g. here), the technology is experimental, its rollout is a grand national experiment in social re-engineering of medicine, there is no patient informed consent, nobody is in control, and nobody is taking responsibility for regulating the domain despite known risks. The results will very likely reflect the Wild West free-for-all that is now extant.

Finally, at a Feb. 2011 post "An Updated Reading List on Health IT" I aggregated recent literature that sheds significant doubts on health IT beneficence.

Is the popularity of health IT as it exists in 2011 a paradox, or not?

When I was dealing with public employees in a transit authority several decades ago, I learned that mysterious circumstances and paradoxes were not paradoxes at all. The assumptions were simply wrong. Often, mysterious circumstances proved to be related to money (as in worker's comp fraud) and/or money related to [illegal] drugs.

Below are some charts and data from a HC Renewal reader, an ED physician, that may answer the mystery of the "irrational exuberance" exhibited by healthcare organizations towards clearly disruptive, medically risky and liability-provoking experimental technology. This is especially so in high risk areas such as an ED.

It should be noted that my recommendations as a CMIO a number of years ago in a very large regional medical center were that an ED would benefit best clinically from paper charts supplemented by document imaging, so that images of past charts could be instantly retrieved. ED's are busy places where clinicians have little time for distractions (such as computer data entry), but where access to past charts can be important. ED charts are also not generally very long or complex documents. Further, document imaging is a mature technology (e.g., Documentum).

Those ED recommendations still stand.

(Incidentally, my mother, injured in mid 2010 via an EHR error started in the ED, would certainly be in far better shape had this been the case. The error would not have occurred. How do I know this? I worked in that very same EHR pre-HIT.)

Note that the data and charts below are estimates only, pointing to a potentially huge problem -- with computer-assisted Medicare fraud implications -- if the projections are reasonable.

First, from the ED physician reader, a graph showing an increase in overall ED billing levels of 2002 versus 2008:

(E/M level increases, 2002 compared to 2008. Click to enlarge)


This appears to indicate a great increase in upper level (level 4 and 5) billings from 2002 to 2008 in ED's with increasing advent of EHR (part of this is likely due to "paper template" charting).

A very terse summary of these billing codes, for laypeople. Claims submitted to managed care organizations and States include the emergency levels of screening and treatment. They range from CPT 99281 ("straightforward medical decision making") to CPT 99285 ("medical decision making of high complexity"). These codes reflect not only the complexity of the treatment but also the time and difficulty of making a diagnosis:

  • 99281 - level 1 - very short visit - low level service, minimal charge
  • 99282 - level 2 - e.g. ankle sprain
  • 99283 - level 3 - limited focus history & physical exam performed, e.g. physisian evaluation with additional tests such as an x-ray.
  • 99284 - level 4 - significant history & physical exam performed, significant or major injury, often admission
  • 99825 - level 5 - significant time, e.g., 45 minutes to an hour spent by physician. (Note that considering that a high % of ED visits are using that code, it might be physically impossible for ED physicians to be spending this much time seeing each patient.)


Second, a spreadsheet with some estimated numbers:


(Click to enlarge)


There appears to be an absolute increase in Medicare/Medicaid payments to ER doctors due to this billing-upcoding phenomenon from 2002 to 2008 (roughly $2 billion in payments, which at 80% allowed Medicare rate represents appx. a $2.6 Billion increase in billing)

The 2006-2008 billing levels for each specialty are public info on the CMS website. The hard part was obtaining earlier numbers. These were found in 2 papers from 2004 and 2006:

Variation in Coding of Evaluation and Management (E&M) Services by Hospital Emergency Departments
Paul Shoemaker, FACHE, President and CEO
American Hospital Directory, Inc.
Jan. 2004
http://www.ahd.com/EMstudy040108.pdf


Variations and Trends in the Coding of Evaluation and Management (E&M) Services by Hospital Emergency Departments
Paul Shoemaker, FACHE
Leatrice Ford RN, BSN, CCS, Founder, ConsultCare Partners, LLC
May 2006
http://www.ahd.com/EMstudy060530.pdf

Apparently Shoemaker and Ford were able to get CMS data for 2002 and 2004 and draw the conclusions that while some hospitals and ER groups were undercoding, there was a significant level of upcoding occurring.

Note that the above projections assume the same Medicare physician allowables for 2002 and 2008; lesser values in 2002 (compared to 2008) would push the 2008 payment increases upward.

Finally, some actual ED screens from a major HIT vendor, McKesson, that "encourage" physicians to upcode.

The "coding engine" of this ED EHR system produces "recommendations" for "additional documentation" to support a higher level of coding. Such incentives themselves might be legitimate, but can create moral hazard - and the means, motive and opportunity for "playing with the coding a little bit" - if physicians' coding levels are monitored by their organizations.


(Click to enlarge. An abscess. ED EHR advising physician to "provide more documentation" in order to upcode from level 2 based on the data entered, to level 5 or maximum, via unknown algorithms. A "Correct Deficiencies" button brings up screens with selections a physician can click on to "support" the cybernetic upcoding.)



(Click to enlarge. In this example an ED physician is begin cybernetically cajoled - uh, "reminded" - to enter data to support upcoding a sprained ankle evaluation from a level 3 to a level 5.)


It should be remembered that increased ED coding levels by ED groups, often contractors to hospitals, may permit hospitals to increase their own billing charges for services rendered to the ED patients.

In summary:

  • With little solid evidence of major clinical benefit, ED physicians seem to be paradoxically "asked" to utilize distracting, often mission hostile, even hateful ED EHR systems that many despise, even in face of the critical nature of an ED environment and increased risks and liabilities that can occur with distractions such as data entry. This is a seeming paradox.
  • The ED physicians, after automated coding algorithms analyze the clinical data entered, may be "reminded" that if they fill in a few more details, the visit might be significantly upcoded.
  • Payers such as CMS have little or no way to ascertain the veracity of the additional entered information that may result.
  • The ED physicians' coding levels may be monitored by their employer(s) i.e., group or hospital compared to the "possible" coding levels suggested by the vendor software algorithms. ED EHR software makes this easy. This would create the means, motive and opportunity for upcoding, so as to prevent negative repercussions on a "low coding" doctor. (The ED physician who submitted this data claims that this is so in his organization, and in others.)
While the above is not a rigorous analysis, if it overall reflects the current situation with ED and other EHR's in the field in 2011, we may be seeing:

  • A significant explanation for "irrational exuberance" around health IT - which would actually not be so irrational after all (just unethical) - with potentially deadly results;
  • A highly systematic, major fraud participated in by major stakeholders, from software vendor to purchaser to clinical user to biller. This would represent a cybernetically-facilitated swindle of major (and growing) proportions; and
  • Claims of purported savings from health IT that we hear about from pundits and government officials being a rather pathetic joke.

Perhaps CMS (and other payers as well) could perform the needed rigorous investigation using their data and resources, to either confirm or disprove these findings. That's what we, the taxpayer, pay the government for.

-- SS

Thursday, February 3, 2011

Brief Comments on the President�s Council of Advisors on Science and Technology (PCAST) Report on Health IT

A short post.

In Dec. 2010 the President�s Council of Advisors on Science and Technology issued a 108-page report entitled:

REPORT TO THE PRESIDENT: REALIZING THE FULL POTENTIAL OF HEALTH INFORMATION TECHNOLOGY TO IMPROVE HEALTHCARE FOR AMERICANS: THE PATH FORWARD

The PCAST health IT report is available from whitehouse.gov in PDF format here.

Rather than reproduce the executive summary of the report, a terse "PCAST Report Summary" can be seen about midway down the page of the HISTalk blog here.

My comments are simple and twofold:

1. The term "risk" is absent from the PCAST report in the context of risk to patients from clinical IT. Instead, the context is largely about the risk to patients of NOT having health IT.

2. The term "safety" is similarly absent in the context of adverse effects of HIT, and only present in the context of how HIT will improve safety, except for one reference (indeed, the match to my search for the term "safety" was in the cited URL itself):

... The Food and Drug Administration (FDA), while not currently regulating EHRs, currently does receive voluntary reports of death and injury associated with EHR malfunctions. FDA officials have suggested possible future regulatory strategies that could include mandatory adverse event reporting, or even classifying EHRs as medical devices, which would make them subject to pre-market regulation. [13]

[13] Healthcare IT News (February 26, 2010) at http://www.healthcareitnews.com/blog/should-fda-regulate-ehr-safety

I find this paucity of a certain context for these terms in a report to the President of the United States striking.

Finally, ONC solicited responses to the PCAST report. The American Medical Informatics Association (AMIA) issued a response that is very good. It outlines practical matters, the need for more Medical Informatics specialists and incorporation of their work into national efforts at a common "language" for medicine, interoperability, and other issues. For instance:

AMIA applauds the PCAST report for addressing important issues regarding interoperability and quality improvement. However, we note that the report highlights potential shortcomings and inadequacies in the current approaches and proposes a number of activities in addition to or instead of those currently underway. AMIA believes that the report would benefit from additional details and specificity to support the recommendations that are proposed. The report does not mention the significant prior work that has been conducted in these areas. Additional consideration of data and lessons learned from the scientific and peer-reviewed literature would help to further support and substantiate the ideas presented.

The AMIA response is available at this link (also PDF).

Also see summaries of other responses to the PCAST report at the Jan. 23, 2011 e-CareManagement blog post aptly entitled "Comments to ONC: PCAST HIT Report Becomes a Political Pi�ata."

-- SS

Tuesday, February 1, 2011

An Updated Reading List on Health IT

�It is only the ignorant who despise education.� - Publilius Syrus, Latin Poet, 43-85 AD

In a March 2009 post "A Primer On Why We Have Busywork Generators Masquerading as EMR's: Health IT Reality vs. the Bernard Madoff Version", I posted a "reading list" of articles about health IT that I consider essential in understanding the issues surrounding the experimental nature of this technology.

I wrote:

Anyone contemplating policy work in health IT should be intimately familiar with these works, as they illustrate the true HIT environment in 2009, not the Bernard Madoff "unlimited returns no matter what" version.

What these articles do not illustrate is the common atmosphere of irrational exuberance and complacency now prevalent about HIT, with healthcare organizations and physicians now being pushed forcefully to adopt these technologies or suffer payment penalties they can ill afford:

I've updated the list.

Updated reading list (not meant to be at all comprehensive; this is just a set I've collected over recent years writing on health IT):

[1] Electronic Health Records and Clinical Decision Support Systems: Impact on National Ambulatory Care Quality. Max J. Romano, BA; Randall S. Stafford, MD, PhD, Arch Intern Med. Published online Jan. 24, 2011. doi:10.1001/archinternmed.2010.527

[2] Electronic Health Record Use and the Quality of Ambulatory Care in the United States. Linder, Ma, Bates et al. Arch Intern Med. 2007;167:1400-1405

[3] The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview. Black AD, Car J, Pagliari C, Anandan C, Cresswell K, et al. PLoS Medicine 8(1): e1000387 (Jan. 18, 2011). doi:10.1371/journal.pmed.1000387.

[4] Electronic Health Records� Limited Successes Suggest More Targeted Uses. DesRoches, Jha et al., Health Affairs, April 2010 vol. 29 no. 4 639-646, doi:10.1377/hlthaff.2009.1086

[5] Health information technology: fallacies and sober realities. Karsh et al. (Oct. 2010). JAMIA 2010 17: 617-623.

[6] Challenges in ethics, safety, best practices, and oversight regarding HIT vendors, their customers, and patients: a report of an AMIA special task force. Goodman et al. JAMIA doi:10.1136/jamia.2010.008946. Nov. 2010.

[7] The Top Ten Health Technology Threats to Patient Safety for 2011. ECRI Institute, Plymouth Meeting, PA, Dec. 7, 2010.

[8] Current Approaches to U.S. Healthcare Information Technology are Insufficient. Computational Technology for Effective Health Care: Immediate Steps and Strategic Directions. National Research Council of the U.S. National Academies, Jan. 2009.

[9] Evaluating the Impact of the Electronic Health Record on Patient Flow in a Pediatric Emergency Department. Mathison, Chamberlain, Children's National Medical Center � Division of Emergency Medicine, Washington, DC. Applied Clinical Informatics, Vol. 2, Issue 1, 2011, 10.4338/ACI-2010-08-RA-0046

[10] A Critical Essay on the Deployment of an ED Clinical Information System - Systemic Failure or Bad Luck?, Professor Jon Patrick, Health Information Technology Research Laboratory, School of Information Technologies, University of Sydney, Australia, Jan. 2010.

[11] E-Health Hazards: Provider Liability and Electronic Health Record Systems. Sharona Hoffman and Andy Podgurski. Berkeley Technology Law Journal (2010). Followup paper on EHR medical and legal risks. This article is a first of its kind, a comprehensive analysis of the liability risks associated with use of clinical IT. The authors point out that the potential benefits of computerization could be substantial, but EHR systems also give rise to new liability risks for health care providers that have received little attention in the legal literature.

[12] Meaningful Use and Certification of Health Information Technology: What About Safety? Sharona Hoffman and Andy Podgurski. Case Research Paper Series in Legal Studies Working Paper 2010-34, October 2010.

[13] Finding a Cure: The Case for Regulation And Oversight of Electronic Health Records Systems, Sharona Hoffman and Andy Podgurski. Harvard Journal of Law & Technology 2008 vol. 22, No. 1

[14] Emerging Trends in Electronic Health Record Liability. Chad P. Brouillard. For the Defense, July 2010 (Defense Research Institute).

[15] Electronic Health Records: Recognizing and Managing the Risks. ClaimsRx: Clinical and Risk Management Perspectives. NORCAL Mutual Insurance Company, Oct. 2009.

[16] Litigation in the Decade of Electronic Health Records. Joel B. Korin and Madelyn S. Quattrone. New Jersey Law Journal, June 11, 2007.

[17] Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors. Ross Koppel, PhD, et al, Journal of the American Medical Association, 2005;293:1197-1203

[18] Workarounds to Barcode Medication Administration Systems: Their Occurrences, Causes and Threats to Patient Safety, Koppel, Wetterneck, Telles & Karsh, JAMIA 2008;15:408-423

[19] Hiding in Plain SIght: What Koppel et al. tell us about healthcare IT. Christopher Nemeth, Richard Cook. Journal of Biomedical Informatics. 38 (4): 262-3.

[20] Health Care Information Technology Vendors' "Hold Harmless" Clause - Implications for Patients and Clinicians, Ross Koppel and David Kreda, Journal of the American Medical Association, 2009; 301(12):1276-1278

[21] Hospital Computing and the Costs and Quality of Care: A National Study. Himmelstein. Wright, Woolhandler. The American Journal of Medicine, Volume 123, Issue 1 , Pages 40-46, January 2010.

[22] Unintended errors with EHR-based result management: a case series. Yackel and Embi; JAMIA 2010 17: 104-107; doi: 10.1197/jamia.M3294, Oct. 2009.

[23] Information Technology: Not a Cure for the High Cost of Health Care: Knowledge@Wharton, Wharton School of Business, University of Pennsylvania, June 10, 2009. (PDF version available at this link).

[24] Contemporary Issues in Medical Informatics: Common Examples of Healthcare Information Technology Difficulties. Teaching website, Scot Silverstein, MD, Drexel University, College of Information Science and Technology, Philadelphia, PA. Site initiated in 1999.

[25] Healthcare Renewal blog. I write on healthcare IT issues at this multi-author website sponsored by the Foundation for Integrity and Responsibility in Medicine (FIRM), a 501(c)(3) advocacy group. FIRM researches problems with leadership and governance in healthcare that threaten healthcare's core values.

[26] Health IT Project Success and Failure: Recommendations from Literature and an AMIA Workshop. Journal of the American Medical Informatics Association. Bonnie Kaplan and Kimberly D. Harris-Salamone.

[27] Joint Commission: Sentinel Events Alert on HIT, Dec. 2008.

[28] Internal FDA memorandum on HIT risks to Jeffrey Shuren MD JD (Director, Center for Devices and Radiological Health). "Not Intended for Public Use." Feb. 23, 2010. (Description/summary of memorandum is at this link).

[29] FDA's Manufacturer and User Facility Device Experience database (MAUDE) and HIT risks. Healthcare Renewal Blog, Scot M. Silverstein MD, Drexel University, Philadelphia, PA, Jan. 2011

[30] Health IT's Mission Hostile User Experience (eight-part essay), Healthcare Renewal Blog, Scot M. Silverstein MD, Drexel University, Philadelphia, PA, Feb. - Mar. 2009.

[31] The National Programme for IT in the NHS: Progress since 2006. Summary points, conclusions & recommendations. UK Parliament House of Commons, Public Accounts Committee, January 2009. (Full report at this link.)

[32] The future of the NPfIT. Program slated for major downsizing and decentralization after poor progress and expenditures exceeding �13 billion. UK Dept. of Health, Sept. 2010.

[33] "Don't Repeat the UK's Electronic Health Records Failure". Stephen B. Soumerai, Professor of Population Medicine at Harvard Medical School, Anthony Avery, Professor of Primary Care at the University of Nottingham Medical School, UK. Huffington Post, Dec. 5, 2010.

[34] Health informatics � Guidance on the management of clinical risk relating to the deployment and use of health software. UK National Health Service, DSCN18 (2009), formerly ISO/TR 29322:2008(E).

[35] Health Informatics � Application of clinical risk management to the manufacture of health software. UK National Health Service, DSCN14 (2009), formerly ISO/TS 29321:2008(E).

[36] The Impact of eHealth on the Quality & Safety of Healthcare, A Report for the NHS Connecting for Health Evaluation Programme. (Note: 7 MB in length). Car et al., Imperial College, London, March 2008.

[37] Medical Project Agency's Working Group on Medical Information Systems: Project summary. Proposal for guidelines regarding classification of software based information systems used in health care. L�kemedelsverket - Swedish Medical Products Agency (MPA), June 2009, revised Jan. 2010.

[38] Tensions and Paradoxes in Electronic Patient Record Research: A Systematic Literature Review Using the Meta-narrative Method. Greenhalgh, Potts, Wong, Bark and Swinglehurst, University College London. Milbank Quarterly, Dec. 2009.

[39] Failure to Provide Clinicians Useful IT Systems: Opportunities to Leapfrog Current Technologies, Ball et al., Methods Inf Med 2008; 47: 4�7,

[40] IT Vulnerabilities Highlighted by Errors, Malfunctions at Veterans� Medical Centers, JAMA Mar. 4, 2009, p. 919-920.

[41] Software hiccups cause drug, treatment errors at VA�, Associated Press, January 14, 2009

[42] Electronic Records System Unreliable, Difficult to Use, Service Officials Tell Congress, Sandra Basu, U.S. Medicine - the Voice of Federal Medicine, May 2009

[43] Unexpected Increased Mortality After Implementation of a Commercially Sold Computerized Physician Order Entry System, Han et al., PEDIATRICS Vol. 116 No. 6 December 2005, pp. 1506-1512 (doi:10.1542/peds.2005-1287).

[44] The Computer Will See You Now, New York Times, Armstrong-Coben, March 5, 2009.

[45] Physician testimony to HHS Standards Committee Implementation Workgroup of the ONC, Jan. 11, 2011. Reproduced with permission at Healthcare Renewal blog.

[46] Bad Health Informatics Can Kill. Working Group for Assessment of Health Information Systems of the European Federation for Medical Informatics (EFMI).

[47] "Dutch nationwide EHR postponed: Are they in good company?", ICMCC.org, Jan. 24, 2009

[48] "The failure rates of EMR implementations are also consistently high at close to 50%", from Proceedings of the 11th International Symposium on Health Information Management Research � iSHIMR 2006

[49] Adverse Effects of Information Technology in Healthcare. This knowledge center presents a collection of information on the adverse effects of information technology in its application to healthcare. It also references sources of information on information security, and related media reports.

[50] The Machinery Behind Health-Care Reform: How an Industry Lobby Scored a Swift, Unexpected Victory by Channeling Billions to Electronic Records - Robert O'Harrow Jr., Washington Post, May 16, 2009

[51] Pessimism, Computer Failure, and Information Systems Development in the Public Sector. Shaun Goldfinch, University of Otago, New Zealand, Public Administration Review 67;5:917-929, Sept/Oct. 2007

The following are dead-serious personal accounts of health IT misadventure:

[52] EHR Problems? No, They're Merely Anecodotal; the Truth Must Be That I Attract Bad Electrons and Stale Bits (personal account, Scot M. Silverstein, MD, Jan. 2011)

[53] This post on leadership changes at ONC might also be worth a read.



I await invective that these sources are "cherry picked" and "anecdotal."

-- SS

Related Posts Plugin for WordPress, Blogger...