Craig Stoltz (previously mentioned) dropped me an email to point out a post I’d missed from The Health Care Blog by Alicia White of Bazian (the company which evaluates stories for the NHS’s Behind the Headlines service).

Says Ms. White:

…we’ve developed the following questions to help you figure out which articles you’re going to believe, and which you’re not.

Questions include:

• Does the article support its claims with scientific research?
• Is the article based on a conference abstract?
• Was the research in humans?
• How many people did the research study include?
• Did the study have a control group?
• Who paid for and conducted the study?
• Did the study actually assess what’s in the headline?
• How can I find out more?

Good stuff. Go read it.

Thanks again for the pointers, Francesca and Craig!

_______________
Feed-only Footer:
Know of something I should blog about? Please let me know!

If you prefer, you can leave me a voicemail here: (315) 876-9574

“Trying to Measure the Quality of Health Information on the Internet: Is It Time to Move On?” [html] | [PDF]

Short answer:

Hell, no.

Longer answer:

Says the article:

“The natural assumption is to believe that there exists a link between the quality of information on the Internet and harm. However, a systematic review attempting to evaluate the number and characteristics of reported cases of harm in the peer-review literature determined that for a variety of reasons, there was little evidence to support this notion.”

It is impossible to quantify why people make bad decisions. For instance, say someone makes foolish financial decisions and loses everything they own: can it be determined if these bad decisions were made based on information they found online?

On the other hand, ask anyone you know who works in an Emergency Room if they’ve seen people who have done harm to themselves with a self-diagnosis or self-treatment based on something they read online. Every one of them will confirm they have. This isn’t necessarily the fault of the information found online, but knowing as we do that people increasingly make decisions about their health that are informed by information they find online, we don’t need evidence to assume, a priori and with confidence, that bad information can lead to harm. It is common sense for everyone in the health community to promote/produce good online information resources and discourage the existence/use of bad online information resources. Most importantly, we must help both health professionals and patients gain the information and health literacy skills to tell the good from the bad.

Authors Deshpande and Jadad argue that the evaluation tools for Web sites suffer…

“…from several limitations, which, in addition to those mentioned by the authors, include uncertain levels of usability, reliability, and validity.”

I won’t argue with that. I don’t think any single evaluation tool can let someone without information literacy skills determine the quality of information found online. I don’t think that even a vast arsenal of such evaluation tools will do the trick.

The authors have questions I’ll answer (I don’t care that the questions were intended to be rhetorical- they need to be answered):

“Will we ever develop an ideal tool that allows individuals to assess the quality of health information?”

Nope. We’ll also never invent a diagnostic tool that can replace a good physician’s experience and judgement- but that doesn’t mean we shouldn’t create tools that help new doctors build these skills or help more experienced doctors flesh out their differential.

What are the determinants of this quality?

That’s a pretty broad question, but the MLA has a good basic guide of where to start.

“Is it possible to assess or measure quality?”

Medical librarians assess the quality of information every single day. So…yeah, it is.

“Even if possible, is the formal assessment of quality even necessary?”

Necessity depends on who the user is and what his/her particular information needs are.

“Does it even matter?”

Kind, encouraging teachers throughout the world often say that there’s no such thing as a stupid question. These extraordinarily compassionate educators are lying. Yes, it matters.

I’m a Web enthusiast who sees a lot of value in lots of online collaborative efforts. There are absolutely places and uses for the wisdom of the crowd- but to hear some people talk about apomediation, you’d think the wisdom of the crowd could replace the judgement of experts.

“For example, Internet users could provide ratings or recommendations based on their own experiences to judge the quality and relevance of health information.”

Huh. So if we’re just gonna’ go with what a crowd of self-selecting amateurs agree upon, I guess we don’t need double blind trials any more, either? We’ll just ask a crowd their opinion. Who needs empiricism? Science, schmience. Why should we take this article more seriously because the authors have two MDs and dotorate between them? I’m not Andrew Keen¹, but neither am I an irrational technotopian.

“Analogous to the peer-review process, aggregation of ratings from many individuals (a form of crowdsourcing) allows “good” information to be highlighted prominently, while “not so good” information gets pushed to the bottom.”

That’s a terrible analogy. In the peer-review process, both the author(s) and the reviewer(s) are credentialed experts in their field. I would not ask a room full of neurologists for advice on my house’s plumbing and I would not ask a room full of plumbers about the treatment of peripheral neuropathy. For the “Digg” model to work at all for health information, the crowd should be very large and very knowledgeable. On the other hand, even widely agreed-upon practices have later been proven wrong by empirical testing and evidence…so shouldn’t we rely on evidence?

The problem with relying only on the wisdom of crowds is that, sometimes, the crowd shows an alarming lack of wisdom regarding health information. Huckster snake-oil salesman Kevin Trudeau’s Natural Cures ‘They’ Don’t Want You to Know About was a New York Times self-help Best Seller! (For more on why this is a great example of an unwise crowd, see this video.)

Let me illustrate with a real life example: When our infant son developed what appeared to be a tremor, we didn’t waste time asking a crowd. We first asked a local pediatric neurologist for his opinion. When he was unable to make a diagnosis, we asked Dr. Marc Patterson, a pediatric neurologist at the Mayo Clinic. Dr. Patterson’s unusual experience and education enabled him to make a diagnosis very quickly.²

I wonder: If Drs. Deshpande and Jadad have family members with worrying illnesses, do they consult the best available clinical expert, or go to the wisdom of the crowd?

But I’m getting off-track. Back to the article.

As seems to be typical for malinformed physician technotopians, these authors point at Wikipedia to support their perspective.

“The interplay of users to collaborate and deal with information overload has already been proven successful in other areas outside the health space. For example, Wikipedia not only allows users to submit content on various topics, but also provides the capability for users to edit the content of others. Although there is the potential for misuse, Wikipedia, which relies on anonymous, unpaid volunteers, seems to be as accurate in covering scientific information as the Encyclopedia Britannica.”

Well, yeah. And I’d trust Wikipedia as a source for health information about as much as I’d trust Britannica….which is to say not very much.³

“Since its inception in 1990 until the present day, the health system has grappled with how to manage potential harm associated with information available on the Internet. Research in this area, for the most part, continues to assume that techniques used to evaluate paper-based information can automatically be applied to online resources, ignoring the added complexity created by the multiple media formats, players, and channels that are brought together by the Internet.”

Someone please explain to me why peer review is less effective with texts distributed online than texts distributed on artifacts of dead trees. Text is text. Someone please explain to me why peer review would be less effective if this text is read aloud and recorded/played online as video or audio (downloadable or streamed in any format). The “added complexity” the authors mention impacts peer review in no way. Perhaps this is why they make such an assertion while providing utterly no support for it.

I like Web technology and have made a nice niche for myself by writing and talking about how it can be useful to health information professionals. I am sick to death, however, of people attempting to make names for themselves with inane prognostication and unsupported technotopianism. When these authors write that “…as the Web continues to evolve, we will likely gain new insights as to how this happens along with a better understanding of how to handle health information from any source…”, I want to beat my head against a wall. To me, this is no different from saying we should go ahead and build lots more nuclear reactors because we have faith that technology will work out a way to dispose of nuclear waste in a safe manner before we are harmed by our inability to dispose of it properly. Such things are too important to take on faith in the future.

“The time has likely come to end our Byzantine discussions about whether and how to measure the quality of online health information. The public has moved on. It is time to join them in what promises to be an exciting voyage of human fellowship, with new discoveries and exciting ways to achieve optimal levels of health.”

Such a perspective would have us ally ourselves with the Jenny McCarthys of the world. Jenny McCarthy believes and popularizes the idea that immunizations cause autism spectrum disorders, despite the fact that there is no scientific evidence correlating immunizations and ASDs.

Laypeople, even brilliant laypeople, do not generally have the information or health literacy skills to know where to find quality information and to know what to trust. My brother is an experienced Web programmer. He has topped out every IQ test he has ever taken. He is a brilliant man and as talented an autodidact as anyone I know. Still, when he needed a medical procedure, he was able to find more information and trust its authority by conferring with me- because I spend my days finding and evaluating health information.

Reliance on science and expertise is not Byzantine. Rationalism is not Byzantine. Empiricism is not Byzantine. Politicians should not make public policy decisions based on polls and clinicians should not make decisions based on the misinformed preferences of their patients. Clinicians have a duty to educate patients and help point them towards good information because the volume of shoddy information is growing at an alarming rate.

_______________
Feed-only Footer:
Know of something I should blog about? Please let me know!

If you prefer, you can leave me a voicemail here: (315) 876-9574

I have believed in the value of the dialectical process since long before I knew there was a term for it and have always believed that honest criticism serves the criticized, the critic, and those witnessing the process.

So I LOVE that Dean decided to specifically address one of my criticisms.

I questioned Dean’s assertion that “Google’s search results are emblematic of an approaching crisis with information overload”.

Dean writes:

Google most certainly is emblematic (a visible symbol) of information overload, and in fact is the information specialist’s laboratory for it.

I see honest disagreement here.

I think Google is emblematic of the way that the clever application of technology overcomes “information overload.” The Web is huge, filled with an insane amount of information that is varyingly good, bad, ugly or [fill in your favorite adjective here]. But if one uses Google to search for Google Scholar Dean, the first four results are about Dean Giustini, the author of the UBC Google Scholar Blog. It took typing three words and I found EXACTLY what I was looking for in about 0.51 seconds. To me, this doesn’t paint an image of Google as a symbol of information overload.

Dean continues:

“It’s well-documented throughout the blogosphere that web 2.0 has resulted in too many RSS feeds, too much data and information from disparate sources with little connection to each other.”

First, there are many popular positions (technical, political, philosophical…) expressed in the blogosphere (and elsewhere) that I believe to be wrong-headed, foolish, unwise or silly. I also believe that decisions based on evidence and direct experience tend to get better results than those built on blogosphere buzz and hearsay.

Second, I’m sincerely flabbergasted to hear a librarian (or any information professional) complain that there is “too much data” or “too many RSS feeds”.

“Web 2.0″ doesn’t cause an information glut. What causes an information glut is being an information glutton, taking on more than anyone can reasonably manage. There aren’t too many RSS feeds. Rather, there are users who subscribe to too many RSS feeds. The solution isn’t for less data to exist, the solution is smarter, more selective use of the data. The tools that help us filter and manage the information that we care most about are continuing to improve in power and sophistication.

The feeds I subscribe to are so carefully chosen and filtered that I started to worry about missing out on serendipitous discovery of information I didn’t know I needed. I remedied this by using feeds from the social bookmarking of medical librarians (and other beloved medical/technology information nerds) to keep an eye on what they find interesting. In this way, the tools that Dean sees as contributing to the “information glut” open me up to new ideas, thoughts, and resources that I would otherwise not have found. With the smart use of my aggregator, I can browse these quickly and easily, discarding what doesn’t suit my interests.

Dean also writes:

“99% of the information that we are finding in Google is irrelevant to medicine.”

Sure, but I’ve never heard a single physician claim Google is irrelevant to Medicine. On the contrary, many talk about how frequently they use it. Meanwhile, 99.99% of what’s findable via PubMed is irrelevant to a particular healthcare information need. So what?

Meanwhile, turnabout is absolutely fair play, and Dean reciprocates by sharing some criticism of my own writing. Because I’m really enjoying the direct discussion and believe in the dialectical process, I’m going to address each of his criticisms.

Dean writes:

“If you have some different ideas on where the web is, why don’t you write your own piece?”

I will absolutely be pleased to write an Op-Ed on the Web and health information at the invitation of the first prestigious medical journal which invites me to do so.

I’d might use the opportunity to demystify buzzwords, clear up popular misconceptions and/or call for a new sort of rigor in the way that writers in medicine and libraries use technology buzzwords in order to better empower each other and those whose needs they serve.

I could alternately suggest practical ways in which existing technologies could be leveraged to expand or improve information services for clinical patrons.

Dean writes:

“One thing about David’s blogging is that he doesn’t explore the social or cultural context for all the tools he introduces.”

That’s usually true. After all, I’m neither a sociologist nor a cultural anthropologist.

I usually talk about how Web tools might be applied in the setting of a medical library. That’s sort of what this blog is for- and I’ve said that from the very first post.

I run a hospital library and I solve problems that hospital-based users experience while using technology. I write about the things that interest me and that I think will interest others who do similar work and face similar problems. That’s what defines the general scope of this blog- and I’m happy with it.

Dean writes:

“It’s much harder to place information technologies in some context than it is to merely announce that you’ve found a new tool worth exploring.”

Although I really do think it is important and useful just to share the existence of tools that I think may be of interest or use to other medical libraryfolk, I often provide context. Some examples:

• The Future of the Hospital Librarian (Pure Op-Ed on the future of hospital librarianship)
• Misdiagnosed Cyberchondriasis (Taking apart a very stupid term that is increasingly popular in mainstream media)
• Veropedia: Like Wikipedia, Except Totally Lame (A review and critique)
• PogoFrog vs. the Dyson Vacuum Cleaner (A review and critique)
• Elsevier’s WiserWiki (A review and critique)
• Rating Medical Blogs (A review and critique)
• Social Search for Health Librarians (A review and critique)
• Disliking “Web 2.0″ and Hating “Web 3.0″¹

Both Dean and Dr. Ves Dimov (at Clinical Cases and Images) seem to think that I want to replace “Web 2.0″ with some other term.

Dr. Dimov writes:

David seems to suggest the alternative term “Web Geekery” which does not sound much better than “Web 2.0.”

I have at no time suggested that my complaints with the (mis)use of “Web 2.0″ would be solved by replacing it with ANY other term. “Web Geekery” is just the phrase I use to describe the stuff that interests me.²

I will point out, though, that not a single person has ever needed me to explain what I mean by “Web geekery.”

For the record, Ves- I don’t think there was anything wrong with your description of “Web 2.0 and Medicine” in 2005. You were talking about applying the trends O’Reilly described with “Web 2.0″ to the needs of medicine. Still, I dearly hope that many of the current uses of “2.0″, “3.0″ and (yes, really!) “Web 4.0″ will go away. The metaphor threatens to outgrow and eat what it was supposed to represent.

_______________
Feed-only Footer:
Know of something I should blog about? Please let me know!

If you prefer, you can leave me a voicemail here: (315) 876-9574

MedlinePlus has a new page on Health Literacy with links to lots of good stuff.

_______________
Feed-only Footer:
Know of something I should blog about? Please let me know!

If you prefer, you can leave me a voicemail here: (315) 876-9574

Testimonials from real patients and physicians on how the Internet has helped them with their health care treatment and delivery.

Above: Embedded video. If you are reading this via your feed aggregator, you may need to visit the site itself to view.

What bothers me about this video is that “the internet” isn’t the best tool for ANY of the purposes mentioned in this video. There are better tools for finding quality health information than Google and “the internet.”

Also: What is “Health Think”? (See opening and closing titles)

_______________
Feed-only Footer:
Know of something I should blog about? Please let me know!

If you prefer, you can leave me a voicemail here: (315) 876-9574

More about McKean here and here.

Also be sure to check out McKean’s blog, Dictionary Evangelist.

_______________
Feed-only Footer:
Know of something I should blog about? Please let me know!

If you prefer, you can leave me a voicemail here: (315) 876-9574

On 7/21/2007, a Harris Poll was released which stated that in the last two years, the percentage of people who “have used the Internet to search for health-related information” has gone from 53% to 71%. This was based on a telephone poll of 1,010 adults between 7/10/07 and 7/16/07. The Harris report refers to these people as “cyberchondriacs.”

The media seems to love this story. BusinessWeek, ars technica and a ton of others have offered articles on it.

I think the study matters and merits coverage, but I object to the term “cyberchondriac.”

Managed care has resulted in physicians not being able to spend as much time educating patients at the same time patients are taking more responsibility for their healthcare decisions. The healthcare consumer who decides to do some research using the most powerful research tool the world has ever seen isn’t a hypochondriac. That consumer is taking responsibility for her healthcare and becoming informed.

Let us say that a patient is told that she had a kind of hypothyroidism. That patient goes to the bookstore and buys a book published under the name of Mayo (or Johns Hopkins or some other very reputable authority) on thyroid disorders. The patient reads the book and makes notes on questions she wants to ask her doctor when she goes in for her next office visit. This isn’t hypochondriasis. This is the sort of patient clinicians should treasure. A patient who self-educates and asks informed questions saves the time of clinicians.

Now let’s imagine that the same patient skips the book and instead searches Medline Plus for information about hypothyroidism, printing some pages out and making notes on questions she wants to ask her doctor when she goes in for her next office visit. This isn’t hypochondria either. This is the exact same behavior in a patient that clinicians should celebrate and encourage.

Hypochondriasis is a very real disorder. The hypochondriac doesn’t need an Internet connection to experience its awful symptoms.

If clinicians want to complain about patients who look up health information online with no regard to the authority of the information or the information provider, that’s fine. Those clinicians would do well to volunteer and donate to help improve outreach and information literacy programs in medical libraries. When the clinician encounters a patient who habitually looks for health information from poor sources, the clinician should refer the patient to the nearest medical library or at least point the patient towards Medline Plus.

I won’t be using the word “cyberchondriac” to describe people who seek health information online.

I won’t use it to describe hypochondriacs who look for health information online, either. It has a glib feel to it that doesn’t sit well with me when describing someone dealing with a disorder as awful as hypochondriasis.

[/soapbox]

The Newest Vital Sign is based on a nutrition label from an ice cream container. Patients are given the label and then asked 6 questions about how they would interpret and act on the information contained on the label.

Specifically, the patient is handed a copy of the nutrition label and then asked a series of 6 questions about it. Patients can and should retain the label so they can refer to it while answering questions. It is not necessary to give the patient time to review the label before asking the questions. Rather, they will review the label as they are asked and answer the questions.

The questions are asked orally and the responses recorded by a clinical staff member on a special score sheet, which contains the correct answers. Based on the number of correct responses, the health care provider can assess the patient’s health literacy level.

You can order yours here.

Yes, it is created by Pfizer- but they’ll send you a copy of the peer-review study of the Newest Vital Sign (or you can snag it yourself here).

More about NVS:

• MedScape article: Newest Vital Sign May Be an Effective Quick Screen for Health Literacy
• Pfizer’s promotional materials

[via BHIC]

Among other things, Ratcatcher mentions this great self-paced tutorial from the Mount Sinai School of Medicine.

In case you missed it, Rachel has moved Women’s Health News to Wordpress.com, so be sure to bookmark it at its new home and to subscribe to its new feed.