When the user actually *is* broken (Anna Kushnir and PubMed)

I have distinct childhood memories of asking my mother what one word or another meant. She would point out that there was a dictionary close at hand designed exactly for that purpose and invite me to make use of it.

I remember asking my father to teach me to program in BASIC. He cheerfully agreed and handed me the big brown manual.

So maybe I’m weird and so are my folks, but these memories inform my take on the chatter in the blogosphere and on MEDLIB-L about this post by Harvard PhD student Anna Kushnir in which she expresses her frustration with PubMed. Kushnir writes (in part):

“I hate PubMed. I hate it with a burning passion. For a site that is as vital to scientific progress as PubMed is, their search engine is shamefully bad. It’s embarrassingly, frustratingly, painfully bad.”

[...]

“Why is PubMed so behind the times? Why? How does it even work? Does it search only the abstract? Does it also search the body of the papers that are available online? Why does it get so massively confused by an author’s initials and last name together, in one search? Why can’t it alert me when papers relevant to my work are published?”

I’m the first to admit that PubMed has problems and much room for enhancement, but if Kushnir had bothered to look at PubMed’s help manual or try some of its excellent tutorials she’d have learned exactly how it works, what PubMed indexes, how she can search by author, and that it can alert the user when papers relevant to her work are published via email or RSS.

So while PubMed has real, legitimate problems, Kushnir’s complaints don’t really touch on any of them. She could’ve resolved the problems she notes by flipping through the well-written, clearly laid-out, easy-to-navigate manual.

A number of helpful people who are much nicer than I am left useful comments for Kushnir.

Medical librarian Kathleen Crea offered a clear explanation of how articles are indexed and what MeSH is.

Medical librarian Rachel Walden even offered to help remotely with specific searches if Kushnir didn’t have a Harvard medical librarian handy.

But Kushnir decided that none of this really helped and later commented:

I don’t think I should have to be, or enlist the services of, a medical librarian in order to do a simple search on a literature search engine. PubMed should be an intuitive search engine such as Google, or others. I don’t know of many researchers, either MDs or PhDs, who have had extensive training in computer science or search algorithms. I am going to go out on a limb and say that I am representative of many other biomedical researchers in my struggles with PubMed. I am trained in Cell Biology and Virology. PubMed should be tuned to my needs and my skill set. I should not have to tune to it. Harsh as it may sound, PubMed is most useful for biomedical professionals, not for medical librarians or for computer scientists. Yes, if I devoted an afternoon or more to learning the system I dare say I would become a proficient, but my question stands – why should I have to?

Huh.

The index of biomedical literature searched from PubMed is a vast and complex set of data. Any tool that will search it effectively for very specific needs will necessarily be complex. If Ms. Kushnir doubts this, perhaps she should perhaps try any other interface for the same data. Some other interfaces work better for some purposes and some users, but all are complex.

Using PubMed does not require “extensive training in computer science or search algorithms,” it requires reading the manual. Kushnir actually admits that if she “devoted an afternoon or more to learning the system” she would “become a proficient,” and yet she fails to recognize her complaints as the whining they are.

Kushnir writes at JOVE:

My rant somehow wound up on a medical librarian listserv and they came out in force defending NCBI and PubMed, listing pages and pages of helpful and warm instructions and hints on how to make it do what I need it to do, pages of suggestions, with offers of hands-on assistance and training, which have all been wonderful. Occasionally though, they were biting and harsh, saying that if I only knew what I was doing (and only if I weren’t so ignorant… yup, ignorant), PubMed would seem to me the greatest thing ever.

I’m not criticizing Kushnir’s ignorance and would take issue with those who did. Ignorance, once identified, should alert the librarian to a teaching opportunity- not an occasion for shaming. Criticizing the extraordinary laziness in her refusal to receive help from a librarian or to take a quick look at the manual, though? That’s fair game.

Kushnir continues:

I am a research scientist by long, hard training. I am a fairly web-savvy research scientist, and still, I have trouble with PubMed.

As a medical librarian friend recently pointed out to me, it requires instruction to learn to drive a car. Kushnir is unwilling to read the manual and wants to blame PubMed/NLM for her difficulties. Kushnir talks about having spent hours trying to get PubMed to do what she wants, but declines help from multiple medical librarians who’re happy to teach her and can’t be bothered to invest 30 minutes in reading from the manual because it should, in her thinking, be possible to do without any effort on her part.

Kushnir continues:

The search engine is not made for medical librarians. It’s not made for computer programmers. It’s made for scientists, to be used by scientists, needed most by scientists.

Actually, Medline’s history is that it was made primarily for medical librarians and secondarily for physicians, but that’s not really important.

It should be easy for scientists, goofy, only moderately-computer literate scientists, to use. It should be intuitive (read: Google), it should not have a ginormous page of inscrutable instructions, it should not require the hour-long training sessions, kindly offered at most medical libraries. It should be plug and chug.

I might just as well argue that the tools of virology research should be intuitive to me. After all, I’m a very computer-literate, Web-savvy biomedical information professional. Why should I need her years of training to understand her work?Hint: Because the work is complex and involves a skill set that grows (with effort) over time.

“Inscrutable?”
Kushnir also describes PubMed’s help documentation as “inscrutable.” When I was teaching myself how to use PubMed, I found the documentation clear and helpful, so this surprised me. I decided to run the PubMed Quick Start document through Google Docs’ analysis:

Let’s review these scores:

Flesch Reading Ease: 62.97
(A score from 60-69 is considered “standard”)

Flesch-Kincaid Grade Level: 5.00
(Fifth grade)

Automated Readability Index: 5.00
(Again, fifth grade)

So it would appear that the help documentation is written at a fifth-grade level. I find it hard to believe that a PhD student at Harvard cannot read at a fifth-grade level, so I’m left with the impression that Ms. Kushnir didn’t actually attempt to read any of the documentation before declaring it “inscrutable.”

Suggestions for Ms. Kushnir and other research scientists who don’t like reading the instructions:

So the tool is necessarily complex because the data it searches is complex and the user refuses to read the well-written help documentation or accept help from a friendly librarian (even when multiple librarians are reaching out across thousands of physical miles of distance and the gulf of the patron’s unwillingness to learn).

I can only conclude that sometimes the user *is* broken.See Karen Schneider’s excellent post, “The User is Not Broken”.

Thank you to the two medical librarian friends who read the first draft of this post and offered comments.

25 thoughts on “When the user actually *is* broken (Anna Kushnir and PubMed)

  1. thank you David!
    many PhD students, even postdocs don’t know how to use PubMed right. This is truth.
    Even more, many of them don’t ask medical librarians about help or even don’t search manuals in the web.
    Thanks for indication to manual and tutorial. Very useful.

  2. I’m overwhelmed how much attention Anna gets. Next time I want to promote my blog I will bash some public service,too ;->
    She is in fact wrong – and she uses a blog service that requires registration to comment. I’d just ignore the angry little girl!

  3. This is a great post. I believe that many medical library staff (para and librarian) have encountered a similar patron. This was a weekly occurrence at Georgetown. It’s heartbreaking and challenging. There is such pride in this line of work it can often get in the way of the bottom line: teamwork. Teamwork of the researcher and librarian to get the needed information. Yet that slight sign of “weakness” by the researcher seems to be the biggest problem we have to solve. How do you do it and not offend them to the point of turning them away from a decent resource or worse yet the library as a whole?

  4. I’ve said this elsewhere, but it is nearly April, so you know what’s on my brain. Using Pubmed is a little like doing your taxes. Sure, it should be easy, because not only do “scientists” (meaning smart people) use it, but everyone. But it isn’t. And we generally don’t have a problem with that. We recognize that tax law is necessarily complex to accomodate the whole citizenry and their divergent needs. Many of us buy software to make it easier, but it still requires a significant investment of time on our part to do it. So why do we expect taxes to be difficult, and are not ashamed of our need to ask for professional help in doing it? If Kushnir hires an accountant to do her taxes, she should view that in the same way she would using a medical librarian to search the literature. If she instead chooses to do it herself, she should read the manual and be thankful it isn’t as impossible to decipher as the average tax instruction.

    And she should be thankful she isn’t in a dual income – dual citizen couple like myself who has to file in two countries. Ha!

  5. David,

    Thanks for the great post. I have patrons from undergraduates to postdocs and even faculty members once in a while. I also teach future Pharm-Ds every months since 2003. When I showed them how to search PubMed effectively, I got lots of wows: “I didn’t that. That’s wonderful!” The point was they would not bother to spend time reading manuals or tutorials and rather depend on their own way of searching. Medical librarians are challenged as to how to provide good service while not offend patrons.

  6. When I saw the student’s post percolating through medical blogs a week ago, I took a different approach and contacted Countway directly about it. They thanked me for the heads up & emailed the student directly about their services. We, the readers, cannot assume that the student was previously aware of the library or its services, no matter how unlikely that seems :) The student apparently chose to decline this personal connection from her home library and other resources, and after the way she was referred to elsewhere I can’t say I blame her. If comments regarding the age and intelligence of our users are posted by us on a publicly accessible listserv, how does that reflect upon us as medical librarians?

    However, I disagree that RTFM is an assumption that we as information professionals should place upon users seeking to navigate information systems. The one question I will never ask during reference interviews is “Did you check the catalog?” because this puts a presumption of prior knowledge upon the user who is actively seeking our assistance. By turning my screen to face the user and navigating with him or her we can then identify which path (if any) was already taken, then go forward from there to identifying the right resources.

    Posting additional information resources still doesn’t get to the heart of the matter: how can we make a human connection regarding online resources? That’s what I sought to do. Human nature prefers the familiar to the unknown. We all rant about or have an expectation that can’t always be met by information systems, that’s part of being human and information-seeking behavior of users in general, let alone specialized populations, is a complex field of study I have a lot of respect for. There are no quick & dirty fixes despite society’s expectations for them.

    I was inspired on Friday by Dr. Joseph Janes, founding director of the Internet Public Library, who spoke at our state medical librarians’ association meeting. In his closing remarks he offered a reminder that librarians make humanity more human, and our task is to organize and keep the human record. We need to be central to the information lives of communities, wherever those communities may be, and in an extended notion of library and librarianship we actually have to be better online than we are in person. Social protocols (usually!) guide traditional reference interaction on the phone and face-to-face, but they do not exist on the internet and users can click wherever they choose for information.

    I don’t think this online reference situation (let alone social protocol, would any of us say the things being said of this user out loud to her face?) worked out very well. What can we learn and carry forward from it? I know it’s unrealistic to expect that everyone will always be connected and instantly satisfied with library services. I’m still thinking through that one myself and would love to hear additional perspectives :)

  7. Nikki, thanks for the thoughtful comment.

    We, the readers, cannot assume that the student was previously aware of the library or its services, no matter how unlikely that seems.

    Agreed that it should not be assumed that “a student” understands what services an academic library offers.

    A PhD student, however, can be expected to have at least a little bit of a clue.

    Putting aside the idea that it is completely reasonable to expect a bio-science PhD to put an afternoon’s worth of effort into learning how PubMed works (whether self-directed or instructed by a librarian or colleague), Kushnir continued to whine even after qualified, generous librarians from at least four institutions offered her assistance and after she has declined their offers. She clearly didn’t want help- she wanted to kvetch.

    However, I disagree that RTFM is an assumption that we as information professionals should place upon users seeking to navigate information systems.

    Okay, but you’re disagreeing with something I didn’t say and don’t believe. Sometimes people need one-on-one assistance and that’s great.

    I don’t think this online reference situation (let alone social protocol, would any of us say the things being said of this user out loud to her face?) worked out very well.

    A library cannot help the user who does not wish to be helped and prefers to whine.

    What can we learn and carry forward from it?

    Sometimes the most important thing we can learn is that extraordinary helpfulness and expertise won’t make every patron happy. In this case, the library didn’t fail. The patron did.

  8. David, in response to “A PhD student, however, can be expected to have at least a little bit of a clue.”

    I have found that bench scientists are *extremely* hard to reach. I’m not sure why this is–maybe the nature of the lab culture lends itself to a certain insularity and self-reliance? And of course, clinicians are so money aware (don’t mean this in a derogatory way–but they have to be aware of reimbursements, patient load, etc), that when you offer to save their time they often can immediately see the bottom-line impact.

    That being said, now that the assistance of librarians has been offered and declined, I think it’s safe to say that she’s now *choosing* to remain frustrated, rather than making lemons into lemonade. Or something.

  9. David, thanks for the venue to discuss this issue :)

    A PhD student, however, can be expected to have at least a little bit of a clue.

    Why should we assume a PhD student should have more of a clue than a student in any other program? We’ve all had users who are knowledgeable in their respective fields and been resistant to accepting the fact that they need research help. I know of entire schools within universities who are resistant to medical library assistance as part of their apparent culture and/or administration. :( She was appreciative of the offers of help, and is expressing her position (whether we perceive it as reasonable or not) regarding her expectations of the system with admitted ranting. I don’t see this as a win/loss situation.

    My take on …if Kushnir had bothered to look at PubMed’s help manual or try some of its excellent tutorials… equated RTFM. My apologies if that’s not what was intended.

  10. Whoops, forgot to add, after my comment about clinicians, that academic biomedical researchers probably don’t see the time = money equation quite so starkly. I think they expect to be able to do all of their own research–even lit. searching–b/c that’s what their job is.

  11. Hey Erika-

    Agreed.

    Nikki-

    I don’t see it as a win/lose situation either- but I’m a little frustrated to see some of the librarian hand-wringing over this particular incident as though the librar* has somehow failed the user.

    I appreciate the people asking “what can we learn from this,” but have yet to hear a single suggestion of what else could have been done to help a user who chooses to remain frustrated.

    On the topic of what might reasonably be expected of a bioscience PhD student:

    When my wife was working on her PhD in Art History, we would have been rightly surprised and a little dismayed if she didn’t know how to load a slide carousel, use image-presentation software, make a slide list, or locate the present location of a great work of art. These skills are essential to her discipline.

    For someone to be a PhD student in the biosciences, to fail to seek out the skills required to search the biomedical literature, and to have rejected all avenues for gaining these skills is beyond my ability to comprehend or appreciate.

    And yes, for a PhD student to lack the wits to look for the “help” link on any Web page surprises me. Looking for “help” or a FAQ is something that a “computer-literate” or “Web-savvy” person does almost reflexively.

  12. I am the medical librarian at Harvard that Nikki contacted regarding the original blog post. Here is the message that I sent to Anna:
    Anna,
    I was pointed to your blog piece on Nature Network about your frustrations with searching PubMed. We are here to help and are happy to give you a one-on-one lesson in the power of PubMed searching if you would like. Please contact me and we can see if we can relieve some of that frustration. We also offer classes if you would rather attend one of those. Go to:
    https://www.countway.harvard.edu/classes#literacy to see if there are any that might fit your schedule. If you have any questions about any information resources here at Harvard please contact me (information below) or the reference staff by choosing the Ask a Librarian link at the top of the Countway home page (http://countway.harvard.edu).
    Here is her response:
    Thank you so much for your kind offer of assistance with PubMed searches. I will be sure to follow up on the resources you listed in your email. I so wish that I could devote the time necessary to learn the system well, however, at the moment I am shorter on time than I have ever been before – my dissertation is due in three weeks! I continue to be amazed by the response the post has generated and am very grateful for your time and attention.
    I agree with much of what has been said on this list but I also know that it is very difficult to convince people that either have Ph.D.s or are earning them that a librarian can teach them anything regarding their chosen area of research (even though I have a Ph.D. in Genetics myself). Partly this is due to the problem with the librarian image that has been around for far too long and partly due to the fact that trust is only earned – not freely given. The upcoming Google generation will be a challenge for all information professionals because it is very hard to teach someone something that they think they already know.

    David L. Osterbur, Ph.D.
    Public and Access Services Librarian
    Countway Library of Medicine

  13. Thanks for this comment, Dr. Osterbur.

    In your view, is there anything that your library or its staff should/could have done differently to better suit this particular patron’s needs?

  14. David I doubt much else, if anything, could be done for this particular user. I also don’t think a single person serves as an accurate measurement for the success or failure of anything regarding a librar*; be it staff, services or facilities.

    What I do hope is that information-seeking behavior studies are being done in the biomed research field (I haven’t gleaned MLA abstracts yet), along with evaluating the efficacy of library outreach programs. I highly recommend the NN/LM Outreach Evaluation Resource Program’s (OERC) publications for the latter but I’m a wee bit biased. ;)

  15. I also don’t think a single person serves as an accurate measurement for the success or failure of anything regarding a librar*…

    Agreed entirely.

  16. Wow – this is scary wrong.

    For example: “The index of biomedical literature searched from PubMed is a vast and complex set of data.”

    - It’s neither vast nor complex. Pubmed is about 18M records of a fairly uniform nature, each of which is semantically well structured. It’s probably no more than about 80GB in totlal – in other words it would fit onto the notebook omputer on which this message is being written.

    - Even if the dataset were complex there is no reason why the search tool should be complex, and no reason why you should need to go an a training course or read a manual to use the dataset.

    - Ms Kushnir’s point is that she doesn’t want to be offered help. She wants the resources to be made easier to use.

    - She’s right and I can’t help but feel some of the contributions illustrate why she’s right.

    - Karen Schneider has a nice phrase which summarises the discussion: “If we continue fetishizing the format and ignoring the user, we will be tomorrow’s cobblers.”

  17. It’s neither vast nor complex.

    Ben, everyone is entitled to his own opinion. Mine is that 18 million records that are indexed with a complex controlled vocabulary and lots of metadata can accurately be described as both “vast” and “complex.”

    Even if the dataset were complex there is no reason why the search tool should be complex, and no reason why you should need to go an a training course or read a manual to use the dataset.

    Please elaborate on how a tool might finely search this data set and not be complex? The NLM, I’m certain, will be anxious to implement any idea which is feasible given present technological capabilities.

    Ms Kushnir’s point is that she doesn’t want to be offered help. She wants the resources to be made easier to use.

    And I wish books to shelve themselves. They don’t and the technology that would allow for book-shelving robots isn’t in my library’s budget. In the meanwhile, it is a reasonable expectation of the person managing a one-person library that he shelve books.

    She’s right and I can’t help but feel some of the contributions illustrate why she’s right.

    You lost me there. What contributions are you referring to?

    Karen Schneider has a nice phrase which summarises the discussion: “If we continue fetishizing the format and ignoring the user, we will be tomorrow’s cobblers.”

    Except I’m not fetishizing the format. Despite the efforts of many to come up with better interfaces for the data PubMed searches, PubMed is still the best for most purposes. Is it not at all unreasonable to suggest that a PhD student should spend the little time it takes to learn to use the primary research tool of her field.

    I’m all for making interfaces better and more intuitive. I caught flack for expressing the belief that a library’s OPAC should be as easy to search as amazon.com. What I find “scary wrong,” Ben, is hearing librarians make sweeping generalizations about “this should be better” without offering a single idea about how to accomplish it.

  18. Wow! There have been a lot of thoughts flying around about that blog post.

    Here are some more.

    First, remind me never to vent on a blog after a particularly frustrating experience.

    Second, I think it is important that people are able to say “this ain’t right” even if they are unable to put forward a solution. For example, even if I can’t produce a photovoltaic cell, I should still be able to say we ought to be able to harness the sun’s energy. Sometimes, I’m in a position to say something is amiss. Other times, I’m in the position where I should be listening to someone saying something is amiss. (I would even go so far as to say people who don’t have a solution may be the best people to point out a flaw in the system as they don’t have any preconceived notions on what the limits are.) I hate to sound managemental, but to ignore or condemn someone pointing out a flaw is to miss out on an opportunity.

    Finally, I think this situation illustrates a mixed message given by libraries. Patrons are steered toward self-reliance. When that goes awry, suddenly the message becomes “why were you trying to do this by yourself?” I realize librarians are put between a rock and a hard place on that, but, from a patron’s perspective, it seems a bit inconsistent.

    Now, I’ll go back to my work space and be quiet.

  19. I think it is important that people are able to say “this ain’t right” even if they are unable to put forward a solution.

    In general terms, sure.

    In this case, no.

    If Anna or Ben have nothing more to offer than “it should be easier,” they should perhaps consider that they’re preaching to the choir.

    Everyone would like PubMed to be easier- but unless they have specific ideas on how to make that happen, their kvetching just makes useless noise.

    Patrons are steered toward self-reliance. When that goes awry, suddenly the message becomes “why were you trying to do this by yourself?”

    A good point, but again, that’s not what happened here. Anna was given both excellent options (documentation available that is clear to the average 5th-grader or instruction from one of many competant, friendly librarians) and rejected each in turn.

  20. Pingback: Llyfrgellydd » Those Harvard Ph.D.s

  21. Anna is an educated, intelligent, and thus incredibly stubborn person incapable of realizing that the world is not designed around her needs and wants. Anna wants the world to be just like Google. She felt she didn’t need the help section because she is a PhD student and researcher.

    She mistakenly wrote her post when angry. The system did not read her mind and do what she wanted it to do, so therefore it’s horrible. As a result, she appears to be a spoiled brat.

    I’m sure she is, in reality, an fairly intelligent woman under a lot of pressure. However, she herself summed up the situation so very well: “The upcoming Google generation will be a challenge for all information professionals because it is very hard to teach someone something that they think they already know.”

    All librarians will face this challenge, and everyone will be approaching it from a different angle. We work in different environments so that’s okay.

    I feel sorry for Anna. She will continue to err by thinking the world should be nothing more than a big Google search engine. This will be a detriment to her if she plans to continue on her chosen career path as a professional writer and researcher.

    Librarians and others have reached out to her. Now, it’s her turn to come to gripes with something the best authors have known for years: assistance is a great thing.

    Thank you for proving a place for us to comment on your article.

  22. Pingback: PubMed blijft de gemoederen bezig houden « Debibliotheker’s Weblog

  23. The analogy of learning to drive a car is good, but I think I have a better one: she probably uses Microsoft Word in her work. I’ve been using it for years, and I still have trouble with it. Does that mean it’s a bad program? No, it means that it has lots and lots of useful features that take a while to learn. Every time I need to put in a footnote or a running head (or show someone, since I rarely do those things for myself), I have to look around to find them. Sometimes I even (*gasp*) use the help menus. At home I use OpenOffice.org, which I prefer to MS Word, but it too takes a while to learn. If you want to use MS Word to its fullest capacity, or use even a few of its advanced features, there’s going to be a learning curve. It’s the same with PubMed. For that matter, even Google has a few tricks up its sleeve.

  24. Given that it’s now routine to examine the web presence of prospective new hires, I’m not sure which would be worse to hear at an interview: “oh yeah, you’re that kid who couldn’t figure out PubMed” or “oh yeah, you’re that kid who couldn’t be bothered to figure out PubMed”.