As we at Silverchair and Semedica see more and more interest in automated tagging solutions (such as our Tagmaster system), we are more frequently encountering questions about how to evaluate their results. Here are a few ideas on the subject:

Evaluation: Humans Required!

It is hard to get around the fact that you will need human editors (or professional indexers) and your human technology team (who will use the tags to create interesting new features) to verify that an automated system is working correctly and that the tagging is accurate and useful. 

Recently, someone asked our CEO Thane Kerner if we had an automated system to verify the accuracy of our automated tagging. Thane replied (rather cheekily, I must say): “If we had an automated review system that could measure tagging accuracy more precisely than the current tagging system, we wouldn’t use it to verify tags, we’d use it to tag the content to begin with!” The lesson: Once you’ve deployed your best automated system to do the tagging, humans are the next logical reviewers. 

Here are four factors your humans should consider in their review:

View inside Semedica's Tagmaster, showing tags automatically inserted at the paragraph level

1.  Expert/Editorial Accuracy Confidence

One key target for evaluation is to assess how much confidence your key stakeholders (journal boards, editors, etc.) express in the output of the system. But confidence is not a linear equation. I posit the following values:

• Impeccable tag placement: +1
• Debatable tag placement: −1
• Debatable tag omission: −1
• Obvious tag omission: −10
• Obvious irrelevant tag placement: −50

The first thing you’ll notice is the weight of positive to negative. In high-stakes fields (including science and medicine), humans are naturally biased to more heavily favor negative experiences.  (Of course, this has aided us well in survival: “Don’t eat that type of berry again, it made you sick last time!”) What that means in terms of confidence is that stakeholders will need a disproportionate amount of positive reassurance to get over negative outcomes. And the impact of a particularly egregious negative outcome (resulting from a particularly poorly placed tag) can be devastating to your stakeholder’s impression of a tagging system. (This is why Silverchair’s system defaults to using conservative methods with very little “guessing” to avoid obvious irrelevant tag placement.) 

2.  Usefulness!

The next key target for evaluation for both editorial and technical stakeholders to assess is usefulness of the tagging applied. Tags should be highly relevant in a domain-specific context and they should drive better discoverability and linking. Primary care, genetics, surgery, and emergency care all take very different approaches to the same topics, and their tagging should reflect their uses. 

The tagging system you are evaluating may have added tagged concepts that are tangential or irrelevant to the use model of the content, and such tags would not be capable of driving innovative site features (in many cases, tangential tagging actually inhibits the ability for new systems to work effectively). For example, it is a nice-to-have if your tagging system can recognize place names and person names, but if it misses or miscategorizes important topics like clinical trial names it doesn’t matter how many people or places it can tag. (Clinical trial acronyms can be particularly tricky to tag―see our post about them.)

3.  Granularity

Does the system still work with “documents” or can it identify topics down to the section/paragraph/figure/table/equation level? At Silverchair we work with many dense medical chapters that may cover more than 200 distinct topics, so we see it as a necessity for our tagging system to break those documents down into smaller parts in order to deliver precise packets of highly relevant information to our users.

4.  Control and Ongoing Improvement

Any system selected is not going to be extremely accurate “out-of-the-box.” (I write that as a realist, not as a pessimist!) So during evaluation you must ask, “How easy is it to make impactful positive changes to the system?” This can take a variety of methods—some systems suggest manually selecting training documents for each topic or category (which can get onerous when you have 20,000 topics), some systems allow your software developers to go in and tinker with the code (you have data classification expert software developers, right?!?), and some systems allow you to load and use a taxonomy or thesaurus to aid in topic identification and tagging (assumes a taxonomy/thesaurus exists or can be created for your domain).

At Silverchair, we work primarily in medicine, which is a taxonomy-rich domain with an ever-growing list of topics. For that reason, we’ve chosen the last method as our control and improvement strategy. Our editors update our Cortex medical taxonomy and its related thesaurus every day to keep pace with the topics being written about and searched for. 

Summary

If you choose a system that 1) is accurate enough to instill confidence in your editorial team, 2) is useful enough to drive meaningful new features and improvements, 3) classifies your data at a granular level, and 4) is flexible enough to allow explicit control and ongoing improvements―you’ve made a wise purchase!

The web servers are designed to execute application logic, store session tracking data, handle user interaction input, and synthesize, parse, and display data from a variety of sources—they are logic processing engines that handle data storage only when necessary. On the other hand, the SAN has one purpose—to store a large amount of data and enable a super-efficient data delivery channel that rapidly responds to content requests from the web servers.

The more I thought about it, the more I realized it was a fitting metaphor for how humans work. We are fantastic logic processing engines. We parse, synthesize, analyze, and use data input from a variety of sources to perform creative problem solving. And most importantly to this metaphor, we only store data internally when absolutely necessary. In the present day, the comprehensiveness and ubiquity of the Internet have allowed us to store an unprecedented amount of collective memory in external sources and access it from wherever we may be.

To be clear, human use of external memory did not arrive with the Internet—it has been around since the beginning of civilization. We are used to storing memory in external sources and freeing up our internal resources. Papyrus eliminated the need to memorize long epic poems. Abaci eliminated the need to memorize multiplication tables. (NB: Don’t try telling that to a 2nd grade teacher.) In modern medicine, drug handbooks store dosage and safety information that is too complex for doctors to memorize in toto. Phone numbers stored in our mobile phones eliminate the need to memorize the phone numbers of friends. We even store memories in our friends and family—I recently asked my wife, “What was the name of that hotel we liked in Chicago?” She knew, and voila, I had accessed my external memory successfully.

Alas, my comparison of human activity to Silverchair’s web farm breaks down at a key point. In many cases, accessing our external memory is not fast and efficient. Currently the external memory sources of humans are not deployed as efficiently as a SAN. Internet content sources can be hard to access, store content in highly variable forms, require a special vocabulary or technique to query, and return data in a way that does not suit our purpose.

This is the fundamental problem that Silverchair’s Semedica division addresses with semantic enrichment of data sources. We’re organizing a specific external memory category (in our case, online medical and health care information) in a way that allows it to be accessed more quickly and to return data in the right form for efficient use by clinicians and researchers. The less data that health care workers need to store internally, the more of their “processing time” can be used toward envisioning creative solutions for preventing and curing diseases. That is something that the Internet cannot do. (Yet.)

Image via Wikipedia

The NIH has rolled out their new RePORT (Research Portfolio Online Reporting Tool) web site for information on funding, grants, and NIH research. As someone who works on government grants and contracts, I’m happy with this new level of transparency and clarity as to what topics (and who!) is being funded. It is a big upgrade from the incumbent system, which was hard to navigate and understand.

The most useful area of the site to me is the categorical spending section. It really gives you an idea of NIH’s funding priorities—it offers over 200 categories of funding.

However, it still has ample room for improvement. Currently it is an alphabetical list that contains items that are hard to compare. Here are some example categories that are not equivalent in scope:

• Allergic Rhinitis (Hay Fever)
• American Indians / Alaska Natives
• Burden of Illness
• Cancer
• Cardiovascular
• Clinical Trials
• Conditions Affecting Unborn Children
• Gene Therapy
• Gene Therapy Clinical Trials
• Genetic Testing
• Genetics

Some are very specific (hay fever), some are broad (cancer), some are ambiguous (cardiovascular), some take a completely different approach than the dominant disease/condition approach (American Indians/Alaska Natives), and some seem to be repetitive.

With a bit of work, this information could be turned from its current flat list expression into a multilevel taxonomy that allows users to slice it up in the ways that appeal to them (conditions or target populations, for example). Silverchair does this for the Agency for Healthcare Research and Quality on their PSNet patient safety clearinghouse. A small amount of classification work can go a long way in creating valuable new features—NIH has proven that with their RePORT upgrade, but I’d like to see them go farther.

I’d be happy to help out with the NIH site, but I’m not sure what category that would be funded under…

• Medical Terminology: A Short Course
• Quick Medical Terminology
• Medical Terminology: The Basics
• Medical Terminology Simplified
• Medical Terminology for Dummies

Anyone else sensing a theme? Considering that the Unified Medical Language System (UMLS) has more than 2,000,000 terms, I’m not surprised simplicity is in demand.

For publishers, taking measures to make medical terminology (and life) easier for health care professionals has a direct payback [re-read list, above]. Consider it mission critical!

Map created by Robert Lewis Stevenson in Treasure Island (image from Wikipedia)

When good authors write, they choose the terminology they want to describe the topics they are addressing and use that terminology consistently throughout the text. This, of course, is good for readers in terms of internal clarity and consistency.

But this authoring strategy is distinctly disadvantageous to discovery (search) and integration (linking) in modern web applications. Why? Because every time an author makes a terminology choice they EXCLUDE other equivalent options. These excluded options could include terminology that other authors have chosen or are the preferred terminology of their potential readers. I’m not blaming the authors, of course—their writing would be nonsense if they included all equivalent choices in their text.

So how do you deal with these missing options? Thesauri to the rescue! Every web search, linking, and categorization system should employ some form of thesaurus behind the scenes. And in specialized areas like medicine, you’ll need a specialized thesaurus rather than a basic broad one. This thesaurus should include synonyms, acronyms, abbreviations, and jargon, and should be based on real-world authoring and searching behavior (rather than academic nit-picking).

In essence, a thesaurus expands the author’s original text into much richer data for automated searching and linking algorithms. Let’s look at an example:

ACTUAL TEXT: Chemoreceptors in the carotid bodies and medulla are activated by hypoxemia, acute hypercapnia, and acidemia.

EXPANDED TEXT¹: Chemoreceptors in the carotid bodies (carotid glomus, glomera carotica, glomus caroticum, glomus caroticus) and medulla (adrenal medulla, medulla oblongata, glandula suprarenalis, suprarenal medulla, adm, metepencephalon, medullary, myelencephalon) are activated by hypoxemia (hypoxaemia, arterial hypoxemia), acute hypercapnia (blood carbon dioxide increased, blood co2 increased, carbon dioxide retention, carbon dioxide, increased level, hypercapnemia, hypercapnaemia, hypercarbia, pco2 increased on arterial blood gas, elevated pco2, retention carbon dioxide, serum carbon dioxide increased), and acidemia (acidaemia).

The actual text as written was 15 words. The expanded text was 71 words, or approximately 4.7 times longer. Humans read the first sentence, and machines read the second.

No matter how a user searches for this text (“hypercapnia” vs. “hypercarbia,” for example) they will match this text with a good thesaurus.

Are readers finding what they want on your web site this easily?

¹Thesaurus Source: Silverchair’s Cortex taxonomy—with references to SNOMED, Read Codes, MeSH, Digital Anatomist, NCI Thesaurus, NeuroNames Brain Hierarchy, MedDRA, WHO Adverse Reaction Terminology, OMIM, DXplain, CRISP Thesaurus, Clinical Problem Statements, and COSTART.