Every once in a great while I find the comments to a post so thoughtful and intriguing I decide to "upstream" them into their own blog post, both to draw more attention to what I deem to be valuable points, and as an attempt to provoke more debate. A few days ago I received an email from a Louisiana-based writer and consultant named Louise Maynard. Having endured contradictory interpretations of her own MRI scans, she wondered whether their might not be a crowdsourcing application in, essentially, amateur radiology. In this case, the comments made for better reading than my original post.
The implications of crowdsourcing the act of diagnosis are troubling, problematic and also tantalizing. To some extent, medical expertise is already being crowdsourced, as the commenters pointed out. Nalts pointed out that the site CrazyMeds runs a very active forum in which people gather to discuss the ups and downs of the medicated life. As Nalts notes, he'd rather trust the consensus regarding a drug formed by 100 patients than a single doctor. (Ed's Note: It's also worth reading the witty and poignant personal essay on the front page.)
Jen McCabe Gorman notes that the site American Well poses an interesting case study in that it vets medical professionals and then allows them to offer their services online, which isn't far from what Maynard is proposing
I was also happy to see Alpheus Bingham—founder of InnoCentive and a veteran scientist (he holds a Stanford PhD in organic chemistry) in his own right—join the conversation. He aptly problematizes the prospect of crowdsourcing diagnosis: "Of course, it goes without saying
that many 'portions' of the overall healthcare process (including
research objectives) can and should be crowdsourced for a host of
reasons that would improve outcome and quality," he writes. As most of the readers of this blog realize, this is an essential component of what companies like InnoCentive, NineSigma and YourEncore do—broadcast research problems to large groups of experts and non-experts alike in the hopes of unearthing novel solutions. However, "the specific issue of crowdsourcing diagnosis or treatment is far less clear."
On the one hand, the numbers would seem to favor such a project. Here's Alph:
An expert (say, right 95% of the time) is wrong 5% of the time. An amateur might be wrong 20% of the time, but the chance that two amateurs are both wrong is only 20%x20% or 4%. So two "informed" amateurs consistently reading a scan or collection of lab results has a pretty good record. Hmm, seems like a clear case FOR the crowd...
Ah, but the numbers can deceive:
Of course, as the number of semi-professionals or informed amateurs goes up, the chance of them ALL being wrong goes down, but the chance of getting mixed diagnoses goes up very fast. (73% say "benign" and 27% say "malignant.") What to do then? Majority rules? Supermajority required?
When is the "vote" compelling enough to stake your treatment on it? It would be well outside the scope of a blog comment to delve deeply into this. But let's briefly return to our two amateurs. When they agree there is a 96% chance of them being right. But how often do they agree? If they are looking at a cancerous scan, with an 80% individual accuracy rate, they agree on the cancer diagnosis only 64% of the time. They split opinions (a very confusing state of affairs since they are equally likely to get it right and now you don't know who to believe) 32% of the time (that's a lot) and they both get it wrong only 4% (as we said already). There may well be some sophisticated statistical analyses that would supplement such crowdsourcing approaches -- BUT -- 'experts' or 'crowds' or 'crowds of experts,' there will remain ambiguity when dealing with judgement calls. Our penchant for certainty is just not going to get fully satisfied.
This is, to some degree, the situation we're already in (albeit with very small crowds) when several professionals are asked to interpret medical data (such as an MRI scan): Ambiguity. That said, in the end Alph makes a strong case for the exploration of a crowdsourcing effort that would entice the semi-professionals to engage in such a project: "Total non-experts (the masses
referred to in the post) do NOT help matters as their input is just
noise -- obscuring a signal. But the crowd of semi-experts could well
be, in my opinion, desirable, and we should investigate appropriate
systems and knowledge aggregation tools for its exploitation."
Finally, Daniel Reda and
Alexandra Carmichael, the co-founders of the very promising CureTogether.com, both posted useful comments. Here's Daniel:
What would happen if you crowdsourced interpretation or even diagnosis? Well, the consensus interpretation of 100 amateurs on your MRI would probably not be at all helpful. What you'd want is a method to select the best interpretations and have them bubble up to the top. How do you select the best interpretations? One way is to keep historical data on how accurate those predictions were once more data became available. Ideally we'd gather data on doctors' performance as well. It's not about credentials - it's about accuracy. If the doctors don't want to participate, then their judgments will look progressively weaker compared with those of a supposed amateur who was proven to be correct 99% of the time on thousands of MRI interpretations.
"It's not about credientials—it's about accuracy." That's as concise a definition of crowdsourcing as it gets. Here's hoping we can keep this conversation going.
