Crowdsourcing Medical Expertise Part Two: Reader Comments

Every once in a great while I find the comments to a post so thoughtful and intriguing I decide to "upstream" them into their own blog post, both to draw more attention to what I deem to be valuable points, and as an attempt to provoke more debate. A few days ago I received an email from a Louisiana-based writer and consultant named Louise Maynard. Having endured contradictory interpretations of her own MRI scans, she wondered whether their might not be a crowdsourcing application in, essentially, amateur radiology. In this case, the comments made for better reading than my original post.

The implications of crowdsourcing the act of diagnosis are troubling, problematic and also tantalizing. To some extent, medical expertise is already being crowdsourced, as the commenters pointed out. Nalts pointed out that the site CrazyMeds runs a very active forum in which people gather to discuss the ups and downs of the medicated life. As Nalts notes, he'd rather trust the consensus regarding a drug formed by 100 patients than a single doctor. (Ed's Note: It's also worth reading the witty and poignant personal essay on the front page.)

Jen McCabe Gorman notes that the site American Well poses an interesting case study in that it vets medical professionals and then allows them to offer their services online, which isn't far from what Maynard is proposing

I was also happy to see Alpheus Bingham—founder of InnoCentive and a veteran scientist (he holds a Stanford PhD in organic chemistry) in his own right—join the conversation. He aptly problematizes the prospect of crowdsourcing diagnosis: "Of course, it goes without saying that many 'portions' of the overall healthcare process (including research objectives) can and should be crowdsourced for a host of reasons that would improve outcome and quality," he writes. As most of the readers of this blog realize, this is an essential component of what companies like InnoCentive, NineSigma and YourEncore do—broadcast research problems to large groups of experts and non-experts alike in the hopes of unearthing novel solutions. However, "the specific issue of crowdsourcing diagnosis or treatment is far less clear."

On the one hand, the numbers would seem to favor such a project. Here's Alph:

An expert (say, right 95% of the time) is wrong 5% of the time. An amateur might be wrong 20% of the time, but the chance that two amateurs are both wrong is only 20%x20% or 4%. So two "informed" amateurs consistently reading a scan or collection of lab results has a pretty good record. Hmm, seems like a clear case FOR the crowd...

Ah, but the numbers can deceive:

Of course, as the number of semi-professionals or informed amateurs goes up, the chance of them ALL being wrong goes down, but the chance of getting mixed diagnoses goes up very fast. (73% say "benign" and 27% say "malignant.") What to do then? Majority rules? Supermajority required?

When is the "vote" compelling enough to stake your treatment on it? It would be well outside the scope of a blog comment to delve deeply into this. But let's briefly return to our two amateurs. When they agree there is a 96% chance of them being right. But how often do they agree? If they are looking at a cancerous scan, with an 80% individual accuracy rate, they agree on the cancer diagnosis only 64% of the time. They split opinions (a very confusing state of affairs since they are equally likely to get it right and now you don't know who to believe) 32% of the time (that's a lot) and they both get it wrong only 4% (as we said already). There may well be some sophisticated statistical analyses that would supplement such crowdsourcing approaches -- BUT -- 'experts' or 'crowds' or 'crowds of experts,' there will remain ambiguity when dealing with judgement calls. Our penchant for certainty is just not going to get fully satisfied.

This is, to some degree, the situation we're already in (albeit with very small crowds) when several professionals are asked to interpret medical data (such as an MRI scan): Ambiguity. That said, in the end Alph makes a strong case for the exploration of a crowdsourcing effort that would entice the semi-professionals to engage in such a project: "Total non-experts (the masses referred to in the post) do NOT help matters as their input is just noise -- obscuring a signal. But the crowd of semi-experts could well be, in my opinion, desirable, and we should investigate appropriate systems and knowledge aggregation tools for its exploitation."

Finally, Daniel Reda and Alexandra Carmichael, the co-founders of the very promising CureTogether.com, both posted useful comments. Here's Daniel: 

What would happen if you crowdsourced interpretation or even diagnosis? Well, the consensus interpretation of 100 amateurs on your MRI would probably not be at all helpful. What you'd want is a method to select the best interpretations and have them bubble up to the top. How do you select the best interpretations? One way is to keep historical data on how accurate those predictions were once more data became available. Ideally we'd gather data on doctors' performance as well. It's not about credentials - it's about accuracy. If the doctors don't want to participate, then their judgments will look progressively weaker compared with those of a supposed amateur who was proven to be correct 99% of the time on thousands of MRI interpretations.

"It's not about credientials—it's about accuracy." That's as concise a definition of crowdsourcing as it gets. Here's hoping we can keep this conversation going.

Comments

[Generic Viagra]

Many people are grappling with obesity problems nowadays. Despite of so many measures this problem still persists. The only reason is laziness. Walking, Jogging and exercise etc. they all require physical exertion and time, today’s people are not ready for it because of their busy schedule.

[Damier Azur Canvas]

I want to take a brief break from Gannett coverage (I'm under slept, having stayed up half the night watching Montana returns roll in.) Last week I appeared in my first virtual Q&A, at a Wired event in our new Second Life digs. The topic, naturally, was crowdsourcing in general and how Linden Lab – the company behind SL – has used crowdsourcing to great advantage. It was great fun, and the audience asked several thoughtful questions, such as Mordant Kepler, who wondered whether minimum wage laws should apply in crowdsourcing. My reply, somewhat abbreviated:

I think there are policy frameworks that we haven't begun to work out. If I'm a turker in Bangladesh, what's my minimum wage? If it's in US dollars and the requesting company is incorporated here, I think the answer would be our minimum wage, but I'm not even sure you could enforce minimum wage on piecework like that, which explains why some people are -- rightfully -- worried about crowdsourcing's implications.

[Caverta Online]

Hey really this is very interesting post here... This is very informative blog found. I have enjoyed to be reading it. keep posting I'm looking forward to your blog.

[New Balance Shoes]

The study found that people with depression or anxiety were more likely to associate their mood with the color gray, while happier people preferred yellow.
http://www.yaahshoes.com/

[Rose Faith Sialonga]

Wow!excellent!keep sharing articles like this..
The Writer is great!Has anyone tried a medical call center??
http://www.pigeonsandracingpigeons.com

[Range Cookers]

You have made some really good points, thanks for sharing this information.

[edegra 100mg]

As I am Medical Student I always like to read about health sector.
I think it is a good place for health articles.

Smith Alan

http://www.raymeds.com/blog
http://healthcaretips.typepad.com/

[Medical Accidents]

Hello
i medical student .I have read your article.It consist of very useful terms.I have just bookmarked your site so that i can regularly see your updates.

[Generic Viagra]

This is where you can ask a nurse questions about minor medical problems.

[Caverta Online]

Hey really this is very interesting post here... This is very informative blog found. I have enjoyed to be reading it. keep posting I'm looking forward to your blog.

Jenny

[fibator]

Thank you for sharing to us.

Please one more blog about that

[supra shoes]

I agree with your idea.You look like very talented.It is very happy to meet you. Thank you!

[Generic Viagra]

Thanks for sharing your ideas. I was desperately waiting for such posts. I really appreciate your efforts and I will be waiting for your further write ups thanks once again.

[dugi's guide]

Hi, great article. The way you explained it is really awesome and makes every one to read till the end. keep posting..

[Magnetic Generator]

Crowdsourcing is just a fancy word for outsourcing, isn't it? I mean the concept is old, but with the advent of the internet it just means you can find people very quickly?

[Generic Viagra]

You portrayed the topic well.. Especially the concluding paragraph and the mid section made a good read... keep posting...All the best.

[bluepharmacyrx]

Ever since I read about the subject in this article, my thirst for knowledge has tripled and I want to know more about it from various online and offline sources. Credit should be given to the article writer for putting across the world in a clear and concise matter and the ability to enthrall us giving noteworthy examples. This is how an article should be written. Great Job.
http://www.bluepharmacyrx.com/generic-viagra.html

[generic viagra]

I surely believe what my friends say, because you have a fantastic insight on the topic. The information is surely very entertaining and even helpful in the real life when it comes to solving any problem.

[elsewyoung]

good .http://www.buyndsl.com/electronics/cell-phones-1605.html

[discountcigarettesbox]

This is, to some degree, the situation we're already in (albeit with very small crowds) when several professionals are asked to interpret medical data (such as an MRI scan): Ambiguity. That said, in the end Alph makes a strong case for the exploration of a crowdsourcing effort that would entice the semi-professionals to engage in such a project

[Online Generic Viagra]

This is a great article post. Nicely explained it's awesome. Thanks for sharing the blog....Facebook enhancement for drunk driver provides notable example of social-media sentencing
Keep Posting.

[savsuns]

So far thousands if not hundreds of thousand Singaporean had complained about housing policy and instead review his own policy, he is pointing the 'GUN' to WP as irresponsible. I think Mr Mah should have a self-reflection and not living in his own world.
Lasix Generic

[Air Jordan 1]

Hello! I ran into your site utterly by mistake, and am really glad I did! This is very interesting, and I can tell you have spent some time studying this. Thanks!

[hcg Drops]

I agree with this sentence- "The implications of crowdsourcing the act of diagnosis are troubling, problematic and also tantalizing."

[open innovation management]

this is really interesting article