Very preliminary concept (Get paid to do science)

[PawShaker]

A quick draft writeup. I think better when I put ideas to keyboard.

https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit

[organofcorti]

Wikipedia is human language, which can contain other things beside truth. If you farm out eg data processing in a picemeal fashion, you'll be able to see if some part of it differs significantly from others. So when you figure out who screwed up you can mark them down or if it's a malicious error throw them out.

You'd need a 'web of trust' type identifying system. Management doesn't have to know who workers are, as long as someone trusted in the chain does. This would also double as a referral system too.

I also see this as being handy for collaboration. Don't have anyone at your job interested in the work you're interested in? Pay a fee and get matched with someone with your interests. For example, if you're great at analysis but crap at writing it up, you might be able to find someone familiar with your area to get the written parts ready for publication, either for a fee or for coauthor credits or both.

[BinaryMage]

A quick draft writeup. I think better when I put ideas to keyboard.

https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit

That looks promising in general, but the power of one SC to affect the final product seems excessive. (Only two must approve) Perhaps a less linear process with a more distributed system would work better. (Multiple SCs submit revisions or "no revision", they all vote on the preferred option)

What system would you suggest for dealing with users with large amounts of rejected work?

[PawShaker]

[...]
That looks promising in general, but the power of one SC to affect the final product seems excessive. (Only two must approve) Perhaps a less linear process with a more distributed system would work better. (Multiple SCs submit revisions or "no revision", they all vote on the preferred option)

What system would you suggest for dealing with users with large amounts of rejected work?

In theory instructions should be so precise that process should be deterministic. Anyone following instructions should (theoretically) arrive at materialy same answer. This is supposed to be a scientific observation, hence reproducible.

In practice human factor is not that easy to account for. Hence need for some arbitration and quality control. SC are supposed to be more experienced and cases when their verdict would be questioned will be so rare that they will not bias the study.

I must admit that I wrote spec with corporate context in mind. I could use some lateral thinking and explore a self-organising network of humans.

Well SC has power to deny reward for inadequate work. In practice such workers drop out very quickly. Of course, repeated offenders will be suspended. Depending on the supply of recruits, suspended users will be retrained or dismissed.

[BTCurious]

Wouldn't it also sometimes be an option for the process not to be deterministic? For example, "Use any software to count the cells on this set of images". The worker would have to say what he did, give the numerical results, and also supply the processed images*. This way the researcher can easily see if the work has sufficient quality, without having to do all the manual tweaking of edge-cases. The idea here is that checking if something is done correctly takes a lot less time than doing it yourself.

*Normally software such as this can show the outlines of the cells it detected, for example.

[bb113]

I am glad there is interest in this idea. Here is an example of a simple task that must be repeated hundreds of times. I used some of the terminology PawShaker provided here:

A quick draft writeup. I think better when I put ideas to keyboard.

https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit

Fullsize here: http://i49.tinypic.com/fwuiz8.png

[bb113]

So I was at the zoo the other day and talking to one of the guys running the research aspect of it. Apparently they are working on something similar to this idea. They are trying to devise a way to have zoo-goers report on animal behaviour they observe and filter out the nonsense. The current strategy is to make it part of an ipad app that provides facts about the animals. It sounded like there was a threshold of "learn more" buttons that need to be clicked before the data would be taken seriously.

Anyway I think it is promising that people are beginning to think along these lines.

[viperzero]

I have been also toying with the idea what happens after the GPU-mining comes totally unprofitable. I mean not to hijack this thread but so called classification etc. "mundane" classification tasks can be in my opinion be rather easily solved with neural networks. So my suggestion is to make rather an distributed computer cluster where "mining rigs" can join. Environment could have ready make neural networks etc. plus ability to upload&train your own. Then any researcher could simply buy cluster time. WIN-WIN solution. Plus pretty affordable.

[someguy123]

This is a pretty awesome idea, people will get to learn more science, get paid, and also contribute further to science at the same time. It's a win:win for both sides almost.

[stochastic]

and get paid $5-$10 for a job well done. Others could donate to this site, somehow invest in it and filter out bad submissions. The payout could be a function of the number of submissions and there would be extra rewards for more productive contributors...

You should lower the payout.  At $5 to $10 a job you are paying graduate student wages.

Of course if you marketed it as a way for graduate students to procrastinate then they could double their paychecks.

[bb113]

I have been thinking more and more about this concept and have been convinced that the existence of something like this would be a huge benefit to humanity. However, I have another year or so of work to complete my current project before I can start working on it, and after that... it depends on what happens. If someone else creates something along these lines I will contribute as much as possible. This is really a $15 billion a year industry just sitting there waiting to be taken from academia.

[bb113]

Also, in my mind... this would just be bringing back the days of "human computers", to do the jobs that noone is currently doing because the people in charge of the projects have neither the time nor resources to really have someone examine their data to the fullest extent. The result is a millions of monkeys on typewriters approach, half assing everything. I think we can do better with a crowdsourced approach, even with fraud and noise to filter out.