Bitcoin Forum

I have been trying to come up with ways to reduce researcher reliance on government funding. Be warned, this is just a really vague concept at this phase.

These would be tasks done at a computer like manipulating images, tracing complex structures, proof reading journal drafts, etc that are not yet easily automated. For example, the researcher would upload image stacks taken from histology slides (for example a brain), and someone would download them along with some software, then spend an hour or two tracing the arbor of a neuron and get paid $5-$10 for a job well done. Others could donate to this site, somehow invest in it and filter out bad submissions. The payout could be a function of the number of submissions and there would be extra rewards for more productive contributors. Alternatively the researchers themselves could put up some money for a task.

Can you change this to a multiple-option poll?

Might be cool to mix education and 'menial' online labor. Like they teach you about brain structures and then you practice your knowledge by actually learning. Maybe use pay to sort out surpluses if needed. It would be cool to have something like coinworker, but more advanced, like maybe you need a some skill and/or to take a 5-60 minute lesson.

I don't see an option to do that in the edit poll menu. You are right that it should be multiple option though...

Thanks for adding more options, that certainly helps.

Absolutely. I love the concept, and I would be happy to volunteer some of my spare time for free to dedicate to a noble cause. Some extra education would be the icing on the cake.

This is a good idea. It would be like the practice questions at the end of the chapter, except they matter... and you get paid for doing it "right".

And I suspect, even for low pay, it would be quite popular. People get very excited at the prospect of "free" money.

Right, even if they don't care about science, once people learn they can go and do something for an hour or two and get 10 dollars when they need it, they will return before going out to the movies or whatever. I agree that a substantial number of people would be willing to do low-paid, short-term "contract"-type work for a noble cause. There would also be a subset that is actually interested in the results and context of what they are doing and be willing to donate time for free to at least rate the submitted work after they have completed the task themselves or attained a certain level or something like that.

Initially there would be seed funding from somewhere (perhaps even a gov't grant), but the goal would be to have it become self-sustaining by gaining funding either through advertisements, donations, lessons/lectures, or some other incentive.

I bet you could garner enough support in the form of donations/advertisements from the Bitcoin community to start it off.

I actually think this type of grant would get funded, and be worth funding, by NIH, etc. There are many experiments that do not get done right because a grad student needs to do what amounts to basically the same thing 1000 times and after 30 they have learned all they can from it and are "supposed" to be publishing and moving on with their career, so they stop at 200 times or so when the study is powered to get p < 0.05. This is accepted because everyone who reviews papers and grants understands that in practice, you need to publish as often as possible to continue your career. In my opinion, this practical, but resulting half-assed approach is honestly a cultural plague beset upon science. This is especially the case for animal research that needs to be replicated with small variations time after time because no one is willing to fully analyze the resulting tissue. You can argue the pros and cons of animal research but currently 99% of the animal gets wasted because noone has time to analyze it. It really is a wasteful, unethical system right now.

I have seen a system for transcribing recording into words. The biggest hassle is constant need to train new people. You will be surprised how hard is to train a person who is willing to be paid low end rates. The retention of people who learn quick is not high. The end of the story was that pay rates went up to retain people.

There is half-joke about an ad for a person with PhD with appropriate level salary to cut leaves into small pieces. When one applicant realized what task he is going to perform, he asked recruiter why they need a person with PhD to do such mandate and repetitive task. The answer was simple: no one else understands how important it is to do this task precisely as documented in the instruction.

However, if you do things that are not culture specific (like language specific in the case I have seen) you may have access to cheep foreign labor. There are places in the world where people will work for one hour get paid one dollar and kiss you on your legs for the privilege. What is low end pay rate in US, may be very good for high IQ person in third world country.

I already think about all these people who would like to subscribe and then subcontract the job keeping $9 per hour in their pocket. Not that it did not cross my mind. ;)

This is interesting, and likely would be tried. What would stop the "third-world" individuals from signing up themselves? Also, how do you explain the number of people that contribute to wikipedia?

Culture differences in most cases (language) and economical differences in remaining ones. After all you need to own a computer with reasonable internet connection.

Wikipedia gets really huge reviewing power and still you can find BS in it. It all depends on how sensitive your study is to human factor artifacts.

Sounds quite interesting and a step up from mechanical turk. Eg lots of people might want pretty charts for their work and not know how, only the chart they want.

You might want to have the option to be paid in btc if that's your preference though.

OoC, I want you to know you were crucial in getting me to finally give in and start messing with R. You and stochastic here along with some people IRL. But you were constantly helpful and suggesting it until I realized how much I was limiting myself by relying on excel and my stubbornness to make what was in my head happen... and if I was going to incorporate something into my "workflow" it may as well be R which anyone can use instead of matlab, or worse, sigma plot and brethren. IMO, R doesn't make pretty graphs, but it can depict exactly the info you want, then you make it pretty in illustrator.

In my experience, the most important thing participating in the scientific process teaches you is to acknowledge the degree of uncertainty to be dealt with and, also, subjectivity in interpreting your results. Never "believe" what you read. Wikipedia is great because this is made obvious, the appeal to authority veil has been pushed aside.

Thanks for the kudos, btcbtc113!

I don't want to derail this thread so I'll pm you, but I'm interested in the illustrator prettifying.

Also, I do think R can produce charts sufficiently pretty to make them easy on the eye. Check this out:

http://organofcorti.blogspot.com.au/2012/04/41-slushs-pool.html

sub

(sounds cool, I have experience in science as well as the kind of image manipulation you're talking about)

There are always "human factor artifacts". I'm not sure what the best way to deal with them would be for this system. Have an odd number of people perform the same task then assess how different the odd man out is vs having the same number of trained researchers do it (techs, interns, grad students, whatever)? Incentivize the other people contributing to score the work of their peers accurately. I agree this is an issue, but it is not unsolvable. I don't think wikipedia bureaucracy is nearly as insidious as the bureaucracy I have observed.

A quick draft writeup. I think better when I put ideas to keyboard.

https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit (https://docs.google.com/document/d/1nPI1tsrVt9QMD6HgesA77PvdQ-cD7m_0wLYuBv00NJk/edit)

Wikipedia is human language, which can contain other things beside truth. If you farm out eg data processing in a picemeal fashion, you'll be able to see if some part of it differs significantly from others. So when you figure out who screwed up you can mark them down or if it's a malicious error throw them out.

You'd need a 'web of trust' type identifying system. Management doesn't have to know who workers are, as long as someone trusted in the chain does. This would also double as a referral system too.

I also see this as being handy for collaboration. Don't have anyone at your job interested in the work you're interested in? Pay a fee and get matched with someone with your interests. For example, if you're great at analysis but crap at writing it up, you might be able to find someone familiar with your area to get the written parts ready for publication, either for a fee or for coauthor credits or both.

That looks promising in general, but the power of one SC to affect the final product seems excessive. (Only two must approve) Perhaps a less linear process with a more distributed system would work better. (Multiple SCs submit revisions or "no revision", they all vote on the preferred option)

What system would you suggest for dealing with users with large amounts of rejected work?

In theory instructions should be so precise that process should be deterministic. Anyone following instructions should (theoretically) arrive at materialy same answer. This is supposed to be a scientific observation, hence reproducible.

In practice human factor is not that easy to account for. Hence need for some arbitration and quality control. SC are supposed to be more experienced and cases when their verdict would be questioned will be so rare that they will not bias the study.

I must admit that I wrote spec with corporate context in mind. I could use some lateral thinking and explore a self-organising network of humans.

Well SC has power to deny reward for inadequate work. In practice such workers drop out very quickly. Of course, repeated offenders will be suspended. Depending on the supply of recruits, suspended users will be retrained or dismissed.

Wouldn't it also sometimes be an option for the process not to be deterministic? For example, "Use any software to count the cells on this set of images". The worker would have to say what he did, give the numerical results, and also supply the processed images*. This way the researcher can easily see if the work has sufficient quality, without having to do all the manual tweaking of edge-cases. The idea here is that checking if something is done correctly takes a lot less time than doing it yourself.

*Normally software such as this can show the outlines of the cells it detected, for example.

I am glad there is interest in this idea. Here is an example of a simple task that must be repeated hundreds of times. I used some of the terminology PawShaker provided here:


http://i49.tinypic.com/fwuiz8.png

Fullsize here: http://i49.tinypic.com/fwuiz8.png

So I was at the zoo the other day and talking to one of the guys running the research aspect of it. Apparently they are working on something similar to this idea. They are trying to devise a way to have zoo-goers report on animal behaviour they observe and filter out the nonsense. The current strategy is to make it part of an ipad app that provides facts about the animals. It sounded like there was a threshold of "learn more" buttons that need to be clicked before the data would be taken seriously.

Anyway I think it is promising that people are beginning to think along these lines.

I have been also toying with the idea what happens after the GPU-mining comes totally unprofitable. I mean not to hijack this thread but so called classification etc. "mundane" classification tasks can be in my opinion be rather easily solved with neural networks. So my suggestion is to make rather an distributed computer cluster where "mining rigs" can join. Environment could have ready make neural networks etc. plus ability to upload&train your own. Then any researcher could simply buy cluster time. WIN-WIN solution. Plus pretty affordable.

This is a pretty awesome idea, people will get to learn more science, get paid, and also contribute further to science at the same time. It's a win:win for both sides almost.

You should lower the payout.  At $5 to $10 a job you are paying graduate student wages.

Of course if you marketed it as a way for graduate students to procrastinate then they could double their paychecks.

http://www.phdcomics.com/comics/archive/phd010408s.gif

I have been thinking more and more about this concept and have been convinced that the existence of something like this would be a huge benefit to humanity. However, I have another year or so of work to complete my current project before I can start working on it, and after that... it depends on what happens. If someone else creates something along these lines I will contribute as much as possible. This is really a $15 billion a year industry just sitting there waiting to be taken from academia.

Also, in my mind... this would just be bringing back the days of "human computers", to do the jobs that noone is currently doing because the people in charge of the projects have neither the time nor resources to really have someone examine their data to the fullest extent. The result is a millions of monkeys on typewriters approach, half assing everything. I think we can do better with a crowdsourced approach, even with fraud and noise to filter out.