Bitcoin Forum

Bitcoin => Project Development => Topic started by: SgtSpike on June 08, 2012, 11:07:23 PM



Title: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 08, 2012, 11:07:23 PM
I was looking for online storage providers last night... they all charge an insane amount if you want to store large amounts of data (most were more than $100/month for 1TB of data... what??).  So I got to thinking, what about integrating Bitcoin into a distributed file storage system?

I believe I have heard of torrent-like file storage systems, where pieces of people's personal files (encrypted, of course) are stored across other users hard drives.  What about taking that sort of system a step further?

Files would be distributed across other users computers, but each user could be paid for volunteering their extra HDD space and bandwidth for the purpose.  They could set their own price (in Bitcoins) per GB of storage and per GB of bandwidth, and anyone needing to store files would automatically be given the best price for storage.  Sort of like a GPUMax for online storage.

Of course, you'd want each piece of the files to be backed up in more that one location, so that they are accessible regardless of whether some of the hosting computers are turned off or not.  Maybe people could pay extra for extra backups too.  2x copies would be standard, 3x copies could be paid for with 50% more on the price, 4x copies 100% more, etc.  And any host falling below a specified amount of uptime during a given time period would be booted from providing and/or not get paid.

The main reason for the project is this:  I think individual users are able to, and much more willing to, sell online storage space for much less than a hosting company would.  For example, I'd be willing to rent out 100GB of space with 500GB of bandwidth for $5/month.  Dropbox will do the same for $20/month.  Add another user like me, who is willing to rent out 100GB of space for $5/month, and you have your two copies, but it only costs the user $10/month instead of $20/month with dropbox.

Thoughts?  Would this be doable/practical/feasible/useful?


Title: Re: [IDEA] Dirt cheap online storage
Post by: Garr255 on June 08, 2012, 11:10:31 PM
This is definitely something I'd participate in both ways.


Title: Re: [IDEA] Dirt cheap online storage
Post by: austonst on June 08, 2012, 11:13:21 PM
So, maybe something like Freenet's data storage scheme (http://en.wikipedia.org/wiki/Freenet#Distributed_storage_and_caching_of_data) (maybe without the possibility of old data being removed), but with financial incentive to provide storage space and a free market system to ensure that using it stays cheap? Sounds like a great idea, and a great place to integrate bitcoin transactions.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 08, 2012, 11:17:24 PM
So, maybe something like Freenet's data storage scheme (http://en.wikipedia.org/wiki/Freenet#Distributed_storage_and_caching_of_data) (maybe without the possibility of old data being removed), but with financial incentive to provide storage space and a free market system to ensure that using it stays cheap? Sounds like a great idea, and a great place to integrate bitcoin transactions.
Right, exactly!


Title: Re: [IDEA] Dirt cheap online storage
Post by: Tittiez on June 09, 2012, 03:45:44 AM
I love the idea, I can sell out the TB's of hdd space I have on a vps.


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 04:00:08 AM
This has been done before, several years before Bitcoin was ever invented. I can't for the life of me remember the name of the program but it was an open source P2P network with its own virtual currency. Obviously it never caught on.


Title: Re: [IDEA] Dirt cheap online storage
Post by: finway on June 09, 2012, 04:08:31 AM
This is an interesting idea.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 09, 2012, 05:32:44 AM
This has been done before, several years before Bitcoin was ever invented. I can't for the life of me remember the name of the program but it was an open source P2P network with its own virtual currency. Obviously it never caught on.
Interesting...

Do you think Bitcoin would help it, uh, catch on easier?


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 05:46:36 AM
Do you think Bitcoin would help it, uh, catch on easier?
The project is dead now but from what I can remember the fundamental economic model was broken. You had to pay upload and to download and the only ways to earn currency was to donate hard drive space or run indexing and directory services. They never could figure out a good pricing mechanism that actually worked.

It's been so long ago that I can't seem to find any reference to it and nobody I've asked can remember what it was called.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 09, 2012, 05:49:30 AM
Ah, ok.  I think Bitcoin would serve the purpose much better!


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 06:03:57 AM
Found it:

Mojo Nation (http://en.wikipedia.org/wiki/MojoNation)

It was extensively covered on Slashdot (https://www.google.com/webhp?q=site%3Aslashdot.org%20mojo%20nation) in the 2000-2002 timeframe.

When the company that was developing it started to run out of money they tried to take it commercial and turn it into a LAN backup product called Hive Cache.

One of the developers forked it as Mnet (http://mnetproject.org/) but without the micropayments. The source for Mnet is still available so someone who was so inclined might be able to revive the project and incorporate Bitcoin. The original Mojo Nation code was LGPL and there are still copies (http://ftp.heanet.ie/disk1/sourceforge/m/project/mo/mojonation/OldFiles/) of it floating around too.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 09, 2012, 07:11:59 AM
Wow, well that sounds like the perfect thing to start from...


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 07:24:07 AM
Wow, well that sounds like the perfect thing to start from...
Given the leanings (http://www.salon.com/2000/10/09/mojo_nation/) of the founder of that project it might even be possible to get the original creators on board.

The idea was good but it was about a decade too early...


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 08:04:10 AM
TL;DR what do you want offsite storage FOR? Do you actually want it at all or are you really wanting gun money, troop payroll financing, candy pocketmoney or what?

I suspect that using an accounting token that has uses other than simply keeping everyone's usage within the bounds of their contribution will provide un-useful incentive to abuse, thus that it is preferable not to resort to such tokens unless it is mission-critical that people who do not use the service themselves provide the service.

For off-site backups, for example, the desired feature is not the volume of storage space but the relocation of storage space from on-site to off-site.

Paying for off-site storage using tokens (currency) that people who are not looking to buy such storage space will actually want for other purposes entirely simply serves to attract every hacker faker freeloader etc who is interested in buying drugs, guns, troops, candy, liquor, or gosh knows what else, desiring neither to provide offsite storage nor to obtain offsite storage but merely to obtain their desired guns, troops, drugs, candy or whatever at other people's expense by fooling/repurposing other-purpose token (currency) networks to the new/different purpose, whether it be the financing of drugs, weapons, troops, candy, or whatever.

If however the only thing the tokens can buy is off-site storage, and the only way to obtain them is by providing off-site storage, the tokens might manage more easily to bypass folks looking to accomplish such re-purposing simply by being not too obscure but simply too limited in liquidity on general token/currency markets to be as attractive a target as other more general-purpose tokens such as bitcoins.

Thus if you are on the demand side, that is, actually wanting the service, directly trading the service itself (storage) as provisioned on your site for the service itself (storage) as provisioned on someone else's site might be a much less attractive setup from attackers / freeloaders / other-purpose-financiers point of view.

Basically, if I want to store N copies of my data, then I need an amount of storage N times the size of my data.

If some of those copies are to be stored off-site, then I still need as much storage, I just happen to need some of it to be located off-site.

Thus it seems reasonable I provision storage N times as large as my data and that for each copy of my data I am able to move off-site I free up as much space on-site as I consume off-site.

I might want N to be larger if third parties are involved, in proportion to the reliability I attribute to such third parties, but basically if we all plan to have a certain amount of storage and are looking to swap on-site storage for off-site storage, not to leech gun-money, gum-money, troop-payroll money or whatever other kind of money from wherever we can whenever we can by making whatever representations those we leech such things off of need to be told / need to hear / need to see in order for us to thus leech... maybe we would be better off simply trading such storage directly instead of trying to make our network a desireable target for repurposing attacks?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: notme on June 09, 2012, 08:10:53 AM
https://tahoe-lafs.org/trac/tahoe-lafs

One of the developers (Zooko) is into bitcoin.  Perhaps starting a discussion about this on the tahoe-dev list would be well received.  There may be a way to easily integrate servers advertising a bitcoin address to encourage them to provide more space.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 08:14:57 AM
Tahoe-LAFS is designed more for deployment oneself than for trading space with others.

That is, if you are going to have multiple sites yourself, Tahoe-LAFS might well be one of the best ways to utilise such sites.

But with multiple parties providing the sites, something more like GNUnet (but maybe able to be deployed on more operating-systems than GNUnet currently supports) would be more suitable.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 06:17:22 PM
I suspect that using an accounting token that has uses other than simply keeping everyone's usage within the bounds of their contribution will provide un-useful incentive to abuse, thus that it is preferable not to resort to such tokens unless it is mission-critical that people who do not use the service themselves provide the service.
The incentives would need to be thought through carefully. If it was a simple matter of paying people to accept data then it would be attractive to scammers who would just throw your data into a bit bucket and take your Bitcoins.

If it is intended to be a distributed alternative to DropBox based on the Mojo model the people offering storage space would need an incentive to keep their nodes online and connected, which means an income stream proportional to the amount data they are holding and their node's past performance.

Block servers could all advertise a daily storage cost per block in an auction and the nodes seeking to upload data could send their blocks to the lowest bidder.

Each day (or whatever time interval is used) before a node issues payment for block storage it should query a percentage of random blocks from each block server to verify that the block server actually has the data. If it fails to deliver the requested data the block server would first not get paid and second that information should be fed to a reputation system.

If a user doesn't pay for storage it would be up to the block server to decide what to do. It could discard the data immediately, or it could retain it and demand back payments to release it, or most likely it would keep it until it was either paid for or until the block server ran out of space and unpaid blocks were overwritten by paying blocks.

So in the end the user wanting to store data tells his node the maximum amount he is willing to pay per day to store his data and how much redundancy/reliability he wants and the node automatically tries to find the most cost effective solution based on the block servers which are advertising free space and their reputations.


Title: Re: [IDEA] Dirt cheap online storage
Post by: conspirosphere.tk on June 09, 2012, 07:12:35 PM
I would like that distributed online storage would be implemented as an option to store the phucking blockchain that is already eating 3 GB per each client that I use, and take hours to update if I don't do it for some weeks.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 07:24:28 PM
A straight swap still seems simpler. Want one of your blocks back? Gimme one of mine. Tell me which of yours you want and I'll reply telling you which of mine I want. Probably first wanting just a specific function of the block, only progressing to gimme the entire block if actually the whole block is needed/wanted not merely indication that the other party has enough of its bits to be able to answer the specific query.

"You want me to tell you what the bits at each of that list of 1024 bit-offets into the file are? Fine, I can do that if you undertake in return to tell me what the bits at each of this list of 1024 offsets into one of my files is".

If you are actually wanting cheap storage you buy, then a bunch of corner-store / mom-and-pop / kid in basement operations are probably not a cost-effective solution. A Wal-Mart approach is more likely to be able to get prices down and keep them down.

If you actually want the small private operations setup for reasons other than actual cost, then buy as much storage as the total of local and offsite storage you want, then trade storage with others who do the same.

I know that arguing not to use bitcoin to account these swaps might seem to imply my response is not bitcoin-related, but bear in mind I am looking to do offsite backups of bitcoin-related things such as bitcoin exchange servers and such. The relatedness / relevance to bitcoin lies not in method of accounting for the swaps of backup-storage-space but, rather, in that it is bitcoin users looking to backup bitcoin-wallets and bitcoin-websites and bitcoin-exchange servers and such.

It seems overly complicated and off-purpose to go looking to sell people storage space on my site to raise funds with which I can then go looking for other sellers to buy storage from on their site. It is simply I want to put N copies of X gigabytes offsite and am prepared to store X * N gigabytes similarly for others.

Of all the software suites I have looked at so far for accomplishing this kind of thing, so far GNUnet is looking like most appropriate. Though I suppose in principle I could also throw backups onto Freenet too if I chose to, what the heck, who knows, I might be able to find them there later if it turns out GNUnet fails me somehow. Maybe use GNUnet and Tahoe-LAFS and, just to be nasty to Freenet folk, throw copies onto Freenet too?

See that last part though, ouch, heck, what stops anyone from just leeching off of Freenet? Hmmm.
 
It might well be that the seemingly high prices the original poster is trying to avoid are not so high really at all when you really look into the nitty gritty of providing such a service.

It might also be that prices are driven crazy-high by involving the ignorant masses while not charging by the hour for help desk or other support services, thus having to do handholding of millions of idiots who do not want to pay by the hour for handholding thus basically force you into charging everyone for some high average number of hours of handholding in order to cover the biggest idiots who require the most-inordinate amount of handholding.

Offering the service only to the qualified I.T. departments of businesses maybe the price per terrabyte-month + price per terrabyte-mile could be held down below what those trying to attract the ignorant as their desired customers are forced to charge?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 07:27:43 PM
I would like that distributed online storage would be implemented as an option to store the phucking blockchain that is already eating 3 GB per each client that I use, and take hours to update if I don't do it for some weeks.

For this we are not really talking about offsite storage so much as we are talking about download rates and volumes.

(One copy of the blockchain can be made available to multitudes, so its more about bandwidth of how many copies are wanted how fast and how simultaneously than how many WAL-BITS can fit in their bit-warehouses.)

-MarkM-

EDIT: A problem exists in the form the blockchain currenty takes normally in the reference client.

If it were a file that many can utilise at once, then one copy could exist on dropbox and everyone could share access to that copy, it could appear on their disk like a normal folder, automatically kept in synch by dropbox.

Unfortunately the reference client does not use a blockchain storage type that would work with such an approach.


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 07:50:45 PM
It seems overly complicated and off-purpose to go looking to sell people storage space on my site to raise funds with which I can then go looking for other sellers to buy storage from on their site.
There's absolutely no reason at all that the user would ever be exposed to those details. Automated trading algorithms are fairly common so there's no reason for the user to need to configure any of it himself.

Someone who is running a block server wants to collect a small income stream from his excess hard drive space. Due to competition it's not going to be a very large income stream but lots of people have excess storage and would enjoy being able to collect a few bitcents every day for it (whatever the market price happens to be). This user shouldn't need to do anything other than install the software and allocate drive space. The program should handle the details of getting the best possible price for his storage space without further intervention.

Likewise the person storing data wants to pay as little as possible for a given amount of data, given amount of reliability and a given amount of bandwidth. His software should also take care of the details entirely behind the scenes. He should be able to upload data, choose a reliability and bandwidth targets and forget about it other than periodic reminders to add more Bitcoins to his node's wallet.

There can a plethora of automated trading action going in behind the scenes, with contracts (https://en.bitcoin.it/wiki/Contracts), escrow and reputation systems but that's all complexity that the user never needs to see or worry about.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 08:10:21 PM
Sure, but chances seem good to me that the best price will turn out to be available to me from all the various nodes for whom I have been faithfully storing data and returning it intact upon demand, and similarly for my peers, so that the only "customers" forced to deal with the people who want some trickle of pennies when they fire up their laptop for a few minutes or hours will be other fly by night laptop-storage-by-the-second-minute-or-hour types like themselves.

In short, why bother to pay even pennies to people who aren't even responsible enough themselves to want off-site backups yet nonetheless want to convince you to pay them to store your backups for you, if you can instead store other people's backups for them in return for them storing yours?

Two different goals though... It sounds like you want to build a mom and pop iTunes or Pirate Bay or Malware Incorporated or Cheap Pr0n Emporium.

It might be better to base your pricing/accounting on the actual contents of the files instead of caring at all about how big or small they are: consider how valuable they are even if small if they are "better quality" in some way.

Everyone could advertise whose personal private data they have copies of, how encrypted it seems to be according to their cracking attempts so far, there could be betting markets on whether there will turn out to be another layer of encryption beneach the outwardly visible layer, and the people who actually gave these files away freely could choose whether to buy a download of a copy of such and such a day's crypted backup from this distributed market or not on any given day, maybe taking into account various threats by various sites to take it off the market if no one buys a copy within the next such and such span of time. Name your backups provocatively so more people will share the file thinking it of particular interest or value. Etc.

Is this to be about the data stored, or the volume of storage, or the bandwidth? What is the obstacle, if any, to running something like Tahoe-LAFS or GNUnet etc yourself to build up mutual reputation with other nodes, why do you need some trickle of pennies for doing what normal participants simply "just do"? Why should I even consider having to pay you to store data for me instead of doing mutual data distribution with others who have storage and a desire to move some of the data stored on it offsite?

-MarkM-

EDIT I guess what I am getting at here is that reliability and volume of storage and bandwidth usage are plenty of currencies already, in fact each could be adjusted separately so that for example if I provide vast amounts of storage for huge lengths of time but at low bandwidth, what I will tend to get in return is lots of space for plenty of time but at low bandwidth. If I want to spend general purpose currency of some kind to increase the speed, I can upgrade my own speed with that money, thereby resulting through reciprocity with my peers in better bandwidth for my own usage. I can buy more storage or more speed or more duration simply by buying it for/from my own site, no need to buy it from others' sites. Simple reciprocity will cause what I get from others to adjust in accordance with what I provision locally aka what I pay to/for my own site.

EDIT2: Even considering it as a prospective seller looking to make a trickle of cents, why would I want to pay you more cents to store data than I am being paid for storing data, whether I am having you store data others pay me thinking I am storing it or having you store data I want stored? If it is all about wanting a trickle of pennies the tragedy seems to arise that until some of us actually obtain some pennies there aren't going to be any to be trickled. I can't give you pennies I do not have, and if I give you more than I earn I will go broke...

(I can however give you blocks of data you asked me to store for you, if will take those instead of pennies maybe we can do business?)




Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 08:32:29 PM
I guess what I am getting at here is that reliability and volume of storage and bandwidth usage are plenty of currencies already, in fact each could be adjusted separately so that for example if I provide vast amounts of storage for huge lengths of time but at low bandwidth, what I will tend to get in return is lots of space for plenty of time but at low bandwidth. If I want to spend general purpose currency of some kind to increase the speed, I can upgrade my own speed with that money, thereby resulting through reciprocity with my peers in better bandwidth for my own usage. I can buy more storage or more speed or more duration simply by buying it for/from my own site, no need to buy it from others' sites. Simple reciprocity will cause what I get from others to adjust in accordance with what I provision locally aka what I pay to/for my own site.
I think the problem you're not seeing is called the double coincidence of wants. Not all users who need large amounts of storage have large amounts to share. A market mechanism is an efficient way to bring those with excess storage/bandwidth/uptime/CPU power together with those who need it.

Why would you think that having different currencies for each scarce quantity is better than using a single currency? Would you want to use a different currencies to buy food with, pay rent, and procure transportation?


Title: Re: [IDEA] Dirt cheap online storage
Post by: Gladamas on June 09, 2012, 08:38:38 PM
Sub, this is an interesting idea. Could we incorporate past file versions, and pay users based on their upload speed as well as HDD space?


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 08:41:39 PM
Actually I think what I am not seeing is what the barriers are to mass storage services such as Amazon S3 getting so cheap from so many suppliers that the original poster's complaint that offsite storage is too expensive ceases to be a real driver for this kind of project.

The original idea at top of thread was basically gosh surely we could provide the same product cheaper just with ad hoc networks of random fly by night laptops.

Now the idea seems rather to be gosh surely there are trickles of pennies to be made by doing that.

A friend told me a few years back that big stores like Wal-Mart can actually be amazingly efficient at providing low prices.

I believe that there are more and more free open source "S3-compatible server" packages out there, would something like that be more suitable for your purpose, allowing people to directly compete with Amazon from home and thus maybe by competition drive down Amazon S3 service prices?

-MarkM-

EDIT: Earlier in speculating how much having to handhold ignorant people who consider it their right to remain ignorant might drive up costs, I forgot to also speculate that marketing might also drive up costs. Paying for enough advertising to obtain enough customers to maintain a trickle of pennies might end up costing more than the number of pennies it brings in, until marketing capital such as subscribed mailing-list members and so on help shift you toward the cost of retaining or up-selling existing customers more than the cost of acquiring new customers.  A huge marketing budget might well be able to make cheaper services seem less attractive than services so ridiculously expensive that they can afford a huge marketing budget...


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 08:45:13 PM
Actually I think what I am not seeing is what the barriers are to mass storage services such as Amazon S3 getting so cheap from so many suppliers that the original poster's complaint that offsite storage is too expensive ceases to be a real driver for this kind of project.
The reason someone would choose a decentralized solution for online storage instead of a centralized solution is the same reason they would be interested in Bitcoin in the first place.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 09, 2012, 08:49:20 PM
Markm, you're COMPLETELY missing the point.

If I wanted to swap space with other users, I'd look at using freenode, or something else.

What this idea aims to accomplish, is to match up people who aren't using all of their storage, with people who are using all of their storage and still need more.  I have a couple TB's of extra HDD space, and wouldn't mind renting it out to other people to use.  I don't have any files that I want to store on their systems.  So, in my case, any kind of a "swap" system is fail.  I don't want it.

Similarly, my sister is into photography, and stores the .RAW files of all pictures taken, which obviously eats up hard drive space fairly quickly.  She only has 750 GB total.  She needs more space.  Rather than deleting the old files permanently, she asked how much online storage would cost.  It was outrageous - to the point where buying a new 1TB drive would cost less than renting 1TB of online storage for 2 months.  So, she needs more storage, and doesn't have any extra storage space to store other people's files as well.  Again, in her case, your "swap" system is fail.

I (and my sisters) don't have any criminal wishes, and I don't know why you want to imply that anyone who wants to make money hosting files is attempting to do so with criminal intent.  That is not at all the purpose here.  This is more about allowing people with lots of extra hard drive space a way to make money with that space, and giving people who need more space an opportunity to purchase it at a price that isn't insane.

Also, S3 is still high priced.  $125/month for 1TB of storage.  I'd be willing to provide 1TB of storage at a far lower price than that.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Gladamas on June 09, 2012, 08:50:07 PM
Actually I think what I am not seeing is what the barriers are to mass storage services such as Amazon S3 getting so cheap from so many suppliers that the original poster's complaint that offsite storage is too expensive ceases to be a real driver for this kind of project.

The original idea at top of thread was basically gosh surely we could provide the same product cheaper just with ad hoc networks of random fly by night laptops.

Now the idea seems rather to be gosh surely there are trickles of pennies to be made by doing that.

A friend told me a few years back that big stores like Wal-Mart can actually be amazingly efficient at providing low prices.

I believe that there are more and more free open source "S3-compatible server" packages out there, would something like that be more suitable for your purpose, allowing people to directly compete with Amazon from home and thus maybe by competition drive down Amazon S3 service prices?

-MarkM-


A decentralized service such as this one would be better for the community IMO, and would be immune to raids, DOS attacks, etc. Regardless of the cost of this compared to centralized services' prices, the community can come up with a better protocol and product than a centralized service can. Just look at Bitcoin, BitTorrent, etc...


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 08:53:09 PM
The reason someone would choose a decentralized solution for online storage instead of a centralized solution is the same reason they would be interested in Bitcoin in the first place.

Yes! Thank you.

But just because I am interested in bitcoin and indeed even devcoin and litecoin and on and on and on does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies. How about some of the people who want to PROVIDE THE PENNIES make some concrete offers of how many pennies per what span of time they are willing to pay to have how many copies of how many bytes distributedly stored?

-MarkM-



Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 08:57:25 PM
But just because I am interested in bitcoin and indeed even devcoin and litecoin and on and on and on does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies. How about some of the people who want to PROVIDE THE PENNIES make some concrete offers of how many pennies per what span of time they are willing to pay to have how many copies of how many bytes distributedly stored?
Ok, you don't want to spend any of your precious bitcoins to rent storage space from other users. So what are you objecting to exactly? If you don't want to use this kind of system than all you have to do is not use it. Are you upset that other people might be willing to spend their bitcoins to buy storage? that other people would be willing to rent out their excess storage for bitcoins? Do you not want bitcoins to be used as a real currency that people regularly spend?


Title: Re: [IDEA] Dirt cheap online storage
Post by: Gladamas on June 09, 2012, 08:57:55 PM
Just because I am interested in bitcoin ... does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies.

Um, isn't the whole point of Bitcoin to replace fiat money with a decentralized commodity?


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 09:28:43 PM
https://tahoe-lafs.org/trac/tahoe-lafs

One of the developers (Zooko) is into bitcoin.  Perhaps starting a discussion about this on the tahoe-dev list would be well received.  There may be a way to easily integrate servers advertising a bitcoin address to encourage them to provide more space.
He might not be able to participate (http://zgp.org/pipermail/p2p-hackers/2005-March/002483.html), although this message from 2005 may be outdated.

Quote
Dear mnet-devel and p2p-hackers:

I am about to accept an exciting job that will preclude me from
contributing to open source projects in the distributed file-system
space.

I will miss the Mnet project!  Good luck without me!

There are some interesting thoughts at the bottom of the message regarding digital cash and how it worked in the Mojo Nation project.

Quote
V.  Persistent Tit-for-Tat == Bilateral Accounting.  Mnet v0.7+ lacks
something -- an incentive mechanism to limit excessive costs imposed on
the network and simultaneously to motivate users to contribute
resources to the network.  I was always deeply impressed with the
simplicity and robustness of Bram Cohen's "Tit-for-tat" incentive
mechanism for BitTorrent.  Suppose we wanted a similar "tit-for-tat"
mechanism for Mnet v0.7++, but we wanted the peer relationships to
extend through time and across multiple files and multiple user
operations.  Then we would probably invent a bilateral accounting
scheme for each node to keep track of how much goodness each of its
peers has done for it, and to reward helpful peers.  This would then
turn out to be more or less identical to the "bilateral accounting"
scheme that was originally invented by Jim McCoy and Doug Barnes in
Mojo Nation [footnote *].

[footnote *]  Once upon a time there was digital cash, as pioneered by
David Chaum.  When Jim McCoy and Doug Barnes invented Mojo Nation, they
used digital cash, and added bilateral accounting so that a pair of
peers wouldn't require a transaction with a central token server in
order to incentivize each other.  The first three employees they hired
to implement Mojo Nation in 1999 were Greg Smith, Bram Cohen, and
myself.  (Greg might have started in 1998 -- I'm not sure.)  Several of
the ideas in BitTorrent -- which Bram started writing in 2001 -- can be
understood as radical simplifications of ideas in Mojo Nation.  One
such perspective is to think of BitTorrent's tit-for-tat incentives as
being time-limited, file-specific, and non-transferrable bilateral
accounting.  This is not condemnation of BitTorrent's ideas, but praise
of them -- they demonstrate the virtue of radical simplification.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 09:33:18 PM
This is more about allowing people with lots of extra hard drive space a way to make money with that space, and giving people who need more space an opportunity to purchase it at a price that isn't insane.

Also, S3 is still high priced.  $125/month for 1TB of storage.  I'd be willing to provide 1TB of storage at a far lower price than that.

If it is not about backups, isn't the cost of it being offsite driving up the price needlessly?

Wouldn't your sister be better served by borrowing or renting a disk drive from you for her to use at her site, thus paying just for one sneakernet movement of the drive to her then someday maybe one sneakernet move of it back to you, instead of paying by the month for remote access to a drive at your site?

Or is this sister at the same site as you?

If you have reliable long term lack of need for storage, and she has reliable long term shortage, that seems a perfect setup for a used disks market even: sell her a spare disk and buy a better one later if some day you find you do afterall need more.

The closer you approach the fly by night laptops scenario the closer you get, it seems to me, to a situation where buying momentary storage instead of buying the media itself that the data is stored on starts to look reasonable. If you only need a few petabytes for a few minutes or hours, it might well make more sense to use it remotely rented by the minute or hour than to buy or rent a few petabyte drives for your own location.

So I think we need to figure that yes we are looking for fly by night laptops, however outre that might sound, and thus look for solutions targeted precisely at making distributed storage reliable even on aggregations of fly by night laptops.

Having to make rather extreme assumptions about the nodes being so utterly unreliable, to the point where we are not trying to insure our data against the occasional failure of a machine somewhere in our network but, rather, are trying to insure it against maybe even a 51% attack in which 51% or more of our nodes are with malicious aforethought carefully co-ordinating and tracking data looking for a moment in time at which all of them agreeing to refuse us service at the same time, deleting all copies of our files they have access to, can do us the most harm (for the lols, even, maybe) we can figure we want to ensure the default number of copies of any item of data should be far far higher on our proposed network than the maybe 3 copies to a dozen copies or so things like GNUnet and Tahoe-LAFS and so on typically figure people will configure their data to demand.

Our disk space is thus likely to make up in sheer volume required to keep many more copies of everything than a more reliable network would need a lot of any cost savings it might experience by imagining the space used is "essentially free due to being not in use" (when in fact just like mining, actually spinning up the drive instead of letting it hibernate might well drive up the electricity bill...)

There definitely are projects aiming toward the desired goals, except for the part about money.

Maybe anonymity might not be that great a feature once money is involved, since if Moriarty offers to store my data I might prefer to pay someone else more just to keep it out of his hands rather than pay him to store it for me? If I have to assume that the vast majority of the copies I am paying for of my data will deliberately vanish just at the very moment when I most need them, the price I will be willing to pay per copy maintained will be far far lower than if I am more confident that the vast majority of the copies will survive and be instantly available to me at high bandwidth from numerous sources...

Projects are working on all this, I have looked into those I could find, visited their Freenode IRC channels, given them a test drive and so on. From what I learned so far I have the impression that the main thing preventing GNUnet for bothering to have a currency yet is they are still tuning exactly what any such currency would actually buy, or maybe a better way of saying it would be, given some kind of currency, what exactly should a node consider how valuable compared to what else in order to intelligently decide what exactly to spend how much currency on.

That is, I got the impression that GNUnet was still in the process of working out what needs to be able to be compared in value in order to make purchasing decisions.

Thus I am still thinking that once GNUnet works well at actually trading the commodities it has at hand then would be the time to look at how best to set up mechanisms by which a node can be configured to give some nodes preferential treatment above and beyond that which is "deserved", so that bitcoins or whatever else could then be used to bribe any nodes of your choice into giving your node resources that your node has not "earned" by its own contributions. At that point markets could compare what bribes various nodes are willing to accept and nodes unwilling or unable to contribute their fair share of work/service would be able to shop around to choose who to bribe for what "unearned" resource-usage.

Bear in mind denial of service attacks, too. A market driven only by pennies might be quite vulnerable to anyone willing to throw dollars at it for a few hours to disrupt it. You might think there are a thousand laptops online at any given instant that have copies of your data safe and sound because each one of them each already learned you pay an entire satoshi per year per copy, but then suddenly one moment find every one of them threw away your data without notice due to hearing of an offer of a hundred million satoshis per decade per gigabyte per copy from someone else who might renege on their offer the very moment they realise their offer accomplished its goal of having everyone delete all copies of your data to make room for theirs...

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 09:35:46 PM
Just because I am interested in bitcoin ... does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies.

Um, isn't the whole point of Bitcoin to replace fiat money with a decentralized commodity?

Yes, but by other currencies I meant currencies such as megabit-hours of storage, megabit per second seconds of bandwidth, persistence of reachability at any hour of the day or night, still having your data next year when you need it, not flying by night with all your data, not conspiring to try to DOS you somehow in your use of the system and on and on like that.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 09:42:42 PM
Yes, but by other currencies I meant currencies such as megabit-hours of storage, megabit per second seconds of bandwidth, persistence of reachability at any hour of the day or night, still having your data next year when you need it, not flying by night with all your data, not conspiring to try to DOS you somehow in your use of the system and on and on like that.
That's why I mentioned reputation systems in a previous post. A laptop that is not connected to the internet constantly will frequently fail to deliver blocks so would have a very low reputation score. Its storage would then be devalued compared to nodes which were always on with a reliable and fast connection and probably wouldn't be able to generate much income at all.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 09:47:34 PM
Yes. This is the kind of thing GNUnet is working toward.

Until this kind of thing itself alone without graft/bribery using an external currency works smoothly, I think adding graft/bribery is probably just going to make it even harder to get it "right".

Rabid free-market folk I suppose would say look if someone offers more money for deleting all data that originated from you than you offer for retaining it, tough luck, man up and offer bigger bribes.

I wonder how much those people are offering not to have them shot on sight?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 09:51:14 PM
Until this kind of thing itself alone without graft/bribery using an external currency works smoothly, I think adding graft/bribery is probably just going to make it even harder to get it "right".

Rabid free-market folk I suppose would say look if someone offers more money for deleting all data that originated from you than you offer for retaining it, tough luck, man up and offer bigger bribes.

I wonder how much those people are offering not to have them shot on sight?

Is this a conversation about how a system could be devised to efficiently allocate excess bandwidth and storage space or it is a conversation about emotional issues that you should be talking to your therapist about?


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 10:50:27 PM
Ah, ok.  I think Bitcoin would serve the purpose much better!
I think this true, not necessarily because of anything inherent to the currency, but because of the accumulated experience in the Bitcoin community regarding just how determined some people will be to cheat the system. The market model must have strong incentives for good behavior and robust methods of detecting and deterring bad behavior because there will be bad actors.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 10:58:00 PM
This is expected to be distributed, possibly somewhat in the spirit that bitcoin itself is, right?

We have seen with bitcoin that securing against attack tends to come down to how much it costs to defend it versus how much it costs to attack it, along with hopefully being able to make defending it seem more rewarding to potential attackers than attacking it so that some potentail attackers might decide the luls plus possibly even profit to be had by some ingenious attack are outweighed by the profits to be had by co-operating with the defenders.

So basically it seems to come down to how much is bid by defenders to have their data actually survive versus how much might an attacker pay to have someone's or everyone's or anyone's data not survive.

Is the ultimate purpose to run an experiment to see whether defenders choose to outbid attackers, or to create one or more markets where speculators and manipulators and brokers and shorters and so on can wheel-and-deal, or to get software working that can "equably" or some such term balance the resource contributions of nodes, onto which can then be added some means by which money  can be applied to influence which nodes that are not contributing are nonetheless provided use of resources?

I am proposing the latter: first make sure the thing actually works fine when nodes are not being paid money to service "nodes that normally would be allocated less resources than their owner desires due to providing less resources than they consume".

Basically the idea is we will not know which nodes have excess resources and which have shortfall of resources a-priori ahead of time.

For example even though a human owner of a node might imagine their node to have an excess of disk space could in practice find the space turns out to be offset by their node having a shortage of bandwidth, or of uptime, or of data integrity in terms of actually providing people with the data people thought it was storing for them at the time that they wish to access it, or whatever.

For example I think GNUnet uses "karma" as its internal currency-type-thing. Merely having a lot of disk space might or might not turn out to result in gaining more karma than you lose in the accounting of various other nodes depending on how karma is actually computed and what exactly actually happens in the way of blocks being stored, blocks being requested, the speed with which requested blocks are delivered compared to the speed from which they can be obtained from other nodes, basically whatever heuristics are in the code.

It might turn out that the way to make a trickle of pennies is not so much to have lots of disk space but, rather, to provide really low latency high bandwidth delivery of data stored on such disk space as you do have.

Once the heuristics the system uses internally to assign karma to other nodes work well, then maybe it would suffice to simply have a BTC<->karma exchange where people can buy karma points for their node on nodes of their choice using bitcoins.

But until karma points work well, maybe such a market wouldn't really be all that useful or great as markets for really great products likely do better than markets for products that don't actually work very well.

-MarkM-

EDIT:

Some distributed storage solutions drift the blocks toward the places that rquest them, so that for example if lots of people at a particular ISP ask for certain files and if the ISP itself was a node along the way for those people, more and more often more and more blocks of the popular-with-them files would tend to have migrated closer and closer to them.

Now imagine if speed were a factor, possibly in addition to how distance is in that solution. Having huge amounts of disk space might be almost worthless, since small-storage nodes with high speed might turn out to be the usual nodes most people get a copy of something from, with the huge disk drives having slow bandwidth becoming places only resorted to if it turns out each and every really fast node turns out to not have a copy of something the vast disk slow bandwidth node has. Thus the huge disks could idle unused for years before some partocular block of some particular ancient file no one has enquired about for years turns out ot be available from the huge slow node...




Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 11:10:45 PM
Basically the idea is we will not know which nodes have excess resources and which have shortfall of resources a-priori ahead of time.

For example even though a human owner of a node might imagine their node to have an excess of disk space could in practice turn out to be offset by their node having a shortage of bandwidth, or of uptime, or of data integrity in terms of actually proving people with the data people thought it was storing for them at the time that they wish to access it, or whatever.
That's the advantage of using a market mechanism to figure it out. We don't need to know ahead of time the best allocation of resources and instead allow the relative price to float according to supply and demand. Most individual users are not going to sit over their nodes and daytrade bandwidth and hard drive space so it makes the most sense from a usability perspective for the trading to be done by automated software agents.
For example I think GNUnet uses "karma" as its internal currency-type-thing. Merely having a lot of disk space might or might not turn out to result in gaining more karma than you lose in the accounting of various other nodes depending on how karma is actually computed and what exactly actually happens in the way of blocks being stored, blocks being requested, the speed with which requested blocks are delivered compared to the speed from which they can be obtained from other nodes, basically whatever heuristics are in the code.

It might turn out that the way to make a trickle of pennies is not so much to have lots of disk space but, rather, to provide really low latency high bandwidth delivery of data stored on such disk space as you do have.

Once the heuristics the system uses internally to assign karma to other nodes work well, then maybe it would suffice to simply have a BTC<->karma exchange where people can buy karma points for their node on nodes of their choice using bitcoins.
There's no reason to duplicate a currency system. Bitcoin has a lot of features, like contracts, that would make it useful for exactly this kind of use. If Bitcoin doesn't have sufficient features to be used as karma then wouldn't it make more sense to add them instead of creating a new currency?
But until karma points work well, maybe such a market wouldn't really be all that useful or great as markets for really great products likely do better than markets for products that don't actually work very well.
This is the risk of all new markets - that there isn't actually sufficient demand for the product being offered. It's not like any bitcoins used in the experiment would be lost though. If more people sign up to offer disk space then there are people wanting to rent then it will be extremely inexpensive to store files. This will tend to cause people selling storage to leave the network because they can't make any money and will attract new customers because the price is so low. If there are no customers willing to buy at the price point that anyone offering storage is willing to sell then the network will fail.


Title: Re: [IDEA] Dirt cheap online storage
Post by: dooglus on June 09, 2012, 11:17:13 PM
https://tahoe-lafs.org/trac/tahoe-lafs

One of the developers (Zooko) is into bitcoin.  Perhaps starting a discussion about this on the tahoe-dev list would be well received.  There may be a way to easily integrate servers advertising a bitcoin address to encourage them to provide more space.

There was a thread on these forums not long ago about integrating bitcoin with tahoe lafs.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 11:21:54 PM
Okay so lets rename karma as bitcoins. Or as satoshis or as millibitcoins or whatever.

Are you also proposing that the nodes mine bitcoins?

If not, how will any node ever actually convince its owner to provide it any bitcoins?

Are you planning to put some bitcoins in your node's wallet? If so, how many?

I figured, maybe wrongly, that most people who install the software to see if it can make them some pennies will most likely initially run it a while to see how many pennies it brings in before considering whether maybe their node simply isn't competetive enough to bring in any and from there whether they have any actual use for distributed storage themselves that might justify providing their node with some bitcoins instead of leaving it bring in bitcoins itself.

So maybe lets intially survey how many bitcoins are on offer, in order to consider whether to even bother firming up the details of the contemplated software so that folks can run it to harvest those offered coins...

...Or, we could use the already existing so called karma points for now, with an eye to figuring out whether in the end we will be calling them satoshis or bitcoins or millibitcoins or what.

Failing that, what exact services do you wish the markets to offer up for sale for bitcoins? Bits per second seconds of data-transfer maybe? Along with bits per second or day or hour or year or whatever of storage? Or, pay per block, you bid X satoshis for block ASghk235bbld6w566bnlwlsl66 and first node to deliver it gets paid? Or what?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 11:25:16 PM
If not, how will any node ever actually convince its owner to provide it any bitcoins?
The same way that any other paid online services work right now - the owner wants to buy the service.

Failing that, what exact services do wish the markets to offer up for sale for bitcoins? Bits per second seconds of data-transfer maybe? Along with bits per second or day or hour or year or whatever of storage? Or, pay per block, you bid X satoshis for block ASghk235bbld6w566bnlwlsl66 and first node to deliver it gets paid? Or what?
I want to store X MB for Y hours so I publish this information and then then let the those nodes who want to sell me storage bid on it. My node uses a set of heuristics to pick the best bid taking into account all relevant information (price, reputation, bandwidth, etc). The system does this for every resource that you want to allocate.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 11:28:12 PM
If not, how will any node ever actually convince its owner to provide it any bitcoins?
The same way that any other paid online services work right now - the owner wants to buy the service.

What service?

DIstributed storage tends to be crypted blocks, so maybe you want to buy blocks? Specific blocks, offering more for hard to find blocks than for blocks you can get swiftly and easily from umpteen providers?

-MarkM-

EDIT: Example: you pay nodes to let you send them a block. They wonder whether the block will ever pay anything more than what you paid to have them let you connect to them and deliver it. Some day you offer them bitcoins for a copy of it. You pay well, and pay them to let you tell them the bits of another block you might or might not want to buy a copy of at some later date. Etc.

I suspect a karma<->BTC market might well work better. A human might not hover all day over their node day-trading karma, but might some friday night upon noticing their node didn't have enough karma to fetch them the data they asked for on monday decide to give it some bitcoins so it can buy some karma to see if that can manage to get the data before the weekend is over...


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 11:33:43 PM
What service?

DIstributed storage tends to be crypted blocks, so maybe you want to buy blocks? Specific blocks, offering more for hard to find blocks than for blocks you can get swiftly and easily from umpteen providers?

-MarkM-

Maybe it would help if you read up on some of the whitepapers of Mojo Nation and other similar projects. There's already a lot of material written on how market-based allocation of computing resources could work and it would save a lot of time to not rehash all that in this thread.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 11:38:37 PM
I have read a lot of such material on many distributed storage systems.

I guess if you are offering to buy a resource and can provide the software or provide a URL to download a free open source implementation of the software to provide you that resource, I'd consider installing it and selling you the resource maybe depending on how much of the resource you want to buy how regularly for how long.

These karma points are basically just another resource, they can be experienced as disk space and/or as bandwidth, a kind of spacetime resource maybe. If you don't want to buy that, point to code that explicates executably the precise resource it is you are looking to buy.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 09, 2012, 11:54:47 PM
I want to store X MB for Y hours so I publish this information and then then let the those nodes who want to sell me storage bid on it. My node uses a set of heuristics to pick the best bid taking into account all relevant information (price, reputation, bandwidth, etc). The system does this for every resource that you want to allocate.

Ok that is strange, I had the impression that since you cannot know until the the information is delivered back to you or some kind of N-knowledge proof (where N might even be zero even???) proves they do still have it when the time you paid for is up, and possibly even had it constantly between the start of the timespan and the end of the timespan, they they actually did store it.

Thus my impression had been that it is better to pay after the fact for performance than to believe bids up front.

Thus I thought an "I will pay X for block Y" much more robust.

I suppose maybe asking for estimates ahead of time for how much the network thinks it might cost to buy such a block some certain timeperiod after you submit it for consideration might help give you hints about how often to buy a copy for how much in order to ensure someone out there will be eager to sell it to you next time you ask for it, but really, if it turns out you can get it cheaper from numerous providers than the initial estimates you were given, why not go the cheaper route?

However, I guess you are more in the market for some vapourware-provided service than for something people can actually fire up an app to provide to you by the sound of it?

Is there any bounty offered for moving this vapourware onward toward people being able to actually store your, uh, how many bytes exactly for, uh, how long a timespan is it you want it stored for? Oh and how much are you offering for storing the data once the software to do this for you has been gotten up and running?

I am trying to move on to actually getting offsite backups up and running, having to wait for some vapourware first isn't very appealing. I'd prefer to get to a point where people can store data and can if they wish use bitcoins to encourage the robustness of the storage solution before waiting for some future development that might make doing that even easier than it already is using already working code.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 09, 2012, 11:58:20 PM
I suppose maybe asking for estimates ahead of time for how much the network thinks it might cost to buy such a block some certain timeperiod after you submit it for consideration might help give you hints about how often to buy a copy for how much in order to ensure someone out there will be eager to sell it to you next time you ask for it, but really, if it turns out you can get it cheaper from numerous providers than the initial estimates you were given, why not go the cheaper route?
That's a pretty good description of the theory of operation of a futures market. Believe it or not such markets exists and function reasonably well.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 10, 2012, 12:18:40 AM
I suppose maybe asking for estimates ahead of time for how much the network thinks it might cost to buy such a block some certain timeperiod after you submit it for consideration might help give you hints about how often to buy a copy for how much in order to ensure someone out there will be eager to sell it to you next time you ask for it, but really, if it turns out you can get it cheaper from numerous providers than the initial estimates you were given, why not go the cheaper route?
That's a pretty good description of the theory of operation of a futures market. Believe it or not such markets exists and function reasonably well.

Okay then great. The store blocks on speculation model might work at least for some data. For example if you don't mind the data being stored in legible form, in the sense that anyone who would like a copy is welcome to buy a copy and is not likely to have any trouble executing it or displaying it or playing it - in general, "using it" - then heck thousands of nodes might eagerly store it for you forever just on the strength of the income they hope to get by selling it on to others.

As we have seen from the difficulty of finding hash collisions though, if the data is encrypted the chances are very low that anyone other than you will ever be paying us for a copy of some random block of an encrypted file only you know the key for, so the future looks kind of dim unless we do have some reason to believe you either will buy it back or might very unlikely buy it back but will pay well for it if you ever do. We'd store your data on speculation, much as some three-letter agencies maybe try to store every phone conversation that ever passes over international lines or some other vast collection of possibly largely/mostly useless information.

But is it important for us to care whether or not that is how karma points actually work inside the code/network?

Can we mere humans not get along fine simply deciding whether or not to buy our node some more karma points instead of worrying about the nitty gritty details of how this version or the next updated version of the software actually does its wheeler-dealer dealing to come up with the karma balances it assigns to its neighbors?

Don't we really just want our backups when our local system crashes? Or want our photographs back that we moved offsite without even keeping a local copy of?

Certainly I want offsite backups, real ones in the sense that I actually can get a copy from offsite if my onsite data ever does get lost thus create a need for restoring from backups and my onsite backups also are lost. It would be nice if some shopping app could shop around and find the best price for sufficiently reliable storage but its not as mission-critical as actually having such backups in place.

Right now I could pay bitcoins for karma points on other people's GNUnet nodes if my own GNUnet node turns out to have insufficient karma to accomplish this critical mission.

It happens though that I have not yet experienced sufficient shortfall of such points to be aware yet of any current need to boost my karma by paying for karma points. Maybe other people have?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 10, 2012, 12:23:30 AM
MarkM, I wish you could be a bit less wordy.  You have good thoughts, but your posts are incredibly difficult for me to follow, for some reason.

Anyhow, if I understand you correctly, your proposal is that people pay for files only when they need them?  So they say, "hey, I might need these files in the future, store them for me and I'll pay you $10 if I need them in the future".  Those with higher amounts of bandwidth would perhaps be able to provide those files more quickly, and thus more likely to receive the bounty for a file download in the future if it is necessary.  That's an interesting twist...  I just wonder if people would be willing to store files if future payment is unknown.

I am not sure how the original idea would work with regards to bandwidth vs storage.  Maybe one provider would only be able to provide 100 GB of bandwidth even though they have 1 TB of storage.  Or it might be vice-versa for another.  Perhaps a "bandwidth/storage" ratio would also be provided, along with how much storage is available.  And of course, an "uptime" indicator and data-checker would be a necessity.d


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 10, 2012, 12:32:16 AM
If karma works right, which it is intended to and thus hopefully in some version of the software will do, and might in fact already do, then it should condense nicely all that complication down to a simple do I need more karma with more other nodes or am I satisfied with the level of performance my existing karma provides.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: giszmo on June 10, 2012, 01:38:09 AM
Not sure if this was addressed before but if I get payed per MB I host, each such server would need a different key to encrypt the data as else I could register 12000 servers with one actual server, mimic to provide redundancy but serve all from the same disc. Those that pay for 12 copies would end up with 11 copies on my "cluster" paying me 11x the money.


Title: Re: [IDEA] Dirt cheap online storage
Post by: mollison on June 10, 2012, 02:58:49 AM
In my opinion, this doesn't solve a real problem that anyone has in a way that could be even remotely useful.

Cloud storage will always be superior and the kind of problems people want to solve with this P2P storage idea could be solved much better and more cheaply by extending the basic concept of cloud storage.

This idea would create more problems than it would solve.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 10, 2012, 03:56:15 AM
If karma works right, which it is intended to and thus hopefully in some version of the software will do, and might in fact already do, then it should condense nicely all that complication down to a simple do I need more karma with more other nodes or am I satisfied with the level of performance my existing karma provides.

-MarkM-

I don't understand why any sort of concept of karma is necessary if people just use Bitcoins for it.  Pay Bitcoins if you need space, receive Bitcoins if you rent it out.

Not sure if this was addressed before but if I get payed per MB I host, each such server would need a different key to encrypt the data as else I could register 12000 servers with one actual server, mimic to provide redundancy but serve all from the same disc. Those that pay for 12 copies would end up with 11 copies on my "cluster" paying me 11x the money.
Why couldn't the client encrypt the data before sending it?  Why does each server need to encrypt it?

In my opinion, this doesn't solve a real problem that anyone has in a way that could be even remotely useful.

Cloud storage will always be superior and the kind of problems people want to solve with this P2P storage idea could be solved much better and more cheaply by extending the basic concept of cloud storage.

This idea would create more problems than it would solve.
If cloud storage is superior, then why is it so expensive?  $125/TB/month?


Title: Re: [IDEA] Dirt cheap online storage
Post by: rjk on June 10, 2012, 04:01:18 AM
If cloud storage is superior, then why is it so expensive?  $125/TB/month?
In the case of AWS, it is costly because it is so extremely redundant to absolutely paranoid levels. You can save 30% by choosing Reduced Redundancy Storage, but even then they store you data across several servers.

For Standard (paranoid) redundancy, they say "Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year. Designed to sustain the concurrent loss of data in two facilities."
And for Reduced Redundancy, they say "Designed to provide 99.99% durability and 99.99% availability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.01% of objects. Designed to sustain the loss of data in a single facility."

So for the first option, you could literally wipe out two whole datacenters with missiles or whatever, and they would still have a backup copy.


Title: Re: [IDEA] Dirt cheap online storage
Post by: giszmo on June 10, 2012, 05:29:40 AM
Not sure if this was addressed before but if I get payed per MB I host, each such server would need a different key to encrypt the data as else I could register 12000 servers with one actual server, mimic to provide redundancy but serve all from the same disc. Those that pay for 12 copies would end up with 11 copies on my "cluster" paying me 11x the money.
Why couldn't the client encrypt the data before sending it?  Why does each server need to encrypt it?

Because ... if each server is allowed to hand out the exact same bytes, the client can not tell 2 IPs pointing to the same disk from 2 distinct servers opening the door for said scammer that claims to do all the redundancy for you while he only has one server but 12000 accounts.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 10, 2012, 05:52:37 AM
I don't understand why any sort of concept of karma is necessary if people just use Bitcoins for it.  Pay Bitcoins if you need space, receive Bitcoins if you rent it out.

Don't get confused by the variable-names or metric-labels:

"Pay bitcoins if you need [RESOURCE_OR_SERVICE], receive bitcoins if you rent it out."

If what you need is, purely and solely, storage space, have you compared the monthly cost per terrabyte of writeable DVDs plus the oneshot upfront cost of installing enough DVD drives to put that much DVD space online all at once at your own location?

If what you need is, purely and simply, offsiteness of space, have you compared the monthly cost per terrabyte of sneakernetting or snail;mailing the DVDs to an offsite location?

If what you need is purely and simply, transport from onsite to offsite, again compare DVD-by-sneakernet.

If what you need is, purely and simply transport from offsite to onsite, again have you compared price of DVD-media transport?

It continues to sound though as if you are actually trying to palm of or fob off some dynamic combination of probably at least three of these pure and simply commodities onto a one-dimensional metric (known as bitcoin) that already carries the burden of also having to be equal to dynamic quantities of an endless number of other pure and simple things and complicatedly dynamic things and indeed any thing or service whatsoever.

If you really want to take it down to how many bits you receive from offsite, how many bits you send offsite, how many copies of each bit reside on each reliability of peer storage reachable at each possible bits per second rate at what percentage chance on any given second, minute, hour, day, week month or year, that is vastly complicated set of markets, where you or your agent needs to determine how many bits per second seconds how many bitmonths of how likely to be accessible when you need it storage is equal in value to you to how much more or less of any of those factors. How much do you value the snappiness of response, as measured in number of bitmonths stored? Does it differ according to how many bitcoins the material requested has been stored? And on and on and on. Is two bytes of storage worth more or less to you than two bytes per timespan for two timespans of transport? Is outgoing transport worth more or less to you than incoming trasport? Etc etc endlessly.

Once all these factors as to your personal preferences as to when you want how many more bits per second rather than how many bitmonths of remote storage in use and so on and so on have been all reduced down to a single scalar numeric value, then how many satoshis or bitcoins are  you willing to pay per how large a value of that scalar metric can be tested by offering you such a scalar on a market.

If however you insist that the scalar itself be labelled bitcoins, then it is reduced to a market in which the commodity being auctioned off is known as bitcoins due to your insistence that the scalar result of the reduction of the commodity to a single scalar must be bitcoins rather than being number of bits of data you have on offsite DVDs available by sneakernet using couriers, or speed of vehicle said couriers will use up to and including putting the DVD into a transporter so it materialises on your desk or directly changes the magenetism of your hard-drive.

It is too vast a field of not-the-same-thing that you are trying to boil down into one thing. Having boiled it down to one thing, whatever one thing you boil it down to so as to have a BTC<->ThatThing market, if you insist that thing ITSELF must ITSELF be known as bitcoins rather than being known as "general scalar metric of offsite-storage-network resource consumption quota" you will just end up with a totally confusing market, a "BTC meaning the digital currency"<->"Bitcoins meaning the metric of usage of network storage resources subsuming upload volume, download volume, storage volume and duration, bits per second sustained of download, bits per second burst of download, bits per second sustained of upload, bits per second burst of upload, and other ancillary factors".

It seems simpler and clearer to use some word other than "BTC" or "Bitcoin" or "Bitcoins" for that second metric, the commodity metric being offered for sale not AS bitcoins but IN RETURN FOR bitcoins.

In GNUnet, if I recall correctly, that commodity-provided-by-this-system metric is stored in a variable some coder or other happened to name "karma".

It is simply a human-and-compiler-readable label meaning "that scalar value which serves as the numeric metric of the commodity provided by this network/application".

Hindu or any other mysticism might only be some strange random reference that coder happened to find useful at coding time as a mnemnonic by which to refer to that variable/metric.

So for your purposes maybe it would suffice to use a preprocessor macro to change the name of that variable to QUANTITY_OF_SERVICE_DESIRED_BY_USER so that you will be able, upon reading the code, to make the mental leap to the aha moment that, in such a network, THAT is the variable you would be best served by spending your bitcoins to buy an increase in numeric value of the bits that variable refers to as interpreted as representing a number.

Or, you could chose to change its name to AMOUNT_OF_STORAGE_DESIRED everywhere instead, while people also interested in how much speed it amounts to rather than how many years bits offsite will survive if they don't request them for years could use some other label for it.

It is a complicated service involving redundancy and latency and bandwidth and on and on and on. Marketers would love to game you by offering a different product, different in any of the details that go into the whole generic concept of getting your bits offsite and being able to get them back onsite, whether by dialup or sneakernet or wireless or TCP or UDP or whatever, so they can claim to be a different product entirely, maybe not even a commodity (generic) product even maybe but rather a prestige boutique product incomparable to commodity offerings so they belong on a different market entirely and should not be compared to commodity offerings, especially not on price, since afterall their vast marketing budgets and ridiculously ignorant user-bases' need for hand-holding's accomodation by vast helpdesk centres precludes any comparison to any raw commodity.

To be a commodity that can be assigned a scalar value in bitcoins,  a thing has to be gotten down as close to a scalar value as possible first in order to have a single value, usually labelled quantity (as in X number of porkbellies, X number of grams of gold, X number of remote storage bitspeedmonthdirections or whatever), so that a market can be a "quantity of money" <-> "quantity of that very very specific commodity" market.

Notice even the grams of gold might be this that or the other fine-ness of gold, so even there there are more variables that in casual chat one takes for granted but that in fact a .999-fine grams of gold market is distinct from a .333-fine grams of gold market and so on.

To commoditise a thing you need to get it condensed down to the point where a market offering a scalar value of it per scalar value of a currency makes sense to the participants.

Already in Open Transactions we are seeing how fragmentation of markets scares off potential customers, with some potential customers thinking that having separate markets for ones of bitcoins, tens of bitcoins, hundreds of bitcoins (or s/bitcoins/porkbellies/ or s/bitcoins/shiploads of porkbellies/ or s/bitcoins/grams of porkfat/ or whatever, an assset is an asset is an asset, all are tradeable each against each) is in some way bad or awkward or "too complicated" or something even though in business it is actually very often the case that the price of grams of something, per gram, differs from the price of tons of that same thing, per gram.

So dont keep getting hung up on the name of the variable that serves in the code as the measure / metric of what it is that is being provided, instead consider whether that which is provided suits your needs and whether you wish to be provided with more of it or less of it than you are currently being provided. If you want more of "the service", get that number increased, whether by paying bitcoins to have it increased or by whatever other means. It is the measure aka metric of the service provided by that software.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 10, 2012, 06:21:52 AM
So you are seriously suggesting DVD's as a potential competitor.  Who wants to wait for 200 DVD's to burn?  On top of that, who actually trusts that, out of 200 DVD's, none of them will have an error when you try to restore from them?  On top of that, there's the convenience issue of not being able to access your data immediately if you need it.  On top of that, what "offsite storage location" is going to offer you safe storage for zero cost?  A family member's house?  What if they lose them?

I agree that the pricing model is complicated.  It seems that at least three metrics would need to be calculated and available.  Storage space, bandwidth, and speed.  And determining a pricing matrix between the three would be quite arbitrary and, well, confusing.  So perhaps, reducing it to one metric (storage space), and imposing certain minimum standards on the other two would be appropriate.  A host much be able to provide download speeds of at least 100 KB/s, and must be able to provide bandwidth equal to or greater than the storage space rented on a per-month basis.  Anything above and beyond that would be a bonus.

Or, looking at it from a different perspective, perhaps a central agency (I know, naughty words here) could match up buyers and sellers as appropriate.  The agency would know who has what storage, bandwidth, and speed capabilities.  For each buyer, they might choose a host with lots of speed, a host with lots of storage, and a host with lots of bandwidth, then serve from each of the three depending on how often the person needed the data.  If they end up needing the data often, work with the host with lots of bandwidth.  If they end up needing the data infrequently, let them download at higher speeds.  Etc, etc.  Yes, it would be complicated, but maybe less complicated than trying to figure out a decentralized way of matching up buyers and sellers.

Certainly, the simplest method of conducting such a project is to have a page where people host files to be downloaded, with a bounty in the future if they need it.  This would mean the host with the most space would have the largest chance at being able to provide a file that is needed, and would also mean the host with the most speed would have the biggest chance at providing the file more quickly than anyone else.  Yes, it is uncertain for sellers of space, but it may only take a handful of recoveries to make up for it entirely.  I actually think this would be the most practical route to go, and it's an interesting idea that hasn't been done before (AFAIK).

I still don't see why you want to move away from Bitcoins though.  How would tracking a karma score be any different from tracking Bitcoins themselves?  They're both just numbers...


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 10, 2012, 06:26:04 AM
Or, looking at it from a different perspective, perhaps a central agency (I know, naughty words here) could match up buyers and sellers as appropriate.  The agency would know who has what storage, bandwidth, and speed capabilities.  For each buyer, they might choose a host with lots of speed, a host with lots of storage, and a host with lots of bandwidth, then serve from each of the three depending on how often the person needed the data.  If they end up needing the data often, work with the host with lots of bandwidth.  If they end up needing the data infrequently, let them download at higher speeds.  Etc, etc.  Yes, it would be complicated, but maybe less complicated than trying to figure out a decentralized way of matching up buyers and sellers.
There's no reason to restrict the market-making function to a single entity. While the network is small you could even use #bitcoin-otc to match up buyers and sellers. That gives you an open order book and a reputation system to work with. When the network grows more exchanges could appear and people (or their software agents) could choose the best ones to use.

Creating a market mechanism is really as simple as automating the steps that you'd use to buy and sell manually. If you were looking to rent hard drive space you'd go to a place like #bitcoin-otc, look at the offers that were available and the reputation of the sellers and choose one or more of them to form a contract with. After the contract ended you'd leave either positive or negative feedback as appropriate and expect the seller to do the same for you. If either party provided inaccurate feedback the other one would attempt to prove his innocence in order to protect his reputation. People with higher reputation scores could charge more for their services than those with low or nonexistent scores.

Automating this is a matter of encapsulating in software a set of rules for the buyer and a different set of rules for the seller.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 10, 2012, 06:27:18 AM
Quote
How would tracking a karma score be any different from tracking Bitcoins themselves?  They're both just numbers...

Lets translate that question a litttle to make it clearer what is being asked:

"How would tracking [quantity of product offered for sale] be any different from tracking [quantity of currency offered in return]? They're both just numbers..."

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 10, 2012, 11:55:12 AM
For example, I'd be willing to rent out 100GB of space with 500GB of bandwidth for $5/month.  Dropbox will do the same for $20/month.  Add another user like me, who is willing to rent out 100GB of space for $5/month, and you have your two copies, but it only costs the user $10/month instead of $20/month with dropbox.

Thoughts?  Would this be doable/practical/feasible/useful?

(...You didn't say how many separately-sited copies of your data dropbox's offer is/contains/represents...)

Okay, I have spent the night digging back into the details of various actually currently working software that I could run, but as you did not indicate what operating system you plan to use to provide the storage you are at least hypothetically offering I do not know if you could run them.

I am finding so far only two systems I actually have managed to get installed and working that seem like one or more of them of them might suffice for an offer such as is quoted above.

That is, putting aside all the theory and vapourware-wishlists and philosophy and looking at that specific offer as a potentially-tangible offer.

Those two systems are GNUnet, which currently requires a unix-type operating system to run, and Tahoe-LAFS, which is python and seems to think it is usable even on Windows.

Part of the differences between them is that GNUnet is not only about storage, so it has the "overhead" of trying to provide some anonymity. Requests sent out do not have to originate at the node your node receives them from, and nodes deliberately keep packets moving as background noise all the time so that timing attacks cannot notice aha look that node talked to that one, without having heard from another one recently, so I bet it originated that request itself.

So if you do not want anonymous routing going on,  Tahoe-LAFS seems cheaper in resources to use as provider, provided your customers don't want the kind of anonymity GNUnet adds into the service-offering.

However, Tahoe-LAFS is organised as "grids", each "grid" being basically the collection of nodes a particular "introducer node" is able to introduce clients to. So in order to offer a Tahoe-LAFS based service you'd either have to have a few machines, probably best not all being in one site; or a number of us would have to team up to make one or more "grids" we would between however many of us be able to populate with enough machines, at enough sites, to be able to present the services of our "grid" as a credible service-offering.

I suggest that if you wish to sell your $5/month offering we should maybe consider teaming up with a third person so we can have at least a three-sites grid running, and consider whether to each use the price shown in the quote, totalling $15/month for someone wanting all three of us to store that much data for them so they have three copies out on the grid; or consider charging slightly less per redundant copy located at separated locations.

Did you happen to notice how many separate sites how many copies of your data the $20/month commercial offer was offering?

Despite the "overhead" of the (possibly extraneous to many people) anonymity that GNUnet provides, I think we would be better advised to go with a GNUnet solution, partly because using Tahoe-LAFS we would be trying to compete very directly against commercial services.

I am thus thinking lets start small, using GNUnet, then as we get customers for that "value (of being anonymous) added" service, revisit the price per gigabyte per separately-sited copy we could offer and whether it would be worth our while at the rates trying to compete with the commercial services would force us to drop our prices to.

WIth GNUnet we not only get to have this "value adding" feature of the anonymity stuff, but also the "it runs itself" feature of quite possibly providing quite a lot of its participants enough offsite storage and anonymous packet routing just in node to node barter that we could all have a useful number of usefully stable offsite backups without having to sit down and figure out how much bitcoin to send each other at all. The system has the virtue that we can all have these things for ourselves by barter up to the point where someone wants to receive more service than they provide. Those people will become thereby the potential "customers" who might go shopping around among those of us who have stably, long term, reliably had service to spare that we can offer them.

Basically it should find out for us automatically who among us actually does have surplus available and who among us is suffering sufficient shortfall of provision to start thinking about spending some bitcoins having in a sense run out of actual service of their own to barter with.

With Tahoe-LAFS, not only are folks not anonymous but also we each have to jump through gosh knows what hoops to figure out how much of our resources which other person is consuming more of than we are consuming of their resources. The administrative overhead thus looks like it would be much higher  not onlly for those actually providing service but also for those wanting to leech or buy service (the potential "customers" among the participants).

-MarkM-



Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 11, 2012, 04:14:47 PM
I do not know how many different sites dropbox has with redundant data.  A cursory google search reveals no further details on it either.

On pricing, I don't think it needs to be a managed item as long as it is built in to the software.  For example, there might be 5 hosts available with 100 GB of free storage space.

Host 1 charges $0.03/GB/month
Host 2 charges $0.04/GB/month
Host 3 charges $0.05/GB/month
Host 4 charges $0.05/GB/month
Host 5 charges $0.06/GB/month

I go to rent 100 GB of space.  I am given 3 options:  2 hosts, 3 hosts, or 4 hosts, with prices for each level.  2 hosts will (currently) cost me $0.07/GB/month.  3 hosts will cost me $0.12/GB/month.  And 4 hosts will cost me $0.17/GB/month.  More hosts will protect my data more securely.  But, if one host goes offline, then another host will take its place, although possibly at a higher price (if no further lower priced hosts are available).

I think it would certainly be possible to utilize a modified version of Tahoe-LAFS, but I do not have the knowledge or ability to make the code changes necessary myself.  That's why I haven't really spoken to what software should be used much - it would be more up to the programmer creating the software than anything else, as to whether he'd like to start from scratch or build on an existing project, and which project that might be.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 11, 2012, 06:30:24 PM
Google Drive actually beats your example quoted price, something like $4.95 or $4.99 per month for 100 gigs of storage, no mention of any upload/download fees on top of that, at least not that I noticed. Like dropbox I guess they are going for simple instead of frightening the common people off with nickel and dime stuff that could add up to some unknown in advance total. If you want larger quotas, prices per terrabyte keep going down in more steps as you upgrade to larger quotas.

Nimbus.io offers $6 for 100 gigs, with also $6 per 100 gigs of transfer. ($0.06 per gig for each), and they are free open source software so that anyone who wants to set up storage servers in clusters of ten for parity-striping of data onto ten servers at a time can do so. (So we know they are based on ten hosts at a time, so lets figure we'd have to charge no more than a tenth what they do if we want to offer one host at a time prices.) Their software obviously is set up to do just the kind of pricing in your example quote so maybe if you set up ten virtual machines on your machine you could use it even for just one machine. It has the website, customer info handling and everything by the look of it.

I have found out more about GNUnet. It actually does not "publish" files out onto the grid at all, replication is only an efficiency thing, in that if anyone actually does ever ask for a copy of your file, then it will tend to persist some time at the mode that requested it and possibly even at places along the way to the requesting node. So you don't "push" your files out, instead someone out there has to request from you. Still, given a way of telling a particular node a file to request from you, it could be useful.

Unfortunately though I think most fly by night random hosts that fire up some software to see if it will earn some pennies are not going to be useful for storage. They could provide nice random topology for complicated routing of transfers for anonymity purposes but there is so much chance of the user turning off the machine when they go to bed, or killing the app after half an hour to a few hours when they get bored of playing with it and have not seen any pennies trickling in, that that there seems little point sending them any blocks to store except maybe a canary - a test block to ask for next week if they even manage to stay online a week, which they likely won't.

Did you see the Retroshare thread? I had 28 or more bitcointalk users as "friends" in that friend to friend network, and within a few days tended to have usually no connections, occassionally seeing one connection and maybe catching a glimpse of two for a moment. And this is bitcoiners, a population among whom many have machines that are running 24/7. Among "normal" people I wouldn't be surprised to see even worse results.

I would have needed every single one of those 28 people to have a copy of my data in order to have found one copy still reachable ofter a few days or maybe a week or two, the short time it took for the vast majority of them to fade into the night.

So your idea of lets buy three sites worth seems tragically low. Better buy 30, and still recognise that just gives you a chance one might survive to the end of the month maybe, if the people involved are as used, on average  to keeping their machines on 24/7 as bitcoiners (or bitcointalk members anyway) on average are.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: PocketCoin on June 11, 2012, 07:44:54 PM
Subscribed so i can come back and read everything later :D
very interested.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 11, 2012, 08:25:00 PM
MarkM, I respectfully disagree with your assessment that fly-by-night random hosts would require more copies of the data.  The whole idea of host uptime is to ensure that fly-by-night random hosts aren't used.  Right from the start, files would only be stored on hosts with more than, say, 99% uptime, and priority of storage would be given to hosts with the greatest uptime beyond that.

Retroshare did not offer your friends from Bitcointalk enough incentive to stay online, so they went offline.  But if someone is paying me to keep my computer online, especially when my computer is already online 24/7 anyway, then I'll keep running the software, even if it is only giving me pennies a day.  That's still more than I had the day before.

Does Google drive charge only for what it used?  That is another manner in which this could be a useful service.  Not forcing users to pay for more storage than they actually use.  That said, GDrive pricing is very competitive!

I'd be willing to go even lower on my personal computer to compete too.  Heck, I'd rent out 100GB for $1 a month!  That'd make use of my currently-unused 1TB WDG drive, to the tune of $10/month if fully rented.


Title: Re: [IDEA] Dirt cheap online storage
Post by: MagnaGen on June 11, 2012, 08:27:27 PM
Cool idea but seems like way too many points of failure


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 11, 2012, 08:48:47 PM
There seemed no point in even considering offering any deals/offers/incentives to the RetroShare people, since long before enough time had gone by for me to start thinking "hey, some of these people seem to have been online 24/7 for a month or more maybe that could be useful" all but one of them had already demonstrated total lack of reliability. Heck it was days I think, though my sense of time is poor so maybe over a week, I suppose it may even have hit two weeks.

You make a good point though, good enough maybe to motivate me to turn on RetroShare every week-or-few and make deliberate note, maybe even written not merely a mental note, if I do happen to see a neighbor, of which one it is; so as to notice over time whether in fact it is the same one every time. If so that one might be worth thinking of for a scheme like this.

Basically the evidence was so overwhelming that the people were not reliable that the idea of offering to buy reliable service from them never even came up.

However I recently read somewhere that the length of time a host has reliably been online isn't actually much use in predicting whether they will stay online. Maybe some insight into whether they are actually making enough to cover expenses could help make such predictions though.

It seems a little like Amazon spot instances though in a way; any time the current bids outbid you, you lose your instances. Maybe the moment someone outbids you for storage, poof as much of your data as a financial-attack attacker chooses to displace with their own could be gone.

So maybe another metric to track would be what happens when you are outbid? Which hosts retain your data for how long in spite of higher bids being made bidding for the space they are providing to you?

Maybe we need time-scaled markets, covering for how long they will retain your data to give you a chance to move it or to increase your bid in the event you are outbid by how much? (If they are offered double the price you offer, how long grace time do you get? What if someone offers 10 times as much? 100 times as much? The ethical/moral equivalent of pool-hoppers will presumably be out there...)

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 11, 2012, 09:08:52 PM
I think you're still missing the mark on the retroshare - those people didn't stay online namely BECAUSE there was no incentive, present or future, offered.  If they knew they could make money by staying online (or similarly, if they could build up host reputation to then rely upon to make money at some point in the future), then many more of them would likely still be running the software.

You do bring up a good point about a person being "outbid".  After all, what host wouldn't just drop their current users in favor of users offering more money for the same storage space?  One way to mitigate that problem might be to use a central entity to receive payments, then pay out those payments equally to all hosts offering storage.  The cost to store would be equal to the highest minimum price paid for storage available.  Expanding on my previous 5 hosts example then:

Host 1 charges $0.03/GB/month
Host 2 charges $0.04/GB/month
Host 3 charges $0.05/GB/month
Host 4 charges $0.05/GB/month
Host 5 charges $0.06/GB/month
Host 6 charges $0.07/GB/month
Host 7 charges $0.07/GB/month
Host 8 charges $0.08/GB/month
Host 9 charges $0.09/GB/month
Host 10 charges $0.09/GB/month

If all of the customers combined only needed enough storage to warrant the use of 6 of those hosts, then the lowest-priced 6 hosts would be used to determine price.  The customer would be charged the rate of host 6 for ALL of their storage (for example, for 3 hosts, they'd be charged $0.07*3 = $0.21/GB), and the payout would be equal to all hosts ($0.07/GB/month).  In this way, there would be no incentive for a host to change whose files they are storing, though it would make it more expensive and variable-cost for the customer.  Again, it'd be like the GPUMax of storage.

Aside from that solution or doing the "pay to recover/download" model, I can't think of any other way to make it work.  I do like the pay to recover method for its simplicity though.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 11, 2012, 09:30:01 PM
I forgot to take into account the fact that of Nimbus.io's ten storage nodes per cluster only any two of the ten can vanish without losing data.

So each such cluster only really amounts to three copies of your data in a way since if three nodes go down ouch, pow, data loss.

They may well customarily keep people's data on more than one cluster at a time though.

I prefer watching the RetroShare folk looking out for potentially reliable ones to advertising a bribe because the bribe might bring all the bottom-feeders swarming in, all desperately trying to find a way to game the system. If I actually planned to attract those people I would try to design a system that would get as much use out of them as it can during the time they spend trying to game the system and failing, basically in effect planning to game those that try to game the system.

Part of why I was attracted to the RetroShare thread was they were not ostensibly just in it for money; it is even possible that what failed really was the backup bitcoin-forum the thread initially proposed. Actually it might not even be a failure, since maybe if bitcointalk forum does go down some day, all those RetroShare people will all fire back up their RetroShare, reconnect, and start actually using the bitcoin forums they created there. So maybe the real test of their reliability would be to turn off bitcointalk and see how many of the people hosting the backup forum on RetroShare actually do bring RetroShare back up.

Amazon's spot instances do use an auction where you only pay the minimum price needed to buy as much as people want-at-that-price. Everyone who gets an instance-hour in a given hour pays the same price per hour.

The why not dump all your users if someone else offers more could maybe be something insurance companies could cover? Post a bond with an insurance company, it then insures your users against you losing the users' data.

However I think maybe we are basically starting to glimpse *why* online storage *is* so darn expensive. :)

Maybe some pass-through apps would be useful, that let people go get their free five gigs of dropbox, google, etc, etc, all kinds of free for life storage, then do passthrough to others...

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 11, 2012, 09:52:31 PM
I am not typically interested in any project that isn't "in it for the money".  Namely because those projects usually fail, either due to lack of developer incentive (who wants to put a bunch of time and effort into something that doesn't give them something in return?), or lack of updates, or lack of moderation, or lack of users, etc.  Money gives project organizers and users incentives to use the project, which is why I like projects in which the developers (and/or users) receive monetary compensation.  As you've seen with retroshare, not many people are inspired to continue to use a product that is virtually useless to them unless they are being paid to do so.

Also remember that a lot of people using Bitcoin are only in it for the money too - either in it for the long run, because they believe Bitcoins will be more valuable in the future, or in it to make money mining, etc.  Some use it because they believe in the fundamentals of decentralized and deflationary currency, but I doubt that the majority of Bitcoin users fit that profile...

I don't think insurance for users' data would be necessary if all customers paid the same price, and all hosts received the same compensation.  Make 2 the minimum number of hosts at which data can be stored with, and if one of those hosts drops offline (which would likely happen very rarely, if 99%+ uptime hosts are chosen), then the 2nd host would simply copy the data over to a 3rd host.

A pass-through app utilizing multiple smaller free hosting solutions would be interesting, but I wonder what the developer incentive would be to create and continually update such an application.  I also wonder if various hosts with free storage would simply block that application from using their storage since it could be seen as an abuse of their free accounts.


Title: Re: [IDEA] Dirt cheap online storage
Post by: punningclan on June 11, 2012, 10:01:19 PM
I'd gladly free up and manage some gigs to be paid some Bitcoin. I'd probably also back up the the data, if the pay were enough I'd invest in a RAID.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 11, 2012, 10:10:17 PM
There is already at least one commercial pass-through app, or something very similar to it:

http://smestorage.com/?p=static&page=about#about-clouds

Google Drive does not mention mounting it as a drive in unix/linux using webdav protocol, but the chap who gave me the above link seemed to be saying that using that service he could use webdav to mount his google drive space.

Unthinkingbit too is concerned that developers of free open source stuff often are not paid well; that is why DeVCoins exist. Maybe DeVCoins would be more suitable than BiTCoins for developing this concept since implicit in DeVCoins is developers get coins. So far in your concept this is the first I have heard of any hint of a whisper of a suggestion that developers should get paid, it was all about users (of the software and the network of users it marshals) getting paid.

I wonder whether we even need random home-users though, maybe we should be looking at operators of virtual machines as our target nodes for acting as host nodes. Home users can be target consumers of storage, no need to use any of their own on site storage at all, what we would want them to contribute is coinage, not storage. Even if they do have lots of storage available, so what, that just means they are not likely to be a paying customer. No need to concern ourselves with them until they do not have enough storage and thus go shopping around for more.

In fact if we could estimate very firmly how many terrabytes of storage a large ISP's home-users would offer to sell and how much bandwidth that would cost the ISP maybe we could even go directly to the ISPs suggesting they sell us the storage cheaper than their users could due to not having the bandwidth cost of sending our data all the way to their users and back...

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 11, 2012, 10:25:11 PM
Devcoins are virtually worthless - I don't know why a developer would work to gain them.  And why not use Bitcoins?  How is Devcoin ANY better suited to pay developers than Bitcoin, aside from name?  I didn't mention developers getting paid because it absolutely depends on who is willing to develop something like this, and whether they would like to be paid or not.  If there is a central entity involved, then it is fairly likely that they would take a cut of the hosting fees gathered, and would be willing to front the costs of having a platform like this developed.  If it is a truly decentralized project with no central entity involved, then either some developer needs to believe in it enough to put it together without pay, or some bounty needs to be posted by users interested in seeing it happen.

I made it all about the users because, ultimately, a platform developed without the users in mind is worthless.  So, first figure out what the users want, then decide on how it will be developed and managed.  Right now, we're still in the "figure out what the users want" bit, which means how the platform is developed and whether developers are paid or not is largely irrelevant.  All I was saying in my previous comment is, I'd like to see a platform that allows developers to be paid, either via a central entity taking a cut of the fees, or via a bounty.  That's not a necessity, and obviously, it depends on what users want and need.

I don't know why home users should be ignored, either.  If they can provide storage at a lower cost than VPS providers (which are typically very expensive), then why not include them?  Reselling professional hosting is not the goal here.


Title: Re: [IDEA] Dirt cheap online storage
Post by: enmaku on June 11, 2012, 10:33:58 PM
I'm intrigued... Sign me up for both sides of the fence if you ever get it up and running.

Also, make damn sure every file uploaded is on more than one or two drives and encrypted with keys that the storage providers don't have ;)


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 11, 2012, 11:02:03 PM
For sure the storage providers should not to be able to read the files without the permission of the owners of the files.

There is plenty of free open source code that does that, in fact I have had the impression that part of what drove open source developers was often precisely the tendency of commercial service providers to set services up in such a way as to allow the providers access to as much of the customers' data as possible.

Just recently I was told that one share of devcoin generation is worth about 15 bitcoins. To me fifteen bitcoins is a nice amount to have flowing in regularly just as a side of effect of the fact I develop free open source stuff, and even a majority of the bitcoins I have made directly as bitcoins I received as bounty for working on devcoin and devcoin-related stuff.

It is true that each individual devcoin is worth far less than each bitcoin, but that was a deliberate design decision; bitcoins were skyrocketing back then, there was talk of having to move the decimal-point because users would find numbers with too many decimals awkward, so devcoin deliberately set out to be worth a thousandth or less of a bitcoin. 50,000 are minted per block accordingly.

The people who want to make coin by renting out space might be potential customers for the developers, but they are not potential customers for the users who supply the coins in the normal mainstream manner of thinking. Yes, bitcoin owners sometimes want to sell bitcoins, so any merchant willing to accept bitcoins is a customer of the person who sells them bitcoins in return for other goods or services, but if we take your view of wanting this project to be a consumer of bitcoins rather than a producer of bitcoins we want to be buying bitcoins not buying storage, right?

The people who have bitcoins do sometimes look for customers to sell them to, so maybe this is about how many of their offered bitcoins we can afford to buy given the amount of development ability and storage we can offer in return for their bitcoins.

Maybe we have to look at this in two layers: one, the person who actually wants storage. Two, the person who wants to sell them some. Three, the facilitator(s), which presumably amount to being markets, offering to bring buyers and sellers together.

To operate such a market, maybe a whole new asset-type, OSA (Online Storage Asset) would be useful. Persons wanting to sell storage could look at a prospective customer's OSA balance to decide whether that customer even has enough OSA to buy the storage the storage-vendor wants to sell, and the people wanting to buy storage could look at vendors BTC (BiTCoin) balance to see if the vendor has enough BTC balance to recompense any loss of data purportedly being stored. The market could act as escrow both for both the data stored and the coins paid. And yes I am thinking I mean here escrow of the actual data, not the abstract OSA asset. If the storage provider fails to provide the storage, the data stored on it can be recovered from the escrow aka the operator of the market? Hmm. Shoot that down for me, I am surely not seeing the problems with it clearly.

EDIT: I see one problem already: I seem to have shifted somewhere in that paragraph from OSA as a thing one buys storage with from storage vendors to OSA as actually stored data. I think that the confusion is that the OSA would amount to being coin/value that is commited to buying storage. Looking at someone's OSA balance is different from looking at their bitcoin balance because with bitcoins they might buy anything, so could spend it on something else. Their OSA balance though is purpose-specific. They already spent bitcoins or dollars or icecreamcones or sticks of gum or whatever to buy OSA units, now they are looking at vendors to decide which vendors to give them to. Something like that. I am getting confused.

...Maybe because of the layers. Coin vendors gave coins to the market/escrow, then the market/escrow is looking at storage vendors to choose which ones to obtain storage from...

-MarkM-





Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 11, 2012, 11:23:01 PM
For sure the storage providers should not to be able to read the files without the permission of the owners of the files.

There is plenty of free open source code that does that, in fact I have had the impression that part of what drove open source developers was often precisely the tendency of commercial service providers to set services up in such a way as to allow the providers access to as much of the customers' data as possible.

Just recently I was told that one share of devcoin generation is worth about 15 bitcoins. To me fifteen bitcoins is a nice amount to have flowing in regularly just as a side of effect of the fact I develop free open source stuff, and even a majority of the bitcoins I have made directly as bitcoins I received as bounty for working on devcoin and devcoin-related stuff.

It is true that each individual devcoin is worth far less than each bitcoin, but that was a deliberate design decision; bitcoins were skyrocketing back then, there was talk of having to move the decimal-point because users would find numbers with too many decimals awkward, so devcoin deliberately set out to be worth a thousandth or less of a bitcoin. 50,000 are minted per block accordingly.
You still haven't answered my question though:  Why pay a developer in Devcoin instead of Bitcoin?
Quote
The people who want to make coin by renting out space might be potential customers for the developers, but they are not potential customers for the users who supply the coins in the normal mainstream manner of thinking. Yes, bitcoin owners sometimes want to sell bitcoins, so any merchant willing to accept bitcoins is a customer of the person who sells them bitcoins in return for other goods or services, but if we take your view of wanting this project to be a consumer of bitcoins rather than a producer of bitcoins we want to be buying bitcoins not buying storage, right?
I have no idea what you mean by this.  Again, the goal of the project is to match up users with too much storage with users with too little storage.
Quote
The people who have bitcoins do sometimes look for customers to sell them to, so maybe this is about how many of their offered bitcoins we can afford to buy given the amount of development ability and storage we can offer in return for their bitcoins.

Maybe we have to look at this in two layers: one, the person who actually wants storage. Two, the person who wants to sell them some. Three, the facilitator(s), which presumably amount to being markets, offering to bring buyers and sellers together.

To operate such a market, maybe a whole new asset-type, OSA (Online Storage Asset) would be useful. Persons wanting to sell storage could look at a prospective customer's OSC balance to decide whether that customer even has enough OSC to buy the storage the storage-vendor wants to sell, and the people wanting to buy storage could look at vendors BTC (BiTCoin) balance to see if the vendor has enough BTC balance to recompense any loss of data purportedly being stored. The market could act as escrow both for both the data stored and the coins paid. And yes I am thinking I mean here escrow of the the actual data, not the abstract OSA asset. If the storage provider fails to provide the storage, the data stored on it can be recovered from the escrow aka the operator of the market? Hmm. Shoot that down for me, I am surely not seeing the problems with it clearly.

-MarkM-
I don't like the idea of assuming the loss of data can have a set price on it.  Lots of people would deem their personal files that they hope to never lose as "priceless", especially pictures, videos, etc.

Having a facilitator manage a market of storage vendors would be interesting.  Again, the issue of having 3 different metrics (size, speed, and bandwidth) would be present.  Paring that down to a single metric, size, would be potentially doable, so long as a minimum speed and bandwidth are prerequisites to becoming a host in the first place.

So, I would imagine a storage asset would represent storage in the following manner:
- 1 storage asset = 1 GB stored for 1 month.
- The host is under contract to provide the stored data on demand one time at any point in that 1 month period in order to be paid.  In other words, bandwidth down and bandwidth up would exactly equal the storage amount.
- The facilitator would act as escrow and hold the funds to be paid to the host.  If a dispute is generated, then the facilitator finds out whether the host still has the files or not, and distributes the funds back to the buyer if the host does not.  Simply put, the host does not get paid if they cannot provide the files at the end of the month long period, or at any period within that month (if the user requests their files for download prior to the end of the month).
- It is up to the user to pay for and store their files on multiple hosts, if they so desire, in order to provide for redundancy.

This would aid in providing a marketplace for storage, where each storage asset is more or less equal, and the user knows what they are getting.

I think it would be possible to eventually expand the marketplace to account for all three metrics (size, speed, and bandwidth) for purposes of hosting large files for other people to download, but obviously, it adds complexity to the accountability of the services, and would perhaps mandate the use of a feedback-like system for host rating.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 11, 2012, 11:58:49 PM
Thanks, I think we are getting somewhere.

However I am still tempted to try for competition and redundancy.

Assuming the market/escrow has the funds secured then how about not committing at the start of the month as to which storage provider to download the data back from during or at the end of the month? Instead, solicit from storage providers up to date indexes of which data blocks they do claim to have stored a copy of, while providing to them up to date counts of how many of their competitors claim to already have copied that block.

The storage providers would thus not be competing on price but on which blocks to store. They should prefer blocks none of their competitors have copies of yet. The more of their competitors who claim to have already stashed a copy of that block, the less desirable choosing that block as the next one to save a copy of ought to be.

The storage providers thus would be trying to get the newest blocks copied to their storage first instead of grabbing a copy of some older block, but also to grab a copy of any blocks that lost a lot of copies due to a lot of storage providers fading into the night.

The testing of whether they really have a block could be ongoing; each time one wants to host a block, they get it from one that has claimed to be storing a copy of that block. If we incite churn somehow, between them they will be in effect constantly checking each other's claims as to having a certain block stored. So lets list for each block not only how many claim to have a copy stored but also exactly when they made that claim or exactly when they made (according to their claim) that copy, and have some accounting mechanism along the lines of any copy that hasn't been copied by another provider in X amount of time is stale. It needs to be checked.

Since barring some clever proofs system the most complete check would be to download the entire block from them, maybe when it is stale it should also expire completely as soon as it has been verified, due to the verifying being simply to take possession of it. That is, maybe we should have the one that "verifies" it become a new holder of it on the list of who has a copy and since when and let the previous holder of it relinquish their claim so on the list it still shows the same number of claims-of-having-it even though someone else just checked one of the earlier claims in the process of grabbing a copy themselves to get themselves on the list of that block's current holders.

There is probably still room for gaming in this, as someone could set up large numbers of virtual machines that would grab blocks and claim not to have received a valid copy of the block, in order to attack a competitor's reputation. Maybe such a claim should therefore be a black mark against both parties: one seems possibly unable to copy a block correctly, the other seems possibly unable to provide a block correctly.

Hmm...

Also, why pay developers in devcoins? Well, devcoins are a side-effect of mining bitcoins. Admittedly namecoins and several other types of coin also are similar side-effects but the point is it costs no bitcoins to create them but they are sellable for bitcoins and fiat on markets so since no one seems to want to pay developers in bitcoins maybe these various side effect coins that happen while you mine bitcoins could be put to use in paying developers. People might feel more open-handed with things they get free as side-effects than with the main ore they are mainly mining for.

Also, the devcoins go out automatically. There is no need for a majority of devcoins minted to go to miners since miners already got paid for their work in bitcoins and in namecoins and maybe also in ixcoins and i0coins and coiledcoins, heck some might also have been paid in rucoins and/or geistgeld too, I certainly was for a while.

Why sell, or even pay people with, any ore you happen upon along side your gold in your goldmine, or alongside your platinum in your platinum mine, etc?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: rjk on June 12, 2012, 12:27:56 AM
Can someone TL;DR this thread? I am not going to read so many walls of text.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 12:34:05 AM
I actually really like that idea you just came up with.

It seemed almost implied, but I'll go ahead and state it:  The hosts who hold the valid data at the end of the given time period (which could be as short as a minute or as long as a month) split the fee paid by the customer.  So if the customer is paying $0.05/day for storing 100 GB, and 5 different hosts hold the data, then each host would receive $0.01/day as long as they are holding the data.

It also allows the customer to pay exactly what they want to pay, and the free market on the hosts side determines how many "backups" would be provided for that payment.  A customer paying only $0.01/day for a certain amount of data might only get 2 backups, while a person paying $0.04/day might get 8 backups.

The biggest downside is bandwidth.  There would be no reasonable way (that I can see) for a host to cap bandwidth on any data that they hold.  So they could potentially be required to serve the data 24/7 indefinitely.  I suppose this is workable - it mainly just means that hosts will have to take into account their bandwidth limits, and only host that space accordingly.

This is starting to sound more like paid torrenting than anything.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 12:34:48 AM
Can someone TL;DR this thread? I am not going to read so many walls of text.
It's a good idea, but there are a variety of ways to implement it, and the best route to go is still being discussed.


Title: Re: [IDEA] Dirt cheap online storage
Post by: nimda on June 12, 2012, 12:38:02 AM
The client (doing the paying) should keep some small amount of the data being hosted. Then, at random, request random bytes of the hosted files and compare them. If they're OK, pay the money.


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 12, 2012, 12:56:27 AM
A market mechanism for buying and selling hard drive space would look a lot like the commodity markets. If you express a contract in a standard form they can be traded on an exchange to achieve price discovery. The following is not a complete specification but just an example of how such a system might work.

Contract trading could be done on #bitcoin-otc or a similar platform (including the reputation function). Contracts will be priced in bitcoins. Storage is expressed in blocks of 1 MiB and the base time interval is one increment of the Bitcoin block chain.

Block servers (nodes that sell space) can list the contracts they are willing to accept in human and machine-readable form as something like this: "BLOCKSTORAGE,O,10,10000,144,0.001,0.0005,0.00025"

This means that the node is offering (O) to contract with any node with a reputation of 10 or more. Up to 10000 blocks are available for 144 time units. The total cost of the contract is 0.001 BTC per block. Of this total cost 0.0005 is due up front and 0.00025 is due at the completion of the contract. The remainder is due in periodic intervals (perhaps as often as every time increment).

When a node that is looking to sell storage can express the contract it is looking for as a bid (B). It could either work as both the block server and the client can scan the order book to look for a matching contract or the exchange can handle the matching. There would need to be a means to express what should happen to the data at the end of the contract. It can either be deleted (the client no longer needs it) or it can be transferred to another block server or it can be returned to the client.

At the end of each contract the client and the block server both leave feedback on each other which will either increase or decrease their respective reputations. Block servers with high reputations will be able to charge more up front because the clients can be more confident they will actually be able to get their data back at the end of the contract. Clients with high feedback won't need to pay as much up front because the block servers can be more confident they will be paid at the end of the contract. In general a higher reputation will lead to better prices for both a client and a block server.

The client should have a method of verifying that the block server actually retained the data instead of just tossing it into the bitbucket. One possible way to do this is to periodically transmit a block number and two offsets and ask for the hash of the data between the two offsets. The client could pre-select this verification blocks before transmitting the data and have enough hashes stored ahead of time for the number of verification it expects to need.

This is just a rough draft and probably still has holes that need to be filled in but how does it look as a basic outline?


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 12:56:55 AM
The biggest downside is bandwidth.  There would be no reasonable way (that I can see) for a host to cap bandwidth on any data that they hold.  So they could potentially be required to serve the data 24/7 indefinitely.  I suppose this is workable - it mainly just means that hosts will have to take into account their bandwidth limits, and only host that space accordingly.

That sounds like an argument in favour of dropping a block once it has been "verified", as then the host knows it will only be downloaded once, either by the "verifying host" or by the actual customer. That would also bring back paying more for more copies, because unless the customer specified, and paid for, two copies there would just be one copy of each block floating around (probably plus one insurance copy the market/escrow has in escrow) unless the customer paid to have it downloaded twice, in which case after it has been "verified" the host who had it could keep it, waiting for a second download.

However since your argument is basically a storage-provider-side reason why the provider might want to drop off the list of hosts the block is available from we might as well leave it to them to choose for themselves whether they want to drop the block due to having already spent quite enough bandwidth on that block.

This is starting to sound more like paid torrenting than anything.

Hmm, I am not really very familiar with that. But if it offers us a market for our service by all means lets explore that.

Usually torrenting tends to imagine the blocks are going to be sought by more people than just the original uploader, right? So hey, if there is a wider market for blocks than merely selling them back to their uploader, and the uploader is willing to provide keys for decrypting the data either to our system / the storage providers (to enable selling the data in useful form) or to other end-users (in order to make the uploader some money to cover the cost of having our system store the blocks), great. More money for the service and maybe also for the uploader(s).

-MarkM-

EDIT: Also, if providers do not know until later whether thousands of people are suddenly going to offer to buy copies of certain blocks, holding some on speculation might be encouraged, since any block might turn out upon release of decryption keys of the file it is a block of to be very popular / in demand.

EDIT2: We could go even further, constructing files whose decryption does not require all previous blocks to decrypt a block, so that someday keys can be released for any specific block, revealing that specific block to be very valuable / popular in some way.

EDIT3: For example we could habitually insert into our files a block containing a bitcoin privkey. Someday, maybe quite some time after the pool-hopper types have long forgotten that block, we could release a key allowing anyone who retained that block on spec to decrypt it and get those bitcoins...

EDIT4: We could also have lotteries, such that each time a bitcoin blockchain block reaches a depth of 120 some deterministic function computes some number of what could maybe happen to be block-hashes, or certain numbers of bytes of some block-hashes with the number of matching bytes being related to some lottery-winnings-distribution setup... If, of course, one does in fact still have a data-block that is at least 120 blockchain block-cycles old that matches...


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 04:48:50 AM
AbelsFire, I like the general line of thinking.  Sounds more like you're talking about backend stuff than a nice functional front-end though.  I always found bitcoin-otc to be quite cumbersome, and only natural to those who find it natural to use a command-line linux box.  Setting up a command-based architecture intended to be used by customers and sellers would only serve to limit the potential crowd of users.

But, that's not to say it wouldn't work as a fine backend for whatever front end web interface is created.

Markm, you're right that the bandwidth spent on blocks is another incentive for hosts to generally stick with a customer they are already serving.

Regarding torrenting, here is what I am picturing:
- A service that pays seeders for seeding particular files using funds sent by people who want those files to stay online.
- The person who wants the files backed up sets a daily rate that they are willing to pay in order to have the file available on demand via torrenting.  Say, $0.05 for 50 GB.
- The service takes those customer funds, and pays them out equally among the seeders.  So, if there were 10 seeders of the full file, then $0.005 to each seeder each day.

It seems like a perfect fit to me.  The customer can pay however much they like, though paying more will get more seeders, guaranteeing longevity of their file.  The seeders (or hosts) can get paid for simply using up some bandwidth and hard drive space.  And the load of any future downloads is split roughly evenly among the hosts.  The customer and/or the service can randomly check each host to ensure they have a complete set of the file.

Heck, perhaps the software could be built in such a way that the customer could just keep a few KB of the data on-hand, and have essentially infinite random checking of the files.  All that needs to be done is record a few bytes here and there of the file, then ask the seeders what is at that particular point in the file.  If the data returned from a particular host does not match the data on hand, then the customer knows there is a mismatch and can take appropriate action.  The hosts might also have incentive to periodically check other hosts too - the fewer hosts there are hosting the same data, the high the payout for each host.

So, the blocks torrented would be sought by more than just the original uploader, if for no other reason than that those seeding the blocks would receive a regular payout from the uploader.

The one thing I am unsure about, and perhaps someone with more knowledge of torrenting/programming can answer, is how the service could know who is seeding and pay them accordingly.  Perhaps it would have to be by IP address (i.e., you head to the website of the service, sign up with a particular IP address, then seed from that IP address)?  Or maybe there is another way?

Beyond that little hiccup, I think this could work perfectly.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Gladamas on June 12, 2012, 06:28:41 AM
Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.

Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 02:19:10 PM
This is again going so far out into in the vapourware domain that now as a person actually looking to store some data offsite, but not needing yet anywhere near a terrabyte nor even, yet, 100 gigabytes, I am back to thinking Freenet is the way to go for at least one copy at the present stage in the discussion.

Right now I am not yet sure if one gigabyte would suffice, but think it quite likely that five gigabytes might, for a possibly short period of time depending on how fast my Open Transactions server aquires actual users doing real transactions.

So right now I am thinking of putting backups on whatever free five gigs for life offers I can use from Linux plus test trying to send a copy, as a modifiable file so replacing it doesn't double the storage requirements, to Freenet to see if once there it can actually be gotten back and if so for how long.

Since freenet has friend-to-friend modes available, possibly I could find some other people looking to start with one to five gigabytes or so, set up friend to friend links with them, and see how that works out for a while as a mutual swap, since my problem is not lack of onsite space but desire to swap onsite space for offsite space.

I also have GNUnet and Tahoe-LAFS installed. (Also Tor and i2p, but they are transport more than storage.)

So, are there any among you who are looking to swap onsite for offsite space?

(We can worry about selling space later, once having mutually assured ourselves there is actually space and that it is actually useable and reliable over some reasonably long term.)

-MarkM-

EDIT: Friend to friend might not even be ideal, if it opens the possibility of being someone's only link to the freenet / GNUnet / Tahoe-LAFS grid / whatever; it might suffice to simply request each other's files while both being on such a net.?.?

EDIT2: Technically, I actually also have RetroShare, I just happen not to bother running it lately due to lack of connections.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 03:16:51 PM
Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.

Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.
Wouldn't you have to keep some data on hand to make sure the checksum or hash matched?  In other words, what would be the difference between checking small bits of data vs checking a checksum of small bits of data?

Sounds like it is possible to tie a bitcoin address to a particular seeder then!


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 04:03:17 PM
I am curious where you people put your offsite backups, and how.

It would be interesting to know not only for ideas I might apply to my own offsite backup strategies and/or tactics but also simply to assure me you are serious about data integrity / safety in the first place.

Meanwhile with this torrent type stuff continuing to come up I am thinking again of actual-block-of-data markets.

If satoshis are small enough units of value then maybe the markets can use satoshis as units of value to use to bid on such markets, if not then some lower value per coin flavour of coin might be better, or some other kind of microsatoshi or millisatoshi scale unit of value.

You seem to be more interested in bitcoins or satoshis than in whatever you might be able to provide in return, so lets drive this from the bitcoin or satoshi or other coin side.

I offer to buy a block of data whose hash is $H, and solicit bids as to how many microsatoshis that specific block is currently available for.

If anyone does have that block of data, great, I'll either buy it or await a better price.

For my personal current application-of-interest, I actually already have that block myself anyway, so I can also offer it myself, possibly at a lower price than the offers I am seeing from others. (Because, it is a block of a backup of data of mine, and I already made and retained a copy of that backup myself.)

Someone else might be interested in buying the block whose hash is $D, and bid higher for that block than I typically bid for blocks of my backups. (Possibly because block $D contains part of a particularly appealing frame of a particularly popular scene of "Dolly Does Dallas XII", not yet out in theatres and DVD stores but reported to have been leaked to the internetz.)

Lots of people might be bidding to buy block $D simply on speculation, figuring that once they have collected enough blocks of that frame of that movie they will be able to resell entire frames or maybe entire clips on other markets.

Torrent end-users might not even be up to date enough on current internetz affairs to even know of this market in blocks of data, they might be eagerly waiting for their favourite torrent-provider to offer a full copy of the entire movie, having no idea of this back room market in blocks between torrent-providers and others whereby such people manage to lay their hands, block by block, on such hot digital assets...

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 04:52:51 PM
That's quite a large tangent from the idea of backing up personal files, markm.  Perhaps a good discussion for another thread though.  ;)

Right now, I do not have a good method of offsite backups.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 05:41:07 PM
It could either work as both the block server and the client can scan the order book to look for a matching contract or the exchange can handle the matching.

My game-players have always wanted to be able to choose which offer to accept, but it does open new "game the system" opportunities. Specifically, volume can be created at high prices between sock-puppets (or, in general, colluders) without any non-sockpuppet bids/offers being accepted.

The ostensible reasons and/or excuses for wanting such a structure tend to be not wanting to buy stuff from enemies, wanting to offer allies better prices than offered to enemies and neutrals, and preferring reliable suppliers or suppliers favoured for the quality of the actual "commodity" provided. (Which goes against the idea that to be a "commodity" a product should be fungible.)

There would need to be a means to express what should happen to the data at the end of the contract. It can either be deleted (the client no longer needs it) or it can be transferred to another block server or it can be returned to the client.

Transferring it to another block server is getting into more complex transactions, but whether the client is paying for delivery to client plus deletion from the provider or just delivery to the client seems a useful distinction. Though it does bring up the corner case or imaginary case of how about the client only wants it deleted from the provider (a takedown request) without the bandwidth overhead of taking delivery of a copy.

At the end of each contract the client and the block server both leave feedback on each other which will either increase or decrease their respective reputations. Block servers with high reputations will be able to charge more up front because the clients can be more confident they will actually be able to get their data back at the end of the contract. Clients with high feedback won't need to pay as much up front because the block servers can be more confident they will be paid at the end of the contract. In general a higher reputation will lead to better prices for both a client and a block server.

This is another form or arising of the "is this really a commodity, since commodities should be fungible" problem.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 12, 2012, 05:54:44 PM
My game-players have always wanted to be able to choose which offer to accept, but it does open new "game the system" opportunities. Specifically, volume can be created at high prices between sock-puppets (or, in general, colluders) without any non-sockpuppet bids/offers being accepted.

The ostensible reasons and/or excuses for wanting such a structure tend to be not wanting to buy stuff from enemies, wanting to offer allies better prices than offered to enemies and neutrals, and preferring reliable suppliers or suppliers favoured for the quality of the actual "commodity" provided. (Which goes against the idea that to be a "commodity" a product should be fungible.)
I can't help but notice that we manage to use exchanges to buy and sell wheat, electricity, currency and other commodities. The empirical evidence suggests that it is possible to do it, although some exchanges are better than others. In that context I don't understand the purpose of your reply. If you've identified a problem are do you have a proposed solution? Are you claiming that no solution exists?

Transferring it to another block server is getting into more complex transactions, but whether the client is paying for delivery to client plus deletion from the provider or just delivery to the client seems a useful distinction. Though it does bring up the corner case or imaginary case of how about the client only wants it deleted from the provider (a takedown request) without the bandwidth overhead of taking delivery of a copy.
Why would a client ever pay for a server to delete data? Any sane implementation of a client would encrypt all data first so that it's useless to anyone else. Why would a block server keep around useless data if no one is paying to store it?
This is another form or arising of the "is this really a commodity, since commodities should be fungible" problem.
In what way is this a problem? What does "commodities should be fungible" mean in practical terms? Will the fact that some block servers can provide a higher quality service than other block servers break the price discovery mechanism? Your objections are a little too vague for me to understand their practical implications.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 06:17:21 PM
There are bitcoin-exchanges in which users pick and choose which offer(s) to accept instead of being matched by the market-operator; off-hand I have the impression they tend to also require the users to then conduct the trade out of band between themselves, thus losing the assurance of delivery by both parties that automatic-match exchanges such as MtGox provide.

You brought up the idea of there being these two different approaches available, I merely tried to respond, mostly by elaborating the implications.

It seems to me that it would be useful to take it that one choice opens up questions as to whether the product is a commodity at all, due to seeming to at least potentially bring the product's fungibility into question.

One might for example avoid picking offers that involve coins trackable back on their blockchain to some theft break-in heist scam etc. I have read various threads in which various people assert that such distinctions challenge the very fungibility of blockchain-based coins.

Maybe both types of "market" are useful and we should have both.

Certainly so far I am in picking and choosing mode, still seeking specific individuals with whom I might consider trading; albeit I am also considering trading offsite for onsite initially too, keeping it all about the product - the possibly-a-commodity item - without having to worry yet about (possibly non-fungible?) money/currency trade.

Maybe part of the picking and choosing would  be which providers of the "commodity" actually seem over time to be providing an actual "commodity" rather than a product not quite exactly like (co-fungible with?) the supposed "commodity" we could end up going on to create a "commodity market" for?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 12, 2012, 06:20:26 PM
Maybe both types of "market" are useful and we should have both.
I agree. If there is a standard API for contracts then it would be possible for different types of exchanges to form, each with their own rules.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 06:40:46 PM
That's quite a large tangent from the idea of backing up personal files, markm.  Perhaps a good discussion for another thread though.  ;)

I think it started from an ambiguity about whether we are talking about back-up storage or primary storage.

By that I mean there is a lot of difference between storing a just-in-case copy of your infinitely valuable irreplaceable data and storing your actual infinitely valuable irreplaceable data.

The case of a sister running out of space for photographs for example sounded as if the plan would be that she "move" photographs to offsite storage rather than that she "copy" them (copy implying she of course still has it herself, she does not delete from her system when she copies to someone else's) to offsite storage.

These are very very very very - it is hard to understate how very - different cases.

One must always act as if each and every piece of media on which your data resides is a diabolically patient, fiendish enemy, always hoping to find itself in the position of being the only surviving copy of the data so that it can strike at you as no other copy up until that moment could do, by maliciously destroying the only copy of your data.

This applies to backups more even than to the files backed up, because if you lose the files you have backups of, you can simply restore from backups. But if the backups are lost or corrupt, that is when you really need backups.

So always back up your backups, as if they are hoping and hoping and hoping you will need them so they can nyah nyah nyah you by failing you.

The idea that someone's sister might give me her precious files without keeping a copy herself made me very very worried about how sure I could be that I would still have them when she actually needs them.

Right now, I do not have a good method of offsite backups.

Maybe we can do some trials/tests between us then, trading onsite to offsite storage, and see how it goes?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 07:00:27 PM
I agree there is a difference between a just-in-case copy of data vs an only copy of data.  But how does that have anything to do with your previous post about selling parts of movies via torrent?

The data in my sister's case is raw files from older wedding photoshoots and such - files that she will likely never need to touch again.  But, better than deleting them entirely would be finding some method of cheap online storage until she can buy some extra storage space.  So, it wouldn't be the end of the world if they were somehow lost forever (since the only other option is to simply delete them at this point anyway to make room for newer photos), but she'd still like to keep them, if possible.

Certainly, though, I believe a service like this would be used more for "just in case" copies of data than anything else.

I'd be fine with doing some testing/trials.  I am not a regular torrent user, so I really know only minimal amounts about it, but I do know enough to click on some menus and create/seed a torrent.


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 12, 2012, 07:01:29 PM
The idea that someone's sister might give me her precious files without keeping a copy herself made me very very worried about how sure I could be that I would still have them when she actually needs them.
It would be very foolish for her to give her only copy to one person, unless the data was of very low value or easily replaced. Using a RAID (or similar) algorithm to distribute the data to three people such that she could recover it from any two providers would be safer. Depending on how valuable and irreplaceable the data was she might want even more redundancy.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 07:29:07 PM
I agree there is a difference between a just-in-case copy of data vs an only copy of data.  But how does that have anything to do with your previous post about selling parts of movies via torrent?

I wasn't the one who brought torrent into this. The thread seemed to diverge between offsite copies of data, actually moving data offsite no longer having it onsite, and whether anyone other than its "seeder" / "publisher" / "uploader" ... (owner, even?) ... might want a copy too, which seems to be the usual use-case for Torrent. (Because Torrent is designed for getting the same data to many many people aka to optimise having many many people all wanting the same data and not having a copy of it themselves already.)

The data in my sister's case is raw files from older wedding photoshoots and such - files that she will likely never need to touch again.  But, better than deleting them entirely would be finding some method of cheap online storage until she can buy some extra storage space.  So, it wouldn't be the end of the world if they were somehow lost forever (since the only other option is to simply delete them at this point anyway to make room for newer photos), but she'd still like to keep them, if possible.

Certainly, though, I believe a service like this would be used more for "just in case" copies of data than anything else.

As you also mentioned earlier having an entire drive free, maybe just sending her that drive to install in her machine would be a good way to tide her over her diskspace-crunch? Or, have her upload her photos to you? Or, download them from her?

I'd be fine with doing some testing/trials.  I am not a regular torrent user, so I really know only minimal amounts about it, but I do know enough to click on some menus and create/seed a torrent.

See, Torrent comes up again. I do not yet see why Torrent. Maybe though it is because of the front end GUI, not what the actual protocol it uses it optimised for?

Can you have your sister install torrent and seed a directory so you can download that directory from her any time you wish, then cryptozip or whatever the photos at risk and put them into that directory for you to grab?

Or, how about RetroShare? About the only thing going for RetroShare seems to be eye-candy GUI, presumably it should be just as GUI-user friendly as Torrent?

I have never used Torrent though doubtless can find code for it that will work on Linux. In fact I have on my GUI menus something called Transmission BitTorrent Client. If I used that to grab files from you, would you be able to grab them back from me? Or would I need a server rather than a client for you to be able to grab from me? I really know very little about Torrent.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 07:42:03 PM
You have to step away from what torrent is used for right now, and look at how it can be used.  It can be used to easily give multiple hosts copies of the same data, which is exactly what we want when looking to make backups of said data.  There will still be demand for these backup files to be downloaded via torrent, but the demand will only exist among the original uploader (potentially), and those who wish to seed it to gain a portion of the daily fee paid by the uploader.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 07:54:40 PM
Hmm, I get the impression it is more like multiple hosts take the data than you give the data to multiple hosts, in the sense that it is pull not push. You can't really just give them the data, you have to somehow cause them to ask you for it?

A push could be useful for cases like you go away for the weekend but your sister still has photos she wants to move or copy to you.

If you can set up an automated pull though that will keep asking her for anything new, that would be fine.

Or maybe you could search for specific filenames, leaving the search constantly running, so any time fridays.batch.zip becomes available your client will grab it, as soon as saturdays.batch.zip becomes available it will grab that too and so on, so she has list of names she can zip a batch as to have your machine grab them even while you are away from keyboard.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 08:20:33 PM
Hmm, I get the impression it is more like multiple hosts take the data than you give the data to multiple hosts, in the sense that it is pull not push. You can't really just give them the data, you have to somehow cause them to ask you for it?

A push could be useful for cases like you go away for the weekend but your sister still has photos she wants to move or copy to you.

If you can set up an automated pull though that will keep asking her for anything new, that would be fine.

Or maybe you could search for specific filenames, leaving the search constantly running, so any time fridays.batch.zip becomes available your client will grab it, as soon as saturdays.batch.zip becomes available it will grab that too and so on, so she has list of names she can zip a batch as to have your machine grab them even while you are away from keyboard.

-MarkM-

Yes, you are exactly right - pull instead of push.  The uploader would volunteer the data to be copied as a torrent, and the hosts could download said data at will, as long as it is up.  Many of the more hardcore hosts would likely find a way to automatically download new data that meets a certain criteria (i.e., current payout above $0.0X/day).  An alternative might be automatic download of all new torrents by the Service, until another host picks it up, allowing the uploader to effectively "push" out the new data whenever they like.

The nice thing about using a torrent protocol for the base of a project like this is that torrents are already very widely used and understood by many of the people.  Perhaps even more importantly is that the group that understands torrenting the most is the group that is most likely to have lots of hard drive space, and (potentially) lots of extra hard drive space.

I suppose the one thing I am most interested in at this point is more detail on how Bitcoin addresses can be tied to hosts.  Gladamas touched on it, but I am not enough of a programmer to understand how it might work.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 08:27:28 PM
You could sign, using a bitcoin client, a message to your sister claiming that your machine and the bitcoin address used to sign the message are owned/controlled by the same person/company.

However, does she even get told who asked for her files? And if she displays a list of what files you have on your end is your machine identified similarly to the way it is identified in her logs of who has downloaded which of her files?

Since IP addresses are not usually stable for many home users, does bittorrent have some kind of node-identity that it uses for such identifications/logs?

When you download a file from her, can you be configured at your end such that what you downloaded is automatically also then available to be downloaded from you?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 08:31:31 PM
You could sign, using a bitcoin client, a message to your sister claiming that your machine and the bitcoin address used to sign the message are owned/controlled by the same person/company.

However, does she even get told who asked for her files? And if she displays a list of what files you have on your end is your machine identified similarly to the way it is identified in her logs of who has downloaded which of her files?

Since IP addresses are not usually stable for many home users, does bittorrent have some kind of node-identity that it uses for such identifications/logs?

-MarkM-
This is the part I am really unsure of.  I know IP addresses identify torrent clients, but beyond that, I am not sure what other information they can make public about themselves or request from others.  Gladamas seemed to indicate that it would be easy to attach a Bitcoin address to a particular torrent seeder, but I don't know the specifics about how that would work.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 08:57:13 PM
Okay well also consider that even Windows users might already have as part of Windows something called something along the lines of Windows Shares, or SAMBA shares, or something like that, which should be able to provide them the ability to specify directories they would like others to be able to share, and to mount other people's such directories so they can just drag and drop stuff. The shared directories are "actually" shared, at least once they are mounted.

In Unix type systems, maybe including OS/X, there is a pretty standard remote mount system (NFS: Network File System) that lets you similarly make directories available for remote mounting, though how exactly you tell it who gets access and how is somewhat different.

There is (or certainly long ago used to be) SAMBA code for Unix type systems so that for example an ISP could offer Windows users mountable home-directories on the ISP's unix/linux/etc based servers.

Hosts that are Linux based even have available quota systems so they can assign quotas to users, and have 'cron' scheduling daemon that runs things on specified schedules. Such hosts could have cron run a script that looks up for each user a bitcoin account the balance of which will be used as the number of units of storage to set that user's quota to, and another, monthly script that deducts bitcoins from that account based on the maximum amount of storage that user had actually used in the last month.

Presumably "we" need not worry about what people who want to sell space would prefer in the way of software to be such a seller, except in proportion to the number of bitcoins they offer as bounty for each specific script or function etc they specify? Folk who can handle the technical stuff themselves can get up and running selling space to bring in coins first, limited possibly by the need for pretty front ends for the buyers of space to use in order to maximise the coins brought in by/to the project; then over time as the sellers find it worth their coins to have prettier software for their end made such software will or will not get made.

Basically if software is to be developed we want first to have coin in hand with which to motivate software developers to develop it?

Meanwhile those who are able to raise coin by setting up as sellers using existing software can attack the problem of what exactly do the people who want to buy space need (other than coin) in order to be able to buy it? Such as ability to tell their operating system to make available one or more directories as "shares"? Coupled possibly with the ability to tell their router to allow people coming in from the net looking for shares/specific-shares to reach which port of which machine to get those shares?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 09:02:10 PM
I still don't see a good reason to deviate from a popular and known platform (torrenting) to something that doesn't even exist yet (Windows Shares?).  That's just making things more complicated for the sake of making things more complicated.

Regarding software development, well, this is only idea exploration at this point.  I am getting close to the point where I'd feel comfortable with dropping some coin on a programmer developing the idea though.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 09:06:14 PM
Windows Shares, or SAMBA, or whatever such capabilities are now called in Windows, existed in Windows 3-point-something or earlier.

What operating system are you using / used-to?

What operating system is your sister using / used-to?

I am talking about built in stuff that should be right in your system setup menus/icons somewhere already just waiting for you to turn it on, or in the case of Linux distributions conceivably for you to click on in the install packages tool to have them installed for you.

Certainly far more normal and standard that strange third-party stuff such as bittorrent or retroshare or whatever. It is native to the system / distribution / operating-system.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 09:15:32 PM
I'm a Windows user, as is my sister.  I suppose I have just not heard of whatever this built-in file sharing capability Windows has though.  Unless you are talking about local area network shares (mapped drives and all of that)?  I've never heard it touted as a good way to share files publicly on the internet though - that seems insecure and a way to make yourself ripe for abuse, if it is even possible.

Also, remember, this isn't about my sister.  I could find a number of solutions for her, including giving her one of my hard drives to use (though it wouldn't be as safe as having multiple people store it online).  This is about finding a general-purpose low-cost online storage solution.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 09:26:57 PM
Okay, so you have local area network sharing but are concerned about its security.

This many years or decades in has it not by now been given an option to encrypt the data on the wire?

If not, then a Virtual Private Network might be required, to set up host and customer with a private local area that happens to extend across the internet. That might well *not* be already right there on your system setup menus/icons on Windows. For Linux there are lots of them, I would ask my package manager aka installer to search then try to choose which one to actually use somehow maybe partly by whether it seems one I have heard of a lot or seems to be the usual vanilla standard or whatever... Aha I see "OpenVPN", the "Open" part has me sold on it at first sight.

Google WIndows VPN:

Configure a VPN Connection Using Windows XP
www.windowsecurity.com/.../configure-vpn-connection-windows-xp...
18 Mar 2005 – In this article we will learn how to configure a connection to a virtual private network (VPN) using Windows XP. This article will show you how to ...

Google goes on list other flavours of WIndows. (Link above is an error 404 but hey, back in 2005 with Windows XP this stuff already existed.)

For disk quotas googling "Windows disk quota" shows immediate results for various windows 2000+, windows server, and windows 7 systems, one might have to dig deeper for older systems and they might well have deliberately tried to keep such "professional" features out of old cheap home-user versions intended to lure professionals into upgrading.

But, okay, that is more suitable to you and your sister than to random strangers, whom you of course wisely do NOT want on your local area/net.

If Bittorrent already does what you want then great. If not I still do not really understand the fascination of it, because previously you have been writing about having one to three hosts host the customer's data. If the goal is to have hundreds or thousands or hundreds of thousands of hosts host it, then yes, I too would look at torrent technology. But for the probably for our purposes more common case of 1 to 99 hosts, with 1 to 25 or less likely far more common, I am not convinced of a need to resort to torrent.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 09:55:49 PM
Eh, I've had some experience with VPN's, and my experience has shown that they are complicated to set up and maintain, they are somewhat unreliable, and they do not allow some requisite features necessary for this type of project (i.e., allowing continuation of file transfer after interruption).  Plus, allowing anyone into your file system, even if it seems to be contained, is a rather glaring security hole.  And I am uncertain that multiple VPN connections can be made at the same time without consequence.

Here's why I think torrenting would work well:
- After multiple hosts have the files, it reduces the load on any single host to make additional copies of the data.
- It already allows resuming interrupted downloads, managing download speeds and bandwidth usage, etc.
- No security holes or issues.
- It is scalable - if someone wants their data to be ULTRA safe, they can pay such a ridiculously large sum of money on a daily basis to each person seeding the file that hundreds or maybe thousands of people will host said file to "get in on the money".
- If using VPN's, the hosts couldn't just allow anyone to download from them, as they wouldn't want to put their VPN information out in public.  This would mean that the burden of creating additional copies of the original content would lie with the uploader.  What happens if the uploader no longer has the content, and one of the hosts decides to move on to other files?  With torrenting, this would be fine - a host might decide it is economical to start seeding a particular torrent, and download from the other hosts already seeding the content.  With VPN's, the uploader would have to re-download the information from one of the VPN's, then reupload it to a new one.
- Torrenting is already cross-platform without any special configuration.

I don't see VPN's working well unless hosts were willing to allow anyone to download from their VPN-hosted files.  In which case, why use a VPN at all, and not a more standard FTP or HTTP protocol?

I'll emphasize this point again though:  Torrenting reduces the load of any particular host, as it (more or less) evenly splits the load between all seeders.  If a new person wants to download a particular file, and 5 hosts are already seeding it, then each host only has to use bandwidth equal to 1/5 of the total size of the file.  With VPN, FTP, or HTTP, there is no easy way to split up the load between different hosts.  With torrenting, it is automatic.

Bittorrent doesn't already do what I want, because it provides no incentive for other people to hold my backups.  But, adding incentive for other people to hold my backups make torrenting a very viable platform.  Also, there is no reason that torrent technology cannot be used for 1 host just as easily as thousands.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 10:24:09 PM
Thank you! Okay, I now agree that bittorrent is a strong candidate.

Quite a few other systems can split the load and also migrate blocks towards regions where those blocks are popular though, which I have not heard of bittorrent doing. My impression has been that with bittorrent you deal with an entire file at a time, so any host that has the file has the file?

And the only migration in response to regional/directional demand is when someone not only downloads the file but also chooses to make downloaded files available for others to get from them?

Actually, can you even have all downloads automatically seeded for others to get from you, or do you (at least with whatever client it is that makes you feel so sure everyone already knows how to use it) have to explicitly pick from among your downloads specific files on a case by case basis that you want to "seed" for others to retrieve from you?

Block by block storage could let more people get in on the money even when the paying customer only wants a few copies. Some kind of RAID type system or even the Nimbus.io system could split the data up in such a way that X out of N nodes would have to die to cause loss of data. We could make some copies be full copies, putting all the blocks of a file on one host - all its eggs in one basket - and then also have some scattered-blocks copies also to share the wealth, basically trying to make sure all our storage-providers get at least some token amount of data to store.

The migration features of some distributed storage systems could also provide trickles for people who are along the way toward regions where a particular file is most popular, as blocks migrate toward the region where they are in high demand.

However okay if bittorrent already provides some of that stuff, by being able to automatically make available anything a node downloads, lets move on for now until I look again more closely at the distributed storage systems that do resemble bittorrent in those ways.

So, the money. Well, disk quotas still are on the table as a means by which someone providing space can limit the amount of space they are providing, if for some reason bittorrent does not already include setting how much disk space mximum you wish it to use. How does bittorrent know who is using how much space though? Maybe we need to look at bittorrent server software which google shows plenty of. Maybe if you run a bittorrent server instead of just a client, you will find it lets you allocate space to various users and stuff like that?

If bittorrent does not maintain some kind of concept of which users are using how much space, does it even maintain some concept of users at all, associating each file with a user?

If not you might have to run one instance per user whose usage you wish to account separately.

I am thus starting to suspect that instead of a peer to peer storage-node model we might have to go for a peer to peer cluster model, in which storage nodes only have one customer, which is their cluster co-ordinator node. Normal people who just want to sell some space could deal directly with their nearest cluster-coordinator or pick some other hopefully not too distant cluster co-ordinator.

We then don't need to worry about normal home-user types having to deal with complicated issues, they can simply try to set up a link of some kind with their chosen cluster-coordinator  and see if they notice enough satoshis coming in at the end of each month to motivate them to stick it out another month or even maybe to do some marketing of the service to try to bring in some customers.

-MarkM-

EDIT: Maybe since security has come up we should from the get-go assume storage nodes will be virtual machines, anyone wanting to be a storage node simply fires up some standard virtual machine specific to the purpose? That gives them a disk space max and CPU usage max right off the bat without having to get into granular by user/group usage quota systems.



Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 11:07:14 PM
I think you're a bit confused about how torrenting actually works.

There would be no concept of users on the seeding side, only files and payments.  The site would organize the torrents available, and show how much (currently) a seeder would be paid per day to host a particular file, assuming no change in the current number of seeders of the file.

Again, torrenting is a pull operation, not a push operation.  A seeder would have to request to download a file from the uploader in order to download it - the uploader couldn't push a file onto the seeder's system.

I think torrent software is usually set up by default to seed anything that is downloaded.  In fact, it usually seeds the portions of the file it has WHILE downloading.

Essentially, this is how it would work:
1) An uploader wishes to back up encrypted group of files b1.rar.
2) The uploader creates a .torrent file for b1.rar.
3) The uploader funds his account at service.com with 1 BTC, and states that he will pay 0.01 BTC/day split between all users seeding b1.rar.
4) Various users visit service.com, notice the uploader's proposal, and downloads the .torrent file.  This .torrent file tells their computers where they can download pieces of b1.rar (which would be from all seeders - note that the uploader is a seeder as well).
5) The hosts start seeding the full file as soon as it is fully downloaded.
5) Once the uploader is satisfied with the number of hosts seeding replicas of his file, he takes his file offline and no longer seeds it.
6) Service.com automatically pays out 0.01 BTC/day split between all users seeding b1.rar, with random data integrity checks to ensure they have the complete file.  If 5 users are seeding b1.rar by the end of day 1, then service.com pays out 0.002 BTC to each of them.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Gladamas on June 12, 2012, 11:32:17 PM
Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.

Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.
Wouldn't you have to keep some data on hand to make sure the checksum or hash matched?  In other words, what would be the difference between checking small bits of data vs checking a checksum of small bits of data?

First someone takes a checksum of the file, storing that on their computer. They upload the file to the network, then delete their local copy (to free disk space). Once the person receives the file again once they need it, they simply take the checksum of the received data and see if it matches the previous one.

I suppose the one thing I am most interested in at this point is more detail on how Bitcoin addresses can be tied to hosts.  Gladamas touched on it, but I am not enough of a programmer to understand how it might work.

I'm not really a developer either, but this is what I understand: Alice has files to upload onto the network, and Bob has disk space to share. Alice encrypts her data with her public key (so only she can see her data). Alice sends the encrypted data and her public key to Bob. Bob then encrypts his Bitcoin address using his private key. He then sends that and his public key to Alice, who can then authenticate the message using Bob's public key.

Anyone else can then prove that Bob was the owner of the Bitcoin address by decrypting the message with Bob's public key. Bob can also prove that he owns that private key by having someone else encrypt a message with Bob's public key, then Bob decrypts it with his private key and verifies the contents of the message with the person who sent it.

I'm curious, though, how to stop Alice from, say, denying that Bob ever sent her his Bitcoin address, or if Alice simply refuses to pay.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 11:40:36 PM
6) Service.com automatically pays out 0.01 BTC/day split between all users seeding b1.rar, with random data integrity checks to ensure they have the complete file.  If 5 users are seeding b1.rar by the end of day 1, then service.com pays out 0.002 BTC to each of them.

Ok, so service.com is the cluster co-ordinator for a bunch of storage nodes, or maybe is even further up a hierarchy than that.

How does the customer know how many copies of his/her file are seeded? Specifically, are storage.com's claims about that believed or does the customer get to see directly peer to peer with the hosts hosting the copies that they claim to be seeding it?

I ask because I am wondering if each host can add a firstbits string to the name by which they seed a copy.

For example say you want to store SgtSpike-Desktop-2012-05-31.bak.rar

I have a firstbits of 1DIgitalis, so I seed it as SgtSpike-Desktop-2012-05-31.bak.rar.1DIgitalis or whatever: basically I put my firstbits in there, or if filename length isn't a problem, then lets forget firstbits and use full bitcoin-addresses.

Now you should be able to ask bittorrent for a list of all files whose name start with SgtSpike? or maybe for any with keywords of SgtSpike and Desktop? Or if wondering what else I am hosting, all files with 1Digitalis in the name, or something like that?

Or does bittorrent have metadata even, like GNUnet and hmm not sure but I think maybe Tahoe-LAFS do, so a seeder can associate a whole bunch of keywords with a file without having to stuff them into the filename?

I am thinking here maybe we could eliminate service.com entirely even, simply by including in the keywords or filenames the offered reward for seeders from some offering seeder, the identity of the offering seeder, and the bitcoin address to which to send rewards for the copy having this name or keywords-set (description?)

Service.com could be useful as escrow though, so storage nodes do not have to trust some random seeder's offer just storage.com's claim that storage.com will make the offer good.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 11:45:55 PM
Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.

Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.
Wouldn't you have to keep some data on hand to make sure the checksum or hash matched?  In other words, what would be the difference between checking small bits of data vs checking a checksum of small bits of data?

First someone takes a checksum of the file, storing that on their computer. They upload the file to the network, then delete their local copy (to free disk space). Once the person receives the file again once they need it, they simply take the checksum of the received data and see if it matches the previous one.
So the only way to check if a node is storing the correct data is to download the whole thing?  I was trying to think of a way that the uploader could continually validate that the nodes are holding the proper data.  Keeping a decently long index of random pieces of the file would enable the uploader to check only a particular part of the file.  If the seeder returns that particular random part of the file correctly, then it can be assumed that they still have the whole file intact.  Then, the next day, try a different random part of the file, etc.  As long as a sufficiently long index is kept (maybe, a few bytes of data for each day that they want to be able to continue checking), then the uploader should be able to continually verify.

Is that overkill?
Quote
I suppose the one thing I am most interested in at this point is more detail on how Bitcoin addresses can be tied to hosts.  Gladamas touched on it, but I am not enough of a programmer to understand how it might work.

I'm not really a developer either, but this is what I understand: Alice has files to upload onto the network, and Bob has disk space to share. Alice encrypts her data with her public key (so only she can see her data). Alice sends the encrypted data and her public key to Bob. Bob then encrypts his Bitcoin address using his private key. He then sends that and his public key to Alice, who can then authenticate the message using Bob's public key.

Anyone else can then prove that Bob was the owner of the Bitcoin address by decrypting the message with Bob's public key. Bob can also prove that he owns that private key by having someone else encrypt a message with Bob's public key, then Bob decrypts it with his private key and verifies the contents of the message with the person who sent it.

I'm curious, though, how to stop Alice from, say, denying that Bob ever sent her his Bitcoin address, or if Alice simply refuses to pay.


That's why I think a central entity needs to be involved to manage payments.  Otherwise, hosts would have to rely on the reputation of the uploader to ensure file seeds are paid.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 12, 2012, 11:52:17 PM
6) Service.com automatically pays out 0.01 BTC/day split between all users seeding b1.rar, with random data integrity checks to ensure they have the complete file.  If 5 users are seeding b1.rar by the end of day 1, then service.com pays out 0.002 BTC to each of them.

Ok, so service.com is the cluster co-ordinator for a bunch of storage nodes, or maybe is even further up a hierarchy than that.

How does the customer know how many copies of his/her file are seeded? Specifically, are storage.com's claims about that believed or does the customer get to see directly peer to peer with the hosts hosting the copies that they claim to be seeding it?

I ask because I am wondering if each host can add a firstbits string to the name by which they seed a copy.

For example say you want to store SgtSpike-Desktop-2012-05-31.bak.rar

I have a firstbits of 1DIgitalis, so I seed it as SgtSpike-Desktop-2012-05-31.bak.rar.1DIgitalis or whatever: basically I put my firstbits in there, or if filename length isn't a problem, then lets forget firstbits and use full bitcoin-addresses.

Now you should be able to ask bittorrent for a list of all files whose name start with SgtSpike? or maybe for any with keywords of SgtSpike and Desktop? Or if wondering what else I am hosting, all files with 1Digitalis in the name, or something like that?

Or does bittorrent have metadata even, like GNUnet and hmm not sure but I think maybe Tahoe-LAFS do, so a seeder can associate a whole bunch of keywords with a file without having to stuff them into the filename?

I am thinking here maybe we could eliminate service.com entirely even, simply by including in the keywords or filenames the offered reward for seeders from some offering seeder, the identity of the offering seeder, and the bitcoin address rewards for the copy having this name or keywords-set (description?)

Service.com could be useful as escrow though, so storage nodes do not have to trust some random seeder's offer just storage.com's claim that storage.com will make the offer good.

-MarkM-

Torrent websites typically display how many people are seeding a particular file.  I am sure it would be relatively easy to verify though.  As it is, one could simply start up the torrent download and see how many connections to seeders they got.  There is likely a way to verify # of seeders without actually starting the download though.

I am not sure if torrents are capable of containing metadata, and if they are, if it would screw up the "copy" status and not just make it a whole new separate file if some of that metadata was changed.  But you're thinking along the same lines as I am in that regard - the seeder would somehow advertise their bitcoin address whenever they are seeding a paid file.  Service.com wouldn't be absolutely necessary, but I think it would be valuable to match up uploaders with those who are looking for the paid torrents, as well as acting as escrow to ensure that the uploader actually paid for the uploads.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 12, 2012, 11:58:07 PM
Content-Addressable Storage (CAS) uses the content to address storage not the location in the world or on disk or whatever at which it is stored nor some nickname some user might find humanly appropriate characterising the file or its content.

This, any change in the content of a block / unit/ file / whatever of storage changes the identity/ID of that block /unit/ file / whatever.

Basically the hash is what one refers to it by, so anything that does not hash to the hash that was requested is not what was requested.

Doing this block by block instead of file means that any block can be requested.

Many systems work this way, including, I think, Freenet, Tahoe-LAFS, GNUnet, Twisted Storage, and camlistore. Maybe even bittorrent. If your bittorrent does not offer you a user-interface for asking for individual blocks or sectors or fragments or somesuch of a file, likely it is merely hiding that functionality from you the poor ignorant human, while using it itself to request specific portions of a file from specific hosts in order to download many portions at once from many hosts.

So maybe you just have to dig a little deeper into some kind of advanced mode or debug mode of your client, or complain to its developers that the user-interface is somehow not making clear to you how to tell it yourself which piece to get from where.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 13, 2012, 12:04:49 AM
You would need the following:

service.com:
Gets Bitcoin payment + has a "upload file or torrent to seed" field on the page + displays a daily rate.
Offers a service.com integrity checker client (can be a fairly simply python script probably).
Offers a page with all files hosted + potential daily rate for seeding them.

uploader/user:
Either creates a torrent file and uploads it or simply uploads a file to be distributed to service.com
Service.com then creates a bunch of random check points for arbitrary points in the file (say: MD5 of Bytes 1337-31337, SHA1 of Bytes 23-42...) and puts it on the offer page, as soon as the BTC payment came in.

seeder/server owner:
Checks the offer pager + installs the integrity checker client and any bittorrent client.
If one or more offers are appealing, he loads the torrent into the BT client as well as into the integrity checker client.
This client then allows him to download the file quickly from service.com if he is the first seeder and handles payments as well as the random integrity/uptime checks (from service.com: "What's the MD5 of Bytes 1337-31337?" - "abc123" - "Correct!").

Something still to be determined though is how to make sure the seeders really seed with some guaranteed bandwidth. Maybe this can be done by having a modified bittorrent client on the leecher side that rates the seeders accordingly... At least the problem of how to make sure someone really has a complete file for a certain amount of time though is solvable this way in my opinion.

Something interesting to look at for potentially scoring seeders: http://en.wikipedia.org/wiki/Glicko_rating_system
Now how can we trust the downloaders though... :P


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 12:17:50 AM
Markm, exactly, and that is where my limited knowledge of torrenting has me hung up.  I am not sure how parts of a file are downloaded and verified.  It possible that the .torrent file contains the checksum or hash information of all the pieces of a particular file, so that each piece can be checked as it is downloaded.  I need to do some reading on "how stuff works: torrenting" or something.

Sukrim, sounds like a decent summary of the conclusions made so far.  Thanks for posting.  ;)

I was thinking about the potential seed speed issue as well... not sure how to solve that one, besides users reporting any hosts that do not meet the minimum speed requirement.  Since torrent seeders can typically set whatever upload speed they like, and the speed set is transparent to any users downloading from more than one host, it would be difficult to track.  But, if a way to see the download speeds from each host is implemented, as well as a way for users to report bad hosts to service.com, the threat of being permanently removed from participating in the network might be enough to ensure hosts at least provide the minimum speed required.

Smart (and malicious) hosts could allow quick uploads to service.com, but slow uploads to anyone else.  That way, if service.com was checking their seed speed, they would appear fast.  Service.com might counter by using multiple IP addresses, but those could become known by users making false reports, then recording any new downloading IP's to find out what service.com's other IP addresses are.  Aside from using IP-changing VPN's around the world, I don't see a good way to counter this besides user feedback.


Title: Re: [IDEA] Dirt cheap online storage
Post by: rjk on June 13, 2012, 12:39:41 AM
Torrents break the files into regular and specific piece sizes, and hash each piece with SHA-1. The list of hashes is included in the .torrent file. This is my understanding based on the wikipedia article on bittorrent.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 13, 2012, 01:26:50 AM
It had been quite a while since last I had read up in Bittorrent, so I am wikipeding it now.

Take a look at http://burnbit.com/

Looks handy for anyone who can upload their file to their website to have their website be their initial-seed host.

I remember this part:

"Routers that use network address translation (NAT) must maintain tables of source and destination IP addresses and ports. Typical home routers are limited to about 2000 table entries[citation needed] while some more expensive routers have larger table capacities. BitTorrent frequently contacts 20–30 servers per second, rapidly filling the NAT tables. This is a common cause of home routers locking up.[42]"

DIstributed key word search sounds interesting and there is  free open source code for it (see for example Tribler).

Maybe we don't need any centralisation at all. Adding money into the equation might be do-able just with a Treasure Hunt script, on top of which some GUI-lover can put a GUI, and a Bury Treasure script likewise which an initial-seeder (person hoping to get copies of a file to persist a while out in torrentland) can use to bury treasure in a file before initial seeding and to advertise a treasure map some time later.

("By the way dutiful retainers of my data, if you take a look at bytes X through Y, you might recognise that there, but for ROT13, is a bitcoin private key. Those of you who know not what ROT13 might signify are welcome to await my next clue, or, if impatient, to perform some research on the topic...")

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 04:03:08 AM
That website is pretty neat!  Looks like an easy way for people to quickly and easily create a torrent and get it seeded at the same time.

I think a consistent payout is better than a treasure map that will pay out only to one person at one point in time.  There's nothing wrong with some centralization to organize and achieve this.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 13, 2012, 05:59:13 AM
More useful-looking code:

https://bitbucket.org/blueluna/transmission_supervisor/wiki/Home

http://www.oneswarm.org/about.html

http://en.wikipedia.org/wiki/Opentracker


I am wondering how much of an advertising budget folks looking to sell storage space are willing to spend to find customers for such a service; what percentage of gross earnings will it take to maintain a given level of gross earnings or even increase gross earnings? What percent commissions should people who actually manage to bring a paying customer get, and should it be only a percentage of the actual payment they convince the customer to make instead of an on-going percent, so that salespeople will be motivated to keep in touch with customers encouraging them to renew when their initial payment has been exhausted?

It is all very well to say lets bring those looking for money together with those who have money and want to spend it, but the competition these days for people who even merely have money or might have money, let alone are actually willing to spend it, has gotten a lot more intense, and more expensive, than back in the gravy-train raining-soup days of the very very late twentieth century.

Thus budgeting for purchase of marketing from outsiders (buying adsense ads facebook ads and so on conceivably, maintaining email capture landing-pages and professional third party mailing-list services, purchase of raw hits, etc etc etc), and/or actual marketing by insiders - surfing manual-surf traffic-exchanges? Running auto-surfs in browsers 24/7? Or merely contributing money to pay for professional marketing?

I guess the implication here is am not seeing a whole lot of "heck yes I'll buy X gigabytes a month at Y bitcoins a month per gigabyte" type of comments so I am dubious of finding much paying demand.

Reverting to the Treasure Map model was largely because of having that impression; I had drifted toward lets see who is still online in a few months of freely sharing and see how many months consistent reliable storage hosts will end up having to be online before starting to see people actually offering money instead of merely offering storage of their own. I have also, y'see, an impression that people are more willing to offer storage than to offer money thus if they are reluctant to even make their storage available it is even less likely they will ever actually make any of their money available.

Maybe those with bitcoins can demonstrate control of addresses with bitcoins in them for a few months to show they not only have coin but are disciplined enough to still have coin a month or few down the line, while those who have space can demonstrate they not only have space but it can be utilised and still has in it in a month or few what was entrusted to it, then maybe we can get a better idea of whether this is going to be a workable matchmaking project?

Bearing in mind that if we can do this with off the rack software and already existing networks which might have "network effects" of their own already, the potential paying customers might well notice that what we have come up with could probably work sufficiently well without paying any actual money. Afterall look at the numbers of people doing bitswarming already, each "seeder" providing storage already if only while their own downloading is happening.

The only example we have encountered so far of someone who actually needs space but has none to offer turns out to be probably better served by begging borrowing or stealing a spare disk drive from her brother, so is not actually a case study in favour of this project afterall...

Maybe in reality most use-cases of wanting/needing off-site storage tend to be not really for storage at all but actually for sharing (with oneself and/or others), the desire to get the data off the site being to have it somewhere all one's friends can look at it and enjoy it and so on, you oneself can access it from anywhere and so on, thus maybe tend to be more about bandwidth in a way than storage?

Maybe those looking to make money with their storage capacity would be best off  stocking it with content many people want but not quite enough people for that specific content to already be 24/7 available for free?

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 06:30:21 AM
Maybe you missed all the responses in the beginning of this thread, which said, "I'd be interested!"

If nothing else, it'd be an interesting experiment.

I think the treasure idea is kind of silly myself though.  I wouldn't take it seriously.  As a host, how would I know that there is actually a treasure there, and the uploader isn't just falsely stating as much?  Or how would I know that the uploader didn't make a copy of the private key holding the treasure, and once he gets bored, he'll just withdraw it himself?


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 13, 2012, 06:34:17 AM
Except for one person who wanted to be counted in on both sides of the fence, I had the impression all the interest was in getting money for their space, not in paying others for use of others' space.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 06:39:20 AM
Except for one person who wanted to be counted in on both sides of the fence, I had the impression all the interest was in getting money for their space, not in paying others for use of others' space.

-MarkM-

If that's the case, then anyone who does use the space will get it extremely cheaply, attracting attention and (likely) more customers.  People who are paying $125/month on Amazon S3 to store 1 TB of data can now store the same data split among 5 hosts for $25/month?  It starts to sound attractive...

Free market at work.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 13, 2012, 07:09:54 AM
TL;DR lets demonstrate and/or provide everyone's free 1 to 5 gigs for [life?!?] first like the professionals do, then see if that is enough proof of capability to attract paying customers.

It should eventually start to look attractive, but only once it has been running long enough to see how many copies one should actually have it keep in order to have a reasonable certainty the data will survive X span of time and so on.

Many among "the competition" give one to five gigabytes for life to all comers, without embarrasing their offer with bandwidth limits let alone dangerous-sounding nickel and dime buildup of charges as bandwidth k'ching k'ching k'ching hits your purse every time the data is actually viewed/delivered.

If we start by setting up a stable, useful "free" network based on sharing, people will be able to simultaneously audition as prospective storage nodes and observe for themselves how many copies of a file need to be out there to make that file's survival likely, then once we have real statistics on how many copies it takes people will have some figures from which to start computing how much someone would have to pay for one surviving copy in order to start with enough copies that there will actually be at least one surviving copy.

If prospective storage-providers are not confident enough that they stand ready to store files reliably for months or years to actually do so long enough to build up the confidence of prospective customers, how likely are they to actually do it once they have money in hand and some other short term scheme to try for a few days that seems to them to make it worthwhile to wipe all customer data to make space for the new scheme, whether it be a cute new automatic-pornsite-builder or yet another spam-the-search-engines-site generator or gosh knows what else that might look like a quicker buck than waiting for a storage customer's next periodic payment of storage fees?

Heck maybe they'll decide they can crack captchas if they fill that space with every captcha image all currently popular captcha generators generate, I am talking about bottom-feeders but a huge fraction of them are likely to go try to scam someone else somewhere else rather than wait out a several month assessment period and by the time such a span of time passed they would very likely have moved on to some other seemingly game-able project or crowd that might offer an easier, faster buck.

Meanwhile, I see the Bounty for adding Accounting to Tahoe-LAFS (preferably or requiredly bitcoin accounting) thread has been revived...

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 07:40:00 AM
There's no "chaching" when you download or view a file.  The very definition of seeding means that the host is making the file available for download 24/7, no strings attached.

I don't see how it is possible to give free storage, since no central agency is offering storage of any kind.  The onus to store the files is on the seeders, and they would have little incentive to give free storage to anyone when they can receive paid storage instead.

The problem with setting up a free network is that it would not be a reliable test to find out how a paid network would fare.  Uploaders and seeders would act completely differently in a paid market than in a free one.

Some people may decide to drop out of hosting other people's files, but that's the beauty of doing it via torrent and splitting the customer fee between the hosts - if one host drops out, then it is likely profitable for another host to take his place, which is done quickly and efficiently via downloading bits of the file from all the other hosts currently hosting a particular person's backup files.

Why not just try it and see what happens?


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 13, 2012, 08:01:27 AM
We have been trying it, and see what has happened so far: no concrete offers of bitcoins or fiat to any of those who have space available so far, resulting in continuing water-cooler and/or back-room-engineer chatter about various proposed engineering or marketing tweaks that might or might not result in more or better offers of actual money sooner/faster.

In other words: we already have at least one terrabyte on offer, we already have lowered its price to $1/month per 100 gigabytes, and we still have no concrete offers from anyone to buy a month at that price even stipulating it is on a terrabyte disk dedicatable to the purpose that can become dedicated to the purpose if/when in fact anyone actually decides that purpose suits his or her purpose and presents coin to follow through on it.

So here we sit chattering about various ways the offer can be manipulated, dressed up, elaborated as to precisely what protocols could be deployed for accessing it if a customer has a preference as to what protocol they would prefer to use to access it and so on and so on and so on.

We could firm it up further maybe by getting more details about other swathes of disk, in case the problem preventing firm bids from coming in happens to be not wanting their data to be hosted on a Windows machine, or doubt as to your technical and systems-administration competence to accommodate a request that FTP be the means of access or for NFS mount to be possible or some such technical concern.

Presumably some among us can provide RetroShare based access if some customers would prefer that?

For blocks, chunks, spans, files, collections etc small enough to be threaded through a needle/bottleneck the size of a gmail account, likely some could provide access-via-email if that would be more attractive. Lets maybe just ask each customer how exactly they want to deliver their data to a storage-facility and how exactly they would like to recover it from a storage-facility, and see if we are able to comply with the customer's requirements? How many gigs do the prospects-so-far want to store and by what protocol do they want to deliver it and retrieve it?

If all their preferred methods are vapourware, then basically they are saying they do not actually want to store any data at this time, they prefer to wait until some hypothetical Year of Condensation rolls around. If so, fine, when the coin materialises to seed the vapour sufficiently to cause condensation we can take this up again, meanwhile I see a vapour trail over in a Tahoe-LAFS thread, nice chatting with y'all.

-MarkM-

EDIT: Hey, anyone else noticing this forum is still here after all this time? Heck even when it got shot down over that bitcoin.org that didn't lose the data, did it? Or did it? How much damage, if any? (Lost posts? Lost user-accounts?) Its nice having this free storage to store my posts. :)


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 13, 2012, 08:47:58 AM
Torrents break the files into regular and specific piece sizes, and hash each piece with SHA-1. The list of hashes is included in the .torrent file. This is my understanding based on the wikipedia article on bittorrent.
True, these hashes can't be used for testing though, as they are public knowledge already.

To request seeds for a torrent file, you send the SHA1 hash of the whole list of SHA1 hashes (and some other data - looks kinda similar to the bitcoin block header) called "info hash" to a server called "tracker" and state that you either have these files available or that you want to have a list of IPs who have these files.

All trackers know is that infohash "abc123" is seeded by IP 1.2.3.4:1337. They know nothing about  file names, how big the files are, if 1.2.3.4 even HAS the file or is a jerk pretending to and they have no means to verify any of this. They are just plain stupid databases on the internet. This gets more and more replaced by DHTs nowadays, which are in the end just a distributed database, meaning some nodes feel responsible for the infohash "123456" and others for "654321" and you get your data from these nodes instead of a central server.

As it would be near impossible to include monetary transactions into the bittorrent protocol itself, we'd rather need to extend it and just use it for transport.

Modifications would have to include:
Ability to allow, traffic shape or ignore hosts on-the-fly based on if they are paying or freeriders.
Central trusted element (service.com) - I'm not 100% sure if it can be distributed without having a server yourself that constantly does these integrity checks. It could be distributable in that sense though, that there are multiple services to choose from and you can host one yourself (just like torrent trackers but with a lot more bandwidth requirements)
Something to verify upload speed. This is the biggest question mark for me, as it would be trivial to have a central server that downloads files from nodes to check on their speed - but it would be trivial for them to just grant this single node the high speed, everyone else has to live with 1 kb/s.

@markm:
How could I make sure that any of these offers floating around are actually worth even the 1 USD/month/100Gb they ask? Going offline, not offering any significant bandwidth, destroying data, taking a run with the money...
On the other hand, how would I deal with the fact that if I offered to host 100GB for 1 USD/month it might be the case that I never get paid or have to deal with 100s of GB upload because suddenly I'm hosting the newest season of Game of Thrones for the whole internet...


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 13, 2012, 09:04:12 AM
For now I think the pick and choose offers type of market makes more sense than an automatic-matching market.

The forum's longevity made me think aha, maybe the folk running this thing might not be too awful a choice to pick if one of the offers on the market / on the table was one of theirs.

As to this SgtSpike chap, hey, lemme go see if he has a .sig, I honestly don't remember, having not even been motivated to look and see yet. Hey he lists a website! I wonder if that has been there ever since I or he arrived at this forum and I simply never noticed? Interesting. Maybe that is a sample of some storage he manages. I wonder if he hosts it from on-site or at an off-site-hosting provider...

As to whether you would get paid, why would anyone even consider paying you in the first place? Oops, I haven't noticed whether you have a sig either yet alone inspected it. Why would I? How much space are you offering at what price? Personally I am not looking to buy, I have onsite storage that, to others, is offsite so am more interested in swapping than buying.

-MarkM-

EDIT: I am moving more and more toward regarding "storage" and "hosting" as very different services. "Hosting providers" are basically selling publication, not storage. Maybe it is kind of like the difference between a warehouse that stores your books and a bookshop that displays/sells them to the public...


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 09:17:00 AM
Markm, we haven't been trying anything.  This is simply a discussion.  The fact that there are MANY services out there providing user backups for X dollars a month proves that the market exists and is very real.  The fact that many people here are willing to offer their extra space for much lower prices than the existing services proves that the potential for P2P paid backups is real.

Sukrim, good thoughts, sounds like we're thinking very similarly about the challenges and practicalities of making something like this a reality.  You have more knowledge about torrenting protocol than I do though.

Markm, I have no idea what you are talking about with regards to sigs and offering storage.  I'll say it once again: paid torrented backups are NOT about people offering hosting - it is about uploaders offering files along with regular payments to keep those files seeded.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 13, 2012, 10:17:19 AM
Maybe a system like this should include a share ratio though - something like "1 MB hosted can generate up to 5 MB upload during the last 7 days, after that you can shape or even deny access" with higher ratios obviously costing more. This would address the "storage vs. hosting" issue, as hosted files have to be stored somewhere and stored files need to be hosted to be available on demand. The difference is that hosted files will probably have a lot more traffic but might be smaller than the other way round.

To circumvent the traffic meter issue, it might be possible to split payments in 2 parts:
1 part for storage (checked via checksums from service.com) paid by service.com
1 part for traffic, measured by downloaders and paid by service.com

The downside would be that downloaders also require an account + deposit at service.com as well as some traffic metering software - the upside would be that there can be no (useful) faking of traffic, as the downloader simply says "I loaded 12 MB from node 1, 16 MB from node 2 and 100 MB from node 3". If he wouldn't run the metering software, he wouldn't be able to get unshaped downloading speed in the first place, the only thig he could do is to lie about ratios (like he loaded 100 MB from node 1 and 12 MB from node 3 in reality) or nodes (he loaded nothing from node 3), but both of these scenarios would not benefit the downloader and he'd have to pay anyways for traffic consumed.

The implementation could be like this that a downloader says at service.com that he wants to download 100 MB of files. Service.com runs a bitcoin-style block chain (probably with faster blocks and not publicly mined) and transfers a number of 1 MB "coins" to the user and charges his account for this. As the user downloads (seeds can check that he actually has some money by requesting a message signed by the private key of the address that holds the coins), the metering software sends out DL-coins to the seeders' addresses. This way they can see after some time, that the user is really paying up and trustworthy, so he should be able to quickly build up trust within the swarm. It's even possible to gain back some money for the user, by seeding the file himself while loading and publishing a seeder address.

Once the download is complete, the user can transfer remaining DL-coins back to service.com (there should be some slight overprovisioning to account for CRC errors, rounding errors etc.) and gets some BTC back to his account there or he just accumulates them for the next time.

Seeds can do the same (redeem DL coins for a fixed price or bandwidth).

The balance of BTC stored at service.com can be requested at any time, service.com would act similar to a mining pool nowadays (collecting lots of tiny transactions of some value - mining shares - and paying out aggregated payments to reduce blockchain spam and bitdust issues).

1 DL-coin could for example be equal to 1 satoshi, this way conversion would be easy and there would be no decimal point numbers for people to worry about.

Why have different coins, not BTC?
* no block chain spam
* nano/picotransactions (1 MB of traffic is far below the treshhold for transaction fees)
* quicker confirmation times (you don't want to wait for an hour until you can see that the leecher is really paying up)
* centralized mining allows for a "1 block = 100% confirmed" style of mining but it could still be expanded more easily for more than 1 service provider compared to a pure server solution

What would be the risk for seeders?
* Some leecher shows that he owns 1000 MB worth of download, starts loading and after a few MB still doesn't pay anything. Solution: Traffic shape him until he pays up (confirmation times would be likely a few seconds or so, so not much to be wasted)

What's the risk for leechers (downloaders)?
* A seeder sends bougs data. Solution: Don't pay after getting the data
* A seeder is too careful and wants to see you pay other seeders first, before offering any data and you can't pay anyone else since they all are too cautious. Solution: Probably people will register seeders for the purpose of giving some MB away and then seeing if you pay up - the big advantage for that is that you are then likely to be a preferred seed by the leecher and can earn more through the traffic. Maybe it can be possible to detect such situations by service.com (simply act as leecher and ask seeds if they are fine with offering files in advance) and flag such files as opportunities to be grabbed. This is somehow an unsilved issue and depends on the ecosystem of seeders that arises... the monetary risk of handing out a few MB for free is however probably smaller than the potential profit on being the first seed on the list.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Cobra on June 13, 2012, 02:29:05 PM
I'm not sure if anyone has seen this site before http://www.symform.com/join-the-revolution/how-symform-works/ (http://www.symform.com/join-the-revolution/how-symform-works/) The concept seems very similar to what the end result should be here but without the central controlling authority. It would be something or progress if a company like this would agree to compensate users in BTC for contributing drive space, granted they do not have the need to store files in the cloud in the capacity they are sharing.

I am excited to see where a project like this may end up, it seems like the perfect companion for use with BTC.



Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 13, 2012, 03:35:26 PM
Symform doesn't seem to meter bandwidth provided, so I could just offer some storage, download fragments and rate-limit the port they use in my router to 1 KB/s. In reality I gain online space for that but don't contribute anything meaningful to the network.

Their approach looks actually quite similar to tahoe or Wuala (back when they were P2P) or the japanese P2P sharing system "perfect dark".

Just like BitTorrent they depend on user's generosity and/or not too many freeriders. Once you bring money (even "internet funny money" like bitcoin) into that equation, it could be an explosive mix...


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 03:39:57 PM
@ Cobra - it is similar, but again, doesn't solve the case for people who have extra storage and don't want to store files, or people who don't have extra storage and want to store files.  It is very similar to the idea markm was throwing around earlier about trading storage though.

@ Sukrim - do you really think masses of people downloading from the hosts would be a problem?  I figured that for popular and public files, people would just use traditional torrent methods - the paid torrents would likely only be necessary for unpopular files, or files that no one would otherwise want.

But, if it would be a problem, instead of making it much more complicated, perhaps the amount of bandwidth used on a particular torrent simply goes into the decision-making process that leads a host to keep or drop a particular torrent in favor of another.  If, at the end of a 24 hour period, the host finds that one particular torrent consuming 1 GB of space has consumed 5 GB of bandwidth, they might consider the bandwidth to be an unreasonable expense compared to the daily payment of the file hosted, and drop that torrent.  Or, they might write up a rule that if a particular torrent exceeds twice its size in bandwidth in any 24 hour period, then drop the torrent.  Or if it exceeds five times its size in bandwidth in a month, then drop it.  Etc, etc.

Of course, such an analysis and response is passive, but would be a lot easier to implement than special coins and double-metric pricing.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 13, 2012, 05:15:53 PM
For popular "files snatch and leave" is more of a problem, slow seeds are more of a problem with files that have only few seeds to begin with. If you upload pictures of that sister to a handful of people and they all decide they would rather seed other files faster, there's no chance for you to get these back quick enough.

Symform and others deal with this by dispersing files wide enough that this doesn't matter (you can afford to load a few tiny parts slowly while you're busy downloading from seeds that are not jerks).

Yes, something to ensure seeds just have the file might already be enough, as the bandwidth might be anyways cheap compared to the cost for storage and too popular files can be dropped as you said. If it isn't though, we have a problem. It might be enough though, especially for initial seeding of a potentially popular torrent. These tend to violate copyright laws sometimes though...
This is actually really a problem for service.com, as it MUST be the only one knowing in advance which parts of the file shall be checksummed and what the sums are. This means service.com has to have every file on it's HDDs at least once. This means, files either have to be encrypted with a key not known to service.com (similar to one click hosters) or filtered by service.com.

10 US cent per GB per month would be ~42 Satoshis per minute by the way, so cheaper rates (e.g. 2 satoshis for 1 minute of seeding 1 GB) are possible and feasible at least. 1 Bitcoin would then seed a bit more than 1 TB for 1 month. Maybe a bit too cheap, but offering a market for that might anyways quickly establish prices.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 13, 2012, 05:23:23 PM
Here's another other question:  Is slow downloading a huge issue for people making backups?  If I back up 50 GB's, then my hard drive crashes the next day and I lose everything, is it that big of a deal if it takes me until the next day to recover all my goods?  For a business, certainly, backups would be crucial to retrieve ASAP.  But I don't think this is a solution that businesses would (or should) be interested in anyway.  Just something to think about...  I really don't have any solutions to seeders capping the download rate besides users reporting it though.

Why couldn't the uploader be responsible for knowing what parts of the file needs to be checksummed?  Perhaps some piece of software could be downloaded that automatically creates a checksum index with random parts of the file.  That index could be uploaded to service.com, then service.com periodically requests each seeder to provide a checksum of one of those particular indexes.


Title: Re: [IDEA] Dirt cheap online storage
Post by: justusranvier on June 13, 2012, 05:24:37 PM
We have been trying it, and see what has happened so far: no concrete offers of bitcoins or fiat to any of those who have space available so far, resulting in continuing water-cooler and/or back-room-engineer chatter about various proposed engineering or marketing tweaks that might or might not result in more or better offers of actual money sooner/faster.

In other words: we already have at least one terrabyte on offer, we already have lowered its price to $1/month per 100 gigabytes, and we still have no concrete offers from anyone to buy a month at that price even stipulating it is on a terrabyte disk dedicatable to the purpose that can become dedicated to the purpose if/when in fact anyone actually decides that purpose suits his or her purpose and presents coin to follow through on it.
All you've proved is that a market for the manual negotiation of storage contracts does not exist. I don't believe that was the original intent of the OP.

The set of people who are willing to sell excess hard drive space when the required effort is just install the software and keep their computer online is larger than the set of people willing to haggle for it on an online forum. The same applies to people on the buy side. If you'd been actually reading my posts you would have noticed that is a large part of my thesis of why Mojo Nation failed. The market mechanism needs to be automated or else there will either be a lack of price discovery or price discovery will be so difficult that nobody will bother to use the software.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 13, 2012, 05:41:23 PM
I suspect that compliance discovery is far more of a challenge than price discovery.

I will probably have to take symform up on their offer if as their online chat support said they will have Linux compatibility in a few months, even though they are not open source.

The idea of throttling the connection to them seems not too bad an attack since if one does not actually provide fast speed then one can simply be paid in slow speed access to one's own data. By paying the storage nodes "in kind", service.com can measure what "kind" they actually are and reciprocate with the same "kind".

Maybe bitcoins could be awarded for "performance above and beyond the norm" or somesuch though folk not planning to strive to be among the very best maybe wouldn't bother trying in such a setup.

It was mentioned in the Tahoe-LAFS thread that only the three fastest nodes would be asked for blocks of a file set at 3-copies redundancy. Just as mining got more and more concentrated into the hands of those building the most powerful rig-farms maybe pretty much everyone whose offered space isn't directly on major internet backbones simply would end up never being asked for a copy thus being largely irrelevant backwaters.

-MarkM-


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 14, 2012, 08:50:21 PM
Why couldn't the uploader be responsible for knowing what parts of the file needs to be checksummed?  Perhaps some piece of software could be downloaded that automatically creates a checksum index with random parts of the file.  That index could be uploaded to service.com, then service.com periodically requests each seeder to provide a checksum of one of those particular indexes.
Because that file could be sent to someone collaborating with the uploader (Seeder X) who doesn't really download the data but now has every possible answer to the challenges. He would only get the money for the files from the uploader, but maybe also some rating as "that awesome seed that has 0 CRC errors + stores multiple TB for theat uploader" that would probably be part of the platform too.

I wouldn't throttle transfers from symform to symform people checking on my upload capacity, only to anyone else - so if they check me out, I'm that 1 MB/s upstream candidate they always want to see. If anyone else actually wants their data though, good luck with that!
Symform can't trust complain reports of these clients that I'm slow, otherwise I could write a client that complains about anyone else and rate these down.

About markets - this could very well be automated. Just set a price you're comfortable storing data at or a price you'd like to charge at least and let the service handle the rest. The hard part is really enforcing these contracts and making sure they are followed, from both sides.

A simpler version that can be done without handing too much risk to service.com but that should still be fairly secure would be to hand the random checksumming still to the uploader (which in return however could provide that data to be used to fake seeds). Uploaders then send a torrent file (or better: magnet link) + a file with the random checksums to service.com, send some BTC + give a timespan of how long this file shall be seeded.
These things (magnet link + BTC pledged) then get published + a verification client, that has to run and that handles the checksumming part (can/should be open source) can be loaded. In the verification client, seeders enter their payout address + a treshhold (say 1 BTC) and similar to mining pools, as soon as their balance reaches the treshhold, they get their money. Also if they don't seed anything for 14 days, and their balance is above "bitdust" levels (say 0.005 BTC?) they get their remaining balance. Service.com only has magnet links and random checksum data and doesn't require any data from it's users other than payout addresses + IPs (for connection checks).

This could be used for any kind of P2P network actually, BitTorrent, eMule (ed2k), several of the anonymous networks like WASTE...

Prices would set themselves because it would be something like: "I offer 1 BTC over 1 month to all the people seeding this torrent". Every hour there's a test for anyone claiming to seed that file by service.com: A random checksum challenge + trying to check out if a bittorrent client is running and seeding the file in question (yes, one can lie about that, but at least it takes some effort). If these 2 criteria are fulfilled, 1/720 BTC (there's 720 hours in a month) gets split between all seeds. If they reach a point where they don't earn enough, they'll just drop out. There can be live public stats and maybe even bonuses/penalties for people (not) failing checksum tests.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 14, 2012, 08:58:07 PM
Even using a verification client wouldn't stop the upload speed problem though.  Also, how would the verification client know that the file was being seeded?  Or is that why you are calling it a simpler version?


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 14, 2012, 09:15:13 PM
That's why I called it simple - service.com would only issue challenges + try to connect via BitTorrent to the seeds. If there is a checksum error or not even a torrent client running, no money for them! The hassle for downloading the torrent files + keeping them is hopefully enough though to let them really seed as well.

Ideally, other clients/downloaders would report from whom they loaded how much. This data unfortunately cannot be trusted though and I still can't really think of anything how to make it trustworthy.

Edit:
Maaaaaybe you can trust "I have loaded x MB from ..." data if the person claiming this can also solve checksum riddles. It would still be trivial for seeders though to solve these riddles and claim high upload rates from some random clients.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 14, 2012, 09:37:09 PM
That's why I called it simple - service.com would only issue challenges + try to connect via BitTorrent to the seeds. If there is a checksum error or not even a torrent client running, no money for them! The hassle for downloading the torrent files + keeping them is hopefully enough though to let them really seed as well.

Ideally, other clients/downloaders would report from whom they loaded how much. This data unfortunately cannot be trusted though and I still can't really think of anything how to make it trustworthy.

Edit:
Maaaaaybe you can trust "I have loaded x MB from ..." data if the person claiming this can also solve checksum riddles. It would still be trivial for seeders though to solve these riddles and claim high upload rates from some random clients.
It's hard to have an honor system when money is involved.  But, I'm kind of with you on it - I don't think it'd be a showstopper either.  If undeniable or overwhelming evidence presents itself of people abusing the network, they could be banned on a case-by-case basis.  It's more of a bridge-to-be-crossed-when-we-get-there problem.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 14, 2012, 09:50:11 PM
A different approach could be kind of a "flattr" model for paying uploaders.

Seeders of anything run a client that replies with their ID and downloaders run a client that asks any connected client for that ID (it either gets ignored or someone replies). Every now and then a list of "I loaded 1 MB from X, 40 MB from Y and 100 MB from Z" is uploaded to service.com.
Service.com then holds funds of downloaders in escrow and pays them proportionally to X, Y and Z. If downloaders don't tell any data, their funds for that time frame get donated or evenly split or split according to some rating (glicko2 sounds really cool) or ???. Stats should be as fast as possible available but aggregated over a week or even longer (2 weeks, 1 month...) to have meaningful data for payouts for most participants.

To get more funds, it is of course better to prefer clients that ask for an ID. It can be even 2-sided, so seeders can also see if they really got paid something from A, where they uploaded 1 GB to or if A was just asking for their ID but never submitted a report.

This would again require some trust from seeders, that someone asking for an ID will really pay something and requires 0 trust from leechers (as they anyways can use the swarm as usual - but have a chance to get some much better seeds if they ask for IDs to pay towards). If the updates run in near-realtime, trust can again be increased over time ("I uploaded 1 MB to A and a few seconds later A rightfully told service.com about it - more bandwidth to A!") and even without some fancy traffic shaping stuff it's simple to just seed some popular torrents and potentially gain a few coins too.

This way it doesn't matter if there are fakes or not (they can only pay anyways) and service.com also doesn't need to care about bandwidth offered as it is i the best interest of seeds to offer more bandwidth to users of service.com.

The above would be the other extreme - big torrents with few leechers would be not very popular, unless there are some leechers that really offer a lot for that traffic. Just keeping the torrent alive though (no leechers atm.) is not really attractive, as there's more money to earn by loading + sharing the next file and actually uploading something.
Maybe these 2 extremes (it's already enough holding a file and not even seeding it vs. upload is everything, even just loading 1 part of a popular file + just sharing it can net you a lot) can/should be somehow combined for different user groups or so?


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 14, 2012, 10:05:11 PM
What I am afraid of with that sort of solution (where they are only paid when people download from them) is that the files, particularly personal backups, wouldn't actually be downloaded very often.

But, perhaps a combination could be used, as you said.  Payments are aggregated for up to 2 weeks.  If no download of the data is made within those two weeks, then as the payments age to 15 days, they are paid split evenly between the seeders.  But, if the data is downloaded, at least once, then all aggregated payments to that point are paid, and are split between the seeders according to whatever share of the file was provided to the downloader.

This would encourage them to make the file available 24/7, and also seed it as quickly as possible in order to gain a larger share of the payment.


Title: Re: [IDEA] Dirt cheap online storage
Post by: lologarithm on June 14, 2012, 10:33:06 PM
What I am afraid of with that sort of solution (where they are only paid when people download from them) is that the files, particularly personal backups, wouldn't actually be downloaded very often.

But, perhaps a combination could be used, as you said.  Payments are aggregated for up to 2 weeks.  If no download of the data is made within those two weeks, then as the payments age to 15 days, they are paid split evenly between the seeders.  But, if the data is downloaded, at least once, then all aggregated payments to that point are paid, and are split between the seeders according to whatever share of the file was provided to the downloader.

This would encourage them to make the file available 24/7, and also seed it as quickly as possible in order to gain a larger share of the payment.

The idea of paying per download is the best way (and by best I mean only 100% sure way) to be sure of the seeders properly hosting the file. The issue as you mentioned is that for files that are rarely downloaded there is low incentive to store them.

Perhaps there is a baseline requirement: If no files are downloaded in a week the person pays a set amount to the people who have been seeding. The baseline would be much less than if the file were actually downloaded but might provide some incentive. Connectivity could be tested a few times a day perhaps to see who was connected throughout the week, then paid proportionally to uptime during the week.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on June 14, 2012, 10:45:08 PM
To determine that somebody has downloaded a file though, service.com needs that checksum quiz file... with the "flattr" model, service.com doesn't even need to know/care which file was downloaded. I get your point though, but I guess I have to think a bit more to come to a solution.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 14, 2012, 10:46:41 PM
What I am afraid of with that sort of solution (where they are only paid when people download from them) is that the files, particularly personal backups, wouldn't actually be downloaded very often.

But, perhaps a combination could be used, as you said.  Payments are aggregated for up to 2 weeks.  If no download of the data is made within those two weeks, then as the payments age to 15 days, they are paid split evenly between the seeders.  But, if the data is downloaded, at least once, then all aggregated payments to that point are paid, and are split between the seeders according to whatever share of the file was provided to the downloader.

This would encourage them to make the file available 24/7, and also seed it as quickly as possible in order to gain a larger share of the payment.

The idea of paying per download is the best way (and by best I mean only 100% sure way) to be sure of the seeders properly hosting the file. The issue as you mentioned is that for files that are rarely downloaded there is low incentive to store them.

Perhaps there is a baseline requirement: If no files are downloaded in a week the person pays a set amount to the people who have been seeding. The baseline would be much less than if the file were actually downloaded but might provide some incentive. Connectivity could be tested a few times a day perhaps to see who was connected throughout the week, then paid proportionally to uptime during the week.
That's an interesting idea as well.  But if files do get downloaded, does the person who is fake seeding get paid, or does he get left out at that point?

It does reintroduce the complexity of tracking two metrics in the marketplace though.  Whoever is hosting has to track both the payment per download and the payment per seed.  OTOH, it would allow people who don't have much bandwidth to spare, but lots of storage space to spare, to better compete with those who have lots of bandwidth to space, but little storage space.


Title: Re: [IDEA] Dirt cheap online storage
Post by: markm on June 15, 2012, 12:53:02 PM
EDIT 2: TL;DR By "dirt cheap", do we mean "cheaper than foisting it off upon the bitcoin blockchain"?


Okay, lets consider something that might be massive overkill of amount of resources being committed and overall certainty of the integrity of the data.

At least maybe try to figure out just how totally over the top the costs would be.

Remember that thing called a blockchain. whereby distributed peers maiintain an ongoing mutual confidence in the validity of past transactions some among them purported to have occurred?

There are algorithms too whereby checking each member of a (spy, etc?) network's reports about each other member, actual path to path inferences as to which path a "weak link" is on. Reliability matrices across many possible paths, making intferences both about which path was or  might have been taken and which nodes are how likely to be the sources of any unreliability that is measured/reported.

The content of the blocks of the chain could be assertions that each peer on the entire network received from each other one a correct merkle value taking into consideration which peers claimed to have recieved which such values from each other peers and what the actual total combined merkle of all bits ever submitted to the network the consensus agrees to be correct based upon the order in which the consensus agrees those bits were submitted (or more likely, committed; as like with bitcoins we'd not count a block of bits to have been submitted uintil the consenses agrees not only with whether it has been submitted but in what order, relative to others).

This could all even maybe serve as some research toward just how much a huge blockchain really does cost a network overall compared to how usefull to the network nodes that fail to store the entire blockchain actually are.

The chain could be pruned of blocks that have already been returned to their original sender undamaged after having been retained the stipulated period of time, and if original-uploaders ("content providers"? "financiers", maybe, as presumably they should bear the entire cost of this project between them?) are not anonymous if they fail to publish cryptographic proof of their agreement that they, the orignal uploader of bts blah blah blach, did in fact receive back intact said bits from nodes this that these those et al and sundry, etc etc etc then some black mark could be held against them too for trying to cheat participants of their deserved repution owed to them for their part in actually performing as contracted to.

This could be massive massive cost, but cxnsider how much people pay for the bits of the bitcoin blockchain, which is precisely a record of what quantities of bits, representing what quantity of bits worth of value, which parties exchanged in a consensually agreed approximate order in time...

-MarkM-

EDIT:  Also, what the heck, why not tie it all to bitcoins from the start, by trying to store the blockchain itself on just such "dirt cheap storage"...

...Or is that precisely what exactly bitcoin IS doing, in effect, and by doing so, is revealing to us the actual cost of storing (such?) data in such a way?

EDIT3: By bidding using various altcoins, might bidders thusly express the certainty level they are budding upon, like "I want to store this as securely as if I had coded it into the ___coin blockchain in some way" for each coin designated by the ___ ? (Taking into consideration the propensity of the developers of the ___coin blockchains to attempt to prune any data from it that they consider no longer necessary to retain after some span of time verus cost of blockchain size.)




Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 15, 2012, 04:00:02 PM
Markm - having everyone store the same data is a bit different than having a handful of people store the same data, and a different handful storing different data, etc.


Title: Re: [IDEA] Dirt cheap online storage
Post by: jfreak53 on June 27, 2012, 01:43:10 PM
See Sarge, this is what you get for not asking me first ;)

We offer dedicated server's with up to 3TB drive's for a one-time fee, not monthly.  The main ATOM server (great for storage) with a 2TB drive is $58.50 a month and $160 one-time fee.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 27, 2012, 03:49:47 PM
See Sarge, this is what you get for not asking me first ;)

We offer dedicated server's with up to 3TB drive's for a one-time fee, not monthly.  The main ATOM server (great for storage) with a 2TB drive is $58.50 a month and $160 one-time fee.
A one time fee?  That's interesting... how much?  What limits are there on bandwidth usage and speed.


Title: Re: [IDEA] Dirt cheap online storage
Post by: jfreak53 on June 27, 2012, 04:36:19 PM
The one time fee is $160 for the 2TB and 3TB I'd have to get you price on since I don't sell a lot of them. Bandwidth is default 10Mbps for that price monthly as stated above.

BUT, we offer 100Mbps and 1Gbps on this type server, we offer more speed's on bigger server's but these are the ATOM's.  The 100Mbps costs $50 a month and comes with 10 IPs and the 1Gbps costs $150 a month. Of course all port's are unmetered ;)

Base price of the ATOM's with 250GB drive and 2GB of ram is $58.50, then you can add drive's for one-time fee's or monthly if you want. The ATOM's can also have up to 2 drive's.  The difference in paying monthly for the drive and one-time is the one-time makes it your's, if it goes bad you have to buy another one, with monthly it's replaced free if it goes out.


Title: Re: [IDEA] Dirt cheap online storage
Post by: jfreak53 on June 27, 2012, 04:41:06 PM
Oh, also the 4GB ram upgrade for $20 a month on the ATOM's, comes with a 1.5 TB drive upgrade for the primary drive from the original 250GB free of charge ;) You can't beat it ;) And on that 4GB upgrade you only pay $15 one time for the 2TB upgrade from 1.5TB ;)

Those prices are not on the order system, unfortunately WHMCS doesn't allow for such sophisticated of pricing, so it's one of those thing's that we have to setup the original order for.  But those are the price's.

Also if you absolutely need it, we do offer speed's of 10Gbps unmetered ;) it's just a bit pricey and only on bigger server's.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 27, 2012, 05:48:45 PM
I misunderstood - I thought you were offering 2 TB of online storage for ONLY a one time fee, no monthly fees!

BTW jfreak, do you do colocation?  I have a server running from my house, and it's a nice high-spec server (would cost hundreds/month to rent something comparable), but I don't have all that much upload bandwidth.  I'd love a good 10Mbps or 100Mbps non-residential connection to run it on!  All the colocation services I've looked in to are very expensive though...


Title: Re: [IDEA] Dirt cheap online storage
Post by: jfreak53 on June 27, 2012, 07:26:59 PM
Well the one time is the Hard drive then your just paying for server and internet, you never again pay for HD, just server usage.

Colo: Yes I do :), just not publicized since we do it very seldom.

What do you have rackmount or tower?


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 27, 2012, 08:20:44 PM
Well the one time is the Hard drive then your just paying for server and internet, you never again pay for HD, just server usage.

Colo: Yes I do :), just not publicized since we do it very seldom.

What do you have rackmount or tower?
It's a full-size tower.  Could stuff it in a 4U though.


Title: Re: [IDEA] Dirt cheap online storage
Post by: jfreak53 on June 27, 2012, 08:56:41 PM
For tower it would be $69.00 a month that would give you:

-10Mbps unmetered (100Mbps unmetered + $25/m)
-electric included / 1 power drop
-Denver DataCenter
-5 Free IPs (10 on 100Mbps)



Title: Re: [IDEA] Dirt cheap online storage
Post by: coga on June 27, 2012, 08:58:59 PM
Markm, you're COMPLETELY missing the point.

I'm reading through the thread, and that is one thing I'm totally agreeing with


Title: Re: [IDEA] Dirt cheap online storage
Post by: coga on June 27, 2012, 09:20:16 PM
I've been contemplating developing something like that, and here is my vision how it must work. There should be 3 types of actors:

- Nodes (provide storage)
- Clients (consume storage)
- Network Controllers, or NCs (used to locate nodes for clients, and manage escrow)

Storage network allocates blocks in chunks, say 1MB each.

Nodes broadcast to NCs their reserves, price per megabyte and advertised avaiability and speed.

Lets say Client wants to store 20MB worth of data, and store each megabyte on at least 3 nodes. Client should have an account on NC, which should already have some BTC. Client would use NCs to locate 60 Nodes that charge requested price or less, and if those are found, upload the data.

Client would also register with NC that it did upload the data to a node, and specify the checksum of the blocks he uploaded, and several random salted checksums. Those would be used later to verify that node indeed has the chunk.

Nodes will be paid by NC daily as long as they store the chunk. NC will use the challenge that client gave it. To prove that Node keeps data, it will have to calculate the checksum of the salted chunk, and come up with the one that matches the challenge. If they don't, the chunk is considered abandoned, and node is no longer paid for this chunk.

Client periodically checks with NC for the list of abandoned chunks and relocates them to keep necessary redundancy.

NC stores the "reputation" for each NC in form of abandoned chunk count. The Nodes with zero or very little dropped chunks are considered higher reputation.

Nodes will get 50% of the bitcoin amount in form of mandatory payments from NCs. The remaining 50% will be distributed as "tips", and the distribution is determined by Client. So, if Client has to pay 2BTC for storage, 1BTC out of 2 will be given to nodes equaly, and another 1BTC will be distributes as per discretion of Client. This would be done so that Client could reward fast or least problematic nodes.






Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 27, 2012, 09:27:58 PM
@ jfreak - how much per month if I converted it to a 4u?  Would that make it cheaper?  The pricerange is close to what I am looking for, so it definitely has potential.  And 5 IP's would be awesome too!

I like your line of thinking coga.  Not sure I agree regarding the tips - that would add extra complication on the client side that they might not want to be involved with.  I still like the idea of the payments being distributed according to how much of the file each node contributes when the file is downloaded.  I am just uncertain as to whether it would be possible to track such a statistic.


Title: Re: [IDEA] Dirt cheap online storage
Post by: nimda on June 27, 2012, 09:39:35 PM
Simple, just download directly from the nodes, and meter how much data comes in from that particular connection.


Title: Re: [IDEA] Dirt cheap online storage
Post by: jfreak53 on June 27, 2012, 10:30:38 PM
Sarge: Sorry, nope, unfortunately it doesn't make it any cheaper. Basically we do what you stated, we take a 4U space and put a shelf adapter in it and it sits there, so it's more or less using up the same space as a 4U :)

Sorry, but it would be the same price.  But, PM me, let me know around what price range you were thinking, maybe we can work something out close to that.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on June 27, 2012, 10:46:25 PM
Sarge: Sorry, nope, unfortunately it doesn't make it any cheaper. Basically we do what you stated, we take a 4U space and put a shelf adapter in it and it sits there, so it's more or less using up the same space as a 4U :)

Sorry, but it would be the same price.  But, PM me, let me know around what price range you were thinking, maybe we can work something out close to that.
Got it, thanks.  ;)


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on July 01, 2012, 01:03:45 PM
Simple, just download directly from the nodes, and meter how much data comes in from that particular connection.
So all you'd need to do is choke all non-NC nodes and profit.

As said, the solution from coga is nice for ensuring somebody does indeed store a certain piece of data and keeps it on his/her HDD. It does NOT ensure that this data can be quickly accessed or accessed at all should the need arise. If I get paid only 50% (I won't get any tips) just for running a HDD with ~10W/hour but no outbound bandwidth cost at all, I'd be fine with that...

Compared to non-monetized P2P systems (e.g. Bittorrent's tit-for-tat) there is a much higher possibilty for fraud and a higher motivation and incentive to game the system (something that only pays out on bandwidth would mean that nodes only share very popular file pieces and delete them soon afterwards when nobody/few leechers want them any more).


Title: Re: [IDEA] Dirt cheap online storage
Post by: coga on July 02, 2012, 05:12:18 AM
Simple, just download directly from the nodes, and meter how much data comes in from that particular connection.
So all you'd need to do is choke all non-NC nodes and profit.

As said, the solution from coga is nice for ensuring somebody does indeed store a certain piece of data and keeps it on his/her HDD. It does NOT ensure that this data can be quickly accessed or accessed at all should the need arise. If I get paid only 50% (I won't get any tips) just for running a HDD with ~10W/hour but no outbound bandwidth cost at all, I'd be fine with that...

Compared to non-monetized P2P systems (e.g. Bittorrent's tit-for-tat) there is a much higher possibilty for fraud and a higher motivation and incentive to game the system (something that only pays out on bandwidth would mean that nodes only share very popular file pieces and delete them soon afterwards when nobody/few leechers want them any more).

The problem with nodes not returning actual data would be resolved using following method: Client would be responsible for monitoring his data for availability, randomly downloading his chunks. NCs would be getting statistics from a Clients and aggregating them, so each node will have %availability. When Client requests allocation, it would specify not only the price, but also the minimum availability rating. So, if you are available less that 10% time, you probably will not get much business, and if you are even able to sell ANY space, it would be for very low price. Probably, another statistical value to collect is how many time node refused a block, for which it correctly calculated hash.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on July 02, 2012, 11:19:09 AM
So you would have potentially huge overhead bandwidth (99 clients download my files just to test to /dev/null and the 100th actually really needs the data) - good for anonymity ("I didn't need the part, my client just loaded it randomly to check!") but really bad if I want to host 5 TB of data with 1 MB chunks:
1 download of 1 MB per week(!) would mean:
(5*1000*1000)/(7*24*60*60) = >8 MB per second constant upload only for making sure the files I claim to store are actually available. If more than 1 client actually wants to check 1 piece (I'd want to have estimates of at least a handful of clients how trustworthy this random storage node is in their experience) you'd even have to multiply that, so a 100MBit line isn't even enough just to make the data available.

This will definitely NOT be "dirt cheap"!

Also you (or NCs) have to trust other clients - then you can trust the storage node itself too, because for all you know the operator of a storage node could also operate lots of sock puppet client nodes that praise him for his great reliability.


Title: Re: [IDEA] Dirt cheap online storage
Post by: coga on July 02, 2012, 01:44:56 PM
So you would have potentially huge overhead bandwidth (99 clients download my files just to test to /dev/null and the 100th actually really needs the data) - good for anonymity ("I didn't need the part, my client just loaded it randomly to check!") but really bad if I want to host 5 TB of data with 1 MB chunks:
1 download of 1 MB per week(!) would mean:
(5*1000*1000)/(7*24*60*60) = >8 MB per second constant upload only for making sure the files I claim to store are actually available. If more than 1 client actually wants to check 1 piece (I'd want to have estimates of at least a handful of clients how trustworthy this random storage node is in their experience) you'd even have to multiply that, so a 100MBit line isn't even enough just to make the data available.

This will definitely NOT be "dirt cheap"!

Also you (or NCs) have to trust other clients - then you can trust the storage node itself too, because for all you know the operator of a storage node could also operate lots of sock puppet client nodes that praise him for his great reliability.

You don't check every chunk by downloading, only small portion. Most chunks would be checked using salted checksum method.

Also, this was never intended to be a 1 Client : 1 Node. What you want to do is to put your data onto as many nodes as possible. This way, if needed, you could download all of it fast, even when individual nodes have suboptimal uplink.

Regarding sock puppet nodes - you can probably do that. I imagine, the nodes that do that would run into the problem of cost associated with running that network. I would also imagine, client would be able to blacklist node by ID or IP address, so if there is a known scammer, it will soon run out of paying clients.

And would it put a strain on node's network and other resources? Hell yeah. But so does bitcoin client, skype, tor node, freenet node, bittorrent client. If it was no cost, why would anybody pay for it.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on July 02, 2012, 02:24:33 PM
You can NOT check if a client will provide a chunk by salted/random checksumming, only if a client stores a chunk. To check for actual availability, you have to transfer + measure accordingly. If I still get money/other benefits for not uploading anything, I won't (legal hassle, upload bandwidth is scarce).

1 chunk --> 5 nodes is even worse, as I'd have to constantly download and measure speed from 5 nodes just for 1 piece.

IP addresses are dynamic (with IPv4) - if I run that on Amazon EC2 (and now Google's service too), good luck with banning all my IPs! IDs are no problem, as they are being randomly generated anyways and can't even be guaranteed to be unique. Currently it would make no/little sense to steal NodeIDs in the BitTorrent DHT for example. It's easily possible however to do so.

The strain on the network should come from actual transactions, not only "show me that you still store this!" traffic imho. Also I would have inbound traffic...
Again, imagine me storing 5 TB of data in this cloud. 1 MB chunks, stored at 5 nodes at the same time to make sure they are still available all the time (in reality I'd use Solomon Reed but just for this example...). I only download a random (on average) 4 kB part of this file and then cancel the transfer (if I only always load 4 kB fixed, they migh choke anyone trying to transfer more than that, as it isn't a test).

25000000 parts * 4 kB = 100000000 kB transfer per week (100 GB) = ~165 kB/s constant download. I didn't even do the 1 kB = 1024 MB etc. thingie, including other overheads you can expect to have about 200 kB/s constant download just to check that your part files are still available once a week, which is very little (on average just ~4 tests a month).

Sorry, but monitoring the download speeds can ONLY be done by the actual downloaders themselves. Everything else is a HUGE waste of time and ressources.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on July 02, 2012, 04:35:14 PM
I still say that the client should create an index file (of sorts) with random salted checksums prior to uploading the file.  Perhaps several thousand of them.  Then each day, the client sends out a request to complete the salted checksum to every node hosting the file.  Those who successfully complete the salted checksum and match the index file get paid - those who don't, do not get paid.

Alternatively, the client could upload this salted index to the central entity, who would then send out the checks.  Clients may not be online 24/7 to verify file integrity, but it'd be a simply job for the central entity to do.

The file wouldn't take up much storage space, but would ensure 1000's of days of file integrity verification.

Also, whenever the file is downloaded (whether it be by the client or by another node wishing to host the file), the hosts are paid for that week by percentage according to how much data they provided during the download.

So node payments:
- Would be weekly.
- The node would have to meet the checksum requirement 100% every day for the week to be paid for it.
- If the file is downloaded at any point in the week, the nodes would split the payment according to how much of the file they provided.  Higher-bandwidth nodes would be paid more for this reason.
- If the file is not downloaded, the nodes would split the payment equally.


Title: Re: [IDEA] Dirt cheap online storage
Post by: nimda on July 02, 2012, 04:46:36 PM
Exactly. You don't need to check the whole file from every node. Just check random chunks of the file.


Title: Re: [IDEA] Dirt cheap online storage
Post by: nedbert9 on July 02, 2012, 04:51:34 PM



I am sooo waiting for this to happen.


Do it. 


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on July 02, 2012, 04:52:21 PM
I haven't found a programmer capable of it so far.  ;)


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on July 02, 2012, 06:54:28 PM
Also, whenever the file is downloaded (whether it be by the client or by another node wishing to host the file), the hosts are paid for that week by percentage according to how much data they provided during the download.

Who pays and how should/could this be enforced?

I'm still thinking mainly about issues with my original plan earlier on:
service.com checks hashlists provided by uploaders and pays some standard fees (quite low) for that
service.com publishes payments downloaders committed to paying to storage nodes, so storage nodes can check that they are trustworthy

If someone wants to check if storage nodes really offer the files they claim, they can anyways try to download them.

One of the few questions that remain for me is which protocol should be used (I'd lean towards WebDav), how downloaders verify themselves (so nobody else claims to be "Alice" and spends her money deposited on service.com) while still remaining anonymous (I'd lean towards PGP public/private key pairs) and some other minor things...


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on July 02, 2012, 07:58:57 PM
The client pays, and it'd be up to a programmer to figure out how to make it work.  Maybe a special torrent client has to be used for hosting the files?  I don't know.

Quote
service.com publishes payments downloaders committed to paying to storage nodes, so storage nodes can check that they are trustworthy
I don't understand what you mean by this.  Client payments shouldn't be on a per-download basis, and all the client fees are paid up front to service.com, who then distributes them to the nodes.  No trust necessary.

It sounds like our goals aren't quite aligned Sukrim.  Why not just have a regular web login system for the clients?  My goal with the project isn't to serve anonymous file storage.  That'd just allow people to upload all sorts of illegal nonsense and put service.com in a world of hurt for organizing it all.


Title: Re: [IDEA] Dirt cheap online storage
Post by: Sukrim on July 03, 2012, 10:04:57 PM
I meant client puts up money up-front to service.com (= centralized service instance that is responsible for handing out money). If a client then downloads something, they will very soon report to service.com how much they downloaded from whom. A seeder can then see if the people loading from him are really reporting their traffic or not. This way he can make sure people actually reporting correct amounts of traffic (he knows how much traffic went out per IP/user).

I don't see how a "regular web login system" should help anything against abuse or copyright claims. I personally really like the idea of at least limited anonymity (service.com would know: IP addresses, SSL IDs, Bitcoin flows + amount of traffic (maybe even per file/piece, though that's not necessary) as well as some meta data like bandwidth estimates). Service.com could even ban certain pieces, just like BitTorrent trackers can blacklist Infohashes.

There's no way though to make sure as service provider that people don't use the system for illegal stuff other than hand checking everything and requiring cleartext transfers. I'd rather go the BitTorrent way, where service.com simply has no chance whatsoever of knowing what's behind a piece hash and can only offer to blacklist these upon a court order.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on July 04, 2012, 12:25:59 AM
Ok Sukrim, everything you said makes sense.  I agree with you.  ;)

Ideally, a special torrent client would be used that automatically reports how much they downloaded from whom to service.com.  And like you said, that communication would happen nearly instantly.


Title: Re: [IDEA] Dirt cheap online storage
Post by: giszmo on August 30, 2012, 05:22:13 AM
Hmm … without reading all this thread allow me to drop three links:

http://www.forbes.com/sites/jonmatonis/2012/07/12/kim-dotcoms-pretrial-legal-funds-would-be-safe-with-bitcoin/
https://torrentfreak.com/new-megaupload-will-be-massive-global-network-to-change-the-world-120828/
https://bitcointalk.org/index.php?topic=101011.0


Title: Re: [IDEA] Dirt cheap online storage
Post by: Insu Dra on August 30, 2012, 10:39:51 PM
Hmm … without reading all this thread allow me to drop three links:

http://www.forbes.com/sites/jonmatonis/2012/07/12/kim-dotcoms-pretrial-legal-funds-would-be-safe-with-bitcoin/
https://torrentfreak.com/new-megaupload-will-be-massive-global-network-to-change-the-world-120828/
https://bitcointalk.org/index.php?topic=101011.0

Looks like we will all be adding hard drives to our btc/ltc miners to make some extra on the side. distributed file systems ftw. The "Mega Api" in combination with tor .onion, namecoin .bit, .... ugh wet dreams about the good old day's.

http://www.youtube.com/watch?feature=player_embedded&v=MokNvbiRqCM



Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on August 30, 2012, 10:43:11 PM
Someone has to program it first!   :(


Title: Re: [IDEA] Dirt cheap online storage
Post by: Insu Dra on August 30, 2012, 10:53:30 PM
Did you read the links ? I'm about 95% sure thats exactly what he has ;)

Quote from: Kim Dotcom
We are building a massive global network. All non-US hosters will be able to connect servers & bandwidth. Get ready.
Developers get ready. The Mega API will provide incredible powers. Our API and your Mega tools will change the world.
The new Mega will offer one-click-encryption of ALL your data transfers, on the fly, easy to use, free of charge, TOTAL PRIVACY!
Developers of up/download & file managers, email & fax tools, VOIP & video apps please email twitter at kim.com for early API access.


Title: Re: [IDEA] Dirt cheap online storage
Post by: SgtSpike on August 30, 2012, 11:03:31 PM
Did you read the links ? I'm about 95% sure thats exactly what he has ;)

Quote from: Kim Dotcom
We are building a massive global network. All non-US hosters will be able to connect servers & bandwidth. Get ready.
Developers get ready. The Mega API will provide incredible powers. Our API and your Mega tools will change the world.
The new Mega will offer one-click-encryption of ALL your data transfers, on the fly, easy to use, free of charge, TOTAL PRIVACY!
Developers of up/download & file managers, email & fax tools, VOIP & video apps please email twitter at kim.com for early API access.

Derp.  Not yet, lol!  Ok so, that is awesome.  :)