[IDEA] Dirt cheap online storage

[nimda]

The client (doing the paying) should keep some small amount of the data being hosted. Then, at random, request random bytes of the hosted files and compare them. If they're OK, pay the money.

[justusranvier]

A market mechanism for buying and selling hard drive space would look a lot like the commodity markets. If you express a contract in a standard form they can be traded on an exchange to achieve price discovery. The following is not a complete specification but just an example of how such a system might work.

Contract trading could be done on #bitcoin-otc or a similar platform (including the reputation function). Contracts will be priced in bitcoins. Storage is expressed in blocks of 1 MiB and the base time interval is one increment of the Bitcoin block chain.

Block servers (nodes that sell space) can list the contracts they are willing to accept in human and machine-readable form as something like this: "BLOCKSTORAGE,O,10,10000,144,0.001,0.0005,0.00025"

This means that the node is offering (O) to contract with any node with a reputation of 10 or more. Up to 10000 blocks are available for 144 time units. The total cost of the contract is 0.001 BTC per block. Of this total cost 0.0005 is due up front and 0.00025 is due at the completion of the contract. The remainder is due in periodic intervals (perhaps as often as every time increment).

When a node that is looking to sell storage can express the contract it is looking for as a bid (B). It could either work as both the block server and the client can scan the order book to look for a matching contract or the exchange can handle the matching. There would need to be a means to express what should happen to the data at the end of the contract. It can either be deleted (the client no longer needs it) or it can be transferred to another block server or it can be returned to the client.

At the end of each contract the client and the block server both leave feedback on each other which will either increase or decrease their respective reputations. Block servers with high reputations will be able to charge more up front because the clients can be more confident they will actually be able to get their data back at the end of the contract. Clients with high feedback won't need to pay as much up front because the block servers can be more confident they will be paid at the end of the contract. In general a higher reputation will lead to better prices for both a client and a block server.

The client should have a method of verifying that the block server actually retained the data instead of just tossing it into the bitbucket. One possible way to do this is to periodically transmit a block number and two offsets and ask for the hash of the data between the two offsets. The client could pre-select this verification blocks before transmitting the data and have enough hashes stored ahead of time for the number of verification it expects to need.

This is just a rough draft and probably still has holes that need to be filled in but how does it look as a basic outline?

[markm]

The biggest downside is bandwidth.  There would be no reasonable way (that I can see) for a host to cap bandwidth on any data that they hold.  So they could potentially be required to serve the data 24/7 indefinitely.  I suppose this is workable - it mainly just means that hosts will have to take into account their bandwidth limits, and only host that space accordingly.

That sounds like an argument in favour of dropping a block once it has been "verified", as then the host knows it will only be downloaded once, either by the "verifying host" or by the actual customer. That would also bring back paying more for more copies, because unless the customer specified, and paid for, two copies there would just be one copy of each block floating around (probably plus one insurance copy the market/escrow has in escrow) unless the customer paid to have it downloaded twice, in which case after it has been "verified" the host who had it could keep it, waiting for a second download.

However since your argument is basically a storage-provider-side reason why the provider might want to drop off the list of hosts the block is available from we might as well leave it to them to choose for themselves whether they want to drop the block due to having already spent quite enough bandwidth on that block.

This is starting to sound more like paid torrenting than anything.

Hmm, I am not really very familiar with that. But if it offers us a market for our service by all means lets explore that.

Usually torrenting tends to imagine the blocks are going to be sought by more people than just the original uploader, right? So hey, if there is a wider market for blocks than merely selling them back to their uploader, and the uploader is willing to provide keys for decrypting the data either to our system / the storage providers (to enable selling the data in useful form) or to other end-users (in order to make the uploader some money to cover the cost of having our system store the blocks), great. More money for the service and maybe also for the uploader(s).

-MarkM-

EDIT: Also, if providers do not know until later whether thousands of people are suddenly going to offer to buy copies of certain blocks, holding some on speculation might be encouraged, since any block might turn out upon release of decryption keys of the file it is a block of to be very popular / in demand.

EDIT2: We could go even further, constructing files whose decryption does not require all previous blocks to decrypt a block, so that someday keys can be released for any specific block, revealing that specific block to be very valuable / popular in some way.

EDIT3: For example we could habitually insert into our files a block containing a bitcoin privkey. Someday, maybe quite some time after the pool-hopper types have long forgotten that block, we could release a key allowing anyone who retained that block on spec to decrypt it and get those bitcoins...

EDIT4: We could also have lotteries, such that each time a bitcoin blockchain block reaches a depth of 120 some deterministic function computes some number of what could maybe happen to be block-hashes, or certain numbers of bytes of some block-hashes with the number of matching bytes being related to some lottery-winnings-distribution setup... If, of course, one does in fact still have a data-block that is at least 120 blockchain block-cycles old that matches...

[SgtSpike]

AbelsFire, I like the general line of thinking.  Sounds more like you're talking about backend stuff than a nice functional front-end though.  I always found bitcoin-otc to be quite cumbersome, and only natural to those who find it natural to use a command-line linux box.  Setting up a command-based architecture intended to be used by customers and sellers would only serve to limit the potential crowd of users.

But, that's not to say it wouldn't work as a fine backend for whatever front end web interface is created.

Markm, you're right that the bandwidth spent on blocks is another incentive for hosts to generally stick with a customer they are already serving.

Regarding torrenting, here is what I am picturing:
- A service that pays seeders for seeding particular files using funds sent by people who want those files to stay online.
- The person who wants the files backed up sets a daily rate that they are willing to pay in order to have the file available on demand via torrenting.  Say, $0.05 for 50 GB.
- The service takes those customer funds, and pays them out equally among the seeders.  So, if there were 10 seeders of the full file, then $0.005 to each seeder each day.

It seems like a perfect fit to me.  The customer can pay however much they like, though paying more will get more seeders, guaranteeing longevity of their file.  The seeders (or hosts) can get paid for simply using up some bandwidth and hard drive space.  And the load of any future downloads is split roughly evenly among the hosts.  The customer and/or the service can randomly check each host to ensure they have a complete set of the file.

Heck, perhaps the software could be built in such a way that the customer could just keep a few KB of the data on-hand, and have essentially infinite random checking of the files.  All that needs to be done is record a few bytes here and there of the file, then ask the seeders what is at that particular point in the file.  If the data returned from a particular host does not match the data on hand, then the customer knows there is a mismatch and can take appropriate action.  The hosts might also have incentive to periodically check other hosts too - the fewer hosts there are hosting the same data, the high the payout for each host.

So, the blocks torrented would be sought by more than just the original uploader, if for no other reason than that those seeding the blocks would receive a regular payout from the uploader.

The one thing I am unsure about, and perhaps someone with more knowledge of torrenting/programming can answer, is how the service could know who is seeding and pay them accordingly.  Perhaps it would have to be by IP address (i.e., you head to the website of the service, sign up with a particular IP address, then seed from that IP address)?  Or maybe there is another way?

Beyond that little hiccup, I think this could work perfectly.

[Gladamas]

Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.

Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.

[markm]

This is again going so far out into in the vapourware domain that now as a person actually looking to store some data offsite, but not needing yet anywhere near a terrabyte nor even, yet, 100 gigabytes, I am back to thinking Freenet is the way to go for at least one copy at the present stage in the discussion.

Right now I am not yet sure if one gigabyte would suffice, but think it quite likely that five gigabytes might, for a possibly short period of time depending on how fast my Open Transactions server aquires actual users doing real transactions.

So right now I am thinking of putting backups on whatever free five gigs for life offers I can use from Linux plus test trying to send a copy, as a modifiable file so replacing it doesn't double the storage requirements, to Freenet to see if once there it can actually be gotten back and if so for how long.

Since freenet has friend-to-friend modes available, possibly I could find some other people looking to start with one to five gigabytes or so, set up friend to friend links with them, and see how that works out for a while as a mutual swap, since my problem is not lack of onsite space but desire to swap onsite space for offsite space.

I also have GNUnet and Tahoe-LAFS installed. (Also Tor and i2p, but they are transport more than storage.)

So, are there any among you who are looking to swap onsite for offsite space?

(We can worry about selling space later, once having mutually assured ourselves there is actually space and that it is actually useable and reliable over some reasonably long term.)

-MarkM-

EDIT: Friend to friend might not even be ideal, if it opens the possibility of being someone's only link to the freenet / GNUnet / Tahoe-LAFS grid / whatever; it might suffice to simply request each other's files while both being on such a net.?.?

EDIT2: Technically, I actually also have RetroShare, I just happen not to bother running it lately due to lack of connections.

[SgtSpike]

Rather than keeping data on hand to check the files, a checksum or hash of random parts of the files could be checked, similar to the current checksum of the received data performed by BitTorrent.

Regarding paying people accordingly, a Bitcoin address could be tied to each seeder, and a public-private keypair could be generated verifying that the host machine for the files matches the Bitcoin address.

Wouldn't you have to keep some data on hand to make sure the checksum or hash matched?  In other words, what would be the difference between checking small bits of data vs checking a checksum of small bits of data?

Sounds like it is possible to tie a bitcoin address to a particular seeder then!

[markm]

I am curious where you people put your offsite backups, and how.

It would be interesting to know not only for ideas I might apply to my own offsite backup strategies and/or tactics but also simply to assure me you are serious about data integrity / safety in the first place.

Meanwhile with this torrent type stuff continuing to come up I am thinking again of actual-block-of-data markets.

If satoshis are small enough units of value then maybe the markets can use satoshis as units of value to use to bid on such markets, if not then some lower value per coin flavour of coin might be better, or some other kind of microsatoshi or millisatoshi scale unit of value.

You seem to be more interested in bitcoins or satoshis than in whatever you might be able to provide in return, so lets drive this from the bitcoin or satoshi or other coin side.

I offer to buy a block of data whose hash is $H, and solicit bids as to how many microsatoshis that specific block is currently available for.

If anyone does have that block of data, great, I'll either buy it or await a better price.

For my personal current application-of-interest, I actually already have that block myself anyway, so I can also offer it myself, possibly at a lower price than the offers I am seeing from others. (Because, it is a block of a backup of data of mine, and I already made and retained a copy of that backup myself.)

Someone else might be interested in buying the block whose hash is $D, and bid higher for that block than I typically bid for blocks of my backups. (Possibly because block $D contains part of a particularly appealing frame of a particularly popular scene of "Dolly Does Dallas XII", not yet out in theatres and DVD stores but reported to have been leaked to the internetz.)

Lots of people might be bidding to buy block $D simply on speculation, figuring that once they have collected enough blocks of that frame of that movie they will be able to resell entire frames or maybe entire clips on other markets.

Torrent end-users might not even be up to date enough on current internetz affairs to even know of this market in blocks of data, they might be eagerly waiting for their favourite torrent-provider to offer a full copy of the entire movie, having no idea of this back room market in blocks between torrent-providers and others whereby such people manage to lay their hands, block by block, on such hot digital assets...

-MarkM-

[SgtSpike]

That's quite a large tangent from the idea of backing up personal files, markm.  Perhaps a good discussion for another thread though. 

Right now, I do not have a good method of offsite backups.

[markm]

It could either work as both the block server and the client can scan the order book to look for a matching contract or the exchange can handle the matching.

My game-players have always wanted to be able to choose which offer to accept, but it does open new "game the system" opportunities. Specifically, volume can be created at high prices between sock-puppets (or, in general, colluders) without any non-sockpuppet bids/offers being accepted.

The ostensible reasons and/or excuses for wanting such a structure tend to be not wanting to buy stuff from enemies, wanting to offer allies better prices than offered to enemies and neutrals, and preferring reliable suppliers or suppliers favoured for the quality of the actual "commodity" provided. (Which goes against the idea that to be a "commodity" a product should be fungible.)

There would need to be a means to express what should happen to the data at the end of the contract. It can either be deleted (the client no longer needs it) or it can be transferred to another block server or it can be returned to the client.

Transferring it to another block server is getting into more complex transactions, but whether the client is paying for delivery to client plus deletion from the provider or just delivery to the client seems a useful distinction. Though it does bring up the corner case or imaginary case of how about the client only wants it deleted from the provider (a takedown request) without the bandwidth overhead of taking delivery of a copy.

At the end of each contract the client and the block server both leave feedback on each other which will either increase or decrease their respective reputations. Block servers with high reputations will be able to charge more up front because the clients can be more confident they will actually be able to get their data back at the end of the contract. Clients with high feedback won't need to pay as much up front because the block servers can be more confident they will be paid at the end of the contract. In general a higher reputation will lead to better prices for both a client and a block server.

This is another form or arising of the "is this really a commodity, since commodities should be fungible" problem.

-MarkM-

[justusranvier]

My game-players have always wanted to be able to choose which offer to accept, but it does open new "game the system" opportunities. Specifically, volume can be created at high prices between sock-puppets (or, in general, colluders) without any non-sockpuppet bids/offers being accepted.

The ostensible reasons and/or excuses for wanting such a structure tend to be not wanting to buy stuff from enemies, wanting to offer allies better prices than offered to enemies and neutrals, and preferring reliable suppliers or suppliers favoured for the quality of the actual "commodity" provided. (Which goes against the idea that to be a "commodity" a product should be fungible.)

I can't help but notice that we manage to use exchanges to buy and sell wheat, electricity, currency and other commodities. The empirical evidence suggests that it is possible to do it, although some exchanges are better than others. In that context I don't understand the purpose of your reply. If you've identified a problem are do you have a proposed solution? Are you claiming that no solution exists?

Transferring it to another block server is getting into more complex transactions, but whether the client is paying for delivery to client plus deletion from the provider or just delivery to the client seems a useful distinction. Though it does bring up the corner case or imaginary case of how about the client only wants it deleted from the provider (a takedown request) without the bandwidth overhead of taking delivery of a copy.

Why would a client ever pay for a server to delete data? Any sane implementation of a client would encrypt all data first so that it's useless to anyone else. Why would a block server keep around useless data if no one is paying to store it?

This is another form or arising of the "is this really a commodity, since commodities should be fungible" problem.

In what way is this a problem? What does "commodities should be fungible" mean in practical terms? Will the fact that some block servers can provide a higher quality service than other block servers break the price discovery mechanism? Your objections are a little too vague for me to understand their practical implications.

[markm]

There are bitcoin-exchanges in which users pick and choose which offer(s) to accept instead of being matched by the market-operator; off-hand I have the impression they tend to also require the users to then conduct the trade out of band between themselves, thus losing the assurance of delivery by both parties that automatic-match exchanges such as MtGox provide.

You brought up the idea of there being these two different approaches available, I merely tried to respond, mostly by elaborating the implications.

It seems to me that it would be useful to take it that one choice opens up questions as to whether the product is a commodity at all, due to seeming to at least potentially bring the product's fungibility into question.

One might for example avoid picking offers that involve coins trackable back on their blockchain to some theft break-in heist scam etc. I have read various threads in which various people assert that such distinctions challenge the very fungibility of blockchain-based coins.

Maybe both types of "market" are useful and we should have both.

Certainly so far I am in picking and choosing mode, still seeking specific individuals with whom I might consider trading; albeit I am also considering trading offsite for onsite initially too, keeping it all about the product - the possibly-a-commodity item - without having to worry yet about (possibly non-fungible?) money/currency trade.

Maybe part of the picking and choosing would  be which providers of the "commodity" actually seem over time to be providing an actual "commodity" rather than a product not quite exactly like (co-fungible with?) the supposed "commodity" we could end up going on to create a "commodity market" for?

-MarkM-

[justusranvier]

Maybe both types of "market" are useful and we should have both.

I agree. If there is a standard API for contracts then it would be possible for different types of exchanges to form, each with their own rules.

[markm]

That's quite a large tangent from the idea of backing up personal files, markm.  Perhaps a good discussion for another thread though.  

I think it started from an ambiguity about whether we are talking about back-up storage or primary storage.

By that I mean there is a lot of difference between storing a just-in-case copy of your infinitely valuable irreplaceable data and storing your actual infinitely valuable irreplaceable data.

The case of a sister running out of space for photographs for example sounded as if the plan would be that she "move" photographs to offsite storage rather than that she "copy" them (copy implying she of course still has it herself, she does not delete from her system when she copies to someone else's) to offsite storage.

These are very very very very - it is hard to understate how very - different cases.

One must always act as if each and every piece of media on which your data resides is a diabolically patient, fiendish enemy, always hoping to find itself in the position of being the only surviving copy of the data so that it can strike at you as no other copy up until that moment could do, by maliciously destroying the only copy of your data.

This applies to backups more even than to the files backed up, because if you lose the files you have backups of, you can simply restore from backups. But if the backups are lost or corrupt, that is when you really need backups.

So always back up your backups, as if they are hoping and hoping and hoping you will need them so they can nyah nyah nyah you by failing you.

The idea that someone's sister might give me her precious files without keeping a copy herself made me very very worried about how sure I could be that I would still have them when she actually needs them.

Right now, I do not have a good method of offsite backups.

Maybe we can do some trials/tests between us then, trading onsite to offsite storage, and see how it goes?

-MarkM-

[SgtSpike]

I agree there is a difference between a just-in-case copy of data vs an only copy of data.  But how does that have anything to do with your previous post about selling parts of movies via torrent?

The data in my sister's case is raw files from older wedding photoshoots and such - files that she will likely never need to touch again.  But, better than deleting them entirely would be finding some method of cheap online storage until she can buy some extra storage space.  So, it wouldn't be the end of the world if they were somehow lost forever (since the only other option is to simply delete them at this point anyway to make room for newer photos), but she'd still like to keep them, if possible.

Certainly, though, I believe a service like this would be used more for "just in case" copies of data than anything else.

I'd be fine with doing some testing/trials.  I am not a regular torrent user, so I really know only minimal amounts about it, but I do know enough to click on some menus and create/seed a torrent.

[justusranvier]

The idea that someone's sister might give me her precious files without keeping a copy herself made me very very worried about how sure I could be that I would still have them when she actually needs them.

It would be very foolish for her to give her only copy to one person, unless the data was of very low value or easily replaced. Using a RAID (or similar) algorithm to distribute the data to three people such that she could recover it from any two providers would be safer. Depending on how valuable and irreplaceable the data was she might want even more redundancy.

[markm]

I agree there is a difference between a just-in-case copy of data vs an only copy of data.  But how does that have anything to do with your previous post about selling parts of movies via torrent?

I wasn't the one who brought torrent into this. The thread seemed to diverge between offsite copies of data, actually moving data offsite no longer having it onsite, and whether anyone other than its "seeder" / "publisher" / "uploader" ... (owner, even?) ... might want a copy too, which seems to be the usual use-case for Torrent. (Because Torrent is designed for getting the same data to many many people aka to optimise having many many people all wanting the same data and not having a copy of it themselves already.)

The data in my sister's case is raw files from older wedding photoshoots and such - files that she will likely never need to touch again.  But, better than deleting them entirely would be finding some method of cheap online storage until she can buy some extra storage space.  So, it wouldn't be the end of the world if they were somehow lost forever (since the only other option is to simply delete them at this point anyway to make room for newer photos), but she'd still like to keep them, if possible.

Certainly, though, I believe a service like this would be used more for "just in case" copies of data than anything else.

As you also mentioned earlier having an entire drive free, maybe just sending her that drive to install in her machine would be a good way to tide her over her diskspace-crunch? Or, have her upload her photos to you? Or, download them from her?

I'd be fine with doing some testing/trials.  I am not a regular torrent user, so I really know only minimal amounts about it, but I do know enough to click on some menus and create/seed a torrent.

See, Torrent comes up again. I do not yet see why Torrent. Maybe though it is because of the front end GUI, not what the actual protocol it uses it optimised for?

Can you have your sister install torrent and seed a directory so you can download that directory from her any time you wish, then cryptozip or whatever the photos at risk and put them into that directory for you to grab?

Or, how about RetroShare? About the only thing going for RetroShare seems to be eye-candy GUI, presumably it should be just as GUI-user friendly as Torrent?

I have never used Torrent though doubtless can find code for it that will work on Linux. In fact I have on my GUI menus something called Transmission BitTorrent Client. If I used that to grab files from you, would you be able to grab them back from me? Or would I need a server rather than a client for you to be able to grab from me? I really know very little about Torrent.

-MarkM-

[SgtSpike]

You have to step away from what torrent is used for right now, and look at how it can be used.  It can be used to easily give multiple hosts copies of the same data, which is exactly what we want when looking to make backups of said data.  There will still be demand for these backup files to be downloaded via torrent, but the demand will only exist among the original uploader (potentially), and those who wish to seed it to gain a portion of the daily fee paid by the uploader.

[markm]

Hmm, I get the impression it is more like multiple hosts take the data than you give the data to multiple hosts, in the sense that it is pull not push. You can't really just give them the data, you have to somehow cause them to ask you for it?

A push could be useful for cases like you go away for the weekend but your sister still has photos she wants to move or copy to you.

If you can set up an automated pull though that will keep asking her for anything new, that would be fine.

Or maybe you could search for specific filenames, leaving the search constantly running, so any time fridays.batch.zip becomes available your client will grab it, as soon as saturdays.batch.zip becomes available it will grab that too and so on, so she has list of names she can zip a batch as to have your machine grab them even while you are away from keyboard.

-MarkM-

[SgtSpike]

Hmm, I get the impression it is more like multiple hosts take the data than you give the data to multiple hosts, in the sense that it is pull not push. You can't really just give them the data, you have to somehow cause them to ask you for it?

A push could be useful for cases like you go away for the weekend but your sister still has photos she wants to move or copy to you.

If you can set up an automated pull though that will keep asking her for anything new, that would be fine.

Or maybe you could search for specific filenames, leaving the search constantly running, so any time fridays.batch.zip becomes available your client will grab it, as soon as saturdays.batch.zip becomes available it will grab that too and so on, so she has list of names she can zip a batch as to have your machine grab them even while you are away from keyboard.

-MarkM-

Yes, you are exactly right - pull instead of push.  The uploader would volunteer the data to be copied as a torrent, and the hosts could download said data at will, as long as it is up.  Many of the more hardcore hosts would likely find a way to automatically download new data that meets a certain criteria (i.e., current payout above $0.0X/day).  An alternative might be automatic download of all new torrents by the Service, until another host picks it up, allowing the uploader to effectively "push" out the new data whenever they like.

The nice thing about using a torrent protocol for the base of a project like this is that torrents are already very widely used and understood by many of the people.  Perhaps even more importantly is that the group that understands torrenting the most is the group that is most likely to have lots of hard drive space, and (potentially) lots of extra hard drive space.

I suppose the one thing I am most interested in at this point is more detail on how Bitcoin addresses can be tied to hosts.  Gladamas touched on it, but I am not enough of a programmer to understand how it might work.