Time for an anonymous BTC based storage network?

[iamsomeoneelse]

If Mega won't take on Bitcoin maybe Bitcoin should take (on) Mega! 

I think that this has been briefly discussed previously but its time to consider it seriously.  It seems to me like a distributed p2p file storage network that uses BTC as the other side of the exchange could be very valuable.  Even if the majority of the storage/bandwidth providers end up being dedicated hosting companies/machines, the existence of a liquid, frictionless, fine-grained market for these services is very valuable.  Just look at ebay... at this point there are very few actual individuals selling as opposed to small companies.  But just like Ebay at first there would be a lot of individuals offering storage on their own PC.  And this could be an easy way to earn (and spend) BTC, as opposed to mining or dealing with an exchange.

Key markets:
  Backup
  Sharing
  Content delivery
  Streaming
 
Each market has interesting issues to solve.  For example, a backup is rarely accessed and so needs some kind of periodic verification to ensure that the provider has not deleted the data but is still taking the BTC.

Of course, a strong focus on privacy and encryption will ensure that untrusted hosting sites cannot access content.  This simultaneously protects these sites from hosting objectionable content (Dotcom's argument).  Is it necessary (possible) to obfuscate the source of the data in a manner similar to TOR?  Or the storage network recommend TOR be used for those who care?

Here's a use case example that is only possible because of the low friction:

To host files for Mega you've got to call them and do a deal which will result in you dedicating a certain # of machines, bandwidth and disk to Mega.  So there a lot of lawyers involved in hammering this out, etc.

Imagine instead that a provider of hosted services (VMs) also runs the BTC storage network nodes on its machines.  Whatever portion of the machine is not providing hosted services (I'm assuming that that would be higher margin) is allocated to the storage network and receiving BTC for services provided.  So the machine is 100% utilized all the time, resulting in a quicker ROI on the hardware.


We need to store the blockchain somewhere! 

[1713946068]

1713946068

[1713946068]

1713946068

[1713946068]

1713946068

[BlackLilac Jordan]

Go for it.   

[bonker]

Go for it.   

+1 it's a great idea

[kangasbros]

This idea has revolved in my idea for a long time. Just that for it it to succeed, I would recommend starting with something really simple, and building from there onwards.

[2weiX]

awesome idea.

could also implement that, for example, streaming is more valuable than just backupping, for instance.
so if you as the service provider have sufficient bandwidth, you can adjust to hosting streams over backups.

the system will self-regulate in sich a way that streaming-hosters that prove to consistently deliver get chosen over those who don't.

incentives, baby!

[Sukrim]

Well, you could check out the code on Tahoe-LAFS and help them implement the accounting backend (as far as I know they plan on integrating BTC in there as well)...

They are doing encrypted secure file storage with an Open Source solution already for years.

[meowmeowbrowncow]

Yep, this is gooooood.


There have been very interesting threads involving p2p storage and BTC.  I'll try to find them.


edit:

Building a fully functional CDN, including robust load balancing, leveraging p2p storage is tantalizing.


Wonders if p2p CDN could be the enabler for DDOS resistance.  hmmmm.....

[iamsomeoneelse]

Let's start at the beginning, but please forgive me if this posting is not well organised.  Right now it is just a bunch of ideas on implementation issues.

Clearly we need a basic protocol for P2P discovery. 

This protocol can use public/private address grabbed directly from Bitcoin source code to uniquely identify entities.  Let's use the terminology "provider" and "subscriber" to distinguish who is offering and who is buying the storage, although the same entity may do both.

It is not (necessarily) useful to force files into standard block sizes because this system is highly abstracted from the underlying block device.  So let's define a block as an arbitrary-sized encrypted chunk of bytes identified by the SHA-256 hash, after encryption.  So an entire file can be stored in a block.  Or a file could comprise of a list of blocks (this may help access times).  Is my assumption that SHA-256 collisions would be essentially impossible acceptable?  So a subscriber does not modify a block; it simply uploads a new block.  For efficiency we can have a protocol message to delete old blocks, but ultimately the blocks are cleaned because the subscriber stops paying maintenance for them.

Of course, the subscriber encrypts/decrypts the blocks.  The provider cannot access the cleartext data.  But just like a bitcoin wallet, any subscriber with the private key can decrypt blocks so files can be shared simply by creating a new private key and giving it to your other devices and/or friends.  This becomes a "group share" key.  The subscriber is welcome to send the blocks to multiple providers to implement data redundancy, or content delivery.  And providers could act as a subscriber, request popular blocks from other providers and then compete on delivery price.


Layered on that, let's add a protocol for service requests and advertisements.

Very simply, this protocol allows a provider to advertise (and a client to request) capacity, bandwidth, access protocols (whether it willing to act as a gateway for protocols like http & streaming, or whether is will only pass encrypted blocks within the native storage protocol), privacy (will it divulge IP addr/location for CDN calculations), and the cost of the services.  Service cost splits out storage from access bandwidth so subscribers can choose the optimum provider for their needs. 

So the first issue is verification that the provider meets its advertised services.  Obviously, it would be unfortunate for a provider to just turn off his computer permanently.
A distributed keepalive mechanism could calculate every provider's uptime.
Subscribers can issue challenges of the form "calculate the SHA-256 of this portion of a block" to a provider to ensure that the provider is retaining the data.
How about a trust network?

This meta-data should be universally accessible and associated with the provider and subscriber's public key.

How shall all the trust attacks be defeated?  A POW blockchain is the obvious answer but that requires CPU.  I am wondering if we cannot rely directly on the Bitcoin blockchain either by requiring transaction fees to modify this meta-data or by determining some level of trust by an analysis of the revenue received by the provider's address.  To defeat fake transactions, this analysis would have to examine the sending addresses and determine that they are generally involved in other providers as well.  I'd love some of the devs opinions on this question!!!

But I do not see this as an app-killer even if unresolved.  We will also allow providers to connect their identity to a real-life identity by piggy-backing on the efforts to do the same with Bitcoin (ultimately by having the public key squirreled away in a https request that is verified by a cert authority, I'd imagine).  So in the worst case subscribers with strong data preservation requirements only deal with identified provider companies that are committed to availability.

How can payments be made?  Perhaps via an automatic 2-of-3 escrow.  I wonder if a proof-of-work network could act as the escrow arbiter.  Or, since we are talking about small transactions here, can failed escrow bitcoin somehow count against both parties and then be raked into transaction fee style fund?  It is important that some mechanism exist to trickle-pay for the maintenance of blocks even if the subscriber is offline.  The best option would be a POW network as a whole acts as an escrow agent, but in the worst case redundant 2-of-3 escrow agents could be picked randomly out of entities that are advertising this service.

Anyway, these are some ideas... I'm interested in your thoughts.

[jl2012]

Just a rough idea

It is a P2P network. A node (A) wants to offer storage space will broadcast messages like "I have 2TB of free space. I will host your files at 0.001BTC/MB/month". B reads the message and sends 10MB of data to A, and sends 0.01BTC to escrow. In the following month, B will send some random request to A, asking for some random bytes of the data (This is to make sure A is really holding all data). B will release the escrow if and only if A gives correct answers. On the other hand, if A does not receive payment after giving correct answers, it will drop B's data.

[iamsomeoneelse]

Just a rough idea

It is a P2P network. A node (A) wants to offer storage space will broadcast messages like "I have 2TB of free space. I will host your files at 0.001BTC/MB/month". B reads the message and sends 10MB of data to A, and sends 0.01BTC to escrow. In the following month, B will send some random request to A, asking for some random bytes of the data (This is to make sure A is really holding all data). B will release the escrow if and only if A gives correct answers. On the other hand, if A does not receive payment after giving correct answers, it will drop B's data.

yes!  that's exactly what I'm talking about in the above message but I was suggesting a SHA-256 checksum over a broader swatch of data.  But in retrospect I can't think of why this would be more effective then just asking for random bytes as you propose.  Also, I'm thinking that it might be desireable to delegate this task (periodically dropping coins in the machine) to some 3rd party "agent" that has greater uptime then the subscriber.

A quick look at Tahoe-LAFS is making me think that it over-structures the data (it pulls issues like redundancy into the cluster, whereas I do not see why those issues cannot be handled at the client) compared to what I want.  Too much structure is going to limit the applicability.  And it seems to have no market, whereas in this proposal the market is just as important as the storage.

[Fiyasko]

I was about to walk in screaming "HOW STUPID ARE YOU!?!!?" And then i started to read, Saving myself from being an idiot.
a P2P bitcoin Trading Market is what we need, None of this MT.Gox and stuff. They can be shutdown!

Its not a P2PStorage network we need, Its a P2P Buy/Sell market that we need

[TraderTimm]

This is a very interesting idea - how about combine aspects of both? (Storage and peer-to-peer decentralized trading)

One thing that came to mind is this for the front-end:

https://en.wikipedia.org/wiki/Osiris_%28Serverless_Portal_System%29

And combining aspects of this for filesharing via the web portal above:

https://en.wikipedia.org/wiki/OneSwarm

Got me thinking here... I really like the idea of offering up bandwidth/storage for BTC, assuming of course inbound/outbound is encrypted and I have no way to decrypt the files. (Common carrier defense.)

As for accepting files for bitcoins - I think you could use a system where waiting for at least three confirmations to start file transfer, checking for the final confirm to retain the file in storage.

[jl2012]

Just a rough idea

It is a P2P network. A node (A) wants to offer storage space will broadcast messages like "I have 2TB of free space. I will host your files at 0.001BTC/MB/month". B reads the message and sends 10MB of data to A, and sends 0.01BTC to escrow. In the following month, B will send some random request to A, asking for some random bytes of the data (This is to make sure A is really holding all data). B will release the escrow if and only if A gives correct answers. On the other hand, if A does not receive payment after giving correct answers, it will drop B's data.

yes!  that's exactly what I'm talking about in the above message but I was suggesting a SHA-256 checksum over a broader swatch of data.  But in retrospect I can't think of why this would be more effective then just asking for random bytes as you propose.  Also, I'm thinking that it might be desireable to delegate this task (periodically dropping coins in the machine) to some 3rd party "agent" that has greater uptime then the subscriber.

A quick look at Tahoe-LAFS is making me think that it over-structures the data (it pulls issues like redundancy into the cluster, whereas I do not see why those issues cannot be handled at the client) compared to what I want.  Too much structure is going to limit the applicability.  And it seems to have no market, whereas in this proposal the market is just as important as the storage.

You may ask for SHA-256 of random fractions of the data, just to prevent the storage nodes from cheating. That could be more efficient because no matter how big the fractions are, the checksum is always 256bits.

You may use something like the so called RAID-96 system: http://www.symform.com/blog/tag/raid-96/

People may also exchange storage space in 1:1 ratio, so they don't need to pay each other

[ahdinosaur]

+1 for purchasing tahoe-lafs storage with bitcoin. https://tahoe-lafs.org/trac/tahoe-lafs/wiki/Ostrom
is this bounty still active? https://bitcointalk.org/index.php?topic=2236.20