[IDEA] Dirt cheap online storage

[justusranvier]

It seems overly complicated and off-purpose to go looking to sell people storage space on my site to raise funds with which I can then go looking for other sellers to buy storage from on their site.

There's absolutely no reason at all that the user would ever be exposed to those details. Automated trading algorithms are fairly common so there's no reason for the user to need to configure any of it himself.

Someone who is running a block server wants to collect a small income stream from his excess hard drive space. Due to competition it's not going to be a very large income stream but lots of people have excess storage and would enjoy being able to collect a few bitcents every day for it (whatever the market price happens to be). This user shouldn't need to do anything other than install the software and allocate drive space. The program should handle the details of getting the best possible price for his storage space without further intervention.

Likewise the person storing data wants to pay as little as possible for a given amount of data, given amount of reliability and a given amount of bandwidth. His software should also take care of the details entirely behind the scenes. He should be able to upload data, choose a reliability and bandwidth targets and forget about it other than periodic reminders to add more Bitcoins to his node's wallet.

There can a plethora of automated trading action going in behind the scenes, with contracts, escrow and reputation systems but that's all complexity that the user never needs to see or worry about.

[markm]

Sure, but chances seem good to me that the best price will turn out to be available to me from all the various nodes for whom I have been faithfully storing data and returning it intact upon demand, and similarly for my peers, so that the only "customers" forced to deal with the people who want some trickle of pennies when they fire up their laptop for a few minutes or hours will be other fly by night laptop-storage-by-the-second-minute-or-hour types like themselves.

In short, why bother to pay even pennies to people who aren't even responsible enough themselves to want off-site backups yet nonetheless want to convince you to pay them to store your backups for you, if you can instead store other people's backups for them in return for them storing yours?

Two different goals though... It sounds like you want to build a mom and pop iTunes or Pirate Bay or Malware Incorporated or Cheap Pr0n Emporium.

It might be better to base your pricing/accounting on the actual contents of the files instead of caring at all about how big or small they are: consider how valuable they are even if small if they are "better quality" in some way.

Everyone could advertise whose personal private data they have copies of, how encrypted it seems to be according to their cracking attempts so far, there could be betting markets on whether there will turn out to be another layer of encryption beneach the outwardly visible layer, and the people who actually gave these files away freely could choose whether to buy a download of a copy of such and such a day's crypted backup from this distributed market or not on any given day, maybe taking into account various threats by various sites to take it off the market if no one buys a copy within the next such and such span of time. Name your backups provocatively so more people will share the file thinking it of particular interest or value. Etc.

Is this to be about the data stored, or the volume of storage, or the bandwidth? What is the obstacle, if any, to running something like Tahoe-LAFS or GNUnet etc yourself to build up mutual reputation with other nodes, why do you need some trickle of pennies for doing what normal participants simply "just do"? Why should I even consider having to pay you to store data for me instead of doing mutual data distribution with others who have storage and a desire to move some of the data stored on it offsite?

-MarkM-

EDIT I guess what I am getting at here is that reliability and volume of storage and bandwidth usage are plenty of currencies already, in fact each could be adjusted separately so that for example if I provide vast amounts of storage for huge lengths of time but at low bandwidth, what I will tend to get in return is lots of space for plenty of time but at low bandwidth. If I want to spend general purpose currency of some kind to increase the speed, I can upgrade my own speed with that money, thereby resulting through reciprocity with my peers in better bandwidth for my own usage. I can buy more storage or more speed or more duration simply by buying it for/from my own site, no need to buy it from others' sites. Simple reciprocity will cause what I get from others to adjust in accordance with what I provision locally aka what I pay to/for my own site.

EDIT2: Even considering it as a prospective seller looking to make a trickle of cents, why would I want to pay you more cents to store data than I am being paid for storing data, whether I am having you store data others pay me thinking I am storing it or having you store data I want stored? If it is all about wanting a trickle of pennies the tragedy seems to arise that until some of us actually obtain some pennies there aren't going to be any to be trickled. I can't give you pennies I do not have, and if I give you more than I earn I will go broke...

(I can however give you blocks of data you asked me to store for you, if will take those instead of pennies maybe we can do business?)

[justusranvier]

I guess what I am getting at here is that reliability and volume of storage and bandwidth usage are plenty of currencies already, in fact each could be adjusted separately so that for example if I provide vast amounts of storage for huge lengths of time but at low bandwidth, what I will tend to get in return is lots of space for plenty of time but at low bandwidth. If I want to spend general purpose currency of some kind to increase the speed, I can upgrade my own speed with that money, thereby resulting through reciprocity with my peers in better bandwidth for my own usage. I can buy more storage or more speed or more duration simply by buying it for/from my own site, no need to buy it from others' sites. Simple reciprocity will cause what I get from others to adjust in accordance with what I provision locally aka what I pay to/for my own site.

I think the problem you're not seeing is called the double coincidence of wants. Not all users who need large amounts of storage have large amounts to share. A market mechanism is an efficient way to bring those with excess storage/bandwidth/uptime/CPU power together with those who need it.

Why would you think that having different currencies for each scarce quantity is better than using a single currency? Would you want to use a different currencies to buy food with, pay rent, and procure transportation?

[Gladamas]

Sub, this is an interesting idea. Could we incorporate past file versions, and pay users based on their upload speed as well as HDD space?

[markm]

Actually I think what I am not seeing is what the barriers are to mass storage services such as Amazon S3 getting so cheap from so many suppliers that the original poster's complaint that offsite storage is too expensive ceases to be a real driver for this kind of project.

The original idea at top of thread was basically gosh surely we could provide the same product cheaper just with ad hoc networks of random fly by night laptops.

Now the idea seems rather to be gosh surely there are trickles of pennies to be made by doing that.

A friend told me a few years back that big stores like Wal-Mart can actually be amazingly efficient at providing low prices.

I believe that there are more and more free open source "S3-compatible server" packages out there, would something like that be more suitable for your purpose, allowing people to directly compete with Amazon from home and thus maybe by competition drive down Amazon S3 service prices?

-MarkM-

EDIT: Earlier in speculating how much having to handhold ignorant people who consider it their right to remain ignorant might drive up costs, I forgot to also speculate that marketing might also drive up costs. Paying for enough advertising to obtain enough customers to maintain a trickle of pennies might end up costing more than the number of pennies it brings in, until marketing capital such as subscribed mailing-list members and so on help shift you toward the cost of retaining or up-selling existing customers more than the cost of acquiring new customers.  A huge marketing budget might well be able to make cheaper services seem less attractive than services so ridiculously expensive that they can afford a huge marketing budget...

[justusranvier]

Actually I think what I am not seeing is what the barriers are to mass storage services such as Amazon S3 getting so cheap from so many suppliers that the original poster's complaint that offsite storage is too expensive ceases to be a real driver for this kind of project.

The reason someone would choose a decentralized solution for online storage instead of a centralized solution is the same reason they would be interested in Bitcoin in the first place.

[SgtSpike]

Markm, you're COMPLETELY missing the point.

If I wanted to swap space with other users, I'd look at using freenode, or something else.

What this idea aims to accomplish, is to match up people who aren't using all of their storage, with people who are using all of their storage and still need more.  I have a couple TB's of extra HDD space, and wouldn't mind renting it out to other people to use.  I don't have any files that I want to store on their systems.  So, in my case, any kind of a "swap" system is fail.  I don't want it.

Similarly, my sister is into photography, and stores the .RAW files of all pictures taken, which obviously eats up hard drive space fairly quickly.  She only has 750 GB total.  She needs more space.  Rather than deleting the old files permanently, she asked how much online storage would cost.  It was outrageous - to the point where buying a new 1TB drive would cost less than renting 1TB of online storage for 2 months.  So, she needs more storage, and doesn't have any extra storage space to store other people's files as well.  Again, in her case, your "swap" system is fail.

I (and my sisters) don't have any criminal wishes, and I don't know why you want to imply that anyone who wants to make money hosting files is attempting to do so with criminal intent.  That is not at all the purpose here.  This is more about allowing people with lots of extra hard drive space a way to make money with that space, and giving people who need more space an opportunity to purchase it at a price that isn't insane.

Also, S3 is still high priced.  $125/month for 1TB of storage.  I'd be willing to provide 1TB of storage at a far lower price than that.

[Gladamas]

Actually I think what I am not seeing is what the barriers are to mass storage services such as Amazon S3 getting so cheap from so many suppliers that the original poster's complaint that offsite storage is too expensive ceases to be a real driver for this kind of project.

The original idea at top of thread was basically gosh surely we could provide the same product cheaper just with ad hoc networks of random fly by night laptops.

Now the idea seems rather to be gosh surely there are trickles of pennies to be made by doing that.

A friend told me a few years back that big stores like Wal-Mart can actually be amazingly efficient at providing low prices.

I believe that there are more and more free open source "S3-compatible server" packages out there, would something like that be more suitable for your purpose, allowing people to directly compete with Amazon from home and thus maybe by competition drive down Amazon S3 service prices?

-MarkM-

A decentralized service such as this one would be better for the community IMO, and would be immune to raids, DOS attacks, etc. Regardless of the cost of this compared to centralized services' prices, the community can come up with a better protocol and product than a centralized service can. Just look at Bitcoin, BitTorrent, etc...

[markm]

The reason someone would choose a decentralized solution for online storage instead of a centralized solution is the same reason they would be interested in Bitcoin in the first place.

Yes! Thank you.

But just because I am interested in bitcoin and indeed even devcoin and litecoin and on and on and on does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies. How about some of the people who want to PROVIDE THE PENNIES make some concrete offers of how many pennies per what span of time they are willing to pay to have how many copies of how many bytes distributedly stored?

-MarkM-

[justusranvier]

But just because I am interested in bitcoin and indeed even devcoin and litecoin and on and on and on does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies. How about some of the people who want to PROVIDE THE PENNIES make some concrete offers of how many pennies per what span of time they are willing to pay to have how many copies of how many bytes distributedly stored?

Ok, you don't want to spend any of your precious bitcoins to rent storage space from other users. So what are you objecting to exactly? If you don't want to use this kind of system than all you have to do is not use it. Are you upset that other people might be willing to spend their bitcoins to buy storage? that other people would be willing to rent out their excess storage for bitcoins? Do you not want bitcoins to be used as a real currency that people regularly spend?

[Gladamas]

Just because I am interested in bitcoin ... does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies.

Um, isn't the whole point of Bitcoin to replace fiat money with a decentralized commodity?

[justusranvier]

https://tahoe-lafs.org/trac/tahoe-lafs

One of the developers (Zooko) is into bitcoin.  Perhaps starting a discussion about this on the tahoe-dev list would be well received.  There may be a way to easily integrate servers advertising a bitcoin address to encourage them to provide more space.

He might not be able to participate, although this message from 2005 may be outdated.

Dear mnet-devel and p2p-hackers:

I am about to accept an exciting job that will preclude me from
contributing to open source projects in the distributed file-system
space.

I will miss the Mnet project!  Good luck without me!

There are some interesting thoughts at the bottom of the message regarding digital cash and how it worked in the Mojo Nation project.

V.  Persistent Tit-for-Tat == Bilateral Accounting.  Mnet v0.7+ lacks
something -- an incentive mechanism to limit excessive costs imposed on
the network and simultaneously to motivate users to contribute
resources to the network.  I was always deeply impressed with the
simplicity and robustness of Bram Cohen's "Tit-for-tat" incentive
mechanism for BitTorrent.  Suppose we wanted a similar "tit-for-tat"
mechanism for Mnet v0.7++, but we wanted the peer relationships to
extend through time and across multiple files and multiple user
operations.  Then we would probably invent a bilateral accounting
scheme for each node to keep track of how much goodness each of its
peers has done for it, and to reward helpful peers.  This would then
turn out to be more or less identical to the "bilateral accounting"
scheme that was originally invented by Jim McCoy and Doug Barnes in
Mojo Nation [footnote *].

[footnote *]  Once upon a time there was digital cash, as pioneered by
David Chaum.  When Jim McCoy and Doug Barnes invented Mojo Nation, they
used digital cash, and added bilateral accounting so that a pair of
peers wouldn't require a transaction with a central token server in
order to incentivize each other.  The first three employees they hired
to implement Mojo Nation in 1999 were Greg Smith, Bram Cohen, and
myself.  (Greg might have started in 1998 -- I'm not sure.)  Several of
the ideas in BitTorrent -- which Bram started writing in 2001 -- can be
understood as radical simplifications of ideas in Mojo Nation.  One
such perspective is to think of BitTorrent's tit-for-tat incentives as
being time-limited, file-specific, and non-transferrable bilateral
accounting.  This is not condemnation of BitTorrent's ideas, but praise
of them -- they demonstrate the virtue of radical simplification.

[markm]

This is more about allowing people with lots of extra hard drive space a way to make money with that space, and giving people who need more space an opportunity to purchase it at a price that isn't insane.

Also, S3 is still high priced.  $125/month for 1TB of storage.  I'd be willing to provide 1TB of storage at a far lower price than that.

If it is not about backups, isn't the cost of it being offsite driving up the price needlessly?

Wouldn't your sister be better served by borrowing or renting a disk drive from you for her to use at her site, thus paying just for one sneakernet movement of the drive to her then someday maybe one sneakernet move of it back to you, instead of paying by the month for remote access to a drive at your site?

Or is this sister at the same site as you?

If you have reliable long term lack of need for storage, and she has reliable long term shortage, that seems a perfect setup for a used disks market even: sell her a spare disk and buy a better one later if some day you find you do afterall need more.

The closer you approach the fly by night laptops scenario the closer you get, it seems to me, to a situation where buying momentary storage instead of buying the media itself that the data is stored on starts to look reasonable. If you only need a few petabytes for a few minutes or hours, it might well make more sense to use it remotely rented by the minute or hour than to buy or rent a few petabyte drives for your own location.

So I think we need to figure that yes we are looking for fly by night laptops, however outre that might sound, and thus look for solutions targeted precisely at making distributed storage reliable even on aggregations of fly by night laptops.

Having to make rather extreme assumptions about the nodes being so utterly unreliable, to the point where we are not trying to insure our data against the occasional failure of a machine somewhere in our network but, rather, are trying to insure it against maybe even a 51% attack in which 51% or more of our nodes are with malicious aforethought carefully co-ordinating and tracking data looking for a moment in time at which all of them agreeing to refuse us service at the same time, deleting all copies of our files they have access to, can do us the most harm (for the lols, even, maybe) we can figure we want to ensure the default number of copies of any item of data should be far far higher on our proposed network than the maybe 3 copies to a dozen copies or so things like GNUnet and Tahoe-LAFS and so on typically figure people will configure their data to demand.

Our disk space is thus likely to make up in sheer volume required to keep many more copies of everything than a more reliable network would need a lot of any cost savings it might experience by imagining the space used is "essentially free due to being not in use" (when in fact just like mining, actually spinning up the drive instead of letting it hibernate might well drive up the electricity bill...)

There definitely are projects aiming toward the desired goals, except for the part about money.

Maybe anonymity might not be that great a feature once money is involved, since if Moriarty offers to store my data I might prefer to pay someone else more just to keep it out of his hands rather than pay him to store it for me? If I have to assume that the vast majority of the copies I am paying for of my data will deliberately vanish just at the very moment when I most need them, the price I will be willing to pay per copy maintained will be far far lower than if I am more confident that the vast majority of the copies will survive and be instantly available to me at high bandwidth from numerous sources...

Projects are working on all this, I have looked into those I could find, visited their Freenode IRC channels, given them a test drive and so on. From what I learned so far I have the impression that the main thing preventing GNUnet for bothering to have a currency yet is they are still tuning exactly what any such currency would actually buy, or maybe a better way of saying it would be, given some kind of currency, what exactly should a node consider how valuable compared to what else in order to intelligently decide what exactly to spend how much currency on.

That is, I got the impression that GNUnet was still in the process of working out what needs to be able to be compared in value in order to make purchasing decisions.

Thus I am still thinking that once GNUnet works well at actually trading the commodities it has at hand then would be the time to look at how best to set up mechanisms by which a node can be configured to give some nodes preferential treatment above and beyond that which is "deserved", so that bitcoins or whatever else could then be used to bribe any nodes of your choice into giving your node resources that your node has not "earned" by its own contributions. At that point markets could compare what bribes various nodes are willing to accept and nodes unwilling or unable to contribute their fair share of work/service would be able to shop around to choose who to bribe for what "unearned" resource-usage.

Bear in mind denial of service attacks, too. A market driven only by pennies might be quite vulnerable to anyone willing to throw dollars at it for a few hours to disrupt it. You might think there are a thousand laptops online at any given instant that have copies of your data safe and sound because each one of them each already learned you pay an entire satoshi per year per copy, but then suddenly one moment find every one of them threw away your data without notice due to hearing of an offer of a hundred million satoshis per decade per gigabyte per copy from someone else who might renege on their offer the very moment they realise their offer accomplished its goal of having everyone delete all copies of your data to make room for theirs...

-MarkM-

[markm]

Just because I am interested in bitcoin ... does not mean I want to part with my precious bitcoins for something I can instead buy with other currencies.

Um, isn't the whole point of Bitcoin to replace fiat money with a decentralized commodity?

Yes, but by other currencies I meant currencies such as megabit-hours of storage, megabit per second seconds of bandwidth, persistence of reachability at any hour of the day or night, still having your data next year when you need it, not flying by night with all your data, not conspiring to try to DOS you somehow in your use of the system and on and on like that.

-MarkM-

[justusranvier]

Yes, but by other currencies I meant currencies such as megabit-hours of storage, megabit per second seconds of bandwidth, persistence of reachability at any hour of the day or night, still having your data next year when you need it, not flying by night with all your data, not conspiring to try to DOS you somehow in your use of the system and on and on like that.

That's why I mentioned reputation systems in a previous post. A laptop that is not connected to the internet constantly will frequently fail to deliver blocks so would have a very low reputation score. Its storage would then be devalued compared to nodes which were always on with a reliable and fast connection and probably wouldn't be able to generate much income at all.

[markm]

Yes. This is the kind of thing GNUnet is working toward.

Until this kind of thing itself alone without graft/bribery using an external currency works smoothly, I think adding graft/bribery is probably just going to make it even harder to get it "right".

Rabid free-market folk I suppose would say look if someone offers more money for deleting all data that originated from you than you offer for retaining it, tough luck, man up and offer bigger bribes.

I wonder how much those people are offering not to have them shot on sight?

-MarkM-

[justusranvier]

Until this kind of thing itself alone without graft/bribery using an external currency works smoothly, I think adding graft/bribery is probably just going to make it even harder to get it "right".

Rabid free-market folk I suppose would say look if someone offers more money for deleting all data that originated from you than you offer for retaining it, tough luck, man up and offer bigger bribes.

I wonder how much those people are offering not to have them shot on sight?

Is this a conversation about how a system could be devised to efficiently allocate excess bandwidth and storage space or it is a conversation about emotional issues that you should be talking to your therapist about?

[justusranvier]

Ah, ok.  I think Bitcoin would serve the purpose much better!

I think this true, not necessarily because of anything inherent to the currency, but because of the accumulated experience in the Bitcoin community regarding just how determined some people will be to cheat the system. The market model must have strong incentives for good behavior and robust methods of detecting and deterring bad behavior because there will be bad actors.

[markm]

This is expected to be distributed, possibly somewhat in the spirit that bitcoin itself is, right?

We have seen with bitcoin that securing against attack tends to come down to how much it costs to defend it versus how much it costs to attack it, along with hopefully being able to make defending it seem more rewarding to potential attackers than attacking it so that some potentail attackers might decide the luls plus possibly even profit to be had by some ingenious attack are outweighed by the profits to be had by co-operating with the defenders.

So basically it seems to come down to how much is bid by defenders to have their data actually survive versus how much might an attacker pay to have someone's or everyone's or anyone's data not survive.

Is the ultimate purpose to run an experiment to see whether defenders choose to outbid attackers, or to create one or more markets where speculators and manipulators and brokers and shorters and so on can wheel-and-deal, or to get software working that can "equably" or some such term balance the resource contributions of nodes, onto which can then be added some means by which money  can be applied to influence which nodes that are not contributing are nonetheless provided use of resources?

I am proposing the latter: first make sure the thing actually works fine when nodes are not being paid money to service "nodes that normally would be allocated less resources than their owner desires due to providing less resources than they consume".

Basically the idea is we will not know which nodes have excess resources and which have shortfall of resources a-priori ahead of time.

For example even though a human owner of a node might imagine their node to have an excess of disk space could in practice find the space turns out to be offset by their node having a shortage of bandwidth, or of uptime, or of data integrity in terms of actually providing people with the data people thought it was storing for them at the time that they wish to access it, or whatever.

For example I think GNUnet uses "karma" as its internal currency-type-thing. Merely having a lot of disk space might or might not turn out to result in gaining more karma than you lose in the accounting of various other nodes depending on how karma is actually computed and what exactly actually happens in the way of blocks being stored, blocks being requested, the speed with which requested blocks are delivered compared to the speed from which they can be obtained from other nodes, basically whatever heuristics are in the code.

It might turn out that the way to make a trickle of pennies is not so much to have lots of disk space but, rather, to provide really low latency high bandwidth delivery of data stored on such disk space as you do have.

Once the heuristics the system uses internally to assign karma to other nodes work well, then maybe it would suffice to simply have a BTC<->karma exchange where people can buy karma points for their node on nodes of their choice using bitcoins.

But until karma points work well, maybe such a market wouldn't really be all that useful or great as markets for really great products likely do better than markets for products that don't actually work very well.

-MarkM-

EDIT:

Some distributed storage solutions drift the blocks toward the places that rquest them, so that for example if lots of people at a particular ISP ask for certain files and if the ISP itself was a node along the way for those people, more and more often more and more blocks of the popular-with-them files would tend to have migrated closer and closer to them.

Now imagine if speed were a factor, possibly in addition to how distance is in that solution. Having huge amounts of disk space might be almost worthless, since small-storage nodes with high speed might turn out to be the usual nodes most people get a copy of something from, with the huge disk drives having slow bandwidth becoming places only resorted to if it turns out each and every really fast node turns out to not have a copy of something the vast disk slow bandwidth node has. Thus the huge disks could idle unused for years before some partocular block of some particular ancient file no one has enquired about for years turns out ot be available from the huge slow node...

[justusranvier]

Basically the idea is we will not know which nodes have excess resources and which have shortfall of resources a-priori ahead of time.

For example even though a human owner of a node might imagine their node to have an excess of disk space could in practice turn out to be offset by their node having a shortage of bandwidth, or of uptime, or of data integrity in terms of actually proving people with the data people thought it was storing for them at the time that they wish to access it, or whatever.

That's the advantage of using a market mechanism to figure it out. We don't need to know ahead of time the best allocation of resources and instead allow the relative price to float according to supply and demand. Most individual users are not going to sit over their nodes and daytrade bandwidth and hard drive space so it makes the most sense from a usability perspective for the trading to be done by automated software agents.

For example I think GNUnet uses "karma" as its internal currency-type-thing. Merely having a lot of disk space might or might not turn out to result in gaining more karma than you lose in the accounting of various other nodes depending on how karma is actually computed and what exactly actually happens in the way of blocks being stored, blocks being requested, the speed with which requested blocks are delivered compared to the speed from which they can be obtained from other nodes, basically whatever heuristics are in the code.

It might turn out that the way to make a trickle of pennies is not so much to have lots of disk space but, rather, to provide really low latency high bandwidth delivery of data stored on such disk space as you do have.

Once the heuristics the system uses internally to assign karma to other nodes work well, then maybe it would suffice to simply have a BTC<->karma exchange where people can buy karma points for their node on nodes of their choice using bitcoins.

There's no reason to duplicate a currency system. Bitcoin has a lot of features, like contracts, that would make it useful for exactly this kind of use. If Bitcoin doesn't have sufficient features to be used as karma then wouldn't it make more sense to add them instead of creating a new currency?

But until karma points work well, maybe such a market wouldn't really be all that useful or great as markets for really great products likely do better than markets for products that don't actually work very well.

This is the risk of all new markets - that there isn't actually sufficient demand for the product being offered. It's not like any bitcoins used in the experiment would be lost though. If more people sign up to offer disk space then there are people wanting to rent then it will be extremely inexpensive to store files. This will tend to cause people selling storage to leave the network because they can't make any money and will attract new customers because the price is so low. If there are no customers willing to buy at the price point that anyone offering storage is willing to sell then the network will fail.