[IDEA] Dirt cheap online storage

[Sukrim]

That's why I called it simple - service.com would only issue challenges + try to connect via BitTorrent to the seeds. If there is a checksum error or not even a torrent client running, no money for them! The hassle for downloading the torrent files + keeping them is hopefully enough though to let them really seed as well.

Ideally, other clients/downloaders would report from whom they loaded how much. This data unfortunately cannot be trusted though and I still can't really think of anything how to make it trustworthy.

Edit:
Maaaaaybe you can trust "I have loaded x MB from ..." data if the person claiming this can also solve checksum riddles. It would still be trivial for seeders though to solve these riddles and claim high upload rates from some random clients.

[SgtSpike]

That's why I called it simple - service.com would only issue challenges + try to connect via BitTorrent to the seeds. If there is a checksum error or not even a torrent client running, no money for them! The hassle for downloading the torrent files + keeping them is hopefully enough though to let them really seed as well.

Ideally, other clients/downloaders would report from whom they loaded how much. This data unfortunately cannot be trusted though and I still can't really think of anything how to make it trustworthy.

Edit:
Maaaaaybe you can trust "I have loaded x MB from ..." data if the person claiming this can also solve checksum riddles. It would still be trivial for seeders though to solve these riddles and claim high upload rates from some random clients.

It's hard to have an honor system when money is involved.  But, I'm kind of with you on it - I don't think it'd be a showstopper either.  If undeniable or overwhelming evidence presents itself of people abusing the network, they could be banned on a case-by-case basis.  It's more of a bridge-to-be-crossed-when-we-get-there problem.

[Sukrim]

A different approach could be kind of a "flattr" model for paying uploaders.

Seeders of anything run a client that replies with their ID and downloaders run a client that asks any connected client for that ID (it either gets ignored or someone replies). Every now and then a list of "I loaded 1 MB from X, 40 MB from Y and 100 MB from Z" is uploaded to service.com.
Service.com then holds funds of downloaders in escrow and pays them proportionally to X, Y and Z. If downloaders don't tell any data, their funds for that time frame get donated or evenly split or split according to some rating (glicko2 sounds really cool) or . Stats should be as fast as possible available but aggregated over a week or even longer (2 weeks, 1 month...) to have meaningful data for payouts for most participants.

To get more funds, it is of course better to prefer clients that ask for an ID. It can be even 2-sided, so seeders can also see if they really got paid something from A, where they uploaded 1 GB to or if A was just asking for their ID but never submitted a report.

This would again require some trust from seeders, that someone asking for an ID will really pay something and requires 0 trust from leechers (as they anyways can use the swarm as usual - but have a chance to get some much better seeds if they ask for IDs to pay towards). If the updates run in near-realtime, trust can again be increased over time ("I uploaded 1 MB to A and a few seconds later A rightfully told service.com about it - more bandwidth to A!") and even without some fancy traffic shaping stuff it's simple to just seed some popular torrents and potentially gain a few coins too.

This way it doesn't matter if there are fakes or not (they can only pay anyways) and service.com also doesn't need to care about bandwidth offered as it is i the best interest of seeds to offer more bandwidth to users of service.com.

The above would be the other extreme - big torrents with few leechers would be not very popular, unless there are some leechers that really offer a lot for that traffic. Just keeping the torrent alive though (no leechers atm.) is not really attractive, as there's more money to earn by loading + sharing the next file and actually uploading something.
Maybe these 2 extremes (it's already enough holding a file and not even seeding it vs. upload is everything, even just loading 1 part of a popular file + just sharing it can net you a lot) can/should be somehow combined for different user groups or so?

[SgtSpike]

What I am afraid of with that sort of solution (where they are only paid when people download from them) is that the files, particularly personal backups, wouldn't actually be downloaded very often.

But, perhaps a combination could be used, as you said.  Payments are aggregated for up to 2 weeks.  If no download of the data is made within those two weeks, then as the payments age to 15 days, they are paid split evenly between the seeders.  But, if the data is downloaded, at least once, then all aggregated payments to that point are paid, and are split between the seeders according to whatever share of the file was provided to the downloader.

This would encourage them to make the file available 24/7, and also seed it as quickly as possible in order to gain a larger share of the payment.

[lologarithm]

What I am afraid of with that sort of solution (where they are only paid when people download from them) is that the files, particularly personal backups, wouldn't actually be downloaded very often.

But, perhaps a combination could be used, as you said.  Payments are aggregated for up to 2 weeks.  If no download of the data is made within those two weeks, then as the payments age to 15 days, they are paid split evenly between the seeders.  But, if the data is downloaded, at least once, then all aggregated payments to that point are paid, and are split between the seeders according to whatever share of the file was provided to the downloader.

This would encourage them to make the file available 24/7, and also seed it as quickly as possible in order to gain a larger share of the payment.

The idea of paying per download is the best way (and by best I mean only 100% sure way) to be sure of the seeders properly hosting the file. The issue as you mentioned is that for files that are rarely downloaded there is low incentive to store them.

Perhaps there is a baseline requirement: If no files are downloaded in a week the person pays a set amount to the people who have been seeding. The baseline would be much less than if the file were actually downloaded but might provide some incentive. Connectivity could be tested a few times a day perhaps to see who was connected throughout the week, then paid proportionally to uptime during the week.

[Sukrim]

To determine that somebody has downloaded a file though, service.com needs that checksum quiz file... with the "flattr" model, service.com doesn't even need to know/care which file was downloaded. I get your point though, but I guess I have to think a bit more to come to a solution.

[SgtSpike]

What I am afraid of with that sort of solution (where they are only paid when people download from them) is that the files, particularly personal backups, wouldn't actually be downloaded very often.

But, perhaps a combination could be used, as you said.  Payments are aggregated for up to 2 weeks.  If no download of the data is made within those two weeks, then as the payments age to 15 days, they are paid split evenly between the seeders.  But, if the data is downloaded, at least once, then all aggregated payments to that point are paid, and are split between the seeders according to whatever share of the file was provided to the downloader.

This would encourage them to make the file available 24/7, and also seed it as quickly as possible in order to gain a larger share of the payment.

The idea of paying per download is the best way (and by best I mean only 100% sure way) to be sure of the seeders properly hosting the file. The issue as you mentioned is that for files that are rarely downloaded there is low incentive to store them.

Perhaps there is a baseline requirement: If no files are downloaded in a week the person pays a set amount to the people who have been seeding. The baseline would be much less than if the file were actually downloaded but might provide some incentive. Connectivity could be tested a few times a day perhaps to see who was connected throughout the week, then paid proportionally to uptime during the week.

That's an interesting idea as well.  But if files do get downloaded, does the person who is fake seeding get paid, or does he get left out at that point?

It does reintroduce the complexity of tracking two metrics in the marketplace though.  Whoever is hosting has to track both the payment per download and the payment per seed.  OTOH, it would allow people who don't have much bandwidth to spare, but lots of storage space to spare, to better compete with those who have lots of bandwidth to space, but little storage space.

[markm]

EDIT 2: TL;DR By "dirt cheap", do we mean "cheaper than foisting it off upon the bitcoin blockchain"?


Okay, lets consider something that might be massive overkill of amount of resources being committed and overall certainty of the integrity of the data.

At least maybe try to figure out just how totally over the top the costs would be.

Remember that thing called a blockchain. whereby distributed peers maiintain an ongoing mutual confidence in the validity of past transactions some among them purported to have occurred?

There are algorithms too whereby checking each member of a (spy, etc?) network's reports about each other member, actual path to path inferences as to which path a "weak link" is on. Reliability matrices across many possible paths, making intferences both about which path was or  might have been taken and which nodes are how likely to be the sources of any unreliability that is measured/reported.

The content of the blocks of the chain could be assertions that each peer on the entire network received from each other one a correct merkle value taking into consideration which peers claimed to have recieved which such values from each other peers and what the actual total combined merkle of all bits ever submitted to the network the consensus agrees to be correct based upon the order in which the consensus agrees those bits were submitted (or more likely, committed; as like with bitcoins we'd not count a block of bits to have been submitted uintil the consenses agrees not only with whether it has been submitted but in what order, relative to others).

This could all even maybe serve as some research toward just how much a huge blockchain really does cost a network overall compared to how usefull to the network nodes that fail to store the entire blockchain actually are.

The chain could be pruned of blocks that have already been returned to their original sender undamaged after having been retained the stipulated period of time, and if original-uploaders ("content providers"? "financiers", maybe, as presumably they should bear the entire cost of this project between them?) are not anonymous if they fail to publish cryptographic proof of their agreement that they, the orignal uploader of bts blah blah blach, did in fact receive back intact said bits from nodes this that these those et al and sundry, etc etc etc then some black mark could be held against them too for trying to cheat participants of their deserved repution owed to them for their part in actually performing as contracted to.

This could be massive massive cost, but cxnsider how much people pay for the bits of the bitcoin blockchain, which is precisely a record of what quantities of bits, representing what quantity of bits worth of value, which parties exchanged in a consensually agreed approximate order in time...

-MarkM-

EDIT:  Also, what the heck, why not tie it all to bitcoins from the start, by trying to store the blockchain itself on just such "dirt cheap storage"...

...Or is that precisely what exactly bitcoin IS doing, in effect, and by doing so, is revealing to us the actual cost of storing (such?) data in such a way?

EDIT3: By bidding using various altcoins, might bidders thusly express the certainty level they are budding upon, like "I want to store this as securely as if I had coded it into the ___coin blockchain in some way" for each coin designated by the ___ ? (Taking into consideration the propensity of the developers of the ___coin blockchains to attempt to prune any data from it that they consider no longer necessary to retain after some span of time verus cost of blockchain size.)

[SgtSpike]

Markm - having everyone store the same data is a bit different than having a handful of people store the same data, and a different handful storing different data, etc.

[jfreak53]

See Sarge, this is what you get for not asking me first

We offer dedicated server's with up to 3TB drive's for a one-time fee, not monthly.  The main ATOM server (great for storage) with a 2TB drive is $58.50 a month and $160 one-time fee.

[SgtSpike]

See Sarge, this is what you get for not asking me first

We offer dedicated server's with up to 3TB drive's for a one-time fee, not monthly.  The main ATOM server (great for storage) with a 2TB drive is $58.50 a month and $160 one-time fee.

A one time fee?  That's interesting... how much?  What limits are there on bandwidth usage and speed.

[jfreak53]

The one time fee is $160 for the 2TB and 3TB I'd have to get you price on since I don't sell a lot of them. Bandwidth is default 10Mbps for that price monthly as stated above.

BUT, we offer 100Mbps and 1Gbps on this type server, we offer more speed's on bigger server's but these are the ATOM's.  The 100Mbps costs $50 a month and comes with 10 IPs and the 1Gbps costs $150 a month. Of course all port's are unmetered

Base price of the ATOM's with 250GB drive and 2GB of ram is $58.50, then you can add drive's for one-time fee's or monthly if you want. The ATOM's can also have up to 2 drive's.  The difference in paying monthly for the drive and one-time is the one-time makes it your's, if it goes bad you have to buy another one, with monthly it's replaced free if it goes out.

[jfreak53]

Oh, also the 4GB ram upgrade for $20 a month on the ATOM's, comes with a 1.5 TB drive upgrade for the primary drive from the original 250GB free of charge You can't beat it And on that 4GB upgrade you only pay $15 one time for the 2TB upgrade from 1.5TB

Those prices are not on the order system, unfortunately WHMCS doesn't allow for such sophisticated of pricing, so it's one of those thing's that we have to setup the original order for.  But those are the price's.

Also if you absolutely need it, we do offer speed's of 10Gbps unmetered it's just a bit pricey and only on bigger server's.

[SgtSpike]

I misunderstood - I thought you were offering 2 TB of online storage for ONLY a one time fee, no monthly fees!

BTW jfreak, do you do colocation?  I have a server running from my house, and it's a nice high-spec server (would cost hundreds/month to rent something comparable), but I don't have all that much upload bandwidth.  I'd love a good 10Mbps or 100Mbps non-residential connection to run it on!  All the colocation services I've looked in to are very expensive though...

[jfreak53]

Well the one time is the Hard drive then your just paying for server and internet, you never again pay for HD, just server usage.

Colo: Yes I do , just not publicized since we do it very seldom.

What do you have rackmount or tower?

[SgtSpike]

Well the one time is the Hard drive then your just paying for server and internet, you never again pay for HD, just server usage.

Colo: Yes I do , just not publicized since we do it very seldom.

What do you have rackmount or tower?

It's a full-size tower.  Could stuff it in a 4U though.

[jfreak53]

For tower it would be $69.00 a month that would give you:

-10Mbps unmetered (100Mbps unmetered + $25/m)
-electric included / 1 power drop
-Denver DataCenter
-5 Free IPs (10 on 100Mbps)

[coga]

Markm, you're COMPLETELY missing the point.

I'm reading through the thread, and that is one thing I'm totally agreeing with

[coga]

I've been contemplating developing something like that, and here is my vision how it must work. There should be 3 types of actors:

- Nodes (provide storage)
- Clients (consume storage)
- Network Controllers, or NCs (used to locate nodes for clients, and manage escrow)

Storage network allocates blocks in chunks, say 1MB each.

Nodes broadcast to NCs their reserves, price per megabyte and advertised avaiability and speed.

Lets say Client wants to store 20MB worth of data, and store each megabyte on at least 3 nodes. Client should have an account on NC, which should already have some BTC. Client would use NCs to locate 60 Nodes that charge requested price or less, and if those are found, upload the data.

Client would also register with NC that it did upload the data to a node, and specify the checksum of the blocks he uploaded, and several random salted checksums. Those would be used later to verify that node indeed has the chunk.

Nodes will be paid by NC daily as long as they store the chunk. NC will use the challenge that client gave it. To prove that Node keeps data, it will have to calculate the checksum of the salted chunk, and come up with the one that matches the challenge. If they don't, the chunk is considered abandoned, and node is no longer paid for this chunk.

Client periodically checks with NC for the list of abandoned chunks and relocates them to keep necessary redundancy.

NC stores the "reputation" for each NC in form of abandoned chunk count. The Nodes with zero or very little dropped chunks are considered higher reputation.

Nodes will get 50% of the bitcoin amount in form of mandatory payments from NCs. The remaining 50% will be distributed as "tips", and the distribution is determined by Client. So, if Client has to pay 2BTC for storage, 1BTC out of 2 will be given to nodes equaly, and another 1BTC will be distributes as per discretion of Client. This would be done so that Client could reward fast or least problematic nodes.

[SgtSpike]

@ jfreak - how much per month if I converted it to a 4u?  Would that make it cheaper?  The pricerange is close to what I am looking for, so it definitely has potential.  And 5 IP's would be awesome too!

I like your line of thinking coga.  Not sure I agree regarding the tips - that would add extra complication on the client side that they might not want to be involved with.  I still like the idea of the payments being distributed according to how much of the file each node contributes when the file is downloaded.  I am just uncertain as to whether it would be possible to track such a statistic.