Solution to Sia/Storj/etc DDOS issues and Sybil Vulnerability

[earonesty]

All cloud storage networks, coin or otherwise, are vulnerable to DDOS attacks, where attackers cause renters or hosts to spend enormous sums of money by repeatedly downloading the same files.

• The typical solution to these DDOS attacks remains "IP blocking".   For TCP/IP, this can be effective, and when coupled with some sort of tracing back to the source, results in DDOS attacks that wane and can be responded to fairly rapidly.
• The practice of IP blocking has spawned an open and somewhat contentious network of public and private IP blocklists.   While it is likely that these blocklists will be relevant for storage coins, there will be undoubtedly be a need for blocklists to be maintained that are specific to cloud storage protocols.
• Blocklists, even if fairly maintained, are centralized and many use DNS for distribution (see https://en.wikipedia.org/wiki/DNSBL).  A blockchain protocol for blacklisting may be ideal, with nominations, evidence, voting and expiration all occurring "on chain".   (It should be obvious why blocklists would greatly benefit from decentralization)

All storage coins are additionally vulnerable to a "not really redundant" Sybil attack. 

• Any attacker can quite simply spin up many instances of the protocol on a single host - increasing his exposure and income - while completely defeating the purpose of redundant storage.
• There is currently no way for a renter or user to know where its hosts physically reside, and whether they are on the same machine or not.
• One way of mitigating this is to publish IP addresses of hosts as a part of file contracts or storage advertisements.   Renters should then prefer IP's that are somewhat distributed across the IP space.   This can, of course be defeated by routing diverse IP's to the same data center.   But this can be trivially detected and violators can then be published via the same IP blocklists above... using either DNSBL or blockchain tech.

In summary:

• Any time you are "hosting" data, you need to deal with DDOS attacks.
• However you solve it, I don't care: this solution will, de facto, provide an analogous solution for Sybil attacks on storage network redundancy

So don't worry about Sia/Storj sybil attacks.   They are just inverted DDOS attacks, and those have to be solved anyway.

[1715361532]

1715361532

[1715361532]

1715361532

[Taek]

I think that you are bringing up a couple of separate concerns, and that they cannot be collapsed into a single concern.

For example, a host or a renter might be subject to a DDOS attack (especially a host) that makes business difficult. But it's much easier to DDOS a single host than it is to DDOS an entire file, because you need to know all of the hosts that have that file (not always public knowledge), and you need to have enough resources to DDOS a large enough number of hosts to make the file inaccessible, and that number could be dozens of different hosts, which is a different level of attack than DDOSing a single host. As far as I am aware, there's not a whole lot that you can do about DDOS attacks (other than work with a service like CloudFlare) - if someone wants to send you network traffic, there's usually a way for them to do it.

The Sybil attack, or 'not really redundant storage' is a different concern. When uploading files to the network, you want to upload to many different hosts (as a part of the security model), and you want to be sure that these different 'hosts' aren't all putting your data on the same physical disk. There are two components to this. We can solve one of the problems pretty easily by encrypting the redundant pieces using different keys. So, we can force the network to store the full redundancy of the file, though this technique is not enough for guaranteeing that the data is not on multiple separate physical drives.

You can introduce a monetary component to make Sybil attacks expensive. On the Sia network, for example, all money transferred between renters and hosts is measured on the blockchain and includes a component of proof-of-burn (in the form of siafund fees). The proof-of-burn component means that this activity is expensive to spoof, mitigating simple Sybil attacks and forcing attackers to spend significant monetary resources executing an attack.

Depending on how much you trust something like an IP address, you can use IP addresses to get a sense of the actual physical location of the data. But you can also do better than that if you've got multiple servers spread over a large geographic area. You can do ping-time challenges to the hosts that are storing your data. If a host claims your data is in China, and another host claims your data is in America, and you've got a server in each, you can do challenges that require the host to respond in under X milliseconds. The laws regarding the speed of light give you a way to be absolutely certain that your data is being stored in both China and America, because the ping time on the challenges from your server in China to the host in China will be very small, and the ping time on the challenges between your server in America to the host in America will also be small.

So, if you are willing to use a more elaborate setup, you can definitely get proofs (based on the fact that data cannot travel faster than the speed of light) that your data is in two different geographic locations. And depending on your trust model, you can outsource these challenges to the network (have someone else do the challenges on your data and trust their results).

The really nice thing about both DDOS and Sybil attack prevention is that it's not a consensus layer problem. Consensus protocols are notoriously difficult to upgrade, and even more difficult to get right, but we don't have to worry about maintaining consensus. We can update nodes and improve techniques in ways that don't require the whole ecosystem to upgrade simultaneously, so as we continue to think of better ways to handle DDOS and Sybil attacks, we can incrementally push them out to the network.

[sine_fuc]

There are bugs in Sia wallet.

[earonesty]

you need to know all of the hosts that have that file (not always public knowledge)

You merely need to download the file over and over... which causes the renter to run out of money.   Same as a DOS attack downloading a web page.

forcing attackers to spend significant monetary resources executing an attack.

Attackers always spend the same money that legitimate hosts spend.   The "attacker" here is running completely legitimate, full copies of the protocol.   The only difference is that they are running multiple docker containers on the same machine... rather than running on multiple machines.   There is simply no way to know that outside of IP.   Just like a DOS attacker runs a fully legitimate client, so too a Sybil attacker runs a legitimate server.

The really nice thing about both DDOS and Sybil attack prevention is that it's not a consensus layer problem.

I think that succinctly sums up what I meant in the original post.

Since DDOS and Sybil are inverted forms of similar attacks (multiple clients vs multiple hosts), these decentralized storage networks haven't introduced a "new" problem to the domain of hosting.  Indeed, any existing cloud hosting provider could, today, "claim redundancy" on your data... but physically locate it on the same machine.   The solution to this problem lies in the transport and social layers... not the consensus protocol.

[Taek]

you need to know all of the hosts that have that file (not always public knowledge)

You merely need to download the file over and over... which causes the renter to run out of money.   Same as a DOS attack downloading a web page.

The person doing the downloading is the one paying for the bandwidth, you pay for them in the same network connection. Downloading a file does not take funds from the renter, otherwise the host could just lie about who had downloaded the file (or download from themselves repeatedly from a cheap local connection). No DoS attack here.

forcing attackers to spend significant monetary resources executing an attack.

Attackers always spend the same money that legitimate hosts spend.   The "attacker" here is running completely legitimate, full copies of the protocol.   The only difference is that they are running multiple docker containers on the same machine... rather than running on multiple machines.   There is simply no way to know that outside of IP.   Just like a DOS attacker runs a fully legitimate client, so too a Sybil attacker runs a legitimate server.

You are misunderstanding how the Sybil defense mechanism works. Having multiple VMs, or even multiple full machines on different IP addresses, is not sufficient. You also have to have a history of burning coins, and there's a linear relationship between how many coins you've burned and how likely a renter is to select you. If you want to be as likely to be selected as a 10,000TB host, you need to burn enough coins to keep up, and they are (per the siafund fee) burning approximately 10% of their income.

And if you actually have 10,000TB, you haven't performed a Sybil attack at all you're outright a legitimate node on the network.

You can leverage identity to manage the Sybil attack as well, but right now the only real solutions to identification that we have are all centralized in some way. Burning coins is decentralized.

[iamnotback]

I wrote those same criticisms many months ago. And you didn't even deal with the copyright issue and the fact that ISPs/hosts will block protocols that promote theft of copyright.

Decentralized file storage is a non-starter. The speculators are willing victims. My popcorn is ready to watch them blow up.

[super3]

All cloud storage networks, coin or otherwise, are vulnerable to DDOS attacks, where attackers cause renters or hosts to spend enormous sums of money by repeatedly downloading the same files.

• The typical solution to these DDOS attacks remains "IP blocking".   For TCP/IP, this can be effective, and when coupled with some sort of tracing back to the source, results in DDOS attacks that wane and can be responded to fairly rapidly.
• The practice of IP blocking has spawned an open and somewhat contentious network of public and private IP blocklists.   While it is likely that these blocklists will be relevant for storage coins, there will be undoubtedly be a need for blocklists to be maintained that are specific to cloud storage protocols.
• Blocklists, even if fairly maintained, are centralized and many use DNS for distribution (see https://en.wikipedia.org/wiki/DNSBL).  A blockchain protocol for blacklisting may be ideal, with nominations, evidence, voting and expiration all occurring "on chain".   (It should be obvious why blocklists would greatly benefit from decentralization)

All storage coins are additionally vulnerable to a "not really redundant" Sybil attack. 

• Any attacker can quite simply spin up many instances of the protocol on a single host - increasing his exposure and income - while completely defeating the purpose of redundant storage.
• There is currently no way for a renter or user to know where its hosts physically reside, and whether they are on the same machine or not.
• One way of mitigating this is to publish IP addresses of hosts as a part of file contracts or storage advertisements.   Renters should then prefer IP's that are somewhat distributed across the IP space.   This can, of course be defeated by routing diverse IP's to the same data center.   But this can be trivially detected and violators can then be published via the same IP blocklists above... using either DNSBL or blockchain tech.

In summary:

• Any time you are "hosting" data, you need to deal with DDOS attacks.
• However you solve it, I don't care: this solution will, de facto, provide an analogous solution for Sybil attacks on storage network redundancy

So don't worry about Sia/Storj sybil attacks.   They are just inverted DDOS attacks, and those have to be solved anyway.

These are issue that thought about and solved early on. We tend towards micropayments, so the attacker would have to pay for all of the downloads. So this would be less of the farmer being attacked, and more of the farmer getting rich.

As far as DDOS attacks, good luck. Only the renter should know where all ~40 of the pieces are. Even the the unlikely event that the renter has exposed the location of all the shards to the farmer the odds are way against the attacker. The farmers will reject connections from the attacker, so the attacker has to execute a pretty massive attack to have any impact. Sia checks the file every ~2 weeks last time I remember, we can check up to 10 minutes or less.  It takes seconds to spin up a new storage contract for DDOSed shards. This is what happens if execute a DDOS attack and the user is using our service in the worst case:

1. You start a massive DDOS on all 40 farmers that store the data.
2. Most of them are behind tunnels, so their tunnels reset. Now only us and the renter know where the new locations are.
3. The ones that you are currently attacking, have already been replaced in a few seconds.
4. No data is lost, although the file might be inaccessible to the first few seconds as we tunnels reset. Once we know a DDOS is happening, we can spin up nodes and tunnels faster that any attacker can find and DDOS.

tldr; DDOS works because its a distributed network attacking a centralized network. Distributed network attacking a distributed network is as effective as trying to drown a glass of water in a lake.

[iamnotback]

No decentralized storage can solve the Sybil attack on storage where many nodes are sharing the same storage, but collecting payments or otherwise deceiving the claimed redundancy as if they have multiple copies of the storage.

It is a fundamental weakness that can not be fixed. Decentralized storage can not work. Period. End of story.

Latency is not reliable enough to be used as an indicator.

[Suggestive Melon]

No decentralized storage can solve the Sybil attack on storage where many nodes are sharing the same storage, but collecting payments or otherwise deceiving the claimed redundancy as if they have multiple copies of the storage.

It is a fundamental weakness that can not be fixed. Decentralized storage can not work. Period. End of story.

Latency is not reliable enough to be used as an indicator.

Even assuming you are right about all of this (which I think you may be), I still don't think this will stop the coming bull market for file storage coins; speculators don't care about functionality, they care about hype!  Now that Ethereum is just about dried up, they need a new fantasy to cling to, I think this might be it.

[super3]

No decentralized storage can solve the Sybil attack on storage where many nodes are sharing the same storage, but collecting payments or otherwise deceiving the claimed redundancy as if they have multiple copies of the storage.

It is a fundamental weakness that can not be fixed. Decentralized storage can not work. Period. End of story.

Latency is not reliable enough to be used as an indicator.

Please read the Storj whitepaper, Section 4.2. "Each shard[of the file] is uniquely encrypted. This means that malicious farmers cant pretend to have multiple redundant copies of a file when they only have one."

You have to encrypt the file anyways, so just make each piece unique. This solves the problem.

[iamnotback]

No decentralized storage can solve the Sybil attack on storage where many nodes are sharing the same storage, but collecting payments or otherwise deceiving the claimed redundancy as if they have multiple copies of the storage.

It is a fundamental weakness that can not be fixed. Decentralized storage can not work. Period. End of story.

Latency is not reliable enough to be used as an indicator.

Please read the Storj whitepaper, Section 4.2. "Each shard[of the file] is uniquely encrypted. This means that malicious farmers cant pretend to have multiple redundant copies of a file when they only have one."

You have to encrypt the file anyways, so just make each piece unique. This solves the problem.

That doesn't prevent a Sybil attack to remove redundancy.

TPTB_need_war explained the technical reasons in February. You can search his archives. I don't have time to dig it up right now.

Edit: super3, you were discussing it with TPTB back at that time:

https://bitcointalk.org/index.php?topic=1354274.msg13833591#msg13833591
https://bitcointalk.org/index.php?topic=1340057.msg13814104#msg13814104
https://bitcointalk.org/index.php?topic=1319681.msg13650450#msg13650450

Edit#2: now I remember my rebuttal to section 4.2 of the Storj whitepaper. Yes you can encrypt files and trick the nodes into being forced to store more than one copy of the file, but this does nothing to prove that all those copies are not stored all on one hard disk. The problem is insoluble. There is no way to insure redundancy. Period. Sorry all decentralized file systems are insolubly flawed.

Again the nodes can Sybil attack the system and all be sharing the same hard disk on some centralized server. That was my entire point.

As I said, I (AnonyMint) invented proof-of-diskspace/storage in 2013. And I also proved it was insolubly flawed. Now you copycats come here 3 years later and don't even check my prior art. So you deserve to lose.

[Come-from-Beyond]

Yes you can encrypt files and trick the nodes into being forced to store more than one copy of the file, but this does nothing to prove that all those copies are not stored all on one hard disk. The problem is insoluble. There is no way to insure redundancy. Period.

I wouldn't say "Period", read https://www.researchgate.net/publication/220797029_Informant_Detecting_Sybils_Using_Incentives and reconsider your position, please.

[iamnotback]

Yes you can encrypt files and trick the nodes into being forced to store more than one copy of the file, but this does nothing to prove that all those copies are not stored all on one hard disk. The problem is insoluble. There is no way to insure redundancy. Period.

I wouldn't say "Period", read https://www.researchgate.net/publication/220797029_Informant_Detecting_Sybils_Using_Incentives and reconsider your position, please.

That only applies in a bounded opportunity cost scenario, because it pays Sybils to reveal themselves in exchange for losing the opportunity they would have gained by not revealing themselves.

The profit potential in Sybil attacking the decentralized file system is unbounded up to the total value of the expenditures for file storage.

Also the research paper quoted does not solve the problem that the Sybils can take the payment and then reestablish themselves as another Sybil identity (because the protocol hinges on the fact that Sybils don't reveal any permanent identity) and take payment again over and over until the system is drained. The paper is nonsense.

[Come-from-Beyond]

The paper is nonsense.

It's a good ending of the convo, no other replies.

[super3]

There is no proof, it comes down to statistics. It boils down to a hypergeometric distribution which you can find the math for here. For an example case, you would have to store 80% of the entire network to get a 50% chance of the attack(for one file). Bitcoin has much lower tolerances and breaks down at around 51%.

So the attack you mentioned is valid, but ignores the math behind it. It goes from being a difficult attack to pull off, to statistically improbable. You can post as much red text as you like but only a hypergeometric distribution showing me otherwise is acceptable.

[iamnotback]

There is no proof, it comes down to statistics. It boils down to a hypergeometric distribution which you can find the math for here. For an example case, you would have to store 80% of the entire network to get a 50% chance of the attack(for one file). Bitcoin has much lower tolerances and breaks down at around 51%.

So the attack you mentioned is valid, but ignores the math behind it. It goes from being a difficult attack to pull off, to statistically improbable. You can post as much red text as you like but only a hypergeometric distribution showing me otherwise is acceptable.

Your system is paying the "attacker" for as many free Sybils as he/she can create in order to fund as much storage as he needs to store as much data as your system wants to pay for. One can rent all the storage they need, e.g. Google's cloud.

Since Sybils (IP address) are virtually free to create, competitors are in an arms race to see who can capture more of the revenue. This is a power vacuum, which means there can only be one winner-take-all, because of economies-of-scale. The little guy gets squeezed and can't get paid hardly ever, analogous ASIC mining in Bitcoin where the more hashrate share the miner has, the more profitable they are. So expect pools to be created when nodes share their hard disks, so they can win more challenge-responses and earn more with less variance.

It is analogous to proof-of-work, in that it leads entirely to centralization.

I hope I don't have to write it in red text for it to not be ignored.

[super3]

There is no proof, it comes down to statistics. It boils down to a hypergeometric distribution which you can find the math for here. For an example case, you would have to store 80% of the entire network to get a 50% chance of the attack(for one file). Bitcoin has much lower tolerances and breaks down at around 51%.

So the attack you mentioned is valid, but ignores the math behind it. It goes from being a difficult attack to pull off, to statistically improbable. You can post as much red text as you like but only a hypergeometric distribution showing me otherwise is acceptable.

Your system is paying the "attacker" for as many free Sybils as he/she can create in order to fund as much storage as he needs to store as much data as your system wants to pay for. One can rent all the storage they need, e.g. Google's cloud.

Since Sybils (IP address) are virtually free to create, competitors are in an arms race to see who can capture more of the revenue. This is a power vacuum, which means there can only be one winner-take-all, because of economies-of-scale. The little guy gets squeezed and can't get paid hardly ever, analogous ASIC mining in Bitcoin where the more hashrate share the miner has, the more profitable they are. So expect pools to be created when nodes share their hard disks, so they can win more challenge-responses and earn more with less variance.

It is analogous to proof-of-work, in that it leads entirely to centralization.

I hope I don't have to write it in red text for it to not be ignored.

But you are forgetting that using hard drives doesn't scale like mining does. You don't have 14nm mining chips inside every single computer. Nodes don't have an incentive to pool, they should make more or less the same if they are "solo-mining" or in a pool.

Sure you can spin up as many IPs as you want, but that doesn't mean you will get better performance than other nodes which is how farmers are actually selected to store data. Plus if you add a deposit to that, it makes spinning up those IPs not free anymore.

[iamnotback]

There is no proof, it comes down to statistics. It boils down to a hypergeometric distribution which you can find the math for here. For an example case, you would have to store 80% of the entire network to get a 50% chance of the attack(for one file). Bitcoin has much lower tolerances and breaks down at around 51%.

So the attack you mentioned is valid, but ignores the math behind it. It goes from being a difficult attack to pull off, to statistically improbable. You can post as much red text as you like but only a hypergeometric distribution showing me otherwise is acceptable.

Your system is paying the "attacker" for as many free Sybils as he/she can create in order to fund as much storage as he needs to store as much data as your system wants to pay for. One can rent all the storage they need, e.g. Google's cloud.

Since Sybils (IP address) are virtually free to create, competitors are in an arms race to see who can capture more of the revenue. This is a power vacuum, which means there can only be one winner-take-all, because of economies-of-scale. The little guy gets squeezed and can't get paid hardly ever, analogous ASIC mining in Bitcoin where the more hashrate share the miner has, the more profitable they are. So expect pools to be created when nodes share their hard disks, so they can win more challenge-responses and earn more with less variance.

It is analogous to proof-of-work, in that it leads entirely to centralization.

I hope I don't have to write it in red text for it to not be ignored.

But you are forgetting that using hard drives doesn't scale like mining does. You don't have 14nm mining chips inside every single computer.

That a regular user has nearer to parity in terms of economies-of-scale cost efficiencies is irrelevant to the game theory economics that forces centralization in this case.

Nodes don't have an incentive to pool, they should make more or less the same if they are "solo-mining" or in a pool.

Incorrect due to variance same as for Bitcoin mining. That is unless every piece of data stored on the network will be accessed every day and in which case you wouldn't even need a challenge and response proof, because reading the data would be a proof.

More importantly, the pool can then lie and create numerous Sybils so that it can be paid multiple times for storing the same data only once. So Storj loses redundancy.

Sure you can spin up as many IPs as you want, but that doesn't mean you will get better performance than other nodes which is how farmers are actually selected to store data. Plus if you add a deposit to that, it makes spinning up those IPs not free anymore.

Selecting on performance locks out the little guy, thus driving centralization. IPs can be created that add no appreciable latency.

Deposits are amortized over all the profits, so just like proof-of-stake they are not an improvement.

Sorry these issues are fundamental. This is the reason I abandoned the idea in 2013, when I was the first person to invent it. Now you copy the broken idea that I invented. Sigh.
(and make a lot of money selling a broken concept to n00bs that I was unwilling to do because I have ethics)

[super3]

"That a regular user has nearer to parity in terms of economies-of-scale cost efficiencies is irrelevant to the game theory economics that forces centralization in this case."
tldr; "I'm just going to ignore the economics because it doesn't help my case."

"More importantly, the pool can then lie and create numerous Sybils so that it can be paid multiple times for storing the same data only once. So Storj loses redundancy."
I missed the part where the pool found all the data, reassembled it in the right order, brute for decrypt it, and de-duplicated.

"Selecting on performance locks out the little guy, thus driving centralization. IPs can be created that add no appreciable latency."
Nope because the centralized farming(3 states away) is not going to beat the little farmer that happens to 3 blocks away.

[iamnotback]

Edit: after writing the following, I took a minute to review super3's GitHub page and I see he is a young man recently graduated and working on his M.S.. I don't wish to discourage him from working on projects & ideas he is passionate about. And I feel empathetic if what I have explained to him is disappointing. Being I am 51 years old, I have a bit more experience than he does, so I am more apt to see wherein a technical idea is fundamentally flawed. I wish him all the best and it is unfortunate that we got entangled on this issue.

"That a regular user has nearer to parity in terms of economies-of-scale cost efficiencies is irrelevant to the game theory economics that forces centralization in this case."
tldr; "I'm just going to ignore the economics because it doesn't help my case."

Liar. That is sleazy to use argumentation that tries to spin what I wrote. State your economic argument instead.

Your tldr; is incorrect, because my point was that it was irrelevant because of the following...

Let me put in simpler terms for you. Google will simply take all your business (hiding behind Sybils with their vast economies-of-scale near hydropower, etc) and say thank you very much. You will have accomplished nothing.

"More importantly, the pool can then lie and create numerous Sybils so that it can be paid multiple times for storing the same data only once. So Storj loses redundancy."
I missed the part where the pool found all the data, reassembled it in the right order, brute for[ce] decrypt it, and de-duplicated.

The pool just routes the requests. It is just a pass-through mechanism. You seem to keep forgetting that latency is not a viable proof of Sybils. I made that point very early in this thread.

"Selecting on performance locks out the little guy, thus driving centralization. IPs can be created that add no appreciable latency."
Nope because the centralized farming(3 states away) is not going to beat the little farmer that happens to 3 blocks away.

Nonsense for several reasons (which I will let you figure out*). Latency is a non-starter. Have fun with your clusterfuck.

* One hint is that a multi-furcating network is always more efficient than a mesh network. That is why we won't run a water main directly to each household.


P.S. if you continue to resort to sleazy tactics of discussion, I will just ignore you and let you go on Dunning-Kruger confident to your foolish failure.