Bitcoin Forum
December 08, 2021, 02:00:15 PM *
News: Latest Bitcoin Core release: 22.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 [16] 17 18 19 20 21 22 23 24 25 »  All
  Print  
Author Topic: Sia - Siafund Redemption Deadline: June 1st, 2015  (Read 68528 times)
Taek
Hero Member
*****
Offline Offline

Activity: 543
Merit: 501



View Profile
September 03, 2014, 04:53:55 PM
 #301

Astor, if you are interested, we'd be willing to interview you and see if you are a match for the team.

The protocol is largely incomplete, where in many places we've chosen the 'proof-of-concept' solution as opposed to the most optimal solution, because there's a lot of work that needs to happen before Sia is a fully optimized protocol. If you have a strong coding background (in any language), we'd be intereted in having you as an employee.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1638972015
Hero Member
*
Offline Offline

Posts: 1638972015

View Profile Personal Message (Offline)

Ignore
1638972015
Reply with quote  #2

1638972015
Report to moderator
1638972015
Hero Member
*
Offline Offline

Posts: 1638972015

View Profile Personal Message (Offline)

Ignore
1638972015
Reply with quote  #2

1638972015
Report to moderator
1638972015
Hero Member
*
Offline Offline

Posts: 1638972015

View Profile Personal Message (Offline)

Ignore
1638972015
Reply with quote  #2

1638972015
Report to moderator
Taek
Hero Member
*****
Offline Offline

Activity: 543
Merit: 501



View Profile
September 03, 2014, 07:07:15 PM
 #302

Going back to the random distrubution problem (it was bothering me enough):

The solution is to forcibly randomize one non-joining person in every single quorum. This is going to be super hand-wavy, but the overall gist is this:

When you join Sia, you are put in a random place where someone already exists. In addition, the quorum that you join has 1 additional person swapped out with someone random in the network.

If you control 1% of the network, and 50.5% of the quorum, all your random trials will only have you maintain 50.5% control of the quorum:

You roll the dice until you get a hit on your target quorum. Your target has 1 removed (the guy you replace), which has a 50% chance of being your own. Then one more is removed randomly. So in total, 2 are removed (-1.01 total for you), and you get +1 (because you are guaranteed to have hit the target quorum), and then you get +0.01 from the new guy (because the new guy has a 1% chance of being you; you control 1% of the network). So essentially by doing this, we prevent someone with 1% of the network from every being able to target a quorum.

What about 33%?

This equilibrium is around 2/3. You gain +1, then you lose 2 random. At 2/3, you lose an expected 4/3 quorum members. But then when bringing in the extra new guy, you have a 1/3 chance of it being yours. So the equilibrium allows a person at 33% to hold a 66% majority on a target quorum, though it's expensive to keep rolling to stay that high. At this point, files are at risk of being corrupted, because 66% is enough to corrupt files stored at the standard network value.

What about 50%?

The equilibrium is 75%, which still isn't enough to hit the 80% that's required to disrupt meta-quorum consensus. Though it's pretty uncomfortably close, requiring only a bit of luck to get to the 80%. Even at 80% though, you still only have a fraction of a percent of a chance of pulling off a corruption attack.

An even safer alternative is to swap out 2 random siblings for each new sibling that enters the network. This is more expensive to the quorum, and results in more overall bandwidth being used, but your equilibrium values fall a lot farther.

1% attack struggles to get past 1/3.
33% attack struggles to get past 55% (which is now ~safe from file corruption attacks)
50% attack struggles to get past 66% (now very safe from meta-quorum disruption)

=====

On the other end of the equation, you can try to kick out honest hosts from a quorum to stack it in your favor. But we have a convenient weapon: you can't kick out arbitrary honest hosts, you have to kick out _all_ honest hosts. It's an all-or-none procedure. When hosts are kicked from the network, you can replace them using the same entry process described above to keep the equilibrium values at the same points.

This does get a little messy, because a malicious party can keep kicking out full sets of honest hosts each time they get the majority in a quorum. That will add lots of load to the network, and is an effective DOS strategy, which we don't have a good solution to yet. Swapping out hosts is expensive, and it's one thing when the new guy is paying for it (like in the first situation), but it's a harder problem when you are kicking 20% (or 45%) of hosts who produced an honest block. Do you charge the whole quorum evenly (for not being well connected?)? You can't be sure which set is dishonest and you have to go with majority. So who pays for turbulence, or how can you force the two honest blocks to merge without introducing more error points into the consensus algorithm?

There might be a way to force multiple honest blocks to merge, but I'm highly wary of this approach because it could also be abusable in a way that results in forked quorums.

=====

I'm glad you brought this up, because our random distrubution algorithm did need a second look.
Taek
Hero Member
*****
Offline Offline

Activity: 543
Merit: 501



View Profile
September 03, 2014, 08:22:37 PM
 #303

Apologizing for the triple post, but I'm happy to announce that we're ready to release the alpha v2!

We believe that we've sorted through all of the major issues, although file sizes are limited to 768kb, and you're not going to be able to be a participant on the network unless you have a public IP address or have manually set up port forwarding on your router. It's also linux only.

You can download the binary at siacoin.com/download.html, or fork the 'release' branch of our github repo.

It's definitely rough around the edges but everything seems to work without any major errors.

Let me know what you guys think and feel free to ask for help. I'll be poking my head into the IRC channel every now and then over the next few weeks, but the forum is still probably the best way to get my attention.
msin
Legendary
*
Offline Offline

Activity: 1414
Merit: 1002


View Profile
September 03, 2014, 09:01:13 PM
 #304

Great, thanks for the update.
Taek
Hero Member
*****
Offline Offline

Activity: 543
Merit: 501



View Profile
September 10, 2014, 07:54:34 PM
 #305

A different update. I completely reworked the way that Sia handles files. This means we're going to need a completely new whitepaper, though only a few of the major concepts have changed.

The biggest benefit is that the total amount of bandwidth required to make Sia work has dropped dramatically. A downside is that now, when you add value to a file (for storing it), you can't remove it ever.

On the other side of things, we're working on building an ncurses-like interface for Sia, because the command line is pretty annoying. I'm hoping that'll be done soon, and we'll have an alpha 2.1, where nothing substantial has changed but at least there's an easier-to-use frontend.

I'm not sure when I'll have the next whitepaper ready. Maybe 2-3 weeks?
Djinou94
Legendary
*
Offline Offline

Activity: 1036
Merit: 1000


View Profile
September 29, 2014, 01:39:43 PM
 #306

Cannot wait for the new whitepaper
marek3ball-orig
Member
**
Offline Offline

Activity: 80
Merit: 10


View Profile
October 01, 2014, 04:11:04 PM
 #307

Some good news? Storj is running fast Smiley.
mladen00
Legendary
*
Offline Offline

Activity: 2127
Merit: 1013


K-ing®


View Profile
October 02, 2014, 05:19:48 AM
 #308

interesting concept

IOTA
Tobo
Hero Member
*****
Offline Offline

Activity: 763
Merit: 500


View Profile
October 02, 2014, 04:04:43 PM
Last edit: October 02, 2014, 04:23:35 PM by Tobo
 #309

CfB (Nxt core developer) is working on a distributed computing project. It might be able to help Sia on the efficiency of network computing.

- Jinn is a general purpose processor based on ternary logic with hardware support for distributed computing. This project also includes the Jiniri Limited (single-node) and Jiniri   Unlimited (multi-node) emulators, in addition we are creating a monetized proof-of-concept game.

- http://www.jinnlabs.com/
Breasal
Hero Member
*****
Offline Offline

Activity: 586
Merit: 500


View Profile
October 03, 2014, 02:57:36 PM
 #310

CfB (Nxt core developer) is working on a distributed computing project. It might be able to help Sia on the efficiency of network computing.

- Jinn is a general purpose processor based on ternary logic with hardware support for distributed computing. This project also includes the Jiniri Limited (single-node) and Jiniri   Unlimited (multi-node) emulators, in addition we are creating a monetized proof-of-concept game.

- http://www.jinnlabs.com/

Wow, this would be incredible if Jinn and Sia could collaborate! It seems that both the tech and their level of "coolness" as disruptors make them a pair made in heaven imo.    Cheesy
msin
Legendary
*
Offline Offline

Activity: 1414
Merit: 1002


View Profile
October 03, 2014, 03:46:16 PM
 #311

CfB (Nxt core developer) is working on a distributed computing project. It might be able to help Sia on the efficiency of network computing.

- Jinn is a general purpose processor based on ternary logic with hardware support for distributed computing. This project also includes the Jiniri Limited (single-node) and Jiniri   Unlimited (multi-node) emulators, in addition we are creating a monetized proof-of-concept game.

- http://www.jinnlabs.com/

Wow, this would be incredible if Jinn and Sia could collaborate! It seems that both the tech and their level of "coolness" as disruptors make them a pair made in heaven imo.    Cheesy

Jinn is a very ambitious project, and if successful, many crypto projects will be using it (Ethereum, NxtAT, etc..)
Taek
Hero Member
*****
Offline Offline

Activity: 543
Merit: 501



View Profile
October 03, 2014, 03:48:53 PM
 #312

I'll be in touch with CfB soon.

I've been looking at our whitepaper and thinking that it might need to be spread into multiple papers. There are a lot of core concepts that don't depend on each other. It's intentionally designed this way to keep complexity low, which is critical for a cryptosystem, but that also means that we can probably split Sia into multiple papers, and this should really help with the credibility of the system as a whole. Additionally, it's a lot easier for someone to read a single paper about Proof-of-Capacity than it is for someone to read the entire Sia paper, especially if they're only interested in the capacity piece.

I've also been reading a bit about Peter Todd's treechains concept, and it seems like that might be a valuable approach to doing things. Treechains are still early on in life but it might be a big step forward for cryptocurrency. As Peter Todd said: (paraphrasing) "Bitcoin is like the telephone. The infrastructure is very smart and the applications are dumb. Treechains are like the Internet. The infrastructure is very dumb and the applications are very smart." Having a much less inherently useful yet much more flexible infrastructure sounds very valuable.
Breasal
Hero Member
*****
Offline Offline

Activity: 586
Merit: 500


View Profile
October 04, 2014, 06:44:06 AM
 #313

https://forum.thesupernet.org/index.php?topic=163.0;topicseen

Tobo
Hero Member
*****
Offline Offline

Activity: 763
Merit: 500


View Profile
October 06, 2014, 12:20:51 AM
 #314

I've also been reading a bit about Peter Todd's treechains concept, and it seems like that might be a valuable approach to doing things. Treechains are still early on in life but it might be a big step forward for cryptocurrency. As Peter Todd said: (paraphrasing) "Bitcoin is like the telephone. The infrastructure is very smart and the applications are dumb. Treechains are like the Internet. The infrastructure is very dumb and the applications are very smart." Having a much less inherently useful yet much more flexible infrastructure sounds very valuable.

This treechains concept sounds like your guys' quorum consensus. Are they similar to each other?
Taek
Hero Member
*****
Offline Offline

Activity: 543
Merit: 501



View Profile
October 06, 2014, 04:30:35 PM
 #315

Treechains are not particularly similar to the quorums that Sia builds, Sia's quorum's [were] built out of a paxos-like message passing system. Treechains are a bunch of merge-mined POW blockchains that follow some special rules to build a tree.

Unforturnately everything that I thought was working for quorums is still broken. The problem is that when you have hundreds or millions of quorums, an attacker with 50% strength will be able to compromise at least a handful of quorums in the 70-80% range, and the attacker can pull successful attacks on those. It'd be easy to solve that problem if you could provide a proof-of-public-data, but the only way to prove that is to have the data yourself. Otherwise you're trusting whoever says the data is public or not public, and you'd have to go with the majority, which sometimes is an attacker.

We could weaken our assumptions to 33% global corruption of storage, and then increase the quorum size, and then it's much less likely for an attacker to break 51% strength in any quorum. But then you have failure modes that might destroy everything. IE if an attacker does manage to break 51% control of a particular quorum, he'll be able to continually inflate (by lying) the amount of storage he has, and eventually take 51% control of every quorum.

The source of the problem comes from the fact that not every node knows about every transaction. With a file storage system, you have to be able to handle many, many, many transactions, or have some way to allow the system to change state without using a transaction. But if you're doing proof-of-storage, and someone edits a file, the rest of the network has to know that the file has been edited so that they can change how they read a storage proof. Therefore, I don't think it's possible to edit a file without having a transaction. If most of the files are computer backups or large static files (videos, music, etc.), it might be okay but the real goal is to support files like web pages. Ideally you'd never need an HDD again (an SDD + large amount of RAM would be sufficient).

So I guess we have two options then:

1. Everyone sees every transaction
2. Not everyone sees every transaction

In the case of 1, the amount of transactions would need to be limited, and you'd have fees and they'd get pretty large. Something like this could potentially be done right on top of Bitcoin. This would probably only be useful for large _and_ important files. I don't really see Netflix using something like this, but it could be a worthwhile alternative to bittorrent (which suffers from the same problem of having files that are very hard to edit).

In the case of 2, you have to have some way of knowing that everything is okay, even though you can't see all of the transactions. The general strategy is to have a large but finite number of people verifying each piece of information, and you randomly assign who gets to verify what. You have some metric for protecting against sybil attacks during your assignment. (storage in our case). The problem is that if you're relying on something like a blockchain, you need to assume that each subset of verifiers is able to stay honest.

Consider the situation where 51% are dishonest. They are the majority, and so they can tell the network that things look like X. The network doesn't know, so it just has to accept X as true. The 49% honest people complain, saying that things actually look like Y. The dishonest group can be completely dishonest (change balances and such), because they don't ever tell anyone what Y is (Y is just a hash to keep things scalable). They just keep it to themselves and successfully corrupt the network.

So say you give the honest hosts some tool to claim that Y doesn't exist and is a made up hash. Then, in honest quorums you can have a dishonest set of hosts claim that the honest hosts are releasing data which doesn't exist. The network now needs to verify somehow that the honest hosts really have made the data public and aren't actually dishonest hosts in disguise. The only way to do this is for the network to download all of the data and verify for itself that the data exists. But now dishonest hosts will always claim that they can't see the data, and the rest of the network is always forced to download the data to verify that it's publicly available. So you end up with a situation that's not any different from a single blockchain.

Moon math aside, there doesn't seem to be any way around this. That's where Sia is stuck right now. The quorum's in the currently available whitepaper suffer from a related problem. Someone who manages to take control of >50% of a large number of quorums can start kicking out honest hosts, which will result in fines for the honest hosts and also result in a higher percentage of the hosts across the network as a whole being dishonest. Sia becomes pretty vulnerable once someone gets around 33% of the network.

=======

So let's say that we accept our limitations of 33%. What happens when someone breaks 33%? Unfortunately, they get a lot of power. In Bitcoin, when someone breaks 50% they control the blockchain for a while, they can attempt double spends and they can block transactions, but they can't spend other people's money. But in a network where the majority of hosts don't see all of the data, someone who manages to break a portion of the data CAN spend money from other wallets. They just lie about the state of the network, because they don't have to provide signatures of accuracy.

Even if we were able to prove that the dishonest hosts hadn't dishonestly spent a wallet's money (eg without a signature) (this proof could be done using snarks/scip), the dishonest hosts could still refuse to reveal what data had made it into the blockchain. The biggest issue here is that honest transactions could make it into the blockchain, and while the snarks would verify that everything was legal, honest hosts might not be able to know if their transactions had gone through. A dishonest party could prove that they had the network in a legal state without ever revealing what that state is. So the rest of the world just sees this giant black void that they can't reason about or make transactions through. The wallets within it might as well not have any coins.

 Undecided

=======

So, what do we do? I'm going to keep looking at this but it doesn't seem like highly scalable storage is going to work in a decentralized way on today's technologies. I don't think it's reasonable to assume that nobody will ever get to 33% control (even without pools) because Bitcoin has a single factory that's got something like 25% of the mining power. Sia might fall to the same thing. It would be highly desirable to have some sort of response to the failure mode, such that if the honest set of people once again got above 67%, the network could quickly restore to a functioning, fully informed state.

Another big issue with proof-of-storage-for-consensus is that it's not progress free. You have to announce that you have the storage, you have to download files (and people have to be willing to upload the files to you - another issue we aren't certain how to solve). A malicous party controlling the network can just refuse to accept you as a contributor, and it won't matter how much storage you have because you will be ignored. This doesn't happen on Bitcoin because any random person can submit a proof of work. In Sia, this isn't currently possible.

=======

Dark days for Sia =/. But filecoin is also broken, permacoin is not efficient, maidsafe is opaque and very difficult to audit, and storj is underspecified. Cryptography is very hard, something I didn't properly appreciate until the past month or two. But Sia is not dead yet, we're just back to square 1 for the time being. The money hasn't run out yet and we won't give up until it does. Our total expenses since doing the crowd sale have been about 1/2 of what we raised. (through the crowd sale. We also have venture funding although that's in a completely different pool of money. Our responsibilities following the crowd sale is to make Sia, our responsibilities following venture funding is to make Nebulous Inc. a succesful corporation).

In the worst case, I think we will be able to create a less powerful system for peer to peer storage. Instead of selling storage to a broader network, you'd sell storage to individual peers. The peers would need to be responsible for picking reliable and honest hosts, which they could achieve by using reputation, randomness, and redundancy.

 Huh sorry guys I hope the next major update has more positive news. As always, happy to walk you through anything and explain exactly what's broken. Also happy to highlight questions you have about other systems, though it's very time consuming to take a close look.
msin
Legendary
*
Offline Offline

Activity: 1414
Merit: 1002


View Profile
October 06, 2014, 05:20:13 PM
 #316

Disappointing to hear, hope you can find a solution.
Tobo
Hero Member
*****
Offline Offline

Activity: 763
Merit: 500


View Profile
October 06, 2014, 07:30:33 PM
Last edit: October 07, 2014, 11:23:22 AM by Tobo
 #317

Thanks for the update. Never thought it would be easy for doing something like this.

Anyway, there is an interesting article - http://www.coindesk.com/ibm-executive-block-chain-internet-of-things/

Another one - http://www.wired.com/2014/10/overstock-com-assembles-coders-build-bitcoin-like-stock-market/
brooklynbtc
Sr. Member
****
Offline Offline

Activity: 336
Merit: 250

AKA jefdiesel


View Profile
October 06, 2014, 08:52:58 PM
 #318

Disappointing to hear, hope you can find a solution.

Agreed, but to be honest this is not the worst thing I could hear. Hope you guys stick with it and see what you can pull out of the ashes.

SN
S   U   P   E   R    N   E   T
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀   
Uniting cryptocurrencies, Rewarding talent, Sharing benefits..

Blockchain Technology.

Taek
Hero Member
*****
Offline Offline

Activity: 543
Merit: 501



View Profile
October 06, 2014, 09:16:23 PM
 #319


That's an interesting article. Why would you ever want to open your house via a blockchain? Why wouldn't you just use a private key and do it _not_ on a blockchain? Or is there value to being able to scan the block history and know that no unauthorized actions have ever succeeded? I think that some of these examples are completely unnecessary though.

=========

Been thinking about alternative ways to construct a system like Sia. One option that was brought up was to have many - thousands - of parallel blockchains that are all merge mined with bitcoin. Each blockchain would manage it's own quorum. The challenge would be trying to figure out how to make sure that each blockchain sufficiently composed of random individuals. They couldn't do anything illegal or hidden, but it would be much easier for an attacker to compromise a large percent of a blockchain. Perhaps you'd have a single 'master' blockchain which managed hosts only, sending hosts of various keys to different blockchains. I'll be looking into this further, but it seems like a possible option.

One way would be to have a single blockchain. I believe this is similar to what storj is doing. Transaction fees would likely be very high, and the blockchain would likely be very heavy. But some decentralized storage is better than no decentralized storage. Proof of upload might be tricky, but that should be the only issue. All the other progress we've made with Sia should be able to apply here.

Finally, we could look at Peter Todd's tree chains and see if they could apply. Treechains are in the baby stages of a protocol, it's nothing that could be defined as usable yet. We could try to accelerate that and turn it into something useful, but I don't think we'd have anything ready any time soon.

The other, _other_ finally is that we could build a centralized system that employs all of the components we've talked about. Proof of upload would be really easy in the centralized case (you just send it to our servers, and then you've proven the upload!), and we'd just apply proof of storage, proof of capacity, and some form of transparent quorum-level consensus that gets validated through a centralized node. This is obviously not a decentralized file system, but it would operate in the same way that Sia originally was going to operate.

We'll be considering all four throughout the week and let you know what happens.
Tobo
Hero Member
*****
Offline Offline

Activity: 763
Merit: 500


View Profile
October 07, 2014, 12:58:18 AM
Last edit: October 07, 2014, 02:14:15 PM by Tobo
 #320

The 4th option makes sense to me. There is no necessary for a storage system to be decentralized as long as it is distributed to lower cost and increase the efficiency.

You always can make it better and more automated and more decentralized when the new technology is available in the future.
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 [16] 17 18 19 20 21 22 23 24 25 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!