Blockchain + DHT = secure email-like messaging.

[tradeaway]

I think you all folks agree that some sort of ubiquitous protected messaging solution which cannot be wiretapped by design is probably what the world needs right now. After recent events it becomes clear that this is not a matter of privacy per se, this is the matter of a better society structure, in which no group of people is able to control society, by having access to basically all information it wants.

Naturally there are tons of privacy tools right now, but really a few of completely decentralized. Decentralization is  crucial to system security on the basic design level.

Naturally Bitmessage is very promising, but my personal gripe with it that it doesn't provide the full email functionality, by deleting messages from block-chain within two days. Also I fear that the use of the blockchain to store all the messages in the system could be not the most optimal design option. In Bitcoin you have only transaction stored in the chain, in my opinion it corresponds to storing in the blockchain only the message sender and recepient address, but not the whole message.

Returning to the email functionality, it seems that one of the most solid points in favor of traditional email is that you store messages for indefinite period of time, and they can be fetched from POP server again if you erase them locally. In spite of all spam problems this is what keeps SMTP protocol afloat with no immediate demise in sight.

So the question arises, if you don't want to store the messages in block-chain, where do you want to store them?

I propose using the old and proven solution of distributed file storage used in Bittorent protocol. In this case blockchain would contain the sender and receiver respective addresses, and the message hash.  When a receiver wants to check the messages sent to her she extracts the hashes of the messages, and does a hash based search through the network. There are established algorithms for this  such as DHT in Bittorent (magnet links) or Kademlia.

So the process of message transfer would be the following:
- Receiver shares its public key
- Sender fetches it, encrypts a message, adds the address of the sender and puts the block containing both addresses and the message hash in the block chain.
- Clients responsible for storing messages in a certain hash range download the message (or its parts) from the sender and store the encypted message.
- When a receiver wants to read the message it obtains its hash from the chain, and finds the peers storing it. After that it downloads the message and decrypts it with its private key.

Naturally tons of technical details are dropped from this simple description, such as provision of permanent message storage, handling huge messages, coping with abusing peers and such.  But hopefully the idea is clear, you use block-chain as the message register, and distributed hash tables as a tool to store the message in a distributed way.  You obtain bullet-proof system able to store messages indefinitely, which cannot be tapped without access to the receiver private key.

Also additional functionality can be easily added, for example  if sender doesn't want to store the message indefinitely  it can be dropped from the system.  Also it's quite obvious that similar set-up can be used for distributed file storage. It's also possible to monetize the system by imposing a fee on large file transfers and their storage.

The main advantages of the approach are:
-Manageable blockchain
-Using cheap storage for using the message data, and using the "expensive" storage (blockchain) for storing only the message information.
-Web-interfaces can be easily constructed, analogous to traditional webmail.
-We get more or less usual email functionality without messing with PGP and key exchange.

We would like to start open-source project along these lines, and welcome everyone interested to join us.  Also any comments on the system design will be greatly appreciated.

[Jaxkr]

Naturally Bitmessage is very promising, but my personal gripe with it that it doesn't provide the full email functionality, by deleting messages from block-chain within two days. Also I fear that the use of the blockchain to store all the messages in the system could be not the most optimal design option. In Bitcoin you have only transaction stored in the chain, in my opinion it corresponds to storing in the blockchain only the message sender and recepient address, but not the whole message.

I don't think you completely understand how Bitmessage works. It doesn't use a blockchain in any way. The nodes store everything they've seen in the past two days, and serve it when requested. Data older than 2 days is automatically deleted by default. However, the client still retains all of your messages and received broadcasts on local storage.
If your message isn't acknowledged by the receiving party within the 2 day window, it is resent. So it will be received even if they don't come online for a while.

Also I fear that the use of the blockchain to store all the messages in the system could be not the most optimal design option.

If Bitmessage in it's current state becomes really big, you're completely right. But this is being planned for with the use of streams. This splits addresses into multiple networks for scalability.

[tradeaway]

All right I have to check on that, but anyway you can't store messages more than two days. So bitmessage is a great idea but in functionality it is somewhere closer to slow chat service than to email.

[favdesu]

All right I have to check on that, but anyway you can't store messages more than two days. So bitmessage is a great idea but in functionality it is somewhere closer to slow chat service than to email.

what if I told you that chatting is just a faster e-mail?

[tradeaway]

I think what is really cool with email is the fact that your messages are not deleted. Otherwise it would have been replaced with IM long ago
Email is extremely old-fashioned and ridden with virtually unsolvable issues, such as spam. But it still exists

[tradeaway]

Started writing the white paper on the project.
Meanwhile I'd like to formulate the problems which the system aims to solve:

1. I want to be able to communicate securely with any counter party for which I know its public key, and our communication cannot be intercepted if the attacker does not have my private key and is able to carry out only polynomial-time computations.
2. My communication data can be accessed by me for indefinite period of time, if I wish so. To that end I don't have to store it locally.
3. I want to minimize unsolicited messages to my address.
4. I want to be able to make general announcement any party can subscribe to and to be able to subscribe to other peers' announcements.
5. I want to be a participant of a strictly peer to peer network, where all participants are equal.

This all may seem quite trivial, but no system with given qualities exist. It would a close cryptographically secure imitation of traditional email

[btharper]

Started writing the white paper on the project.
Meanwhile I'd like to formulate the problems which the system aims to solve:

1. I want to be able to communicate securely with any counter party for which I know its public key, and our communication cannot be intercepted if the attacker does not have my private key and is able to carry out only polynomial-time computations.
2. My communication data can be accessed by me for indefinite period of time, if I wish so. To that end I don't have to store it locally.
3. I want to minimize unsolicited messages to my address.
4. I want to be able to make general announcement any party can subscribe to and to be able to subscribe to other peers' announcements.
5. I want to be a participant of a strictly peer to peer network, where all participants are equal.

This all may seem quite trivial, but no system with given qualities exist. It would a close cryptographically secure imitation of traditional email

1. This is a given for any private messaging system, it's also generally a more easily solved issue given existing systems like PGP or more generally with most modern asymmetric crypto.
2. An admirable trait, but to avoid storing it locally while keeping it accessible means that it still needs to be stored somewhere. Generally this is easiest when done by a central server or something of the sort, but to keep it decentralized means that someone else has to store their messages, your messages, as well as everyone elses. Not only that but since you want to be fault tolerant of a single peer leaving it has to be stored across multiple peers. All this storage also has to be pushed onto people that derive no inherent benefit from storing your unreadable messages. And how long is indefinitely? Until the account is inactive for a certain period (this also opens the can of worms that everyone has to know if the identity is active), until the account asks to be removed (if ever?), until someone runs out of storage space (highly random)? This is likely a large portion of the reason that bitmessage only keeps things for 2 days.
3. Spam is the bane of any messaging system, but to allow for number 1 anyone with your key must be able to message you. Generally publicly publishing your private key is the simplest way to allow others to communicate with you. To preserve secrecy of identity they have to be able to send it without disclosing too much information about who they are until you receive their message. All these together (and with number 2) mean that you're open to several different DDoS attacks (sending a large number of messages for you to decrypt and check if they're worth receiving, even if all you end up doing is running against a spam filter. Storing spam mail. Manually tuning a spam filter of some sort. Transmitting spam for yourself and others. And I would guess several others not mentioned here).
4. This generally sounds like a useful feature, but be care has to be taken with implementation to avoid possible abuses. Very possible though.
5. This just makes all the rest of it more difficult as you have to make sure other people are willing to volunteer the resources, requires certain tolerances to peers leaving the swarm, and means that you can't depend on a lot of the nice features a central server allows you (while avoiding the problems a central server creates of course).

All that said, you may want to see if an existing setup would work just as well, throwing the blockchain idea at everything doesn't always help anything at all. One existing setup that you may want to have a look at is Freenet (a P2P censorship resistant, data storage network where data can include text messages of course) and the various forum implementations therein (FMS, Freetalk, Frost, and the email-like plugin that I forget the name of). They're all fairly different but may help you skip past past issues they've already encountered and build upon areas they've had success in. Some of the things in particular to take away are tradeoffs in storage, availability, duration of messages, and privacy. Web of trust may also be a useful thing to look at, and is featured in most of the forums.

Naysaying aside, good luck if you go forward.

[tradeaway]

Great post!
Yes it's all very generic, what makes my proposal reasonable (hopefully) is the combination of features. Once again, I want to build a secure messaging analogous to email with all its features but without any centralization and asymmetric encryption ingrained in the system.

1. You have to exchange keys with current PGP setup. Also you need to have SMTP and POP servers to actually process your messages. If they want to mess with your messages they can.
The proposal is to use public keys as receiver addresses, so no keys should be exchanged. And build a strict peer-to-peer system.
2. The problem with storage can be solved more or less as it's sometimes done with Bittorent private trackers - if you wanna use the system you have to give back, that is you have to store messages too. People store tones of multimedia files at their home PC's, probably storing 100 mb 1 GB of data is all right. Bitcoin block chain is 6 GB now.
Another way to approach this is the nodes who store the data for a fee, this also would be a way to monetize the system. For example heavy users would be required to pay a fee through bitcoin which will be redistributed between storing nodes. 
3. The way to approach the spam problem would be PoW, analogous to the way its made in Bitmessage. Also a peer would be allowed to block messages from certain nodes. in this case the messages will be dropped from the system
4. This is more like crypto twitter. Actually the spam problem won't be so drastic here since you need to know the public key of the node if you want to subscribe to its messaging stream, it can't message you directly.
5. Appropriate DHT setup is able to solve the problems of peers going offline, as it is proven in practice by various P2P systems. As for central servers - I believe that complete decentralization is the way to go, Internet is inherently decentralized system which certain parties always try to centralize.
There have to be countermeasures to this. Just imagine a sheer amount of private information any webmail provider has access to. I'm not sure that this is the ideal setup of human communication, when a third-party always has access to anything it wants. It creates a new hierarchy, which actually is not necessary, all communication CAN be peer-to-peer indeed.

Freenet is cool but it's more file sharing application. Also there won't be an explicit blockchain here, everything can be done through correct DHT setup.

once again thank you for very insightful comments!

Started writing the white paper on the project.
Meanwhile I'd like to formulate the problems which the system aims to solve:

1. I want to be able to communicate securely with any counter party for which I know its public key, and our communication cannot be intercepted if the attacker does not have my private key and is able to carry out only polynomial-time computations.
2. My communication data can be accessed by me for indefinite period of time, if I wish so. To that end I don't have to store it locally.
3. I want to minimize unsolicited messages to my address.
4. I want to be able to make general announcement any party can subscribe to and to be able to subscribe to other peers' announcements.
5. I want to be a participant of a strictly peer to peer network, where all participants are equal.

This all may seem quite trivial, but no system with given qualities exist. It would a close cryptographically secure imitation of traditional email

1. This is a given for any private messaging system, it's also generally a more easily solved issue given existing systems like PGP or more generally with most modern asymmetric crypto.
2. An admirable trait, but to avoid storing it locally while keeping it accessible means that it still needs to be stored somewhere. Generally this is easiest when done by a central server or something of the sort, but to keep it decentralized means that someone else has to store their messages, your messages, as well as everyone elses. Not only that but since you want to be fault tolerant of a single peer leaving it has to be stored across multiple peers. All this storage also has to be pushed onto people that derive no inherent benefit from storing your unreadable messages. And how long is indefinitely? Until the account is inactive for a certain period (this also opens the can of worms that everyone has to know if the identity is active), until the account asks to be removed (if ever?), until someone runs out of storage space (highly random)? This is likely a large portion of the reason that bitmessage only keeps things for 2 days.
3. Spam is the bane of any messaging system, but to allow for number 1 anyone with your key must be able to message you. Generally publicly publishing your private key is the simplest way to allow others to communicate with you. To preserve secrecy of identity they have to be able to send it without disclosing too much information about who they are until you receive their message. All these together (and with number 2) mean that you're open to several different DDoS attacks (sending a large number of messages for you to decrypt and check if they're worth receiving, even if all you end up doing is running against a spam filter. Storing spam mail. Manually tuning a spam filter of some sort. Transmitting spam for yourself and others. And I would guess several others not mentioned here).
4. This generally sounds like a useful feature, but be care has to be taken with implementation to avoid possible abuses. Very possible though.
5. This just makes all the rest of it more difficult as you have to make sure other people are willing to volunteer the resources, requires certain tolerances to peers leaving the swarm, and means that you can't depend on a lot of the nice features a central server allows you (while avoiding the problems a central server creates of course).

All that said, you may want to see if an existing setup would work just as well, throwing the blockchain idea at everything doesn't always help anything at all. One existing setup that you may want to have a look at is Freenet (a P2P censorship resistant, data storage network where data can include text messages of course) and the various forum implementations therein (FMS, Freetalk, Frost, and the email-like plugin that I forget the name of). They're all fairly different but may help you skip past past issues they've already encountered and build upon areas they've had success in. Some of the things in particular to take away are tradeoffs in storage, availability, duration of messages, and privacy. Web of trust may also be a useful thing to look at, and is featured in most of the forums.

Naysaying aside, good luck if you go forward.

[BadBitcoin (James Sutton)]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

[btharper]

Great post!
Yes it's all very generic, what makes my proposal reasonable (hopefully) is the combination of features. Once again, I want to build a secure messaging analogous to email with all its features but without any centralization and asymmetric encryption ingrained in the system.

1. You have to exchange keys with current PGP setup. Also you need to have SMTP and POP servers to actually process your messages. If they want to mess with your messages they can.
The proposal is to use public keys as receiver addresses, so no keys should be exchanged. And build a strict peer-to-peer system.
2. The problem with storage can be solved more or less as it's sometimes done with Bittorent private trackers - if you wanna use the system you have to give back, that is you have to store messages too. People store tones of multimedia files at their home PC's, probably storing 100 mb 1 GB of data is all right. Bitcoin block chain is 6 GB now.
Another way to approach this is the nodes who store the data for a fee, this also would be a way to monetize the system. For example heavy users would be required to pay a fee through bitcoin which will be redistributed between storing nodes. 
3. The way to approach the spam problem would be PoW, analogous to the way its made in Bitmessage. Also a peer would be allowed to block messages from certain nodes. in this case the messages will be dropped from the system
4. This is more like crypto twitter. Actually the spam problem won't be so drastic here since you need to know the public key of the node if you want to subscribe to its messaging stream, it can't message you directly.
5. Appropriate DHT setup is able to solve the problems of peers going offline, as it is proven in practice by various P2P systems. As for central servers - I believe that complete decentralization is the way to go, Internet is inherently decentralized system which certain parties always try to centralize.
There have to be countermeasures to this. Just imagine a sheer amount of private information any webmail provider has access to. I'm not sure that this is the ideal setup of human communication, when a third-party always has access to anything it wants. It creates a new hierarchy, which actually is not necessary, all communication CAN be peer-to-peer indeed.

Freenet is cool but it's more file sharing application. Also there won't be an explicit blockchain here, everything can be done through correct DHT setup.

once again thank you for very insightful comments!

<Snipped for brevity>

While such a combination of features would be useful, I just think that they may be impractical together.

1. I'm not sure what you mean here, if a server alters an encrypted message it (should be, depending on the encryption scheme) is rendered unreadable. It's possible to cause the message to be unreceivable, but not to alter a message after encryption during transit. If you mean MITM because of poor key exchange, then that's a fundamental problem with key distribution not PGP in particular. The same problem (having to make sure key exchange is secure) would exist in any scheme.
2. Bittorrent users derive direct benefit from storing and sharing material. On public trackers you have the perpetuated availability of material the peer wanted in the first place. On a private tracker it maintains ration in addition to the previous. While there is also altruism (which would be all I can see this relying on), it's not infinite, and given only 100mb (or any finite quantity) per peer you will still eventually run out of space. The blockchain is useful to anyone who stores it because that's how you prove what has happened and already there a large number of "light" nodes that only store the block hashes and verify things separately as needed and don't store many blocks at all as well as several implementations (including the satoshi client) that are looking to prune the spent tx data to reduce the storage requirement (pruned data would be ~120 MB right now if memory serves). For larger file storage clients like Freenet (while file storage you can just think of files as binary messages (see: Usenet file sharing)) the incentive is that you store some extra data so others will save your extra data, though no strict parity is enforced and there is a certain reliance on altruism (and for this reason I don't think this sort of project is anywhere close to impossible, just harder). Since I brought up Usenet, there are basically superpeers that act as peers with each other and central servers for end users; end users normally pay for large data storage provided by central servers.
3. PoW does help, but the higher you make it the harder (relatively) it is for a normal user to send a message, the lower you set the bar the easier it is to spam. So while there is a tradeoff but it's definitely a solid place to start; allowing a user to establish how hard it is to send them messages might work. Rejecting messages from a node may be somewhat meaningless, you want to allow new users to be able to participate reasonably quickly and there's no way to select between a new user and a new identity for a malicious node. In addition to all that routing messages through the p2p portion means the physical source node might not be important, the identity of the sending node is all you can conserve on a practical level.
4. I was thinking this was along the lines of a PM system with optional twitter-like broadcasts, but if it's twitter and includes direct messaging a few things shift priority, but overall allowing any messaging TO an identity opens up all the same problems if it's the most common behaviour or not. Sounds kind of like most of the forums on Freenet though, everyone publishes their board posts to their personal broadcast stream and based on your direct trust level or indirect trust through WoT you choose whether or not to download their messages. The expensive part of creating a new identity is bootstrapping yourself into the network and announcing your presence so other people begin to download your messages to begin with. If you wanted to apply a similar mechanic at all you could start with a heavy PoW problem to announce your identity to other peers.
5. I'm used to seeing DHT as a lookup method, but you can easily tie that to looking up where to find/who to ask about a certain message certainly. I'd say the central servers archetype evolved before p2p was necessary or practical; if you wanted to share something you hosted it, if you wanted mail you sent it to the mail server (similar to the post office I suppose) and it was also easier to manage and there was no reason for it to go away. A lot of hosted platforms (especially closed source hosted sites like google, centralized mining pools, or anything requiring any central authority or a protected/hidden database to run) are either easier or only possible inside a centralized environment. Some things can be moved easily to a decentralized setup (file sharing from FTP to Bittorrent, central mining pools to P2Pool, etc) and some things are harder to move over (Anything abusable, anything that requires strong input checking, required hidden or protected databases). I'd also say that part of the reason webmail providers have so much information they don't need to have access to is that it's easier and established. There's no established default email or text encryption standard for transmission and the default is plaintext. If you could switch everything to encrypted transmission by default the webmail providers wouldn't have anything to begin with (other than sender/receiver and timing, which is also what the NSA was collecting on phone calls and is still powerful information and not eliminated in a p2p setup) but it's harder for end users, grandma doesn't care about PGP and grandma couldn't use PGP and wouldn't be able to use email if it were the default and the only option for her would likely be a webmail provider either receiving plaintext or decrypting it for her server side which means they have the contents anyway (you could do it client side, but then grandma has to carry her private key around and she can't do that either, also it's easier server side).

All told you've definitely addressed a lot of things, I'm having to insert a fair amount of qualifiers to things now, and a lot of problems can be considered corner cases (that I'd still personally like to see addressed in an active implementation, now is too soon for some of them before larger details are finalized). The biggest problem I still have is storage, I'm not sure the balance between indefinite holding and total potential storage is there yet, at some point it just runs out. Though I don't know that a two day window is enough for everyone either :-) If nothing else I can keep playing devil's advocate at least if you want to try to flesh things out more.

[btharper]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

[BadBitcoin (James Sutton)]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.

[btharper]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.

Any data stored in the blockchain is stored there forever, I would however argue that it's wasteful to do. There also wouldn't be any way to charge a fee for such a service except as a service to automate the task, anyone can submit a transaction and anyone can read the blockchain freely.

[BadBitcoin (James Sutton)]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.

Any data stored in the blockchain is stored there forever, I would however argue that it's wasteful to do. There also wouldn't be any way to charge a fee for such a service except as a service to automate the task, anyone can submit a transaction and anyone can read the blockchain freely.

currently thats true, however the blockchain is having some serious problems with bloat (as the multitude of threads can attest to) and one of the solutions is paying a few nodes who will have the rank 0 (or the current fullly downloaded blockchain) to keep the blockchain and keep archiving away.

The reason for this permanent storage is to allow unfettered access by any party wishing to see our digital contract for eternity, nothing is required to visit, its completely on public record.

https://en.bitcoin.it/wiki/Smart_Property

[btharper]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.

Any data stored in the blockchain is stored there forever, I would however argue that it's wasteful to do. There also wouldn't be any way to charge a fee for such a service except as a service to automate the task, anyone can submit a transaction and anyone can read the blockchain freely.

currently thats true, however the blockchain is having some serious problems with bloat (as the multitude of threads can attest to) and one of the solutions is paying a few nodes who will have the rank 0 (or the current fullly downloaded blockchain) to keep the blockchain and keep archiving away.

The reason for this permanent storage is to allow unfettered access by any party wishing to see our digital contract for eternity, nothing is required to visit, its completely on public record.

https://en.bitcoin.it/wiki/Smart_Property

I don't think anyone paying for blockchain access would be functional either in a practical sense or politically given the threat of centralization and the fact that how can I pay you coins from the chain without seeing the chain?

Running stuff through the script portion of the payment address has been theoretically possible from the beginning (though not enabled in the satoshi client I don't believe). Throwing more stuff on top of it won't improve the situation any.

If you were to do anything like that it's easier just to submit the hash of the document to the blockchain and host it yourself, if you go under there's no point to the contract anyway; or just use one of several legal services setup for that sort of thing who can do it much more effectively than shoving it into the blockchain ever could.

My guess for the eventual order of clients though is a large handful (or more) of rank 0 nodes who still do it for free, possibly governments and other larger bodies. Several well distributed copies held as small chunks of the chain across a large bittorrent-like swarm, and most people using light clients that only keep track of their information and a list of all header hashes back to the genesis block or wallet services (that would likely be based on a rank 0 node).

[BadBitcoin (James Sutton)]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.

Any data stored in the blockchain is stored there forever, I would however argue that it's wasteful to do. There also wouldn't be any way to charge a fee for such a service except as a service to automate the task, anyone can submit a transaction and anyone can read the blockchain freely.

currently thats true, however the blockchain is having some serious problems with bloat (as the multitude of threads can attest to) and one of the solutions is paying a few nodes who will have the rank 0 (or the current fullly downloaded blockchain) to keep the blockchain and keep archiving away.

The reason for this permanent storage is to allow unfettered access by any party wishing to see our digital contract for eternity, nothing is required to visit, its completely on public record.

https://en.bitcoin.it/wiki/Smart_Property

I don't think anyone paying for blockchain access would be functional either in a practical sense or politically given the threat of centralization and the fact that how can I pay you coins from the chain without seeing the chain?

Running stuff through the script portion of the payment address has been theoretically possible from the beginning (though not enabled in the satoshi client I don't believe). Throwing more stuff on top of it won't improve the situation any.

If you were to do anything like that it's easier just to submit the hash of the document to the blockchain and host it yourself, if you go under there's no point to the contract anyway; or just use one of several legal services setup for that sort of thing who can do it much more effectively than shoving it into the blockchain ever could.

My guess for the eventual order of clients though is a large handful (or more) of rank 0 nodes who still do it for free, possibly governments and other larger bodies. Several well distributed copies held as small chunks of the chain across a large bittorrent-like swarm, and most people using light clients that only keep track of their information and a list of all header hashes back to the genesis block or wallet services (that would likely be based on a rank 0 node).

hmm, I was under the impression that the blockchain bloat was getting pretty rediculous.

To be honest I think my contract(s) if I go ahead with what I hypothesise will be moderate at best, just for a 10-20 year investment I would like to have an extremely reliable/universially trusted storage medium for such a document, and I cannot think of any better than either the bitcoin blockchain or an altcoin like litecoin, or maybe all of them.

[btharper]

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

no, the conventional bitcoin blockchain, personally I'm interested in the "institutional rank 0 fee" for all people wanting to store data in the blockchain forever, a fee that would be paid monthly (variable depending on what the rank 0 clients think is fair), so that the blockchain can be used to store text information that could be extremely useful for both communication and as a legal "bulletin board" of contract information by using the script field in each tx.

Any data stored in the blockchain is stored there forever, I would however argue that it's wasteful to do. There also wouldn't be any way to charge a fee for such a service except as a service to automate the task, anyone can submit a transaction and anyone can read the blockchain freely.

currently thats true, however the blockchain is having some serious problems with bloat (as the multitude of threads can attest to) and one of the solutions is paying a few nodes who will have the rank 0 (or the current fullly downloaded blockchain) to keep the blockchain and keep archiving away.

The reason for this permanent storage is to allow unfettered access by any party wishing to see our digital contract for eternity, nothing is required to visit, its completely on public record.

https://en.bitcoin.it/wiki/Smart_Property

I don't think anyone paying for blockchain access would be functional either in a practical sense or politically given the threat of centralization and the fact that how can I pay you coins from the chain without seeing the chain?

Running stuff through the script portion of the payment address has been theoretically possible from the beginning (though not enabled in the satoshi client I don't believe). Throwing more stuff on top of it won't improve the situation any.

If you were to do anything like that it's easier just to submit the hash of the document to the blockchain and host it yourself, if you go under there's no point to the contract anyway; or just use one of several legal services setup for that sort of thing who can do it much more effectively than shoving it into the blockchain ever could.

My guess for the eventual order of clients though is a large handful (or more) of rank 0 nodes who still do it for free, possibly governments and other larger bodies. Several well distributed copies held as small chunks of the chain across a large bittorrent-like swarm, and most people using light clients that only keep track of their information and a list of all header hashes back to the genesis block or wallet services (that would likely be based on a rank 0 node).

hmm, I was under the impression that the blockchain bloat was getting pretty rediculous.

To be honest I think my contract(s) if I go ahead with what I hypothesise will be moderate at best, just for a 10-20 year investment I would like to have an extremely reliable/universially trusted storage medium for such a document, and I cannot think of any better than either the bitcoin blockchain or an altcoin like litecoin, or maybe all of them.

The total dataset is around 6GB and the working set is around 120MB (I hope I'm not just making those up), the biggest issue is that the rate of accumulation is increasing. Sticking extra stuff that doesn't need to be there inflates it and is generally a bad idea and bad etiquette. An altcoin designed for it might work, but then you're just asking other people to host your stuff for free. Throwing the blockchain at problems for fun doesn't solve them. Throwing the blockchain at things that aren't even problems is just silly. If remembering a contract for 10-20 years was a problem the whole world would be in chaos, hell a bank loan for 30 is downright normal. If you want something reliable/safe/trusted/etc go buy a safe with a built in filing cabinet or rent a safety deposit box, I doubt more than a handful of people outside the bitcoin community would trust the blockchain anyway.

[tradeaway]

While such a combination of features would be useful, I just think that they may be impractical together.

1. I'm not sure what you mean here, if a server alters an encrypted message it (should be, depending on the encryption scheme) is rendered unreadable. It's possible to cause the message to be unreceivable, but not to alter a message after encryption during transit. If you mean MITM because of poor key exchange, then that's a fundamental problem with key distribution not PGP in particular. The same problem (having to make sure key exchange is secure) would exist in any scheme.

Yes, exactly, I mean key exchange problem. If you use a public key as a messaging address then you don't have to exchange anything.

2. Bittorrent users derive direct benefit from storing and sharing material. On public trackers you have the perpetuated availability of material the peer wanted in the first place. On a private tracker it maintains ration in addition to the previous. While there is also altruism (which would be all I can see this relying on), it's not infinite, and given only 100mb (or any finite quantity) per peer you will still eventually run out of space. The blockchain is useful to anyone who stores it because that's how you prove what has happened and already there a large number of "light" nodes that only store the block hashes and verify things separately as needed and don't store many blocks at all as well as several implementations (including the satoshi client) that are looking to prune the spent tx data to reduce the storage requirement (pruned data would be ~120 MB right now if memory serves). For larger file storage clients like Freenet (while file storage you can just think of files as binary messages (see: Usenet file sharing)) the incentive is that you store some extra data so others will save your extra data, though no strict parity is enforced and there is a certain reliance on altruism (and for this reason I don't think this sort of project is anywhere close to impossible, just harder). Since I brought up Usenet, there are basically superpeers that act as peers with each other and central servers for end users; end users normally pay for large data storage provided by central servers.

Yes, there are two ways to solve this, actually I don't quite disagree with you here - either you make users contribute in order to use the system,
or you make them pay supernodes. Supernode would be a usual peer with unusual storage capacity, but no other additional features. Also these solutions can be combined.

3. PoW does help, but the higher you make it the harder (relatively) it is for a normal user to send a message, the lower you set the bar the easier it is to spam. So while there is a tradeoff but it's definitely a solid place to start; allowing a user to establish how hard it is to send them messages might work. Rejecting messages from a node may be somewhat meaningless, you want to allow new users to be able to participate reasonably quickly and there's no way to select between a new user and a new identity for a malicious node. In addition to all that routing messages through the p2p portion means the physical source node might not be important, the identity of the sending node is all you can conserve on a practical level.

Once again, I mostly agree. Actually it's a subtle trade off between system usability and spam protection. Let's take bitmessage - you need 4 minutes to send the message. Looks like it's fine, but it actually is not - from a large botnet you can send millions of messages a day still. So probably the solution would be combining PoW with ability to reject messages from certain nodes. Also a node should have the ability to receive messages only from trusted notes if it wishes so.

4. I was thinking this was along the lines of a PM system with optional twitter-like broadcasts, but if it's twitter and includes direct messaging a few things shift priority, but overall allowing any messaging TO an identity opens up all the same problems if it's the most common behaviour or not. Sounds kind of like most of the forums on Freenet though, everyone publishes their board posts to their personal broadcast stream and based on your direct trust level or indirect trust through WoT you choose whether or not to download their messages. The expensive part of creating a new identity is bootstrapping yourself into the network and announcing your presence so other people begin to download your messages to begin with. If you wanted to apply a similar mechanic at all you could start with a heavy PoW problem to announce your identity to other peers.

Yes, I'm thinking mostly about twitter-like system for now. That is you announce your address and interesting parties subscribe to it. So this is the messaging FROM identity so to say. General network-wide announcements to all peers are too prone to spamming I fear.

5. I'm used to seeing DHT as a lookup method, but you can easily tie that to looking up where to find/who to ask about a certain message certainly. I'd say the central servers archetype evolved before p2p was necessary or practical; if you wanted to share something you hosted it, if you wanted mail you sent it to the mail server (similar to the post office I suppose) and it was also easier to manage and there was no reason for it to go away. A lot of hosted platforms (especially closed source hosted sites like google, centralized mining pools, or anything requiring any central authority or a protected/hidden database to run) are either easier or only possible inside a centralized environment. Some things can be moved easily to a decentralized setup (file sharing from FTP to Bittorrent, central mining pools to P2Pool, etc) and some things are harder to move over (Anything abusable, anything that requires strong input checking, required hidden or protected databases). I'd also say that part of the reason webmail providers have so much information they don't need to have access to is that it's easier and established. There's no established default email or text encryption standard for transmission and the default is plaintext. If you could switch everything to encrypted transmission by default the webmail providers wouldn't have anything to begin with (other than sender/receiver and timing, which is also what the NSA was collecting on phone calls and is still powerful information and not eliminated in a p2p setup) but it's harder for end users, grandma doesn't care about PGP and grandma couldn't use PGP and wouldn't be able to use email if it were the default and the only option for her would likely be a webmail provider either receiving plaintext or decrypting it for her server side which means they have the contents anyway (you could do it client side, but then grandma has to carry her private key around and she can't do that either, also it's easier server side).

Yes, it would be cool if everyone used PGP, but it exists since decades and didn't really pick up with mainstream audience. Also key exchange problem still exists here. Also central servers make things easier techically that's for sure. Probably you'll never be able to approach features which
central architechure can bring, like fast access to huge mediafiles and such. But we can try to make it as usable as possible, and the benefits are obvious - you do not allow everyone and his dog to eavesdrop on you. I'm not a privacy freak, actually don't care much for privacy per se,
this is the question of society structure and equal access to information.

Also DHT setups can be very different, basically you can achieve very much with DHT, this approach has already been tested practically, and it sure
works.

All told you've definitely addressed a lot of things, I'm having to insert a fair amount of qualifiers to things now, and a lot of problems can be considered corner cases (that I'd still personally like to see addressed in an active implementation, now is too soon for some of them before larger details are finalized). The biggest problem I still have is storage, I'm not sure the balance between indefinite holding and total potential storage is there yet, at some point it just runs out. Though I don't know that a two day window is enough for everyone either :-) If nothing else I can keep playing devil's advocate at least if you want to try to flesh things out more.

Your input is very insightful, I would be glad if you could continue providing it! As for the storage - intuitively I feel that it's doable, storage is cheap and gets even cheaper now. Also, a feature when a sender wants to drop its messages from the network after ceratain time would also help to reduce the load. If you want to store the messages indefinitely - you can do it by default, if not - choose for how long they should be stored after the receiver has read it.

[tradeaway]

Probably one of the reasonable approaches would be starting a new fork of Bitcoin with messaging.
In this case you could monetize the messaging (fee for heavy users and such) right from the start.
What I would probably have against this - that would be another clone, there's quite a number now.
But it sure has some attractive points, for example the exchange rate for the currency would be determined by how much users want to pay for messaging, that is it is pegged to something beyound pure supply and demand.

tradeaway, you sir have been PMed, I think we can assist each other with our two projects and the issues we face with "blockchain bloat"

On the bitcoin blockchain or on the general cryptocurrency styled blockchain? I know there's some work to get pruning into the satoshi client and other clients who don't store much of the blockchain by default. Although for bitcoin there should always be at least a handful of copies of the entire chain somewhere all the way to the genesis block, at least under the current trust model for bitcoin.

[btharper]

While such a combination of features would be useful, I just think that they may be impractical together.

1. I'm not sure what you mean here, if a server alters an encrypted message it (should be, depending on the encryption scheme) is rendered unreadable. It's possible to cause the message to be unreceivable, but not to alter a message after encryption during transit. If you mean MITM because of poor key exchange, then that's a fundamental problem with key distribution not PGP in particular. The same problem (having to make sure key exchange is secure) would exist in any scheme.

Yes, exactly, I mean key exchange problem. If you use a public key as a messaging address then you don't have to exchange anything.

I'm not sure that fixes anything, you still have to get the address somehow. If the service giving you the address is compromised or malicious it could just give you an address/key it controls and forward the mail through using another address of its own as the reply-to address and intercept communication both ways. Alice -> Eve as Bob -> rewrite reply address to Eve as Alice -> Send to Bob. Return works the same way but switching the addresses back again. Eve still comprises all the back and forth through MITM. Generating enough addresses on the fly gets a little harder if creating a new identity is expensive (cost or computation), but would probably still be possible.

2. Bittorrent users derive direct benefit from storing and sharing material. On public trackers you have the perpetuated availability of material the peer wanted in the first place. On a private tracker it maintains ration in addition to the previous. While there is also altruism (which would be all I can see this relying on), it's not infinite, and given only 100mb (or any finite quantity) per peer you will still eventually run out of space. The blockchain is useful to anyone who stores it because that's how you prove what has happened and already there a large number of "light" nodes that only store the block hashes and verify things separately as needed and don't store many blocks at all as well as several implementations (including the satoshi client) that are looking to prune the spent tx data to reduce the storage requirement (pruned data would be ~120 MB right now if memory serves). For larger file storage clients like Freenet (while file storage you can just think of files as binary messages (see: Usenet file sharing)) the incentive is that you store some extra data so others will save your extra data, though no strict parity is enforced and there is a certain reliance on altruism (and for this reason I don't think this sort of project is anywhere close to impossible, just harder). Since I brought up Usenet, there are basically superpeers that act as peers with each other and central servers for end users; end users normally pay for large data storage provided by central servers.

Yes, there are two ways to solve this, actually I don't quite disagree with you here - either you make users contribute in order to use the system,
or you make them pay supernodes. Supernode would be a usual peer with unusual storage capacity, but no other additional features. Also these solutions can be combined.

Two ways is probably a minimum, there's always plenty of ways to work it out. I'd argue that supernodes at any point does or can lead to centralization and all the things you're looking to avoid by making it decentralized. So of those two solutions just relying on donated resources might be required. Implicit supernodes that are just well connected with lots of space and bandwidth might pop up, but are a nice possibility not a requirement of the protocol.

3. PoW does help, but the higher you make it the harder (relatively) it is for a normal user to send a message, the lower you set the bar the easier it is to spam. So while there is a tradeoff but it's definitely a solid place to start; allowing a user to establish how hard it is to send them messages might work. Rejecting messages from a node may be somewhat meaningless, you want to allow new users to be able to participate reasonably quickly and there's no way to select between a new user and a new identity for a malicious node. In addition to all that routing messages through the p2p portion means the physical source node might not be important, the identity of the sending node is all you can conserve on a practical level.

Once again, I mostly agree. Actually it's a subtle trade off between system usability and spam protection. Let's take bitmessage - you need 4 minutes to send the message. Looks like it's fine, but it actually is not - from a large botnet you can send millions of messages a day still. So probably the solution would be combining PoW with ability to reject messages from certain nodes. Also a node should have the ability to receive messages only from trusted notes if it wishes so.

I'm not sure how much filtering would be possible using a DHT/broadcast/everything public system. The source node not being connected when the message is eventually retrieved is nearly a protocol requirement I would think. Storing spam forever is also a PITA because people who aren't receiving it can't spam check or do anything to figure out if anyone wants the mail. Someone sending mail back and forth to themselves could theoretically DoS the network, or at least take up a lot of space. You can control who you connect to, but in a practical sense someone has to be willing to connect to new peers and most people will eventually be connected through some series of nodes to every other peer, friendly or not.

4. I was thinking this was along the lines of a PM system with optional twitter-like broadcasts, but if it's twitter and includes direct messaging a few things shift priority, but overall allowing any messaging TO an identity opens up all the same problems if it's the most common behaviour or not. Sounds kind of like most of the forums on Freenet though, everyone publishes their board posts to their personal broadcast stream and based on your direct trust level or indirect trust through WoT you choose whether or not to download their messages. The expensive part of creating a new identity is bootstrapping yourself into the network and announcing your presence so other people begin to download your messages to begin with. If you wanted to apply a similar mechanic at all you could start with a heavy PoW problem to announce your identity to other peers.

Yes, I'm thinking mostly about twitter-like system for now. That is you announce your address and interesting parties subscribe to it. So this is the messaging FROM identity so to say. General network-wide announcements to all peers are too prone to spamming I fear.

I do think that's a more manageable system overall, everyone polls for messages of users they care about. Messages are signed for public messages or encrypted to the receiver and then signed (all the headers on the inside, source may still be leaked/public possibly) for private messages. Doesn't work like any traditional system I'm familiar with, but there's no central server so it works. Does sound pretty similar to FMS on Freenet though, could be a good reference.

5. I'm used to seeing DHT as a lookup method, but you can easily tie that to looking up where to find/who to ask about a certain message certainly. I'd say the central servers archetype evolved before p2p was necessary or practical; if you wanted to share something you hosted it, if you wanted mail you sent it to the mail server (similar to the post office I suppose) and it was also easier to manage and there was no reason for it to go away. A lot of hosted platforms (especially closed source hosted sites like google, centralized mining pools, or anything requiring any central authority or a protected/hidden database to run) are either easier or only possible inside a centralized environment. Some things can be moved easily to a decentralized setup (file sharing from FTP to Bittorrent, central mining pools to P2Pool, etc) and some things are harder to move over (Anything abusable, anything that requires strong input checking, required hidden or protected databases). I'd also say that part of the reason webmail providers have so much information they don't need to have access to is that it's easier and established. There's no established default email or text encryption standard for transmission and the default is plaintext. If you could switch everything to encrypted transmission by default the webmail providers wouldn't have anything to begin with (other than sender/receiver and timing, which is also what the NSA was collecting on phone calls and is still powerful information and not eliminated in a p2p setup) but it's harder for end users, grandma doesn't care about PGP and grandma couldn't use PGP and wouldn't be able to use email if it were the default and the only option for her would likely be a webmail provider either receiving plaintext or decrypting it for her server side which means they have the contents anyway (you could do it client side, but then grandma has to carry her private key around and she can't do that either, also it's easier server side).

Yes, it would be cool if everyone used PGP, but it exists since decades and didn't really pick up with mainstream audience. Also key exchange problem still exists here. Also central servers make things easier techically that's for sure. Probably you'll never be able to approach features which
central architechure can bring, like fast access to huge mediafiles and such. But we can try to make it as usable as possible, and the benefits are obvious - you do not allow everyone and his dog to eavesdrop on you. I'm not a privacy freak, actually don't care much for privacy per se,
this is the question of society structure and equal access to information.

Also DHT setups can be very different, basically you can achieve very much with DHT, this approach has already been tested practically, and it sure
works.

It's usually a tradeoff between easy and private. No one writes down notes and encrypts them in public because it's a pain, you just talk, the best you can do is whisper or go somewhere more private.

Files, including video, audio, etc, can all be represented in printable characters across text messaging systems, and give it ten minutes of existence and I'm sure someone will be transferring files and they'll slowly get as big as people care to send. yEnc on usenet, base64 more generally. Binary data is still just data.

All told you've definitely addressed a lot of things, I'm having to insert a fair amount of qualifiers to things now, and a lot of problems can be considered corner cases (that I'd still personally like to see addressed in an active implementation, now is too soon for some of them before larger details are finalized). The biggest problem I still have is storage, I'm not sure the balance between indefinite holding and total potential storage is there yet, at some point it just runs out. Though I don't know that a two day window is enough for everyone either :-) If nothing else I can keep playing devil's advocate at least if you want to try to flesh things out more.

Your input is very insightful, I would be glad if you could continue providing it! As for the storage - intuitively I feel that it's doable, storage is cheap and gets even cheaper now. Also, a feature when a sender wants to drop its messages from the network after ceratain time would also help to reduce the load. If you want to store the messages indefinitely - you can do it by default, if not - choose for how long they should be stored after the receiver has read it.

Storage is cheap by some metrics, a large company can have exabytes (Megaupload, Google, Facebook, etc), but the home user can and will fill up his harddrive and want space eventually, or may only care to dedicate so much of it. Add to that junk mail, large file transfer, data redundancy, and all the rest and eventually it will run out. Even if it's not in the plans, assume any corner case will happen and have some plan for it. The problem with opt-in deletion (sender or receiver explicitly deleting things) is that if they forget to or lose they keys to do so then it's stuck there. Theoretically the receiver could delete it once received or the sender could put a destroy-at timestamp, though I don't think you could do it based on time read; you don't really want to let the public know when a message is read, I don't at least. Being able to re-send a message into the swarm (as the sender, or receiver even) isn't too difficult I'd imagine, just keep passing around the block of data containing the message. I would certainly be highly in favor of some mechanism that eventually expires data being included from the beginning, even if there's no expectation to use it.