i simplified the code before posting:


it uses the ascii char val  to choose the cipher to use. so if you have a 4 char pin code, this will map to using 4 of the ciphers. I apply it iteratively to the output of the previous cipher.

not sure how even 18 times becomes over 100 times slower, maybe AES is 10 times faster? if you like AES just use a one char pin code that maps to AES. it is up to the user. anyway the code is already done and it didnt bloat things that much.

James

here is the inner loop that applies the ciphers:

without BBR you cant use the BBR.
SuperNET is NOT a coin. do you understand this?
it is the network that binds the coins
if people on the SuperNET want to use cryptonote functionality, then they would use BBR in this process. How can you take BBR out of this equation?

does having different ciphers hurt? it allows short (user rememberable passwords to "salt" things)
unless any of these ciphers have been cracked, using them means that all the ciphers would need to be cracked.
using just AES means if AES is cracked then all the files are cracked.

so is having a sucky RC2 and a weak 64 bit layer in addition to AES worse than just AES?

you really dont understand...

I am supporting BBR for what it is.

One part of SuperNET is as a network and connecting different crypto's is what it does. Since BBR is a crypto, connecting to it is part of what SuperNET does. Once it is properly connected, then all of SuperNET is able to connect to BBR.

so, think of it like a cellphone network where each network can only call phones in its network. vodaphone vs ATT vs Tmobile
imagine that you cant call from vodaphone to ATT, or from any network to another. this is what crypto is now.

now the SuperNET allows all the networks to talk to each other. So this makes the vodaphone, ATT and Tmobile phones all get the ability they didnt have before. Regardless of what vodaphone thinks of ATT, how is it worse off by having the option of being able to call users from the other network?

this is what "intertwined with SuperNET" is. Do you not want BBR to be connected to the other users? You want BBR island?

James

when your build environment is working, you should go:
cd
git clone https://github.com/jl777/btcd
cd btcd/libjl777
./onetime
<go into nacl directory, build, lib, until you find the randombytes.o and libnacl.a>
cp randombytes.o libnacl.a ~/btcd/libjl777/libs
cd ~/btcd
./m_unix
<make a SuperNET.conf>
./BitcoinDarkd

"There needs to be vetted math, science, crypto and code for all to see."
Well the code is here: https://github.com/jl777/libjl777
I use standard crypto: nacl and libtom

Right now I am debugging on  a test scale network. Cassius will be documenting the API. I have a technical roadmap, but I do not have the skills to explain it to nontechnical people, so I will just push forward and implement it. I dont have time to deal with baseless accusations that I am some snake oil guy.

###
https://forum.thesupernet.org/index.php?topic=154.msg1111#msg1111

I finished debugging all the DHT calls. Ended up adding data compression to the loop, along with out of band binary data beyond the JSON in the onion packets. Also, a lot more difficult that I thought it would be, but finally got all packets to be the same size, thus removing a leak of info about the type of commands you are sending.

./BitcoinDarkd SuperNET '{"requestType":"getpeers"}'
./BitcoinDarkd SuperNET '{"requestType":"getPservers"}'
./BitcoinDarkd SuperNET '{"requestType":"ping","destip":"209.126.70.156"}'
./BitcoinDarkd SuperNET '{"requestType":"findnode","key":"<nxtaddress>"}'

./BitcoinDarkd SuperNET '{"requestType":"store","name":"<name of data>","data":"<hexstr>"}'
./BitcoinDarkd SuperNET '{"requestType":"findvalue","name":"<name of data>"}'

This weekend I implemented a variant of http://www.cs.rice.edu/Conferences/IPTPS02/109.pdf, using 64bit NXT addresses for the hash, eg. least significant 64bits of sha256. collision resolution is something for a layer above this level to do. I plan to add some simple file API on top of this so we can get a nice decentralized storage along with sending of files.

of course all this is under heavy encryption, but still the IP addresses are not shielded yet. The sendmessage API appears to be working in simple topologies, should work with more layers, but need a larger network to test this.

./BitcoinDarkd SuperNET '{"requestType":"sendmessage","dest":"<nxtaddress>,"msg":"<text of message>","L":<maxonionlayers>}'

Using the sendmessage api, it should be pretty easy to make an encrypted chat application.

####

yesterday I did this:

Since there was no large network for me to test with today, I decided to make two new API calls that allow for cloud storage of files. They are massively encrypted and also M of N is supported to deal with hash collisions, sybil attacks, offline nodes, etc. With the proper M and N settings, I think this will be quite a resilient file storage appropriate for the files you just cant lose. The comms with the cloud are via the DHT API from this weekend and the L parameter is for the max number of onion layers to use and all the packets are the same size, so there is no leakage based on packet size.

char *savefile[] = {  "filename", "L", "M", "N", "usbdir", "password", 0 };
char *restorefile[] = { "filename", "L", "M", "N", "usbdir", "password", "destfile", "sharenrs", "txids", 0 };

./BitcoinDarkd SuperNET '{"requestType":"savefile","filename":"<file to save>","L":0,"M":1,"N":1,"usbdir":"<dir for backups>","password":"<can be 4char PIN>"}'

The savefile will print (and save in usbdir) the required sharenrs and txids JSON fields to use for the restorefile.
The "destfile" field is where the file will be reconstructed.

If the "usbdir" parameter is set, then local backups are made (highly recommended!) and it is used to check the data coming back from the cloud. After you verify that the cloud has a proper copy, then you can partition the various parts from the usbdir directory to various places to have two full backups, one under your local control and one in the cloud.

The max value for N is 254 and M has to be less than or equal to N. The M of N parameters are independent of the "password" field. If you are using M of N, then unless the attacker gets a hold of M pieces, they wont be able to reconstruct the file. Without the txid list, the attacker wont know how to reconstruct the file.

But why take any chances. so I made the password field use an iterative method to create what I think is a pretty practical encryption method, which is based on the name of the file, your pubNXT acct passphrase and the password itself. The length of the password determines the number of ciphers that are applied

        namehash = calc_txid(name,strlen(name));
        len = strlen(password);
        passwordhash = (namehash ^ calc_txid(keygen,strlen(keygen)) ^ calc_txid(password,len));
        for (i=0; i<len; i++)
        {
            expand_nxt64bits(key,passwordhash);
            cipherids = (password % NUM_CIPHERS);  // choose one of 18 ciphers
            privkeys = clonestr(key);
            if ( i < len-1 )
                passwordhash ^= (namehash ^ calc_txid(key,strlen(key)));
        }
   
Since the keygen is the pubNXT password, which in turn is a dumpprivkey for a BTCD address, this assures high entropy and the filename being encrypted is added to the passwordhash so that different files will have different encryption keys. By using the password to modify the initial password hash and to determine the number of ciphers and their sequence creates a lot of impact from even a short password, like a PIN

When M of N is combined with password, the attacker would need to get a hold of the name of the file, M fragments, the list of txids, the randomly generated sharenrs array and the password you used. Unless your computer is totally compromised and you divulge your short password, this seems like a pretty good level of security.

Now with the DHT there is the chance of collision, sybil attacks, inaccessible nodes, etc. I think using M of N side steps all of these issues. Also, the txid (calculated like NXT does) is based on the contents being stored, so it would take a lot of computation to be able to even get control of the nodes needed to block access to any specific content and near impossible to spoof anything. Maybe someone can come up with a sybil attack that can be done? However, without knowing the hash values of all the fragments, where will the sybils setup their attack? And will they be able to invalidate M copies that they dont know the txid for?

The following are the ciphers:
    "aes","blowfish","xtea","rc5","rc6","saferp","twofish","safer_k64","safer_sk64","safer_k128",
    "safer_sk128","rc2","des3","cast5","noekeon","skipjack","khazad","anubis","rijndael"

####

I apologize for only working 16 hours per day and not having the proper skills to explain everything perfectly clearly.

I am just a simple C programmer

James

P.S. The fundraising idea did not work out, but this has nothing to do with the tech. SuperNET is not a one dimensional thing.

[ANN - MofNfs: store files in the SuperNET cloud using fully encrypted M of N shared secret fragments]

Since there was no large network for me to test with today, I decided to make two new API calls that allow for cloud storage of files. They are massively encrypted and also M of N is supported to deal with hash collisions, sybil attacks, offline nodes, etc. With the proper M and N settings, I think this will be quite a resilient file storage appropriate for the files you just cant lose. The comms with the cloud are via the DHT API from this weekend and the L parameter is for the max number of onion layers to use and all the packets are the same size, so there is no leakage based on packet size.

Now I am not sure what all the other decentralized storage projects are doing and I am sure what I did today is just a small portion of a full system. Still, after I debug it tomorrow, it will be an easy way to safely put things in the cloud.

char *savefile[] = {  "filename", "L", "M", "N", "usbdir", "password", 0 };
char *restorefile[] = { "filename", "L", "M", "N", "usbdir", "password", "destfile", "sharenrs", "txids", 0 };

./BitcoinDarkd SuperNET '{"requestType":"savefile","filename":"<file to save>","L":0,"M":1,"N":1,"usbdir":"<dir for backups>","password":"<can be 4char PIN>"}'

The savefile will print (and save in usbdir) the required sharenrs and txids JSON fields to use for the restorefile.
The "destfile" field is where the file will be reconstructed.

If the "usbdir" parameter is set, then local backups are made (highly recommended!) and it is used to check the data coming back from the cloud. After you verify that the cloud has a proper copy, then you can partition the various parts from the usbdir directory to various places to have two full backups, one under your local control and one in the cloud.

The max value for N is 254 and M has to be less than or equal to N. The M of N parameters are independent of the "password" field. If you are using M of N, then unless the attacker gets a hold of M pieces, they wont be able to reconstruct the file. Without the txid list, the attacker wont know how to reconstruct the file.

But why take any chances. so I made the password field use an iterative method to create what I think is a pretty practical encryption method, which is based on the name of the file, your pubNXT acct passphrase and the password itself. The length of the password determines the number of ciphers that are applied

        namehash = calc_txid(name,strlen(name));
        len = strlen(password);
        passwordhash = (namehash ^ calc_txid(keygen,strlen(keygen)) ^ calc_txid(password,len));
        for (i=0; i<len; i++)
        {
            expand_nxt64bits(key,passwordhash);
            cipherids = (password % NUM_CIPHERS);  // choose one of 18 ciphers
            privkeys = clonestr(key);
            if ( i < len-1 )
                passwordhash ^= (namehash ^ calc_txid(key,strlen(key)));
        }
  
Since the keygen is the pubNXT password, which in turn is a dumpprivkey for a BTCD address, this assures high entropy and the filename being encrypted is added to the passwordhash so that different files will have different encryption keys. By using the password to modify the initial password hash and to determine the number of ciphers and their sequence creates a lot of impact from even a short password, like a PIN

When M of N is combined with password, the attacker would need to get a hold of the name of the file, M fragments, the list of txids, the randomly generated sharenrs array and the password you used. Unless your computer is totally compromised and you divulge your short password, this seems like a pretty good level of security.

Now with the DHT there is the chance of collision, sybil attacks, inaccessible nodes, etc. I think using M of N side steps all of these issues. Also, the txid (calculated like NXT does) is based on the contents being stored, so it would take a lot of computation to be able to even get control of the nodes needed to block access to any specific content and near impossible to spoof anything. Maybe someone can come up with a sybil attack that can be done? However, without knowing the hash values of all the fragments, where will the sybils setup their attack? And will they be able to invalidate M copies that they dont know the txid for?

The following are the ciphers:
    "aes","blowfish","xtea","rc5","rc6","saferp","twofish","safer_k64","safer_sk64","safer_k128",
    "safer_sk128","rc2","des3","cast5","noekeon","skipjack","khazad","anubis","rijndael"

Having a place to store things reliably will help with managing telepods, especially since with the privateblockchains, any loss of data on your computer would not be good. So I think I will default to cloud backup for the telepods. Unless your computer is compromised and your divulge your passwords, it wont do anybody any good to get some telepod fragments. In any case, if they have compromised your computer, they would have access to the same data.

James

soon we should have end user testable builds, but I am not doing that so not sure when it will be exactly

tested findvalue, worked the first time:

./BitcoinDarkd SuperNET '{"requestType":"findvalue","name":"jl777"}'
STORE.({"result":"kademlia_store key.(4322723274388863363) data.(deadbeef) len.4 -> txid.0"})

./BitcoinDarkd SuperNET '{"requestType":"findvalue","name":"jl777"}'
{"result":"deadbeef"}

the first time, it terminates the search when another node sends a store API, which is exactly what happened. The second search just returns the value that it has locally.

So, as I predicted, I was able to get the DHT working and this is all the current calls: ping (pong), findnode (havenode), store, findvalue (havenodeB). To do it right required adding outof band binary data beyond the JSON inside the onion packet and that was a good time to get all the packets to the same size and also get the data compression in the loop.

Now I need a big network and bug reports. These are things I cannot do alone!

James

I am only debugging the easy cases, but once the bigger network is in place it will hopefully just work.
Since I cant do much without a bigger network, I figured might as well get distributed storage working
with the compression being tested with live data, I think 1kb stores can be handled, with some rare cases that have to be split up

so the out of band data was quite important for this. A day to code up the file system and a day to test for some simple setup with encrypted files in the cloud.

Of course, if a network can be built up so I can test the DHT with more than a few servers, I can do that and get Teleport validated. I say validated, as it worked over a month ago in loopback and now the network is getting stable, I dont expect much problems. I remember I had to do a bunch of accounting API, but that's just tedious work, nothing difficult like doing a DHT system from scratch over the weekend

James

[05:16] <jl777> I pushed another release. this one with the store command debugged
[05:16] <jl777> ./BitcoinDarkd SuperNET '{"requestType":"store","name":"jl777","data":"deadbeef"}'
[05:18] <jl777> it was a bit tricky to deal with the out of band data, but it now works for submitting the command with a "name" field and internally with the "key" field, which is the lower 64bits of sha256 of the string
[05:18] <jl777> and it at least parse it right and gets it to another node.
[05:19] <jl777> I am out of things to test as the findvalue really requires more than a few nodes to verify, will be going offline for a bit, hopefully when I am back online there will be many servers running

[04:13] <jl777> I just pushed another release
[04:14] <jl777> to support data store, I had to increase the capacity as 256 bytes per UDP packet is too small
[04:14] <jl777> so I had to add data after the JSON string to maintain compatibility
[04:15] <jl777> this affects the onion packet and it was great trouble to get it always 1400 bytes
[04:15] <jl777> the reason is that there cannot be any size information in plaintext
[04:15] <jl777> the only way to know if it was encrypted for you is if you can decrypt it
[04:16] <jl777> however to even have a chance of that, you need to know the size it was encrypted at
[04:16] <jl777> so this has to be in the plaintext or implicit by the size and if the real size was used, it would defeat the purpose of doing this
[04:17] <jl777> so it is not as simple as just padding out the packet as there are unknown random number of onion layers
[04:18] <jl777> plus every hop has to reencrypt an extra layer to pad it out to max size and of course this creates the need for double decryption at each node
[04:18] <jl777> as it could very well have to forward to itself
[04:18] <jl777> anyway, I hope I got all paths working properly. with the latest version any packet that isnt the same size is a bug
[04:19] <jl777> Oh, I also put a jsoncodec in the code path
[04:19] <jl777> this compresses and decompresses to make sure it isnt getting any errors
[04:20] <jl777> every 10 DHT commands it should print out stats, for small commands the compression isnt so good, and since we are padding anyway, it seems silly
[04:20] <jl777> however I want to be able to send 1024 store commands, and with the tokenization + onion layers, this will not easily fit, so I am getting ready for this
[04:21] <jl777> please upgrade to the latest, you can tell since it uses port 7777

[22:05] <jl777> I just pushed a version with findnode tested
[22:05] <jl777> ./BitcoinDarkd SuperNET '{"requestType":"findnode","key":"14139201960657020621"}'
[22:05] <jl777> the "key" is the NXT address of the destination acct, must be the privacyserver, eg public one
[22:06] <jl777> the private one is still inaccessible, I want to get Teleport working with the public privacy servers first
[22:07] <jl777> doing a findnode will automatically update the routing tables. theoretically if any node that you are connected to, knows of the node you are looking for, this should start a sequence that gets you in contact with that node
[22:07] <jl777> of course, this is what needs to be tested!
[22:08] <jl777> now all that is left to debug is store and findvalue.

lots of projects for SuperNET at forum.thesupernet.org

nice
do you do webdev work?

very cool!
looks like blockchain.info

[18:01] <jl777> just pushed a new version
[18:01] <jl777> ./BitcoinDarkd SuperNET '{"requestType":"ping","destip":"209.126.70.159"}'
[18:02] <jl777> you can now do a DHT ping, which should update internal routing table according to kademlia, but most importantly open a path to sendmessage to the other node
[18:02] <jl777> only ping is tested now, but I finally got it stable after a couple dozen builds so good time to push a release
[18:03] <jl777> please update with ./m_unix or you can just build the libjl777.so and update /usr/bin with make_shared in btcd/libjl777

James

latest status: https://forum.thesupernet.org/index.php?topic=66.msg1030#msg1030

tl:dr new DHT method should allow much more reliable and scalable connections with peers and this means reliable onion routing and all app layers.

I coded most of the Kademlia DHT API today:

http://www.cs.rice.edu/Conferences/IPTPS02/109.pdf

I have ping, pong, findnode, havenode coded and about to do the store and findvalue API

I am hoping someone will be able to make a GUI that uses this API and allows people to make money by letting others use their HDD. I want to use the auction process to set the pricing. For example, people that want to store data would bid B per month to store their data. People that want to rent space would set the amount of space they can make available. At the end of the month, the total amount paid would get allocated to the people whose HDD got used.

Something like that. Anyway, I dont have time to do this, especially if it has to be all GUI fancy.

Also, I want to build this right into the SuperNET itself, so no new asset for this. The revenues from this would be shared between the user and SuperNET. The GUI would be an extra tab in the SuperNET GUI.

I will pay reasonable bounty for the GUI and billing code

James

Got the kademlia API added to the parser:

    static char *ping[] = { (char *)ping_func, "ping", "V", "NXT", "pubkey", 0 };
    static char *store[] = { (char *)store_func, "store", "V", "NXT", "pubkey", "key", "value", 0 };
    static char *findnode[] = { (char *)findnode_func, "findnode", "V", "NXT", "pubkey", "key", 0 };
    static char *findvalue[] = { (char *)findvalue_func, "findvalue", "V", "NXT", "pubkey", "key", 0 };

http://www.cs.rice.edu/Conferences/IPTPS02/109.pdf is the reference doc. Sort of the classic DHT algo and solves the reliable routing issue. I just have the front end and the k_bucket code implemented, next is the actual kademlia protocol

James