Data-stream encryption

[Anti-Cen]

in my quest to find what i was looking for I took the code here, nice sweet and simple it is and it used maths.mod to encrypt data
and I got it up and running in seconds ready for a bit of benchmark testing on 1GB of data (1mb blocks).

https://www.codeproject.com/Articles/17703/Stream-Based-Encryption-for-NET

I stopped the program after about sixty seconds and it had only managed to encrypt/decrypt
500mb of data so i think you can have good encryption or fast performance but you cannot have both.

BigINT are useful for encryption but again I just cannot get the performance I want using this method.

My current plan is to semi encrypt the data, tweak the key and pass it over to Microsofts black box
encryption but I won't be using AES encryption because the .NET version leaks memory which you only get
to see if you call it half a million times or so which might sound a lot but a proxy server can reach these
type of number in little over an hour.

 

[1715536133]

1715536133

[1715536133]

1715536133

[1715536133]

1715536133

[1715536133]

1715536133

[Ix]

To send encrypted data over a vulnerable protocol (like the internet) you need:

1) key exchange (diffie-hellman usually)
2) a cipher (like salsa20 + initialization vector per chunk)
3) data integrity checks/authentication (hmac / poly1305)

You generally need to use TCP and you need a small maximum message size (chunk) - around 8kb or so typically - otherwise you have to resend huge chunks in case of corruption. You can use UDP which it sounds like you want, but then you need to reinvent a lot of TCP on top of UDP.
You can't use some hardcoded key because then anyone can look at your software and decode all communications. You need a different IV per message or observers can easily determine your key. You need authentication of the data or you will not know if the peer sent bad info or if it was accidentally/maliciously changed in transit.

The NaCl and LibSodium libs implement most of this for you in "secretbox". It's also available in the Go "x/crypto" library. There might be a C# implementation, I don't know.

[cellard]

satoshi was a Windows developer, the early Bitcoin releases were only released in Windows... so don't underestimate people that are good coders in Windows, as long as the project is open source and other developers can keep developing in Linux later on as well then it should be good. I recommend that people learn programing in Linux for projects of such nature tho.

Isn't AES used for the wallet.dat encryption? I would like more options for the encryption of the wallet.dat file, give us the ability to select cascaded encryption Veracrypt style and the option to mask the funds too so you need to enter the password to see the funds.. anyway this is off topic I will create a thread later.

[Anti-Cen]

I don't think people are quite picking up the point here or maybe I didn't put it across correctly but we are not talking about
encrypting small amounts of data like a 2k whats-app message but something more like 100k needed for the HTML on a web-page
and then six javascript files within the page followed up by say ten 2mb .PNG/Gif images files and a CSS file

Performance is critical here because if these requests are not serviced within 2-5 seconds then the browser will try sending the request again
using multi threaded requests so both the client and server nodes need to also be multithreaded also and the exit nodes need to start pumping the data
back across the network even before they have the whole reply from the web-site.

Things are made worse because often data is chunked which more or less means streamed and you have relay nodes possibly in the middle
plus having to service 100 requests all at the same time so buffering up the data to be encrypted in memory and keeping much state information
simply is not an option.

Data can be sent over TCP/IP as a 30K message but can be received as three 10k sections with a second delay between them all
but you still have to start feeding the web-browser as soon as you receive the first 10k or else it might timeout so I have no option
but to use on-the-fly stream encryption and it must perform like Greece lightning.

What i do know is that Microsoft AES encryption works nearly as fast as I can iterate over a byte array
using un-safe code from C# with C++ style pointers so I don't know if specialist hardware is being used to
get the performance for AES encryption but that's the type of speed that I need and using Maths.Mod or BigINT's just won't
give it to me

Microsoft's AES encryption memory leaks and I am not trusting back-door bill gates with any keys or data given that
I don't know whats being copied and uploaded back to the NSA/CIA servers all night long.

To send encrypted data over a vulnerable protocol (like the internet) you need:

1) key exchange (diffie-hellman usually)
2) a cipher (like salsa20 + initialization vector per chunk)
3) data integrity checks/authentication (hmac / poly1305)

Yes got safe key exchange using Bitcoins Secp256k1 on a control channel and the data-streams have good encryption
on the HTTP Requests send to the server nodes but these requests are only between 400-2000 bytes in size so no trouble to encrypt.

You generally need to use TCP and you need a small maximum message size (chunk) - around 8kb

8K chunks are no use when
sending 2mb images over the internet and often using relays, not going to work that's not.

You can use UDP which it sounds like you want, but then you need to reinvent a lot of TCP on top of UDP.

Google is trying to use UDP instead of TCP/IP for web-pages with the Chrome browser but I am not doing anything apart from
DNS using UDP:53

I keep the control channel and a number (Six just now) of these bi-directional streams open per client so that a node sitting behind a firewall
can still act as a server but in general the number of streams needed to service a web-browser can reach as high as 30 steams at a time

[Ix]

8K chunks are no use when
sending 2mb images over the internet and often using relays, not going to work that's not.

Sure it will, and it will work much better than attempting to encrypt it all in one chunk. The maximum packet size is around ~1500 bytes as it is. If you send a 2MB file and one bit gets flipped somewhere along the way, without chunking the entire thing will have to be resent. There is very little overhead to doing this.

It sounds like you want to do something similar to Tor, maybe you should investigate what it does?

[Anti-Cen]

Sure it will, and it will work much better than attempting to encrypt it all in one chunk. The maximum packet size is around ~1500 bytes as it is. If you send a 2MB file and one bit gets flipped somewhere along the way, without chunking the entire thing will have to be resent. There is very little overhead to doing this.

The lower levels of the ISO reference model deal with bit's being flipped, it happens all the time but all a socket can receive in one chuck is 65535 bytes
so really I am sending images and bigger files just like web-site pump out data and that works.

It sounds like you want to do something similar to Tor, maybe you should investigate what it does?

Yes I know my way around Tor, been an exit node myself before now plus hosted a onion site however it's
slow, has no redundancy to host sites if the server node is down plus a few other little problems like only having a few well known
exit nodes and they are all black-listed plus something I don't want to go into here.

Down the road I will be dealing with Whats-app type messages, file sharing, hidden sites but most of it's based on end to end encryption
using a public key generated for each node without needing any centralization however I don't really think this will be possible if it becomes
popular but at least I admit it up front.

One day people will laugh that they had emails addresses instead of a public key knowing that big brother is reading everything we say.

[Ix]

The lower levels of the ISO reference model deal with bit's being flipped, it happens all the time but all a socket can receive in one chuck is 65535 bytes
so really I am sending images and bigger files just like web-site pump out data and that works.

I'm not sure what you mean by ISO reference model. TCP has a 2 byte checksum which can easily be accidentally fooled. Malicious bridge nodes (in Tor terminology) can easily switch all kinds of bits that TCP cannot detect or they can just modify the checksum to match. That is why you need authentication.

https://en.wikipedia.org/wiki/Maximum_transmission_unit

"For example, a 1500-byte packet, the largest allowed by Ethernet at the network layer (and hence over most of the Internet), ties up a 14.4k modem for about one second. Large packets are also problematic in the presence of communications errors. If no forward error correction is used, corruption of a single bit in a packet requires that the entire packet be retransmitted, which can be very costly."

Now imagine you sent your 2MB image without chunking and you have the same problem as a TCP packet.

[Anti-Cen]

I'm not sure what you mean by ISO reference model.

http://www.itprotoday.com/management-mobility/isoosi-ieee-8022-and-tcpip

TCP has a 2 byte checksum which can easily be accidentally fooled. Malicious bridge nodes (in Tor terminology) can easily switch all kinds of bits that TCP cannot detect or they can just modify the checksum to match.

But 2mb are still sent as 2mb blocks from sockets regardless of the trouble in keeping a thread open to send data in little block or encryption
and the lower levels deal with it most of the time and there is not much i can do about it.

14.4k modem   When i started we only had 1200 baud and being able to send faxes involved an upgrade to the modem and these
things were expensive I can tell you but still it was better than Prestel and it's teletext type graphics.

[Ix]

But 2mb are still sent as 2mb blocks from sockets regardless of the trouble in keeping a thread open to send data in little block or encryption
and the lower levels deal with it most of the time and there is not much i can do about it.

The point is you do have to deal with it with a stream cipher, if you want a protocol that is in any way reliable. TCP checksums are not sufficient, and without chunking and authentication you won't know something is wrong until after the whole file is sent, and then the whole file must be sent again.

[bob123]

The lower levels of the ISO reference model deal with bit's being flipped, it happens all the time but all a socket can receive in one chuck is 65535 bytes
so really I am sending images and bigger files just like web-site pump out data and that works.

But 2mb are still sent as 2mb blocks from sockets regardless of the trouble in keeping a thread open to send data in little block or encryption..

While 65535 bytes is the theoretical maximum for a 'tcp packet', which by the way is stream based (why do you care about packet size when using TCP?),
lower layers are not able to process such big packets without errors.

In the end a single packet sent over a tcp connection is always lower than those 65535 byte.

[Anti-Cen]

why do you care about packet size when using TCP?

I don't really care about the TCP packet size, just the chunk/stream size received for decryption which might not be the same size as what was sent
unless you hang around waiting to see if more data arrives a few seconds later.

This makes encrypting large volumes of data sent over the wire more difficult on top of the performance problem of eating up the CPU  and
frankly I don't have the answer yet.