First, here's the prototype of the immutable datastruct its based on:
Immutable Sparse Wave Trees (WaveTree)
Realtime bigdata tool for bit strings up to 2^63 based on AVL forest
https://sourceforge.net/projects/wavetree version 0.2.0 is a 84 kB jar file containing its own source code.
Opensource GNU LGPL 2+
Realtime bigdata tool at the bit level based on immutable AVL forest which can be run in memory or, in future versions, as a merkle forest like a blockchain. Main object is a sparse bit string (Bits) that efficiently scales up to 2^63 bits normally compressed as forest has duplicated substrings. Bits objects support reading bit, byte, short, int, or long (Java primitives) at any bit index in 64 bit range. Example: instead of building a class to hold a header and then data, represent all of that as Bits, subranges of them, and ints for sizes of its parts. Expansion ability for other kinds of compression, since Bits is a Java interface. Main functions on bits are substring, concat, number of 0 or 1 bits, and number of bits (size). All those operations can be done millions of times per second regardless of size because the AVL forest reuses existing branches recursively. Theres a scalar (originally for copy/pasting subranges of sounds) and a bit Java package. Sparse n dimensional matrix.
AVL tree balancing avoids deep and slow forest
Bits substring, concat, and count 1 bits in any subrange or combination costs only log time and memory (millions of times per second on average computer)
Versioning on N dimensional matrix object (Multidim) since its only a view of Bits object. I've tested this on 10000 images from MNIST OCR data.
Scalar and Bit versions - Originally was scalar for copy/paste subranges of sound. Same operations work for bit strings
Can store sounds that are years long since its sparse. Same works for bit strings up to 2^63.
-----
I too often get lost in excessive abstraction, but I'm getting back to keeping it real. This is 1 of my tools that I only came to understand the need for after years of research. It will be at the core of my game, AI, and science network, along with my new kind of mindmap and statistical tools like boltzmann machines and bayesian networks. All those things will be represented using this foundation of bits.
-------------
The possible computing grid:
These computers would not be like Von Neuman motherboards. They would not support mutable memory locations. They're more like lisp machines, except just a data layer, similar to bittorrent except more direct and for streaming near lightspeed (theres already many optical wires, or wireless mesh networks could also be a transport layer).
A memory address is, what I've thought of so far:
256 bits of SHA256, concat 64 bits of length (in bits), concat 64 bits of number of 1s, concat 8 bits of AVL forest height, concat 2 time headers.
The time headers are on any arbitrary scale, probably are 64 bit (or maybe 60 if want it to total 512?).
* minimum delete time
* maximum delete time
So every piece of data in the network is, by agreement of those in the network, required to be deleted by a certain time (in microseconds or nanoseconds, maybe) and is required to be near instantly available from whoever claims it exists (by it being referenced in a higher AVL forest node from them) until minimum delete time. Failure to put in a reasonable effort to do what you agreed (by rebroadcasting or first broadcasting a leaf), I recommend to those who think its a good strategy to make the network work overall, should be disconnected from anyone who discovers this about them, but dont tell anyone else in the network about it because then you'd have to consider incoming claims from many others that various nodes in the network did similar bad thing, and its much simpler in the gametheory of it to accept connections until you have personally seen the evidence of them not giving you what they agreed to (any part of any avlBitstring they sent you a merkle branch of). A balance should evolve of which data nodes will accept based on it having unreasonable min or max delete times, because if you accept it and rebroadcast you are responsible for routing until the min delete time and not routing after the max delete time. You can always hash it again with a different time, so competition should form.
All addresses are the hash of 2 child addresses, including length, 1s, forest height, and min and max time headers for this new branch.
You cant hack a memory space that doesnt support modification ever for any reason, unless SHA256 is cracked and in that case shouldnt have chosen that secureHash function. It makes no sense to create or delete a number, but if you ask for the mapped value (2 childs and times of parent) which you have never received (or not within time range claimed) data claiming it exists, it is no error to not get its value. You can always ask since others need not track what forest nodes you have, and in that case should be ignored with no penalty at first and gradually increasing penalty (of chance of being disconnected) with each repeated request for what was not offered (like noninvited port sniffing or nonimplied URL guessing). If somehow it is implied a data exists and is within time, then go ahead and ask for it without wasting the formality of someone hashing it into something you dont want, but this is not the main function of the network.
Such computers could be any ordinary computer running SHA256 by Java API which may in some implementations call to a GPU or do it in CPU, for example, or what I'm considering is a new kind of computer entirely which has many SHA256 chips, maybe redesigned as layers of NAND gates (as its constant number of steps), so each clock cycle can start a new SHA256 instead of waiting few hundred cycles for it to finish hashing 2 addresses which happens in most clock cycles.
Hash whenever 2 avlBitstring are concat or substring.
Read recursively into hash when you want to count the number of 1 bits in any range (in logBase2 time, not 2^64 cycles if it was a mutable memory, so huge optimization) or concat logBase2 times during substring.
These computers would need extreme cooling. How is the nuclear cooling tech progressing? Not to rotate any massenergy (in the data layer, but thats one of the things computers are for organizing) but to stop it from melting. For example, I have a theory that the thermodynamic saddlepoint of water, a few degrees above freezing where it gets bigger if either cools or heats might be able to find some parts which are heavier (the heavywater, with 1 proton and 1 neutron per hydrogen, and attached to oxygen). But I hope I dont have to go to such extremes. Some computing grids are embedded in metal walls in buildings on icebergs.
This computer would, other than its large address size, support the standard 64 bit integer and floating point math, but pointers work different. You dont write to a pointer. You substring and concat immutable avlBitstrings to include a pointer into itself. This is why the bitstrings can be up to 2^63 bits and all ops on them take logBase2 time and memory. I'm not sure if some ops do that logBase2 times, depending on large difference in tree height, but that can at least be optimized if it becomes an issue.
Think of it like anonymous Subversion for every piece of data and state change anywhere in the global network, and when you want to write a petabit avlBitstring, its potentially everywhere on Earth 1/23 of a second later, available but not cached until it starts being requested.
Prediction of what should be routed where is a layer on top of this, but in general routing should be by physical distance first since light doesnt make exceptions for trips to the internet backbone and back. Would you go to town and back to reach your refridgerator? You would if IPv4 doesnt give you a direct path there.
Target audiences: Internet2.edu superfast scientific grid consumers, video streaming which is currently done on patches on top of patches through bittorrent, high bandwidth massively multiplayer VR, static caching layer optimization, competition for Cassandra, Hadoop, Bigtable etc, supercollider data streams and you wouldnt believe how much data per nanosecond they generate since physics waits for noone, global voxel streams like is planned for live sports events to see from any angle even while pausing such waslive stream, memory mapping of neuromorphic, statistical, and wave based systems. Realtime high precision brain scans for networking minds together of those who choose it (which I cant wait for, and no I'm not the borg, opensource is by choice and therefore keeps the competition fair)... but if someone like the borg collective wants to join the network, I think they'll find it just barely compatible enough and annoying at having to hash when they would prefer more direct access to waves and statistics. Boltzmann machines are an important use case.
Its rediculous that a database would only let you make 1000 or so queries per second. Too much sync when everyone has to agree on the contents of that variable which we call that database. If you want to agree with many others on the value of some bitstring as a name, thats what we have blockchains, namecoin, and domain names for. If you want to download a pointer to a petabit and reliably count how many 1 bits are in a randomly chosen substring of it (which may even be over half its size) in a small fraction of a second, then you would use such an avlBitstring computing grid instead of a name system. Its so much simpler to push all the data by value, than to deal with the gametheory of who agrees on name value pairs.
What do you all think about the difficulty and volume prices for such chips and new kinds of computing grid, if I was to pursue this at a hardware level later? Some of you are specialists in SHA256 chips, but I'd want chips that has the whole concat of 2 childs and time headers (so even bigger than a single hash and padding). And then theres many way addressing to get the 2 childs of any address or find its not in the cache, and by cache I mean main memory. The difference between fast cache slower caches and Internet is just number of hops. Makes no difference if its in the same motherboard or across Internet. Its all 1 big computer to me, but some people like to say they're different things so they can regulate general computing without calling it a computer. I'm not trying to break laws, but keep that slow mess away from me.
