Just a quick comment about progress: No, it's not done yet. But I am finally getting some time to do it. And I'm doing it... slowly.
Just wanted to check in on progress -- I am gladly waiting for the upgrade and glad to see you are taking the time to improve the foundation.
Making tons of progress on it. As I keep saying in this thread and others: I had some critical priorities crop up, but I have still had a bit of time to work on it, and I think I'm getting closer to something that works. I keep trying to make estimates of when I'll have it, but my estimates have been pretty much useless due to the scope of changes and unpredictability of other things.
So I will not put a timeline on it. But what I will say what is done: the DB foundation appears to be functional and working. I have also incorporated google-test into the project and written 5,000+ lines of testing code
(a lot of it is for the pre-existing codebase that didn't have proper automated unit-tests, but at least half of it is for the new DB utilities). I'm juggling some of the fine details of hooking up the new DB engine to the BlockUtils.cpp which is where all the magic happens -- for instance I didn't anticipate I would have to switch to a headers-first pipeline due to the way I chose to key the databases (all blocks are stored by their height, which means I can't put them into the DB until I know their height). Headers-first is probably better anyway, but it does require rewriting some solid code that I had hoped to leave in place a bit longer and not complicate this upgrade. Oh well.
I already have a solid re-org unit test
which is basically the ultimate did-I-get-it-right test. That test took me like a week to create, 18 months ago. Now that it's done, I don't have to spend any more time on it than just running it through.
Unfortunately, I may have made an extremely inefficient design decision, that will have to be fixed before any official releases go out (though it won't take me more than a day or two to fix it). Mainly the way I store address histories is going to choke on the SatoshiDice addresses. It should still work, it will just be super slow when new blocks are received. I'll get it working without fixing that, and with luck it will just work fine, anyway. If not, I'll have a unit-test-passing version that can be slowly morphed into the optimized version. I like having solid unit-tests...
By the way, if you do any C++ development, I can't speak highly enough about googletest. It's remarkably easy to use, yet has an extraordinary amount of flexibility if you feel it necessary to get creative/advanced.