An optimal engine for UTXO db

[hhanh00]

To answer the OP, it is difficult to decide what makes an optimal engine for the UTXO set because as usual one has to make design trade offs.

1. If you are interested in super fast import, i.e. processing the block chain in shortest time, you are dealing with a large amount of insert/delete of key value pairs where the key contains a crypto hash. They are essentially random values and will fall all over the place. However, the net amount fits in an average development machine so you can work entirely in RAM. However, bitcoin core can't assume that you have several GB of RAM available!

Here, the main problem of leveldb is that its engine uses skip lists. Their average search complexity is ln(N). You'd be better off with a hash table which is O(1). For the record, skip lists and B-Trees are used when we need to retrieve the keys in order and it's not the case here.

Because blocks depend on previous blocks, processing them in parallel isn't easy. I'm not sure it helps even. (Block validation excluded)

2. If you are building a browser, then you can precalculate the hell out of your data set. The usual database tricks apply: lineralize the key space, compress values, use bloom filters, partition your data, etc. You are working with data than changes only every 10 mn, so very quite static. If you need the unconfirmed tx, add them on top. This case isn't what a db is really used for anyway. A lot of effort is put into concurrency and durability. When they are not crucial, faster implementations exist - especially if you can choose the hardware.

3. If it's for checking transactions in a full node, there are relatively few of them. Once you have extracted the UTXO, we are talking about a few million records which is peanuts in today's standards.

In conclusion, leveldb is fine for its usage it bitcoin core. The choice of a skip list vs hash table engine is questionable, but I don't think that makes a big difference for most of the users.

Maybe, you could describe your usage?

[2112]

If it has nothing to do with DBMS, then why did you keep doing the database class thing?

Because the title of this thread is about DBMS a.k.a. DB engines.

Because of the fact that the data stored is actually about money.

You just have zero experience dealing with financial data processing, and actually less-than-zero as far as legally-responsible fiduciary-duty-bound handling of money for other people.

Therefore what you are doing are just educational toys. I have nothing against you designing and playing with those toys. But if you thinking that you are making something that will become a real financial product you set yourself for a CIYAM-level disappointment.

https://bitcointalk.org/index.php?topic=1375689.msg13995422#msg13995422

The CIYAM project has got a lot of criticism about being extremely hard to understand and I'd like to explain exactly why this is.

Initially the project was a closed source one originating from Australia (where I grew up) but once I moved to China in late 2006 I decided that due to the problems of enforcing copyright in China (i.e. no chance) that I would need to make the software "hard to understand" (a sort of security by obscurity approach).

So when I created the initial open source version of CIYAM it inherited years of such development that I had worked on in China (around 6 years).

I no longer actually support the ideas of patent and copyright so I actually don't care too much now about that - what I worked out how to do was to just make a software system so difficult to understand that those things are irrelevant (no-one has even bothered to try and fork my project and modify it).