Databases

[hack_]

Why do we still use an old version of BDB ? Also why do we use both BDB and levelDB?

[1713579876]

1713579876

[domob]

BDB (particularly the old version) is only used for the wallet file since 0.8 (I think), which is done for backwards-compatibility.  In other words, the wallet format was never changed so that people could still access their old wallets (which kind of makes sense).

LevelDB has replaced BDB for performance (and maybe other) reasons for everything else, i. e., the actual "blockchain data" relevant for operations (block index and UTXO set).

[achow101]

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

[domob]

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea.  At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

[hack_]

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea.  At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

Ok, the backwards compat makes sense up to a point....but i should want to believe that instead of full compat, why not just update/ move to a better version and maintain a function that can import the data from old version?
 
I know at least one alt that is already testing SQLite in a relational model, but if it really is faster, i'll tip the dev and maybe we can have a case study for the key/data storage model.

What is wrong with SQL ?

[DumbFruit]

A conversation was had last month, about changing from LevelDB, and the gist of it is; it is fast and it works for what Bitcoin needs it to do.

The backend can be swapped out for something else that provides the
same properties, but doing so does not give you any of the
inspection/analytics that you're looking for.  Systems that do that
exist, and they require databases taking hundreds of gigabytes of
storage and take days to weeks to import the network data.  They're
great for what they're for, but they're not suitable for consensus use
in the system for space efficiency, performance, and consensus
consistency reasons.¹

BerkeleyDB is used to manage the majority of data and log files.² LevelDB is used to store the "chainstate" (Including UTXO) and the "block index".³

As to why one database is used for some things as opposed to others, it's certainly because they are designed for different purposes. BerkeleyDB is slower, but scales well, is robust, and supports read/write concurrency. LevelDB has a very limited feature set but is several times faster than BerkeleyDB⁴.


¹http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011633.html
²https://en.bitcoin.it/wiki/Data_directory
³http://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair
⁴http://highscalability.com/blog/2012/11/29/performance-data-for-leveldb-berkley-db-and-bangdb-for-rando.html

Edit: Fixed wrongness, I think I have it right now..

[droark]

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea.  At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

This. There's been talk, and I believe Jeff Garzik made an experimental fork with a new DB, but I don't think this is being taken seriously just yet. I'd love to see a switch to LMDB but the lack of real 32-bit support appears to make it a non-starter.

[hack_]

A conversation was had last month, about changing from LevelDB, and the gist of it is; it is fast and it works for what Bitcoin needs it to do.

The backend can be swapped out for something else that provides the
same properties, but doing so does not give you any of the
inspection/analytics that you're looking for.  Systems that do that
exist, and they require databases taking hundreds of gigabytes of
storage and take days to weeks to import the network data.  They're
great for what they're for, but they're not suitable for consensus use
in the system for space efficiency, performance, and consensus
consistency reasons.¹

BerkeleyDB is used to manage the majority of data and log files.² LevelDB is used to store the "chainstate" (Including UTXO) and the "block index".³

As to why one database is used for some things as opposed to others, it's certainly because they are designed for different purposes. BerkeleyDB is slower, but scales well, is robust, and supports read/write concurrency. LevelDB has a very limited feature set but is several times faster than BerkeleyDB⁴.


¹http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-October/011633.html
²https://en.bitcoin.it/wiki/Data_directory
³http://bitcoin.stackexchange.com/questions/28168/what-are-the-keys-used-in-the-blockchain-leveldb-ie-what-are-the-keyvalue-pair
⁴http://highscalability.com/blog/2012/11/29/performance-data-for-leveldb-berkley-db-and-bangdb-for-rando.html

Edit: Fixed wrongness, I think I have it right now..

Thanks

[samson]

Also, there are plans to replace LevelDB with something that works faster and better like SQLite.

As far as I know, this has been proposed by some developers from time to time, but I don't think that any kind of "majority" of the core devs agrees this is a good idea.  At the very least, it should not be SQLite but rather some no-SQL key/value database like LevelDB and BDB are.

Ok, the backwards compat makes sense up to a point....but i should want to believe that instead of full compat, why not just update/ move to a better version and maintain a function that can import the data from old version?
 
I know at least one alt that is already testing SQLite in a relational model, but if it really is faster, i'll tip the dev and maybe we can have a case study for the key/data storage model.

What is wrong with SQL ?

It's not fully buzzword compliant if it doesn't use NoSQL. SQL just isn't 'trendy' enough any more.

There's nothing wrong with SQLite for what we need it for.

When I started using databases some 20+ years ago it was all NoSQL.