Berkeley Database version 4.8 requirement unsustainable

[Cryddit]

Berkeley Database 4.8 isn't even available any more from "current" sources (package managers, etc).  It's been obsoleted and replaced.  Debian is even moving away from bdb all together and will soon cease to have *any* bdb package.

Yet Bitcoind relies on it for a format for a portable wallet.

This wallet is only "portable" for as long as bdb4.8 is available; it's already impossible to build a client that takes a standard wallet without hunting through archives for dusty old code.

We can migrate to the new version (5.3) - I can build that, easily.  But I probably won't be able to build it two years from now.

Or we can have the bitcoin source take charge of its own wallet format and nail it down to eliminate a dependency.

[1714200295]

1714200295

[1714200295]

1714200295

[1714200295]

1714200295

[gmaxwell]

Changing to our own append only format has been the tentative plan for a long time.  BDB is not acceptable going forward for many reasons, and we've already eliminated it for non-wallet needs.  Breaking wallet compatibility is awkward though so no need to rush into it.

Your concerns about "unavailability" however are not interesting to me. The software is freely licensed, and we can just package and bundle it ourselves, and will likely have to as part of a wallet importing tool for the foreseeable future.

[Cryddit]

eh.  It's a build requirement that's getting hard to satisfy.  I thought I'd point it out.

[wumpus]

Version 4.8 is not a requirement, just a recommendation. You can use newer versions of BDB by passing --with-incompatible-bdb to configure.

The main reason it is discouraged is because it breaks compatibility with the pre-built binaries. If you're building from source you may not care.

But even if you do, it is trivial to convert your wallet back to a 4.8 wallet if you're stuck with a 5.1 or 5.3 one. Use the bdb tools (packages db4.8-util db5.1-util on Ubuntu):

Code:

    db5.1_dump wallet.dat.db5 | db4.8_load wallet.dat.db4

[Cryddit]

Oh, good.  I hadn't seen the db utilities. 

I had been thinking that what I build from source wouldn't be able to use my existing wallet, and downloading the prebuilt binaries instead for that reason.

Thank you for pointing that one out. 

But, trivial conversions or not, it's still only what's available this month.  Or this year. 

And the next version and the next, and the next.  It's just bad practice to rely on an external utility whose save format changes unpredictably when we're talking about taking care that people's money remains available to them.

[wumpus]

And the next version and the next, and the next.  It's just bad practice to rely on an external utility whose save format changes unpredictably when we're talking about taking care that people's money remains available to them.

You're preaching to the choir there.
I'd say "scratch your itch" that's how open source works.

[apetersson]

if there is any remote plan to move to a new format, it is a good idea to include that change early, and already migrate.

so if you some day drop BDB completely from the standard distribution, you can say, wallets v0.XX or later do not need a migration.

[Nagle]

Don't build your own database. You will get it wrong. It is really, really hard to build a reliable database system.

SQLite is a good option for a wallet. It's free, it's a single-file system, it requires no installation if built into a program, it has a good reputation, it's very widely used, it can recover from a crash, and read performance is good. It's not a good choice for a system with lots of concurrent updates, which is why it's not used in high-volume back-end systems, but for a single-user wallet, it should be fine.

Also, if the wallet code uses SQL, big sites can revise the wallet to use other SQL databases, which a high-volume site might need.

[Cryddit]

Don't build your own database. You will get it wrong. It is really, really hard to build a reliable database system.

I agree with you, it's really hard.  I know that it is.  But, I think a database is overkill here. 

The set of things a wallet needs to keep track of is actually fairly simple.  And the append-only Pattern is a good robust way of achieving it, and far simpler (less prone to error) than a full database.  With a database, you complicate matters immensely by indexing and putting data in the middle of other data. 

The only thing that goes beyond the bare capabilities of a filesystem here is multithreaded access.  You really don't want to lock on something as slow as file access, but you absolutely cannot let multiple threads have write access at once.   But you don't need a full-on database to solve that problem.

What you need is a 'listener' that launches threads that can talk to application threads trying to do a transaction, and reports them back to the listener.  The listener would keep track of multiple requests from multiple children, but it would have sole (serial) access to write, flush buffer, read back to make sure it happened correctly, and notify whichever child process that handed it that transaction of whether it did or not.  The listener (and its child threads) should never see the plaintext of a privkey, so the security requirements are manageable. 

[Nagle]

Don't build your own database. You will get it wrong. It is really, really hard to build a reliable database system.

I agree with you, it's really hard.  I know that it is.  But, I think a database is overkill here. 

Many people think that when designing data storage. But they they find they need dump/restore capability, or audit capability, or they want to generate reports from the data. With a real database, even SQLite, you can do all that from the user interface or API. If you have your own file format, someone has to modify low level code, with the risk of breaking it. Also, all the serious databases handle things like recovering the database after a crash. If you're just appending to a file, you may get back a truncated file or a last page of junk after a reboot. Real databases  are able to recover from such situations.

[wumpus]

Many people think that when designing data storage. But they they find they need dump/restore capability, or audit capability, or they want to generate reports from the data. With a real database, even SQLite, you can do all that from the user interface or API. If you have your own file format, someone has to modify low level code, with the risk of breaking it. Also, all the serious databases handle things like recovering the database after a crash. If you're just appending to a file, you may get back a truncated file or a last page of junk after a reboot. Real databases  are able to recover from such situations.

But does "Real databases" include SQLite here? Does SQLite gives any assurances in the case of sudden reboots? At least with an append-only file (and regular syncing) you can be fairly sure that only record of your last activity before the reboot will be lost, and that not some management structure in the middle broke and made it impossible to parse the entire file.

In any case: because of stability of the file format we don't want an external dependency for the database anymore. Relying on an external database already caused one long-running train wreck as you can see in this very thread.

Of course, embedding SQlite would be possible, unlike leveldb it is only one (very large) C file, but that'd be only worth it if that database would be a good match for a wallet, and will not bring us from one ditch into another.

Personally I would prefer a simple wallet format that is fully under our control.

[wheatstone]

But does "Real databases" include SQLite here? Does SQLite gives any assurances in the case of sudden reboots? At least with an append-only file (and regular syncing) you can be fairly sure that only record of your last activity before the reboot will be lost, and that not some management structure in the middle broke and made it impossible to parse the entire file.

Transactional. Atomic commits. ACID.

Yes, SQLite does well with power failures.

[Cryddit]

In any case: because of stability of the file format we don't want an external dependency for the database anymore. Relying on an external database already caused one long-running train wreck as you can see in this very thread.

Of course, embedding SQlite would be possible, unlike leveldb it is only one (very large) C file, but that'd be only worth it if that database would be a good match for a wallet, and will not bring us from one ditch into another.

Personally I would prefer a simple wallet format that is fully under our control.

+1, all three points.  Complexity and security are very hard to mix, and something as complex as the linkage to an external database is a well of impending disasters.  Embedding a particular version of SQlite would be acceptable - it would get all the necessary code into the project and under strict control at the very least - but I agree that the simpler the save format, the easier it is to audit and bugfix in the code that reads/writes it. And the Append-only Pattern is very simple and does achieve transactional integrity. 

[wumpus]

Transactional. Atomic commits. ACID.

Note that BerkelyDB has the same buzzword bingo, but does fail in ugly ways sometimes when the process is suddenly terminated.

It may make more sense to link to the sqlite page: https://www.sqlite.org/howtocorrupt.html .

"An SQLite database is highly resistant to corruption. If an application crash, or an operating-system crash, or even a power failure occurs in the middle of a transaction, the partially written transaction should be automatically rolled back the next time the database file is accessed."

They claim to be fairly resistant to this. OTOH I doubt that there is any magic going on. That's pretty much the same with an append-only file that syncs regularly and ignores partial records at the end.

If you're worried about data loss it's more pressing to switch to deterministic wallets. At least then there is no time frame in which keys can 'disappear', they can always be generated again.

[Cryddit]

Even if we believe that we need a full-on database for live wallets, we should at minimum save a serialized file (a CSV table with encrypted individual values, for example) that can be written by any version of bitcoin and read by any other.  Using that as the wallet file, we could upgrade or change the database back-end without breaking wallet compatibility. 

[CoinBuzz]

why not define an specific database for bitcoin to decrease the blockchain?

I believe BerkeleyDB save a lot of metadata for it's own work/features that maybe we dont need them at all.

[gmaxwell]

why not define an specific database for bitcoin to decrease the blockchain?
I believe BerkeleyDB save a lot of metadata for it's own work/features that maybe we dont need them at all.

BDB is _only_ used for the wallet. The blockchain itself is stored as raw blocks.

[alandsidel]

Personally, I'd rather see a move to something standard as well.  Maintaining some proprietary format is silly unless libraries to use that format are going to be released as well.

BDB won't work indefinitely, obviously.  The sharp edges are already poking through when trying to share a wallet between processes, or when the number of transactions in the wallet gets large.  An append-only file is a nice idea, but precludes nice-to-have things like removing/archiving old transactions.  In the current format this is already desirable since the wallet gets very slow with tens or hundreds of thousands of transactions in it.

The three available wallet hooks are suboptimal for maintaining synchronization with an external database thanks to use of txid's in the notification and behavior during a chain fork and resync.  Watching for wallet transactions that reach a specific confirmation level requires polling the (growing, slowing) wallet once you've received the initial confirmation level 1 notice, and you must check that all previously recorded transactions are still valid.

IMHO as a 'newbie' to bitcoin stuff but a developer with a few decades experience in software development, the 'right thing to do' is to abstract the wallet storage functionality out within the client, and allow the user to decide at build time what backend to use.  If the wallet only needs append(), get(account), and so forth, it is not precluded from using a more advanced backend to satisfy those calls through an abstraction layer.

[wumpus]

Even if we believe that we need a full-on database for live wallets, we should at minimum save a serialized file (a CSV table with encrypted individual values, for example) that can be written by any version of bitcoin and read by any other.  Using that as the wallet file, we could upgrade or change the database back-end without breaking wallet compatibility. 

This already exists. See dumpwallet/importwallet RPCs in 0.9.0.

[wumpus]

FYI I've created an issue for this. Please help implementing it

https://github.com/bitcoin/bitcoin/issues/3971