Reviving post: Where's the -walldir option in bitcoind?

[Binary Finery]

This is an attempt to revive Stephen Gornick's post of over 18 months ago

I can't believe this feature has not yet been added or am I missing something?

This makes basic security common sense - a lack of a -walletdir option is a major bug.

If basic security dictates that you maintain a hot-wallet for day to day transactions and a secure offline wallet for your bank account then surely you need the ability to secure and hide your account wallet.

The obvious way to do this is in a Truecrypt hidden volume - that way you're secure (with reasonable precautions) against trojans, hackers etc AND (and this is what no-one seems to be addressing, in your ethereal coder bubble) against physical attacks where some guy threatens to break your legs if you don't reveal your truecrypt password to your off-shore account wallet.

Hiding a small wallet.dat file in a truecrypt hidden volume within a gb volume is impossible to detect, however having to haul the entire blockchain in there as well, kinda makes it obvious there's a hidden volume there.

I'm no coder but this must be two lines of code: if walletdir is NULL then walletdir="%data%\bitcoin" else walletdir=<-walletdir>

Please sort this out or show me a version of bitcoind that uses it already, but not electrum - I like bitcoind, I'm comfortable with it. I've no problem with downloading the complete blockchain, but a copy with each wallet is too much, especially if I need to encrypt public data just to secure my 1mb of private data which then leaves me open to thumb-screw questions about why I have so much empty space in my truecrypt volume.

This option makes so much sense, it would reduce this page : https://en.bitcoin.it/wiki/Securing_your_wallet to:
Download TrueCrypt;
create hidden volume;
stick hot wallet in the outer volume;
stick account wallet in hidden volume;
make several back ups with different encryption headers;
set bitcoind -walletdir=TrueCryptVol:\MyWallet;
mount whichever wallet you wish to use;
boot bitcoind;
make sure no trojan steal your mounted wallet, which is encrpted on bitcoin-qt anyway.
unmount wallet when finished.

Can anyone offer a plausible explanation why this isn't already a feature??

I use TrueCrypt on Linux to create and mount just a small encrypted volume that holds only the wallet.dat.  That wallet.dat is well under 1MB now, but 5MB seems to be a ample yet convenient upper bound as far as size goes.  I'ld like to ensure that the wallet.dat has not and will not ever be stored on a non-encrypted volume.

For me to be able to ensure that a new wallet is created on an encrypted volume, I created a much larger encrypted volume (e.g, 300MB or larger, depending on volume of logging kepts) and launch the Bitcoin client there.  Once I have the wallet.dat, I then move it to the smaller 5MB encrypted volume.

The protected 5MB volume is not normally mounted and is only used sporadically.   When I do wish to use it, I mount the volume and then create a symlink to its wallet.dat before launching bitcoin.

It would be more convenient if, instead of having to create a symlink, I was able to specify which directory to use just for the wallet.dat.  e.g.,
  $ bitcoin -walletdir=/media/truecrypt1 -datadir=/usr/local/bitcoin/data

Something like a -walletdir option would allow me to create the wallet in a separate directory as well.  An added benefit for that would be that I could then skip having the 300MB encrypted volume as well.

Does anyone have a better solution, or other thoughts on this suggestion?

[1714106342]

1714106342

[1714106342]

1714106342

[Pieter Wuille]

The problem is slightly more difficult than just choosing the location of the wallet file.

The problem is BDB: our wallet file is not just a standalone file, but lives in a Berkleley Database environment, which consists of several files (including log files the database files refer to). Being able to have the wallet be stored in a placed independent from the other databases requires splitting the database environment in two. There are several difficulties with doing this, and it's not a real solution.

The real solution is moving away from BDB for wallets. It was a bad design decision to use BDB for wallets, and has caused us numerous problems already. We've been working on implementing a new wallet format that doesn't depend on a database before, but there have been more urgent issues recently.

In short: yes this needed, yes it will be implemented, but no it's not that easy, and it probably won't be done very soon.

[n8rwJeTt8TrrLKPa55eU]

Binary Finery, I share your pain, and this was the single dealbreaking issue that caused me to move from the full client to Electrum.  You might have to bite the bullet and do likewise.

[gmaxwell]

Can anyone offer a plausible explanation why this isn't already a feature??

Because you haven't submitted a pull request for it, of course.

As Pieter points out, it's not a two line change at all.

And although it would very much be a welcome feature I think you're overestimating it's usefulness. It's quite hard to hide the evidence of a second wallet on a partially encrypted system— shell history, log entries, etc would point to the location of another wallet. We have integrated encryption already for basic security, and the concept of a not-online wallet really requires the not online-wallet be actually not-online— trojans/malware/and compromises will _happily_ steal your displaced wallet when you _do_ access it, as it's absolutely trivial to just watch the bitcoin process and take whatever wallet it accesses.

I find the admonishment to use electrum in response to a message apparently concerned about security to be quite perplexing. Electrum is useful and interesting software, but it has a _much_ weaker security model.

[JompinDox]

Can anyone offer a plausible explanation why this isn't already a feature??

Because you haven't submitted a pull request for it, of course.

As Pieter points out, it's not a two line change at all.

And although it would very much be a welcome feature I think you're overestimating it's usefulness. It's quite hard to hide the evidence of a second wallet on a partially encrypted system— shell history, log entries, etc would point to the location of another wallet. We have integrated encryption already for basic security, and the concept of a not-online wallet really requires the not online-wallet be actually not-online— trojans/malware/and compromises will _happily_ steal your displaced wallet when you _do_ access it, as it's absolutely trivial to just watch the bitcoin process and take whatever wallet it accesses.

I find the admonishment to use electrum in response to a message apparently concerned about security to be quite perplexing. Electrum is useful and interesting software, but it has a _much_ weaker security model.

Huh... Care to elaborate?
What are the security issues with the Electrum model, and can they lead to coin loss or theft? I'd really like to know, as I've switched to it as my main wallet and it seems to be quite well designed - even though it's still a work in progress.

[gmaxwell]

Huh... Care to elaborate?
What are the security issues with the Electrum model, and can they lead to coin loss or theft? I'd really like to know, as I've switched to it as my main wallet and it seems to be quite well designed - even though it's still a work in progress.

Electrum depends on servers to not lie to it about the existence or validity of transactions. You could learn over the network that you were paid a million bitcoin, but its all a fabrication. Worse, the value of your transactions could be _understated_, causing you to sign transactions when you spend which give away all your coin into fees (which then the attacker would presumably mine themselves to collect). These attacks don't require any computing power, they only require that your software is talking to the attacker.  Electrum gives security similar to a JS web wallet, except it has the improvement that the operator can't swap out your client code right from under you.

[n8rwJeTt8TrrLKPa55eU]

I find the admonishment to use electrum in response to a message apparently concerned about security to be quite perplexing. Electrum is useful and interesting software, but it has a _much_ weaker security model.

GMaxwell, to address your perplexity: under some scenarios and for some people, the advantages of small on-disk footprint, undetectability, plausible deniability, and quick startup might justify the risk entailed in connecting to a potentially rogue Electrum server.

I have lived in (and still travel to) countries where buying and selling foreign currency is illegal.  Not all Bitcoin users live in the USA or Western Europe, where currency controls are minimal and property rights are somewhat respected.  Bitcoin's legal status is still up in the air, and it's likely to be eventually demonized or treated as a foreign currency by the more oppressive governments worldwide.  For some people in some locations, it might be critical that forced searches and seizures of their computer do not reveal meaningful Bitcoin holdings or activity.

You can draw an analogy to holding gold.  You can hold it yourself, or trust a 3rd party to hold it for you somewhere else.  The former is undeniably more secure under peaceful conditions in a civilized country with strong rule of law and respect for property rights.  But if you're a Jew in 1940 Germany with stormtroopers knocking on your door, untraceability and plausible deniability might be much more important.

Personal physical security is, for many of us, a higher-order need than transactional security.

BTW, unrelated to the above, there is another reason why, statistically, nonstandard clients might be a more secure option than the reference client, even with the risk of bad servers: they are less likely to be targeted by client-side viruses, trojans, and keyloggers.  Perhaps virus writers have already started looking for Electrum installations and vulnerabilities...but perhaps not quite yet.  For computers whose risk of getting infected is high (e.g. a Windows system shared among family members), it might be logical and prudent to always run the least popular and most lightweight client possible, even if it exposes you to the (arguably smaller?) risk of the "bad server" scenario you described.

PS: note that I have nothing but gratitude and admiration for the developers of the reference client, and understand that the goals and priorities of that project have to be carefully chosen given limited time and resources.

[Binary Finery]

The problem is slightly more difficult than just choosing the location of the wallet file.

The problem is BDB: our wallet file is not just a standalone file, but lives in a Berkleley Database environment, which consists of several files (including log files the database files refer to). Being able to have the wallet be stored in a placed independent from the other databases requires splitting the database environment in two. There are several difficulties with doing this, and it's not a real solution.

The real solution is moving away from BDB for wallets. It was a bad design decision to use BDB for wallets, and has caused us numerous problems already. We've been working on implementing a new wallet format that doesn't depend on a database before, but there have been more urgent issues recently.

In short: yes this needed, yes it will be implemented, but no it's not that easy, and it probably won't be done very soon.

Thanks Pieter for your patient response, but just humour me a little further:

So are you saying wallet.dat effectively comprises a table in the BDBE - does BDB not support linked tables?
If wallet.dat is linked in with these other database files, how come it can be renamed and replaced with a new wallet.dat without any discernible difference apart from I can now send BTC from my new wallet -what difference does the BDB see if this file is outside the data folder? If Access can manage linked tables... - I know, bad example ;-)

n8rwJeTt8TrrLKPa55eU - cheers for pointing out that BTC security is not all about trojans and keystroke loggers. Having a mounted outer with a hot wallet or hidden volume with account wallet, if done right, should be indistinguishable to any shell history or log entry, and thus retain plausible deniability. Trojan wallet stealers should be thwarted by the wallet encryption, but someone physically forcing you to reveal your passwords on any visible file can only be addressed by hidden volumes and only then if they are much smaller than the outer volume.

gmaxwell - your holier than thou, coder superiority is not helpful, with you pull requests and shell histories. The -walletdir thread was posted over 18 months ago by Stephen and promptly died. We all appreciate the amazing work you guys are doing getting the bitcoind to it's current state, but most of us don't understand what's behind the system and that's only going to grow as adoption increases - embrace the eternal September dude, most people don't have the time to understand the ins and out of BDBE, we just see features that we would like or things we'd like to change. Bitcoind is still a terrifying prospect to install and use for most mortals, when it comes to storing actual wealth on your computer. Making it clear which files actually give you (or any thief) access to your cash, should be one-syllable-a-minute, crystal clear to the most sausage-fingered retard able to switch on a PC. If someone would like to point me to the pull request list, I'd happily hammer it up there.

[Binary Finery]

Ah, just spotted, someone helpfully bumped this to the top of the tech discussion:
https://bitcointalk.org/index.php?topic=4571.0
Thanks...

[gmaxwell]

gmaxwell - your holier than thou, coder superiority is not helpful, with you pull requests and shell histories.

It was not intended to be a holier than anything response. It was the frank, honest reason. Why isn't it done?— because no one who wanted it done has done it, people doing things have considered other things more important. It isn't an idea whos usefulness had escaped attention, though that happens sometimes.

On the contrary, I could argue your tone here sounds rather holier-than, with your imperative "Please sort this out or show me a version of bitcoind that uses it already", and your essays on access, linked tables, and two line changes. .... but sometimes tone doesn't come across clearly online and I had no more intention of offending you than you did of offending me, I'm sure.

what difference does the BDB see if this file is outside the data folder?

You can only switch between the files when they've been cleanly closed. Desync of the wallet.dat and the other database coupled with an unclean shutdown results in database corruption which can not be automatically repaired.

most people don't have the time to understand the ins and out of BDBE, we just see features that we would like or things we'd like to change. Bitcoind is still a terrifying prospect to install and use for most mortals,

Not an argument against it, as I said I think it woudl be great (personally I'd use it to have wallet files on more reliable media than the block databases), but a walletdir is far outside of a "most mortals" feature in any case. As generally is plausible denyability. I'm always a little weary about brittle security: People are better off not having denyabilty and knowing they don't have it, then they are not having it and thinking they have it. Moving the wallet is a fine feature, but it does not achieve the goal of hiding bitcoin usage and shouldn't be advertised as such.

[kjj]

So are you saying wallet.dat effectively comprises a table in the BDBE - does BDB not support linked tables?
If wallet.dat is linked in with these other database files, how come it can be renamed and replaced with a new wallet.dat without any discernible difference apart from I can now send BTC from my new wallet -what difference does the BDB see if this file is outside the data folder? If Access can manage linked tables... - I know, bad example ;-)

BDB doesn't work the way you are thinking.

BDB records extra data outside of the database files in an attempt to ensure consistency across (some types of) crashes.  For historic reasons, BDB expects database files all to be in the same place, and all of the files in that place share a single journal set.  The log.* files in database/ are the BDB journal, FYI.  As long as the files are closed properly and marked as clean ("detached"), you can swap them back and forth with no problems.  It simply doesn't support having files in multiple places, and there is nothing that bitcoin's developers can do about it.

The only option is to work around it by creating a whole second BDB instance, doubling the overhead, doubling the development effort needed, and bloating the code.  Oh, and then the client would have to rescan after even trivial "clean" crashes (maybe even on normal starts too, now that I think about it), because there would be no way to do transactions across both instances and so the code could never be sure that both databases were synchronized, even if both were marked clean.

I'm not even sure if Access does any journaling, it's been a long time since I used it, and google results weren't very helpful.  If it does do any, it does it inside the file, not externally like BDB.  Also, Microsoft has gigantic systems (OLE, DDE, etc) for data access abstraction hidden in the operating system so that their products can interact for "free".  The Bitcoin devs get no such help, they have to use the lowest common denominator between Windows, Macs and Unix, which is nothing.  Which means that they need to bring along their own third-party database library, and it needs to be freely distributable, plus work across all three platforms.  BDB fits that bill, not much else does.

gmaxwell - your holier than thou, coder superiority is not helpful, with you pull requests and shell histories. The -walletdir thread was posted over 18 months ago by Stephen and promptly died. We all appreciate the amazing work you guys are doing getting the bitcoind to it's current state, but most of us don't understand what's behind the system and that's only going to grow as adoption increases - embrace the eternal September dude, most people don't have the time to understand the ins and out of BDBE, we just see features that we would like or things we'd like to change. Bitcoind is still a terrifying prospect to install and use for most mortals, when it comes to storing actual wealth on your computer. Making it clear which files actually give you (or any thief) access to your cash, should be one-syllable-a-minute, crystal clear to the most sausage-fingered retard able to switch on a PC. If someone would like to point me to the pull request list, I'd happily hammer it up there.

Shit, September is right around the corner isn't it...  By the way, pull requests are for patches, not feature requests.

[Pieter Wuille]

So are you saying wallet.dat effectively comprises a table in the BDBE - does BDB not support linked tables?
If wallet.dat is linked in with these other database files, how come it can be renamed and replaced with a new wallet.dat without any discernible difference apart from I can now send BTC from my new wallet -what difference does the BDB see if this file is outside the data folder? If Access can manage linked tables... - I know, bad example ;-)

BDB is not like SQL databases. A BDB environment consists of database files and log files, each database file consisting of one or more tables, and each table consisting basically just of key-value pairs. The environment is used to allow atomic transactions that span multiple files, and manage access from multiple processes to the tables without conflicts - all features we don't use. I'm not sure what you mean by linked tables - I'm not talking about the tables/values represented by the database files, but rather their actual implementation. The log files are necessary to recover from crashes (or at least intended for that), to be able to redo or undo partially completed transactions.

Also, for the wallet, we ask BDB to detach the database file from the environment at shutdown, to be sure it can be backed up and copied around. This works, but is certainly not how BDB is intended to be used.

[Pieter Wuille]

The only option is to work around it by creating a whole second BDB instance, doubling the overhead, doubling the development effort needed, and bloating the code.  Oh, and then the client would have to rescan after even trivial "clean" crashes (maybe even on normal starts too, now that I think about it), because there would be no way to do transactions across both instances and so the code could never be sure that both databases were synchronized, even if both were marked clean.

This is not entirely true, to be honest. Splitting the environment would mean some extra code, but after recent cleanups, it would certainly not be too hard anymore. Wallets store information about which blockchain they last saw (since 0.3.21 iirc), and are effectively rescanned at startup when there is a mismatch already, and we don't really do transactions across database files.

The only reason that hasn't been done, is because it is not the final solution. We've had so much troubles with BDB, because we don't use it in a way it was intended for (big servers with power generators, managing large database files simultaneously being accessed by many processes, with all log files backed up regularly to tapes, and software upgrades being performed manually by a database administrator; in this setting, it is rock solid).

As I said, the only real solution - in my opinion - is moving away from BDB and use a key-value store format that guarantees consistency in a single file. As the wallet is loaded entirely in memory anyway, that shouldn't be hard at all. I've already worked on this for a while, but right now, there are more urgent problems, and we're all volunteers that need to choose what to spend time on.

[Binary Finery]

Cool - you've convinced me it's not a trivial issue. A little knowledge is a dangerous thing.

Apologies for hassling you guys - I'll just have to maintain multiple blockchain databases until someone takes on the challenge.

Thanks for clearing it up. I know it's kind of bike shed considering the intricate work you guys do, do, but do post a bitcoin address whoever does post a -walletdir pull that is accepted, as I like to donate for features I can understand.

[Mike Hearn]

Clients based on bitcoinj don't use a database for the wallet, they use a single file. As a result you can, with a bit of effort, maintain wallets in any directory you want. MultiBit lets you do that, for example. There's also a command line tool called wallet-tool in the bitcoinj distribution that lets you do things like this:

./wallet-tool --wallet=/foo/foo.wallet --chain=/foo/foo.chain --action=DUMP
./wallet-tool --wallet=/foo/foo.wallet --chain=/foo/foo.chain --action=SEND --output=1AbCdE....:2.0

and so on.

The main caveat with this is that at the moment, bitcoinj does not support automatic rescanning if the wallet gets out of sync with the chain file. That's why you need to specify and store the .chain (it's a few megabytes) alongside the wallet.

Obviously at some point soon we'll fix that and then you can just have a single chain file somewhere else that's used with multiple wallets.

[Binary Finery]

Clients based on bitcoinj don't use a database for the wallet, they use a single file. As a result you can, with a bit of effort, maintain wallets in any directory you want. MultiBit lets you do that, for example. There's also a command line tool called wallet-tool in the bitcoinj distribution that lets you do things like this:

./wallet-tool --wallet=/foo/foo.wallet --chain=/foo/foo.chain --action=DUMP
./wallet-tool --wallet=/foo/foo.wallet --chain=/foo/foo.chain --action=SEND --output=1AbCdE....:2.0

and so on.

The main caveat with this is that at the moment, bitcoinj does not support automatic rescanning if the wallet gets out of sync with the chain file. That's why you need to specify and store the .chain (it's a few megabytes) alongside the wallet.

Obviously at some point soon we'll fix that and then you can just have a single chain file somewhere else that's used with multiple wallets.

This option seems to provide a reasonable solution to the problem - thanks Mike, and great talk at Bitcoin2012 btw.

This Matonis article highlights the need to be able to protect wallets from physical coercion.
http://www.forbes.com/sites/jonmatonis/2012/09/12/key-disclosure-laws-can-be-used-to-confiscate-bitcoin-assets/