Putting the database on a USB key

[grondilu]

I've recently tried to install my OS (debian sid) on a USB key (ext4 formatted).   As this key has not much storage room, I thought of using a second removable media (SD card or USB) where to put the bitcoin database.

Problem:  it never actually work well enough.  At first, everything's fine:  the database and the index get filled, but eventually everything is stuck and no more block is added in the block chain.  And I get a Warning in the client saying:

WARNING: Displayed transactions may not be correct!  You may need to upgrade, or other nodes may need to upgrade.


I look in the debug.log and to me it doesn't seem like a connection problem.  I get a repetitive pattern of messages like:

askfor tx 5fce383170099b52a632   0
sending getdata: tx 5fce383170099b52a632
askfor tx 5fce383170099b52a632   1346243977000000
ERROR: FetchInputs() : 5fce383170 mempool Tx prev not found 04e6a51ecb
ERROR: CTxMemPool::accept() : FetchInputs failed 5fce383170
stored orphan tx 5fce383170 (mapsz 10001)
mapOrphan overflow, removed 1 tx


My guess is that with low bitrate media, things screw up at some point.  When I use my internal, non-SSD hard drive,  everything works fine and I can update the database up to the latest block.

I used ext4 for my USB key and I suspect that it would work better with vfat but then I could not use symbolic links to point to my wallet.

Has anyone succeeded in using a usb key to store the bitcoin database ?  If so, which format ?

[wabber]

well actually the error says your client version is too old. Try to get the latest from bitcoin.org

[Vladimir]

Has anyone succeeded in using a usb key to store the bitcoin database ?  If so, which format ?

I run my miners in January 2011 from USB pen drives each with own bitcoind. Had a simple script that loaded database to a memory based fs at the start and some daemon that once a week or so dumped database back to USB drive. wallet.dat was symlinked direct from USB drive.

If you run db direct from USB drive your drive will die painful and agonizing death rather quickly. (too much writing).

[grondilu]

I run my miners in January 2011 from USB pen drives each with own bitcoind. Had a simple script that loaded database to a memory based fs at the start and some daemon that once a week or so dumped database back to USB drive. wallet.dat was symlinked direct from USB drive.

If you run db direct from USB drive your drive will die painful and agonizing death rather quickly. (too much writing).

Ok.  Good to know.    But if you use a memory based fs, you need quite a lot of RAM, right?  I only have 1Go so I guess that won't do.

[Vladimir]

RAM is dirt cheap these days

[kjj]

Has anyone succeeded in using a usb key to store the bitcoin database ?  If so, which format ?

I run my miners in January 2011 from USB pen drives each with own bitcoind. Had a simple script that loaded database to a memory based fs at the start and some daemon that once a week or so dumped database back to USB drive. wallet.dat was symlinked direct from USB drive.

If you run db direct from USB drive your drive will die painful and agonizing death rather quickly. (too much writing).

I have serious doubts about killing flash drives.  Modern flash chips can take a pretty hefty number of writes and should last for a decent number of years, even if written to at full interface speed nonstop.

That said, for running a full node, USB flash drives are too damn slow.  At least the cheap ones I tried were.  I use RAM drives for everything now too.

[grondilu]

I have serious doubts about killing flash drives.  Modern flash chips can take a pretty hefty number of writes and should last for a decent number of years, even if written to at full interface speed nonstop.

That said, for running a full node, USB flash drives are too damn slow.  At least the cheap ones I tried were.

I'm pretty sure it's not a problem with the flash storage device, but rather with the USB connection which just can't transmit data fast enough.  I guess the problem would be the same with any kind of USB external disk drive.

[Vladimir]

Nope, trust me, even the best MLC USB drives do not live in servers long. With all the trashing bitcoind gives em they will not last long at all. It is of course unless the system is built so that writes are rare, but this certainly not the case with bitcoind. Also USB interface is surely not the bottleneck, erasing/rewriting blocks is what takes time AFAIK.

SLC USB drives is another matter of course, but try to find one and even if you do, it would cost an arm and a leg and whatever your miner mines during 2013. 

[kjj]

Hard to say if the problem is the USB bus or the NAND speed (or bus).  I'll do a little research, see if I can find a fast USB 3.0 drive to test with.

[racerguy]

USB is plenty fast enough for bitcoin, it's just that lots of writes are the achilles heel of usb flash storage.

[Raize]

Wow, this is bizarre, I was just working on converting my ACARD RAM drive to be a standalone db store as I started reading this thread. I assume those of you setting up new nodes are using more powerful RAM drives, so what are you using? Anyone planning on something that can do 128 GB or more in anticipation of the blockchain increasing vastly in size in a short period of time?

[randy-waterhouse]

Hard to say if the problem is the USB bus or the NAND speed (or bus).  I'll do a little research, see if I can find a fast USB 3.0 drive to test with.

It's not that hard really .... bitcoin absolutely thrashes the disks ... it's like running Windows instead of Linux