Most likely not safe. The LevelDB isn't really 100% ACID, especially the chainstate directory is prone to corruptions if the node crashes.
Edit: on the other hand if you have a tight backup schedule or the underlying file system supports snapshots then it is OK to run them over NFS because neither the blocks nor chainstate LevelDB databases are really critical and can be always rebuilt after the corruption. It all depends on what exactly is your tolerance profile for outages.
I have 3 servers that host nodes for several different block chains (altcoins) and 1 NFS server used to store the blockchain and wallet data files.
I've certainly starting to run into issues with the altcoins based on 0.8.x and 0.9.x codebase. So far Bitcoin 0.10.1 seems to do okay. The NFS server kernel panicked while the nodes where still running and it seems that the nodes continue to 'write' to the databases even though the mount is offline. The operating system has 'paused' all operations to that mount yet the running mode does not seem to honor that.
The end result is devastating at times and fits in to these 3 categories:
- wallet.dat will become damaged, so will backups
- "Large valid fork found" error
- "Error loading blkindex.dat"
In all cases you don't know something is wrong until
after you restart the node, which make matters worse.
wallet.datI run wallet.dat backups every hour thinking I would be safe. However when wallet.dat gets damaged, the backup is also damaged. This is strange and the worst case. Thankfully I've only seen it once and happen to have had an older backup that was not damaged.
With both the original wallet.dat and the backup version you will see something like this in debug.log indicating it's hosed.
2015-05-21 00:39:12 block index 5175ms
2015-05-21 00:39:12 init message: Loading wallet...
2015-05-21 00:39:12 ERROR: CTransaction::CheckTransaction() : vout empty
2015-05-21 00:39:22 Error reading wallet database: CPrivKey pubkey inconsistency
2015-05-21 00:39:24 Error loading wallet.dat: Wallet corrupted
wallet 12226ms
2015-05-21 00:39:24 init message: Rescanning.
2015-05-21 00:42:24 AddToWallet ....<redacted txid>
2015-05-21 00:42:24 AddToWallet ....<redacted txid>
2015-05-21 00:42:25 AddToWallet ....<redacted txid>
...
2015-05-21 00:43:27 rescan 242591ms
2015-05-21 00:43:27 init message: Loading addresses...
2015-05-21 00:43:27 Loaded 19079 addresses from peers.dat 62ms
2015-05-21 00:43:32 Shutdown : In progress...
2015-05-21 00:43:32 Flush(false)
2015-05-21 00:43:32 wallet.dat refcount=0
2015-05-21 00:43:32 wallet.dat checkpoint
2015-05-21 00:43:32 wallet.dat detach
2015-05-21 00:43:33 wallet.dat closed
2015-05-21 00:43:33 DBFlush(false) ended 301ms
2015-05-21 00:43:33 StopNode()
2015-05-21 00:43:33 Flushed 19079 addresses to peers.dat 47ms
2015-05-21 00:43:33 Committing 252 changed transactions to coin database...
2015-05-21 00:43:33 Flush(true)
2015-05-21 00:43:33 wallet.dat refcount=0
2015-05-21 00:43:33 wallet.dat checkpoint
2015-05-21 00:43:33 wallet.dat detach
2015-05-21 00:43:33 wallet.dat closed
2015-05-21 00:43:33 DBFlush(true) ended 240ms
2015-05-21 00:43:33 Shutdown : done
and in db.log
Page 1229: unreferenced page
Page 1229: totally zeroed page
Page 1230: unreferenced page
Page 1230: totally zeroed page
Page 1231: unreferenced page
Page 1231: totally zeroed page
wallet.dat: DB_VERIFY_BAD: Database verification failed
The question is, why is the backup corrupted?
Large valid fork foundThis error I've only seen with 0.9.x version, it's not the worst thing in the world as long as you catch it. As with all cases the error does not appear until after the node is restarted.
2015-05-21 00:18:20 CheckForkWarningConditions: Warning: Large valid fork found
forking the chain at height 2887908 (0000000000017225f9c08e76062d60aa616a14781e6e89a82febe6bd90167052)
lasting to height 2888067 (000000000000f6f16950d0af208bf8087a4c726cb7d935329bd74985f02a9b75).
Chain state database corruption likely.
Warning: The network does not appear to fully agree! Some miners appear to be experiencing issues.
The node continues to run in this state until you do a reindex. Very scary.
Error loading blkindex.datThis has been a very common issue, especially with older versions (0.8.x and whatever Peercoin is based off of). In some versions I can delete blkindex.dat and it will reindex on it's own, in other version I have to run with the 'reindex' command. Worst case here is I need to re-download the chain.
debug.log will look like this, the node will attempt to start then shutdown:
05/21/15 00:10:28 Verifying last 2500 blocks at level 1
05/21/15 00:10:28 ERROR: CheckProofOfWork() : nBits below minimum work
05/21/15 00:10:28 ERROR: CBlock::ReadFromDisk() : errors in block header
05/21/15 00:10:28 ERROR: LoadBlockIndex() : block.ReadFromDisk failed
05/21/15 00:10:28 block index 88932ms
05/21/15 00:10:28 Loading wallet...
05/21/15 00:10:28 nFileVersion = 80501
05/21/15 00:10:28 Error loading blkindex.dat
wallet 282ms
05/21/15 00:10:28 Loading addresses...
05/21/15 00:10:28 Loaded 11294 addresses from peers.dat 72ms
05/21/15 00:10:28 mapBlockIndex.size() = 1509863
05/21/15 00:10:28 nBestHeight = 1508834
05/21/15 00:10:28 setKeyPool.size() = 101
05/21/15 00:10:28 mapWallet.size() = 0
05/21/15 00:10:28 mapAddressBook.size() = 1
05/21/15 00:10:28 Done loading
05/21/15 00:10:28 Emerald: Error loading blkindex.dat
Here is another example where I had to run reindex manually
2015-05-20 23:56:10 Opened LevelDB successfully
2015-05-20 23:56:49 LoadBlockIndexDB(): last block file = 4
2015-05-20 23:56:49 LoadBlockIndexDB(): last block file info: CBlockFileInfo(blocks=72806, size=32576240, heights=1128729...1199945, time=2015-04-17...2015-05-20)
2015-05-20 23:56:49 LoadBlockIndexDB(): transaction index disabled
2015-05-20 23:56:49 LoadBlockIndexDB(): hashBestChain=000000000000521f11b8f41aa943f28e6ff5f2bd304d0dc7cef217c442812790 height=1199945 date=2015-05-20 23:38:48
2015-05-20 23:56:49 init message: Verifying blocks...
2015-05-20 23:56:49 Verifying last 288 blocks at level 3
2015-05-20 23:56:52 ERROR: CheckProofOfWork() : nBits below minimum work
2015-05-20 23:56:52 ERROR: ReadBlockFromDisk(CBlock&, CDiskBlockPos&) : errors in block header
2015-05-20 23:56:52 ERROR: VerifyDB() : *** ReadBlockFromDisk failed at 1199799, hash=000000000000befd4121befe5272aff3767133e6205d073a87086876273a55c7
2015-05-20 23:56:52 : Corrupted block database detected.
Do you want to rebuild the block database now?
2015-05-20 23:56:52 Aborted block database rebuild. Exiting.
2015-05-20 23:56:52 Flush(false)
2015-05-20 23:56:52 DBFlush(false) ended 0ms
2015-05-20 23:56:52 StopNode()
2015-05-20 23:56:52 Flushed 0 addresses to peers.dat 23ms
2015-05-20 23:56:52 Committing 161 changed transactions to coin database...
2015-05-20 23:56:52 Flush(true)
2015-05-20 23:56:52 DBFlush(true) ended 0ms
With one coin there was no blkindex.dat file present (even with a valid data directory) yet it complains it can't load it. This one I had to re-download the chain.
Amazingly (and thankfully) the Bitcoin blockchain has not suffered corruption as of yet.
The worst part about all this is that not even your backups are safe.
Besides NOT using NFS (not an option for the time being), is there a way to minimize the damage? Would taking hourly volume snapshots on the NFS server be of any value? It runs ZFS.
Show Posts
or he will dump and kill it early. Either way with 10% premine there is going to be only one winner with this coin...the dev. Best chance to make money is mine and sell before pos...and hope dev don't dump.
