Bitcoind crashes after logging off of SSH session

[bigbeninlondon]

Ok,

I've been around and around and I have no earthly idea what's going on here. 

I built bitcoind ("version" : 99900) from git on a headless ubuntu 13 machine.  Running the following command:

Code:

sudo bitcoind -datadir=/home/ben/.bitcoin/ -conf=/home/ben/.bitcoin/bitcoin.conf -daemon

Everything works great, as long as I remain connected to my ssh session.  Logging off for a few seconds doesn't interrupt, but if I leave it disconnected for more than a few seconds, I get the following error:

Code:

2014-08-23 06:02:13 LevelDB read failure: IO error: /home/ben/.bitcoin/chainstate/000174.sst: No such file or directory
2014-08-23 06:02:13 IO error: /home/ben/.bitcoin/chainstate/000174.sst: No such file or directory
2014-08-23 06:02:13 *** System error: Database I/O error
2014-08-23 06:02:13 Error: System error: Database I/O error
2014-08-23 06:02:13 ERROR: ProcessBlock() : AcceptBlock FAILED
2014-08-23 06:02:13 Loaded 754 blocks from external file in 4783ms
2014-08-23 06:02:13 Reindexing block file blk00006.dat...
2014-08-23 06:02:13 addcon thread interrupt
2014-08-23 06:02:13 opencon thread interrupt
2014-08-23 06:02:13 msghand thread interrupt
2014-08-23 06:02:13 dumpaddr thread stop
2014-08-23 06:02:13 ProcessBlock: ORPHAN BLOCK 0, prev=00000000000007169e562472fb20f0e940fc2b17b8aa27be620c49f0da3abc14
2014-08-23 06:02:13 net thread interrupt
2014-08-23 06:02:13 Shutdown : In progress...
2014-08-23 06:02:13 StopNode()
2014-08-23 06:02:13 Shutdown : done

Different "sst" file every time I restart.  Doesn't matter how far along I am in the chain.

For some reason, it's as if the chainstate sst files just "disappear" after I log off my ssh session.  Restarting gives me:

Code:

Error opening block database.

Do you want to rebuild the block database now?

Running again with "-reindex" re-indexes just fine and starts it back up; that is until I exit my ssh session again.

I've rebuilt bitcoin from source, downloaded the blockchain about 40 times, moved the data directory, etc.

Any suggestions?

[1714234906]

1714234906

[1714234906]

1714234906

[1714234906]

1714234906

[Justin00]

Not sure of the actual fix, but a short term solution to get around the issue could be to use "screen"

[cr1776]

Not sure of the actual fix, but a short term solution to get around the issue could be to use "screen"

Yes, running with screen should help.

[bigbeninlondon]

I haven't really pursued that option just because what is a "daemon" actually for if it doesn't daemonize?

I plugged in a monitor and keyboard and logged in to the machine directly, ran the daemon and left it logged in.  That's kept it from crashing so far this morning... 

[Foxpup]

You need to use the -daemon argument to daemonise it. Though, it should exit cleanly on hangup otherwise.

[btchris]

As others have said, screen is your friend.

Ubuntu Server 13 also has byobu (installed by default I believe), just run byobu-enable, and then choose your activation key (press Ctrl-a, and choose an option, I prefer the first). Then instead of logging out, use Ctrl-a d (on either an SSH session or the console).

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

[bigbeninlondon]

You need to use the -daemon argument to daemonise it. Though, it should exit cleanly on hangup otherwise.

I am using -daemon to daemonize it.  It's the last flag on the command.

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

I'll give that a try and see what happens.

[bigbeninlondon]

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

Forgot to mention though, this just started happening.  I had the full node running for months (with p2pool on top) without a problem, and I haven't enabled home directory encryption. 

I've moved the directory and short exits from ssh seem to be working.  I'm wondering if a restart has re-enabled home drive encryption.

[Foxpup]

I am using -daemon to daemonize it.  It's the last flag on the command.

D'oh. I need more caffeine, apparently.

[cr1776]

I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this.

Forgot to mention though, this just started happening.  I had the full node running for months (with p2pool on top) without a problem, and I haven't enabled home directory encryption. 

I've moved the directory and short exits from ssh seem to be working.  I'm wondering if a restart has re-enabled home drive encryption.

I've been running a similar node too and just moved it to a new server. I didn't see that with or without screen (just as a datapoint -ubuntu 14.04). 

[bigbeninlondon]

Update:

Moved data directory, looking at 6 hours of uptime with no loss of connection.  It appears that the ~ directory is doing something fishy.  I had about 3 months of uptime, and it looks like a reboot may have sent a configuration off the deep end.  Off to hunt it down.  Will update again in 24 hours if its still up.

[dserrano5]

Moved data directory, looking at 6 hours of uptime with no loss of connection.

Now it's a good time to drop that dangerous 'sudo' you have there.

[bigbeninlondon]

Moved data directory, looking at 6 hours of uptime with no loss of connection.

Now it's a good time to drop that dangerous 'sudo' you have there.

Without it I get permission denied errors.  I have to do everything in Ubuntu with sudo...

[Foxpup]

Without it I get permission denied errors.  I have to do everything in Ubuntu with sudo...

Let me guess, you ran something as root for whatever reason, it created a file owned by root, which means other programs can't access that file, so you ran those programs as root, and you ended up with half your files being owned by root and nothing not running as root will work. Is that pretty much how it went? If so, this should fix it (at the very least you'll need to do it for your Bitcoin data directory):

Code:

sudo chown -R --from root:root ben: /home/ben

It's either that or your path's messed up...

[Justin00]

Been using chown for ~15 years, Didn't know you could do --from (Never bothered to look, but still.. )

thanks

[bigbeninlondon]

Without it I get permission denied errors.  I have to do everything in Ubuntu with sudo...

Let me guess, you ran something as root for whatever reason, it created a file owned by root, which means other programs can't access that file, so you ran those programs as root, and you ended up with half your files being owned by root and nothing not running as root will work. Is that pretty much how it went? If so, this should fix it (at the very least you'll need to do it for your Bitcoin data directory):

Code:

sudo chown -R --from root:root ben: /home/ben

It's either that or your path's messed up...

Yea that's probably how it went.  Most of this is really a foray into more advanced linux topics (trying to build from source, running a python p2pool node, setting up init.d scripts, that kind of thing).

All ubuntu literature prepends "sudo" to just about everything.  I assumed it was standard to invoke root access to do anything of importance in Ubuntu.

What's the importance (if any) of leaving off the "group" in your chown command above?

[Foxpup]

All ubuntu literature prepends "sudo" to just about everything.  I assumed it was standard to invoke root access to do anything of importance in Ubuntu.

That's a dangerous way of thinking, and one of the reasons I hate Ubuntu.

What's the importance (if any) of leaving off the "group" in your chown command above?

It's not left off, it's left blank (note the colon). Leaving it blank sets it to the login group of the specified user, which may just be the user itself, or it may be an actual group (the groups command will tell you which). (And leaving the group off entirely (by omitting the colon) will just leave the group set to "root", which isn't what you want at all.)

[bigbeninlondon]

All ubuntu literature prepends "sudo" to just about everything.  I assumed it was standard to invoke root access to do anything of importance in Ubuntu.

That's a dangerous way of thinking, and one of the reasons I hate Ubuntu.

I agree; I was also a bit put off by the process as well at first.  I guess I just saw it so many places I got used to it.