|
Title: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 23, 2014, 06:13:52 AM Ok,
I've been around and around and I have no earthly idea what's going on here. I built bitcoind ("version" : 99900) from git on a headless ubuntu 13 machine. Running the following command: Code: sudo bitcoind -datadir=/home/ben/.bitcoin/ -conf=/home/ben/.bitcoin/bitcoin.conf -daemon Everything works great, as long as I remain connected to my ssh session. Logging off for a few seconds doesn't interrupt, but if I leave it disconnected for more than a few seconds, I get the following error: Code: 2014-08-23 06:02:13 LevelDB read failure: IO error: /home/ben/.bitcoin/chainstate/000174.sst: No such file or directory Different "sst" file every time I restart. Doesn't matter how far along I am in the chain. For some reason, it's as if the chainstate sst files just "disappear" after I log off my ssh session. Restarting gives me: Code: Error opening block database. Running again with "-reindex" re-indexes just fine and starts it back up; that is until I exit my ssh session again. I've rebuilt bitcoin from source, downloaded the blockchain about 40 times, moved the data directory, etc. Any suggestions? Title: Re: Bitcoind crashes after logging off of SSH session Post by: Justin00 on August 23, 2014, 09:45:34 AM Not sure of the actual fix, but a short term solution to get around the issue could be to use "screen"
Title: Re: Bitcoind crashes after logging off of SSH session Post by: cr1776 on August 23, 2014, 10:39:07 AM Not sure of the actual fix, but a short term solution to get around the issue could be to use "screen" Yes, running with screen should help. Title: Re: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 23, 2014, 12:41:22 PM I haven't really pursued that option just because what is a "daemon" actually for if it doesn't daemonize?
I plugged in a monitor and keyboard and logged in to the machine directly, ran the daemon and left it logged in. That's kept it from crashing so far this morning... Title: Re: Bitcoind crashes after logging off of SSH session Post by: Foxpup on August 23, 2014, 12:49:04 PM You need to use the -daemon argument to daemonise it. Though, it should exit cleanly on hangup otherwise.
Title: Re: Bitcoind crashes after logging off of SSH session Post by: btchris on August 23, 2014, 12:54:01 PM As others have said, screen is your friend.
Ubuntu Server 13 also has byobu (installed by default I believe), just run byobu-enable, and then choose your activation key (press Ctrl-a, and choose an option, I prefer the first). Then instead of logging out, use Ctrl-a d (on either an SSH session or the console). I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this. Title: Re: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 23, 2014, 01:25:28 PM You need to use the -daemon argument to daemonise it. Though, it should exit cleanly on hangup otherwise. I am using -daemon to daemonize it. It's the last flag on the command. I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this. I'll give that a try and see what happens. Title: Re: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 23, 2014, 01:34:05 PM I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this. Forgot to mention though, this just started happening. I had the full node running for months (with p2pool on top) without a problem, and I haven't enabled home directory encryption. I've moved the directory and short exits from ssh seem to be working. I'm wondering if a restart has re-enabled home drive encryption. Title: Re: Bitcoind crashes after logging off of SSH session Post by: Foxpup on August 23, 2014, 02:05:05 PM I am using -daemon to daemonize it. It's the last flag on the command. D'oh. I need more caffeine, apparently.Title: Re: Bitcoind crashes after logging off of SSH session Post by: cr1776 on August 23, 2014, 02:12:32 PM I'll bet that the underlying cause is that you enabled home directory encryption -- your home directory will only remain decrypted while you are logged in. You can relocate the datadir and start bitcoind with -datadir=/new/dir as another way to get around this. Forgot to mention though, this just started happening. I had the full node running for months (with p2pool on top) without a problem, and I haven't enabled home directory encryption. I've moved the directory and short exits from ssh seem to be working. I'm wondering if a restart has re-enabled home drive encryption. I've been running a similar node too and just moved it to a new server. I didn't see that with or without screen (just as a datapoint -ubuntu 14.04). Title: Re: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 23, 2014, 09:38:04 PM Update:
Moved data directory, looking at 6 hours of uptime with no loss of connection. It appears that the ~ directory is doing something fishy. I had about 3 months of uptime, and it looks like a reboot may have sent a configuration off the deep end. Off to hunt it down. Will update again in 24 hours if its still up. Title: Re: Bitcoind crashes after logging off of SSH session Post by: dserrano5 on August 23, 2014, 09:51:05 PM Moved data directory, looking at 6 hours of uptime with no loss of connection. Now it's a good time to drop that dangerous 'sudo' you have there. Title: Re: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 24, 2014, 12:26:08 AM Moved data directory, looking at 6 hours of uptime with no loss of connection. Now it's a good time to drop that dangerous 'sudo' you have there. Without it I get permission denied errors. I have to do everything in Ubuntu with sudo... Title: Re: Bitcoind crashes after logging off of SSH session Post by: Foxpup on August 24, 2014, 01:52:56 AM Without it I get permission denied errors. I have to do everything in Ubuntu with sudo... Let me guess, you ran something as root for whatever reason, it created a file owned by root, which means other programs can't access that file, so you ran those programs as root, and you ended up with half your files being owned by root and nothing not running as root will work. Is that pretty much how it went? If so, this should fix it (at the very least you'll need to do it for your Bitcoin data directory):Code: sudo chown -R --from root:root ben: /home/ben It's either that or your path's messed up... Title: Re: Bitcoind crashes after logging off of SSH session Post by: Justin00 on August 24, 2014, 04:45:40 AM Been using chown for ~15 years, Didn't know you could do --from (Never bothered to look, but still.. )
thanks :) Title: Re: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 24, 2014, 11:27:01 AM Without it I get permission denied errors. I have to do everything in Ubuntu with sudo... Let me guess, you ran something as root for whatever reason, it created a file owned by root, which means other programs can't access that file, so you ran those programs as root, and you ended up with half your files being owned by root and nothing not running as root will work. Is that pretty much how it went? If so, this should fix it (at the very least you'll need to do it for your Bitcoin data directory):Code: sudo chown -R --from root:root ben: /home/ben It's either that or your path's messed up... Yea that's probably how it went. Most of this is really a foray into more advanced linux topics (trying to build from source, running a python p2pool node, setting up init.d scripts, that kind of thing). All ubuntu literature prepends "sudo" to just about everything. I assumed it was standard to invoke root access to do anything of importance in Ubuntu. What's the importance (if any) of leaving off the "group" in your chown command above? Title: Re: Bitcoind crashes after logging off of SSH session Post by: Foxpup on August 24, 2014, 12:19:17 PM All ubuntu literature prepends "sudo" to just about everything. I assumed it was standard to invoke root access to do anything of importance in Ubuntu. That's a dangerous way of thinking, and one of the reasons I hate Ubuntu.What's the importance (if any) of leaving off the "group" in your chown command above? It's not left off, it's left blank (note the colon). Leaving it blank sets it to the login group of the specified user, which may just be the user itself, or it may be an actual group (the groups command will tell you which). (And leaving the group off entirely (by omitting the colon) will just leave the group set to "root", which isn't what you want at all.)Title: Re: Bitcoind crashes after logging off of SSH session Post by: bigbeninlondon on August 25, 2014, 11:56:14 AM All ubuntu literature prepends "sudo" to just about everything. I assumed it was standard to invoke root access to do anything of importance in Ubuntu. That's a dangerous way of thinking, and one of the reasons I hate Ubuntu.I agree; I was also a bit put off by the process as well at first. I guess I just saw it so many places I got used to it. |