[solved] bitcoind upstart problem

[whywefight]

Not 100% if this is the right section but lets see.

I am running the latest BTC daemon on a Raspberry Pi 3 with Stretch Lite and i followed this guide: https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_30_bitcoin.md

Basically, everything works except for one strange thing:

If i reboot the system, bitcoind starts up on boot, runs and is "doing" something but if i want to use bitcoin-cli i get:

Code:

bitcoin@pinode:/home/admin $ bitcoin-cli getblockchaininfo
error: couldn't connect to server: unknown (code -1)
(make sure server is running and you are connecting to the correct RPC port)

If i kill the btc daemon and wait till it gets upstarted by systemd again, everything is fine and working. i am sure the service config is good. Otherwise it shouldnt work with my hard kill.

I changed my conf to:

After=network-online.target
Wants=network-online.target

Cause i assumed the network stack is not ready but still the same. I have no clue why its not working from boot, well lets say why its only working half.

Any suggestions?

[1715087881]

1715087881

[1715087881]

1715087881

[starmyc]

bitcoin-cli is not able to reach the bitcoind's rpc port, which should be located at port 8332. You should try the following to check the port is correctly opened:

Code:

# netstat -nap|grep bitcoin|grep LISTEN
tcp        0      0 127.0.0.1:8332          0.0.0.0:*               LISTEN      13199/bitcoind 
tcp        0      0 0.0.0.0:8333            0.0.0.0:*               LISTEN      13199/bitcoind 
tcp6       0      0 ::1:8332                :::*                    LISTEN      13199/bitcoind 
tcp6       0      0 :::8333                 :::*                    LISTEN      13199/bitcoind

Also, make sure that bitcoind is started with the "-server" flag:

Code:

# ps auxw|grep -i bitcoind|grep -v grep
coins    13199  8.2  3.7 1966860 619076 ?      Ssl  Mar10 1459:29 /home/coins/bin/bitcoin/bin/bitcoind -server -daemon

[whywefight]

bitcoin-cli is not able to reach the bitcoind's rpc port, which should be located at port 8332. You should try the following to check the port is correctly opened:

Code:

# netstat -nap|grep bitcoin|grep LISTEN
tcp        0      0 127.0.0.1:8332          0.0.0.0:*               LISTEN      13199/bitcoind 
tcp        0      0 0.0.0.0:8333            0.0.0.0:*               LISTEN      13199/bitcoind 
tcp6       0      0 ::1:8332                :::*                    LISTEN      13199/bitcoind 
tcp6       0      0 :::8333                 :::*                    LISTEN      13199/bitcoind

Also, make sure that bitcoind is started with the "-server" flag:

Code:

# ps auxw|grep -i bitcoind|grep -v grep
coins    13199  8.2  3.7 1966860 619076 ?      Ssl  Mar10 1459:29 /home/coins/bin/bitcoin/bin/bitcoind -server -daemon

Thats all correct. As i said, if i kill the daemon and let systemd start it up again it works as it should. Config is set accodring to https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_30_bitcoin.md

Everything is working except when its starting during system boot.

[starmyc]

Thats all correct. As i said, if i kill the daemon and let systemd start it up again it works as it should. Config is set accodring to https://github.com/Stadicus/guides/blob/master/raspibolt/raspibolt_30_bitcoin.md

Everything is working except when its starting during system boot.

Ok sorry, I read too much quickly.
What's the state of listening ports of bitcoind after the boot (before killing it) ?
Did you try to force to bind, say, 127.0.0.1 with -rpcbind ?
Did you check debug.log of bitcoind after the boot for any suspicious related messages ?
Did you try to replace bitcoind by a shellscript to dump the network state at the bitcoind startup ? I'm thinking the network started by the dhcp process could not be completed yet.

[whywefight]

After boot:

Code:

tcp        0      0 127.0.0.1:29000         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10009         0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 ::1:18332               :::*                    LISTEN      -
tcp6       0      0 :::18333                :::*                    LISTEN      -

After kill:

Code:

tcp        0      0 127.0.0.1:29000         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:10009         0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:18332         0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 ::1:18332               :::*                    LISTEN      -

Ahhhhh, what i now see is there is a binding on 18333 after boot (i am on testnet) and after my kill it switches to 18332.

EDIT: Okay, i dont see why but somehow btcd is binding it self to 18333 for some reason. ill check my configs again

[starmyc]

After boot:

Code:

tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      -
tcp6       0      0 ::1:18332               :::*                    LISTEN      -
tcp6       0      0 :::18333                :::*                    LISTEN      -

After kill:

Code:

tcp        0      0 127.0.0.1:18332         0.0.0.0:*               LISTEN      -
tcp6       0      0 ::1:18332               :::*                    LISTEN      -

Ahhhhh, what i now see is there is a binding on 18333 after boot (i am on testnet) and after my kill it switches to 18332.

I'm reproducing the exact same problem on a raspberry pi. Interesting. I'll take a deeper look at it.

Code:

root@r01:/home/mycroft# netstat -nap|grep bitcoin|grep LISTEN
tcp        0      0 127.0.0.1:29000         0.0.0.0:*               LISTEN      573/bitcoind        
tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      573/bitcoind        
tcp6       0      0 ::1:18332               :::*                    LISTEN      573/bitcoind        
tcp6       0      0 :::18333                :::*                    LISTEN      573/bitcoind        
root@r01:/home/mycroft# 

Relevant debug lines:

Code:

2018-03-22 18:02:25 libevent: getaddrinfo: address family for nodename not supported
2018-03-22 18:02:25 Binding RPC on address 127.0.0.1 port 18332 failed.

EDIT: Okay, i dont see why but somehow btcd is binding it self to 18333 for some reason. ill check my configs again

It should listen both 18332 & 18333 ports on both ipv4 & ipv6. I think this is not a configuration problem, but a bug somewhere.

[whywefight]

After boot:

Code:

tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      -
tcp6       0      0 ::1:18332               :::*                    LISTEN      -
tcp6       0      0 :::18333                :::*                    LISTEN      -

After kill:

Code:

tcp        0      0 127.0.0.1:18332         0.0.0.0:*               LISTEN      -
tcp6       0      0 ::1:18332               :::*                    LISTEN      -

Ahhhhh, what i now see is there is a binding on 18333 after boot (i am on testnet) and after my kill it switches to 18332.

I'm reproducing the exact same problem on a raspberry pi. Interesting. I'll take a deeper look at it.

Code:

root@r01:/home/mycroft# netstat -nap|grep bitcoin|grep LISTEN
tcp        0      0 127.0.0.1:29000         0.0.0.0:*               LISTEN      573/bitcoind        
tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      573/bitcoind        
tcp6       0      0 ::1:18332               :::*                    LISTEN      573/bitcoind        
tcp6       0      0 :::18333                :::*                    LISTEN      573/bitcoind        
root@r01:/home/mycroft# 

Relevant debug lines:

Code:

2018-03-22 18:02:25 libevent: getaddrinfo: address family for nodename not supported
2018-03-22 18:02:25 Binding RPC on address 127.0.0.1 port 18332 failed.

EDIT: Okay, i dont see why but somehow btcd is binding it self to 18333 for some reason. ill check my configs again

It should listen both 18332 & 18333 ports on both ipv4 & ipv6. I think this is not a configuration problem, but a bug somewhere.

Yeah you are correct, it should ofc bind to both. I just took a break and checked again and then i noticed rpc isnt binding it self. At least someone confirms the issue thanks this issue drives me crazy

[starmyc]

Yeah you are correct, it should ofc bind to both. I just took a break and checked again and then i noticed rpc isnt binding it self. At least someone confirms the issue thanks this issue drives me crazy

You are not alone. This is a bug in libevent, but it seems the debian package on raspberry pi doesn't have the fix.
Other people already reported it:

  * https://github.com/litecoin-project/litecoin/issues/351
  * https://bitcointalk.org/index.php?topic=2345585.0 (that crontab fix is ... well ... kinda shitty).
  * .. and a lot of other links I already closed.

Try this, in /etc/systemd/system/bitcoind.service:

Code:

[Unit]
Description=Bitcoin daemon
After=multi-user.target

Instead of "After=network.target", use "After=multi-user.target". It will start the service after the boot sequence. Not sure it will work on your raspberry pi, but it worked on mine:

Code:

root@r01:/home/mycroft# netstat -nap|grep bitcoind|grep LISTEN
tcp        0      0 127.0.0.1:29000         0.0.0.0:*               LISTEN      706/bitcoind        
tcp        0      0 127.0.0.1:18332         0.0.0.0:*               LISTEN      706/bitcoind        
tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      706/bitcoind        
tcp6       0      0 ::1:18332               :::*                    LISTEN      706/bitcoind        
tcp6       0      0 :::18333                :::*                    LISTEN      706/bitcoind     

[whywefight]

Yeah you are correct, it should ofc bind to both. I just took a break and checked again and then i noticed rpc isnt binding it self. At least someone confirms the issue thanks this issue drives me crazy

You are not alone. This is a bug in libevent, but it seems the debian package on raspberry pi doesn't have the fix.
Other people already reported it:

  * https://github.com/litecoin-project/litecoin/issues/351
  * https://bitcointalk.org/index.php?topic=2345585.0 (that crontab fix is ... well ... kinda shitty).
  * .. and a lot of other links I already closed.

Try this, in /etc/systemd/system/bitcoind.service:

Code:

[Unit]
Description=Bitcoin daemon
After=multi-user.target

Instead of "After=network.target", use "After=multi-user.target". It will start the service after the boot sequence. Not sure it will work on your raspberry pi, but it worked on mine:

Code:

root@r01:/home/mycroft# netstat -nap|grep bitcoind|grep LISTEN
tcp        0      0 127.0.0.1:29000         0.0.0.0:*               LISTEN      706/bitcoind        
tcp        0      0 127.0.0.1:18332         0.0.0.0:*               LISTEN      706/bitcoind        
tcp        0      0 0.0.0.0:18333           0.0.0.0:*               LISTEN      706/bitcoind        
tcp6       0      0 ::1:18332               :::*                    LISTEN      706/bitcoind        
tcp6       0      0 :::18333                :::*                    LISTEN      706/bitcoind     

AWESOME! That did it! Thanks for helping me out