Core 0.21.1 on AWS ubuntu just dies

[Ingo Konito]

Bitcoin Client Software and Version Number: 0.21.1 build from source
Operating System: ubuntu 20.04 LTS
System Hardware Specs:  AWS EC2 EC2 T3 Micro Instance
Description of Problem:
deamon keeps exiting.

Ok, trying to spin up a lightning node on AWS following the Run LND guide.
https://docs.lightning.engineering/lightning-network-tools/lnd/run-lnd#system-requirements

Compiled bitcoin from source, all fine. Runs. But after a while, it just terminates and command line reads
"killed" if daemon=0 That wasn't me tough. I did not interrupt it.

If deamon=1, then it also dies but without being verbose.

Here's my config:

$ cat bitcoin.conf
# Run as a daemon mode without an interactive shell
daemon=1

# Set the data directory to the storage directory
datadir=/data/.bitcoin/data

# Set the number of megabytes of RAM to use, set to like 50% of available memory
dbcache=3000

# Add visibility into mempool and RPC calls for potential LND debugging
debug=mempool
debug=rpc

# Turn off the wallet, it won't be used
disablewallet=1

# Don't bother listening for peers
listen=0

# Constrain the mempool to the number of megabytes needed:
maxmempool=100

# Limit uploading to peers
maxuploadtarget=1000

# Turn off serving SPV nodes
nopeerbloomfilters=1
peerbloomfilters=0

# Don't accept deprecated multi-sig style
permitbaremultisig=0

# Set the RPC auth to what was set above
rpcauth=

# Turn on the RPC server
server=1

# Reduce the log file size on restarts
shrinkdebuglog=1

# Set testnet if needed
testnet=0

# Turn on transaction lookup index
txindex=1

# Turn on ZMQ publishing
zmqpubrawblock=tcp://127.0.0.1:28332
zmqpubrawtx=tcp://127.0.0.1:28333

# put under [main] section
# Some mainnet peers
addnode=gyn2vguc35viks2b.onion
addnode=kvd44sw7skb5folw.onion
addnode=nkf5e6b7pl4jfd4a.onion
addnode=yu7sezmixhmyljn4.onion
addnode=3ffk7iumtx3cegbi.onion
addnode=3nmbbakinewlgdln.onion
addnode=4j77gihpokxu2kj4.onion
addnode=546esc6botbjfbxb.onion
addnode=5at7sq5nm76xijkd.onion
addnode=77mx2jsxaoyesz2p.onion
addnode=7g7j54btiaxhtsiy.onion
addnode=a6obdgzn67l7exu3.onion
addnode=ab64h7olpl7qpxci.onion
addnode=am2a4rahltfuxz6l.onion
addnode=azuxls4ihrr2mep7.onion
addnode=bitcoin7bi4op7wb.onion
addnode=bitcoinostk4e4re.onion
addnode=bk7yp6epnmcllq72.onion
addnode=bmutjfrj5btseddb.onion
addnode=ceeji4qpfs3ms3zc.onion
addnode=clexmzqio7yhdao4.onion
addnode=gb5ypqt63du3wfhn.onion
addnode=h2vlpudzphzqxutd.onion

# Only use Tor
onlynet=onion

# Connect to Tor proxy
proxy=127.0.0.1:9050

Any Related Addresses: None
Any Related Transaction IDs: None
Screenshot of the problem: https://imgur.com/a/zMnKk
Log Files from the Bitcoin Client:
tail end:
2021-10-06T11:07:52Z Pre-allocating up to position 0x700000 in rev00018.dat
2021-10-06T11:07:55Z Pre-allocating up to position 0x4000000 in blk00018.dat
2021-10-06T11:07:58Z Pre-allocating up to position 0x800000 in rev00018.dat
2021-10-06T11:08:05Z Pre-allocating up to position 0x900000 in rev00018.dat
2021-10-06T11:08:09Z Pre-allocating up to position 0x5000000 in blk00018.dat
2021-10-06T11:09:06Z Socks5() connect to am2a4rahltfuxz6l.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:10:22Z Socks5() connect to azuxls4ihrr2mep7.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:11:27Z Socks5() connect to bitcoin7bi4op7wb.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:14:06Z Socks5() connect to bitcoinostk4e4re.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:14:39Z Socks5() connect to bmutjfrj5btseddb.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:16:08Z Socks5() connect to ceeji4qpfs3ms3zc.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:17:01Z Socks5() connect to clexmzqio7yhdao4.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:17:19Z Socks5() connect to 6kte4lyslsmogrckujvljnzm52bivioo3ibdg7hwon6ms5k3lq2naead.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:18:16Z Socks5() connect to p2qp5vh5rjwk5cql.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:18:25Z Socks5() connect to h2vlpudzphzqxutd.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:21:28Z Socks5() connect to kvd44sw7skb5folw.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:21:43Z Socks5() connect to fwnbxr3l2dznzfvwzlm5llpqcu7zxypopnfwno4dyavxen75tqiryiyd.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:22:07Z Socks5() connect to nkf5e6b7pl4jfd4a.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:22:31Z Socks5() connect to g44i6jwsutkwmspz.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:22:43Z Socks5() connect to yu7sezmixhmyljn4.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:23:29Z Socks5() connect to 3ffk7iumtx3cegbi.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:23:49Z Socks5() connect to kq4e6p23sn4bvmf7yojrftdwkg6szypsnrzoy3knlsne6om3ighk2mid.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:24:03Z Socks5() connect to 3nmbbakinewlgdln.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:24:46Z Socks5() connect to 4j77gihpokxu2kj4.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:25:07Z Socks5() connect to n4ol4kh3qahmtbbsq5673ahd3ce2taiezui2vjhsf4d6myth3cllvyqd.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:25:20Z Socks5() connect to 546esc6botbjfbxb.onion:8333 failed: InterruptibleRecv() timeout or other failure
2021-10-06T11:26:00Z Socks5() connect to 5at7sq5nm76xijkd.onion:8333 failed: InterruptibleRecv() timeout or other failure

[1714626504]

1714626504

[1714626504]

1714626504

[1714626504]

1714626504

[ABCbits]

System Hardware Specs:  AWS EC2 EC2 T3 Micro Instance

Looking at https://aws.amazon.com/ec2/instance-types/t3/, is it correct that you use instance with 2 vCPU and 1GB RAM?

Compiled bitcoin from source, all fine. Runs. But after a while, it just terminates and command line reads
"killed" if daemon=0 That wasn't me tough. I did not interrupt it.

Most likely it's triggered by Linux Out of Memory Killer (OOM Killer), but you should check your system log to be sure. I would recommend you to configure LND and Bitcoin Core to use less memory. But i'm not sure if 1GB RAM is enough for both application.

# Some mainnet peers
addnode=gyn2vguc35viks2b.onion

Onion v2 address is dead, you should add onion v3 address instead.

[DaveF]

Most likely it's triggered by Linux Out of Memory Killer (OOM Killer), but you should check your system log to be sure. I would recommend you to configure LND and Bitcoin Core to use less memory. But i'm not sure if 1GB RAM is enough for both application.

No, 1GB is not even close to enough.
Yes, you can run the daemon in that much free RAM, but by the time you have the OS and LND you cannot fit them in there.
I tried a while ago to do it by using massive swap space but eventually just gave up.

I do not know why the guide says you can use 1GB, if it does it's really doing a disservice to people.
2GB at a bare minimum 4GB+ is best.

Edit to add: TOR also needs a bit of RAM to so that does not help the situation.

-Dave

[NotATether]

Also the CPU specs are not enough either, the default configuration of t3.micro only gives you 10% of the computing power of both cores which will severely slow down the execution of Bitcoin Core. You should either turn on T3 Unlimited (at an extra cost) or run the ARM64 Bitcoin Core binaries on an A1 instance which doesn't have this type of CPU limitation.

[ABCbits]

Most likely it's triggered by Linux Out of Memory Killer (OOM Killer), but you should check your system log to be sure. I would recommend you to configure LND and Bitcoin Core to use less memory. But i'm not sure if 1GB RAM is enough for both application.

No, 1GB is not even close to enough.
Yes, you can run the daemon in that much free RAM, but by the time you have the OS and LND you cannot fit them in there.
I tried a while ago to do it by using massive swap space but eventually just gave up.

That's what i expect, 1GB is barely enough for Bitcoin Core on older Raspberry Pi.

I do not know why the guide says you can use 1GB, if it does it's really doing a disservice to people.
2GB at a bare minimum 4GB+ is best.

I guess T3 Micro had bigger RAM in past or the author wrote wrong instance type.

[DaveF]

I guess T3 Micro had bigger RAM in past or the author wrote wrong instance type.

This brings up a bit of another rant that I have with a lot of guides.

1) Why the hell do people put in minimum requirements that are so minimum that stuff will barely run and if it does run the user experience is so miserable that it's a joke.

2) Put a dates in the guide. I don't know if this was written in 2017 or 2021 and as you make updates DATE THOSE TOO. This way we can tell how and when what was updated.

3) When recommending stuff tell us what the stuff has. Much like you said, we don't know if the author messed up and wrote the wrong instance or AWS changed it.

It's not just BTC stuff, I was working on a BMS whos controller PC died and I went for the 'recommended' PC and did not even see that on page 12 out of 40 in the guide that it was from 5 years before. And that was the ONLY date printed in the guide.  All it said was 4 cores 4GB RAM. So I figured the 4 core 8GB RAM would be fine. Nope, now that was now the base and 16 GB was recommended.

-Dave

[NotATether]

I guess T3 Micro had bigger RAM in past or the author wrote wrong instance type.

T3 micro had always had 1GB of RAM.

1) Why the hell do people put in minimum requirements that are so minimum that stuff will barely run and if it does run the user experience is so miserable that it's a joke.

I guess they made the benchmarks with *only* that application running without a GUI shell. It could be accounted for bough, since Bitcoin QT is a GUI application anyway and it's not just the daemon backend.

[DaveF]

1) Why the hell do people put in minimum requirements that are so minimum that stuff will barely run and if it does run the user experience is so miserable that it's a joke.

I guess they made the benchmarks with *only* that application running without a GUI shell. It could be accounted for bough, since Bitcoin QT is a GUI application anyway and it's not just the daemon backend.

GUI / No GUI does not matter with 1GB so many things would be swapping to disk that the performance in general is just going to be miserable.

If you want to haul lumber you buy a pickup truck, if you want good gas mileage you get a small hybrid. You can haul lumber with the hybrid and drive everywhere in the pickup at 25 mph to get better gas mileage but it's not going to be practical.

There are discussions that keep popping up here about how to run BTC nodes in the smallest cheapest way. And for people that cannot afford to do it a better way that is fine. If all you have is a small hybrid and you have to haul lumber you have to use it. But, we should never loose site of telling them that it's not optimal and there will be more issues doing it that way. And possibly it might be better to save up a little more and do it a better way.

-Dave

[HCP]

But, we should never loose site of telling them that it's not optimal and there will be more issues doing it that way. And possibly it might be better to save up a little more and do it a better way.

I concur... just because "you can", it doesn't mean that "you should".

I got as far as "AWS EC2 EC2 T3 Micro Instance" in the OP and already knew what the likely cause of this issue was. Running on anything less than a "T3.medium" would be asking for all sorts of trouble.

[DaveF]

I concur... just because "you can", it doesn't mean that "you should".

I got as far as "AWS EC2 EC2 T3 Micro Instance" in the OP and already knew what the likely cause of this issue was. Running on anything less than a "T3.medium" would be asking for all sorts of trouble.

https://docs.lightning.engineering/lightning-network-tools/lnd/get-started-with-lnd

LND generally has low system requirements and performs well on small single-board devices with 2 GiB and a CPU with over 1 GHz quad core, such as a Raspberry Pi. Generally the system requirements for Bitcoind are higher than those for LND

So even in other parts of their website they say go for 2GB. Which as we all have concluded is still too little for good performance.

Makes you wonder if anyone does any real technical proofreading anymore.
That or if it's people trying to prove anyone can run a BTC node and a lightning node for almost no money, so they are showing how you can shove it into limited hardware just to prove that point.

The OP has not come back since that 1 post, so we don't know if they have read this and gotten it working or figured something else out. But, I now have some text sitting in notepad++ waiting for a quick copy / paste about the real world requirements vs what is printed out there.

-Dave