Title: Exploring alternative full node implementations - Gocoin Post by: ABCbits on July 26, 2022, 01:13:55 PM Intro
Inspired by tiltedIceCream reply[1] on thread "Why aren't alternative implementations encouraged?"[2], i'm inspirited to try alternative Bitcoin full node implementation. Based on Jameson Lopp benchmark[3], Gocoin seems to have good performance so i decide to give a try. Basic info
My system
Compile and installation The documentation[6] only mention i need git and golang on my system. Since it doesn't mention required golang version, i just use golang provided my distro. I receive golang 1.18.1 which is only slightly older than newest version (1.18.4[7]). Code: sudo apt update && sudo apt install golang-go When i read tweak page[8], Gocoin also support using external secp256k1 library from Bitcoin Core. So i decide to install secp256k1 on my system and clone Gocoin 1.10.1. Code: sudo apt-get install gcc automake autoconf libtool make git libgmp-dev Using Gocoin built-in benchmark, external secp256k1 library have 40% faster performance on my system. So i decide to use external secp256k1 library by copying sipasec.go to parent directory. Code: git clone --depth 1 --branch 1.10.1 https://github.com/piotrnar/gocoin Finally it's time to compile Gocoin. You could just compile the client to run full node, but i try to compile everything. It's straightforward and only took few seconds. Both client and wallet compiled with no problem, although 5 external Gocoin failed to compile. I only managed to fix utxo_compress and utxo_decompress after using latest code from Gocoin latest commit. Code: cd /path/to/gocoin Code: # command-line-arguments Initial Block Download (IBD) Here's my gocoin.conf which mashed together from Jameson Lopp blog[3] and Gocoin documentation[9-10]. I also configure Gocoin only connect to my Bitcoin Core node. gocoin.conf should be located on directory where you run/launch client file. Code: { The sync process took about 6.8 hours. As expected, HDD is the only bottleneck on sync process where my Bitcoin Core report sent speed 11-18MB/s speed. Gocoin RAM usage after IBD is 23.1GB, although it's reduced to 19GB after i close and open Gocoin again. Directory size of Gocoin is about 354GB, while Bitcoin Core is about 493GB (txindex, blockfilterindex, coinstatsindex enabled). I have to say Gocoin web UI is rather nice. https://i.ibb.co/vw19Fbt/g.png [1] https://bitcointalk.org/index.php?topic=5406961.msg60608073#msg60608073 (https://bitcointalk.org/index.php?topic=5406961.msg60608073#msg60608073) [2] https://bitcointalk.org/index.php?topic=5406961.20 (https://bitcointalk.org/index.php?topic=5406961.20) [3] https://blog.lopp.net/2021-bitcoin-node-performance-tests-2/ (https://blog.lopp.net/2021-bitcoin-node-performance-tests-2/) [4] https://bitcointalk.org/index.php?topic=199306.0 (https://bitcointalk.org/index.php?topic=199306.0) [5] https://github.com/piotrnar/gocoin/tree/aae8e1774fdfe8bae99bccb5c3fd6113ad7fbc95 (https://github.com/piotrnar/gocoin/tree/aae8e1774fdfe8bae99bccb5c3fd6113ad7fbc95) [6] https://gocoin.pl/gocoin_installation.html (https://gocoin.pl/gocoin_installation.html) [7] https://go.dev/dl/ (https://go.dev/dl/) [8] https://gocoin.pl/gocoin_tweaks.html (https://gocoin.pl/gocoin_tweaks.html) [9] https://gocoin.pl/gocoin_performance.html (https://gocoin.pl/gocoin_performance.html) [10] https://gocoin.pl/gocoin_manual_config.html (https://gocoin.pl/gocoin_manual_config.html) Title: Exploring alternative full node implementations - Gocoin Post by: ABCbits on July 26, 2022, 01:14:28 PM WebUI
To access WebUI, just type 127.0.0.1:8833 or localhost:8833 on your Browser. You'll see homepage of Gocoin which show some statistic about your node. Most parts is easy to understand if you already use Bitcoin Core with primary intention to run full node. I don't what SPB refers though. https://i.ibb.co/vw19Fbt/g.png "Transactions" tab shows your node mempool. Compared with Bitcoin Core, Gocoin show more detail information about your node mempool statistic. You can enable/disable memory pool and transaction relay here. You also can upload and broadcast transaction as well. I haven't test this feature yet, so i don't know accepted transaction file format. https://i.ibb.co/zGmQWrL/j.png If you scroll down, you also can see list of transaction on your node mempool. It's quite detailed where it shows total signature operation and time to verify the transaction in milliseconds (ms). You also can choose how much transaction to be shown and how it's sorted. https://i.ibb.co/tccRMdr/b.png "Blocks" tab show some newest blocks. It's quite interesting to see total signature operation of each block, which rarely shown on Bitcoin block explorer. The rest is something you can see on most block explorer. https://i.ibb.co/Wt8sJ97/p.png "Miners" tab show statistic about miner during last 24 hours. I like how Gocoin show how much BTC earned by miner/pool from transaction fees. https://i.ibb.co/vcLv0nh/q.png "Counters" tab show lots of statistic about your node. Most of you won't find it's interesting, unless you're trying to analyze something. https://i.ibb.co/6NtdkT3/e.png TextUI TextUI is available after you start Gocoin from terminal, load all UTXO/mempool to RAM and optionally start WebUI server. When you enter help, there are 56 available commands. But i'll just show few command here. By the way, the TextUI doesn't support key up to show previous typed command. Code: Gocoin client version 1.10.1 allbal or ab shows Bitcoin distribution on each address. I just found out there are at least 11500 Bitcoin stored on Taproot address. Code: > ab balance or a show balance of an address or raw public key. The result is quite detailed which show when and how much an address receive Bitcoin. I really like this feature since it's fast and not exist on Bitcoin. But i wonder why the developer chose a as short hand option rather than b, bl or bal. Code: > a bc1qu8ujv03v0g4h4dpvempm9snkrrmrue4m5dsmtf richest or r show Bitcoin address with most Bitcoin. It also can show richest Bitcoin addressed based on address type. I wouldn't use it much, but it's good this feature exist. Code: > richest help Gocoin configuration As i said earlier, Gocoin can be configured through gocoin.conf file. But i just realized you can edit it from WebUI homepage. But what surprise me is there are few field which isn't mention on Gocoin config documentation[1] which are RPC., TXPool.Debug and Memory.FreeAtStart. The RPC is limited to experimentation mining API though[2], so it's not interesting feature for me. https://i.ibb.co/g4B3KjV/x.png Reducing RAM Since Gocoin RAM usage is quite high, i tried to reduce it's RAM consumption. Based on Gocoin documentation page[1,3]. there are few action i could do. First action is compressing UTXO records. With tools i compiled earlier, it only reduce UTXO.db file from 8.5GB to 8.2GB on the HDD. Second action is modifying gocoin.conf which involve memory, you can see my modification below. Code: { I managed to reduce RAM consumption to ~14.9GB. IMO it's rather small reduction since Gocoin usually use 17-19GB RAM. But there's nothing much i could do since Gocoin still store all UTXO to RAM. https://i.ibb.co/swvbn6S/f.png Conclusion Aside from RAM usage, Gocoin appear to be decent Bitcoin full node implementation. It's very impressive single developer write Gocoin alone. I don't need instant access to UXO though, so Gocoin isn't suitable for me. But even if i do, the RPC currently limited to mining-related feature. Some things i like, * Minimum dependency with easy compilation process * Lots of features on TextUI * Neat WebUI, although some feature only available on TextUI * Lots of external tools * Fast sync time, which only hold back by my HDD Some things i don't like, * Load all UTXO to RAM * Incomplete documentation * RPC only support mining-related feature, even though many feature exist on TextUI [1] https://gocoin.pl/gocoin_manual_config.html (https://gocoin.pl/gocoin_manual_config.html) [2] https://gocoin.pl/gocoin_issues.html (https://gocoin.pl/gocoin_issues.html) [3] https://gocoin.pl/gocoin_tweaks.html (https://gocoin.pl/gocoin_tweaks.html) Title: Re: Exploring alternative implementations - Gocoin Post by: n0nce on July 26, 2022, 01:47:39 PM Thanks for the experiment and the great writeup. I've been toying with the idea of spinning up another node with an alternative implementation a while ago.
The sync process took about 6.8 hours. As expected, HDD is the only bottleneck on sync process where my Bitcoin Core report sent speed 11-18MB/s speed. Gocoin RAM usage after IBD is 23.1GB, although it's reduced to 19GB after i close and open Gocoin again. Directory size of Gocoin is about 354GB, while Bitcoin Core is about 493GB (txindex, blockfilterindex, coinstatsindex enabled). I have to say Gocoin web UI is rather nice. This is the only thing worrying me a bit: 19GB of memory usage is a lot. Not that RAM is expensive, but some nodes simply don't have that much built-in. Do you know whether Gocoin bases its RAM usage on how much total RAM is available or would it be creating a 3+GB pagefile on a 16GB machine? And can you compare with Bitcoin Core's memory usage on the same system? Title: Re: Exploring alternative implementations - Gocoin Post by: PawGo on July 26, 2022, 02:50:36 PM Thanks for the experiment and the great writeup. I've been toying with the idea of spinning up another node with an alternative implementation a while ago. The sync process took about 6.8 hours. As expected, HDD is the only bottleneck on sync process where my Bitcoin Core report sent speed 11-18MB/s speed. Gocoin RAM usage after IBD is 23.1GB, although it's reduced to 19GB after i close and open Gocoin again. Directory size of Gocoin is about 354GB, while Bitcoin Core is about 493GB (txindex, blockfilterindex, coinstatsindex enabled). I have to say Gocoin web UI is rather nice. This is the only thing worrying me a bit: 19GB of memory usage is a lot. Not that RAM is expensive, but some nodes simply don't have that much built-in. Do you know whether Gocoin bases its RAM usage on how much total RAM is available or would it be creating a 3+GB pagefile on a 16GB machine? Exactly my question. I assume it uses what is available, as project has already several years I am not sure if having more than 16GB was so obvious years ago. I really wonder how it would behave on machine with 8GB total (which means it would have around 6GB free, to use). Would it be possible for you to launch it with limited memory settings? (maybe using cgroups?) Title: Re: Exploring alternative implementations - Gocoin Post by: DaveF on July 26, 2022, 03:06:24 PM According to their github page yes, you need AT LEAST 16GB of RAM
Quote Hardware client: 64-bit architecture OS and Go compiler. File system supporting files larger than 4GB. At least 16GB of system RAM. Seems very excessive. Yes it does a lot but still 16GB for most people that makes it a non starter. If I'm running it on my 'main PC' that would mean I need more then 16GB so I could do other stuff. If I am running it on an older spare PC, that probably means I need to buy more RAM or just run core. Which can run fine on a much lower end system. Still nice to have more options. -Dave Title: Re: Exploring alternative implementations - Gocoin Post by: n0nce on July 26, 2022, 08:54:48 PM According to their github page yes, you need AT LEAST 16GB of RAM Interesting, thanks for looking it up. A bit rushed from my side not to check myself.Quote Hardware client: 64-bit architecture OS and Go compiler. File system supporting files larger than 4GB. At least 16GB of system RAM. Seems very excessive. Yes it does a lot but still 16GB for most people that makes it a non starter. If I'm running it on my 'main PC' that would mean I need more then 16GB so I could do other stuff. One of my nodes was a laptop with 4GB of RAM and upgrading it to 8GB gave the initial block download an amazing speed boost. But for normal operation it would get along just fine with 4. I know people for whom 8GB is totally fine on their daily driver machine, so yeah, for maximum accessibility (think: lower income countries, etc.) requiring 16GB is definitely too high. My belief is that low system requirements are needed for the best shot at true decentralization.If I am running it on an older spare PC, that probably means I need to buy more RAM or just run core. Which can run fine on a much lower end system. But I guess it comes with the choice of programming language (Go). It would be great seeing a Rust implementation, since that should in theory give Title: Re: Exploring alternative implementations - Gocoin Post by: pooya87 on July 27, 2022, 03:34:06 AM But I guess it comes with the choice of programming language (Go). It would be great seeing a Rust implementation, since that should in theory give Technically speaking speed and efficiency is not decided based on what language was used (at least for most modern languages this is true). The implementation and the developer's expertise decides whether the code is fast or not.Title: Re: Exploring alternative implementations - Gocoin Post by: NotATether on July 27, 2022, 08:29:34 AM Why is all that RAM needed, to make one giant dbcache? The majority of servers have only 16GB of RAM, so I don't see this client being used by a lot of nodes.
Title: Re: Exploring alternative implementations - Gocoin Post by: ABCbits on July 27, 2022, 09:15:58 AM Do you know whether Gocoin bases its RAM usage on how much total RAM is available or would it be creating a 3+GB pagefile on a 16GB machine? But I guess it comes with the choice of programming language (Go). It would be great seeing a Rust implementation, since that should in theory give Why is all that RAM needed, to make one giant dbcache? The majority of servers have only 16GB of RAM, so I don't see this client being used by a lot of nodes. Gocoin simply load all UTXO to RAM. Quote from: https://github.com/piotrnar/gocoin The client (p2p node) is an application independent from the wallet. It keeps the entire UTXO set in RAM, providing an instant access to all its records and - in consequece - an extraordinary blochchain processing performance. And can you compare with Bitcoin Core's memory usage on the same system? I don't see the point, since it's heavily based on your dbcache value. IIRC it only use 1-2GB RAM although dbcache value is far higher. Would it be possible for you to launch it with limited memory settings? (maybe using cgroups?) If you could show tutorial without installing lots of application or recompiling kernel, i could do a quick test. Title: Re: Exploring alternative implementations - Gocoin Post by: n0nce on July 27, 2022, 04:09:18 PM But I guess it comes with the choice of programming language (Go). It would be great seeing a Rust implementation, since that should in theory give Technically speaking speed and efficiency is not decided based on what language was used (at least for most modern languages this is true). The implementation and the developer's expertise decides whether the code is fast or not.So in general, I wouldn't say that speed and efficiency are completely unrelated to chosen language. I am not knowledgeable enough about Go to explain well why it performs worse than C, but in a similar task (running a Lightning node), the Go implementation has been causing lots of issues for people and I was under the impression that Go is harder to optimize than C or Rust since it's more higher level / more abstract. But I may be wrong, that's just what I read online. Gocoin simply load all UTXO to RAM. Okay, I mean it does have a benefit then. I guess it will depend on your application. It's going to be rare, but whoever runs services that need 'instant access to all UTXOs' may want to actually run Gocoin. Especially since if they are optimizing Bitcoin Core for similar UTXO access performance, they may increase dbcache to a similar value anyway.Quote from: https://github.com/piotrnar/gocoin The client (p2p node) is an application independent from the wallet. It keeps the entire UTXO set in RAM, providing an instant access to all its records and - in consequece - an extraordinary blochchain processing performance. Title: Re: Exploring alternative implementations - Gocoin Post by: ABCbits on July 28, 2022, 01:04:08 PM I finally finished writing my experiment with Gocoin, it took longer than expected. Does anyone have suggestion what alternative implementations i should check next? But please don't suggest one of these,
1. Mako. It's still on heavy development, https://github.com/chjj/mako#development-status (https://github.com/chjj/mako#development-status). 2. Bitcoin Knots. It's fork of Bitcoin Core, so i don't expect major difference with Bitcoin Core. 3. Libbitcoin version 4. I have negative experience with it, https://bitcointalk.org/index.php?topic=5329445.msg56811006#msg56811006 (https://bitcointalk.org/index.php?topic=5329445.msg56811006#msg56811006). Gocoin simply load all UTXO to RAM. Okay, I mean it does have a benefit then. I guess it will depend on your application. It's going to be rare, but whoever runs services that need 'instant access to all UTXOs' may want to actually run Gocoin. Especially since if they are optimizing Bitcoin Core for similar UTXO access performance, they may increase dbcache to a similar value anyway.Quote from: https://github.com/piotrnar/gocoin The client (p2p node) is an application independent from the wallet. It keeps the entire UTXO set in RAM, providing an instant access to all its records and - in consequece - an extraordinary blochchain processing performance. That makes sense, but currently Gocoin RPC only limited to mining-related feature. People who need instant access to all UTXO need to find workaround with Gocoin's Text UI. Title: Re: Exploring alternative implementations - Gocoin Post by: NotATether on July 28, 2022, 04:20:46 PM Why is all that RAM needed, to make one giant dbcache? The majority of servers have only 16GB of RAM, so I don't see this client being used by a lot of nodes. Gocoin simply load all UTXO to RAM. Quote from: https://github.com/piotrnar/gocoin The client (p2p node) is an application independent from the wallet. It keeps the entire UTXO set in RAM, providing an instant access to all its records and - in consequece - an extraordinary blochchain processing performance. I can see this becoming an operational problem if the UTXO set doubles or triples at any given time, then GoCoin will simply run out of memory before it can load all the UTXO sets. Why not just load the most recent UTXOs, if short of RAM? It's more likely that recent coins will be spent before dormant coins anyway. Title: Re: Exploring alternative implementations - Gocoin Post by: n0nce on July 28, 2022, 11:53:29 PM Why is all that RAM needed, to make one giant dbcache? The majority of servers have only 16GB of RAM, so I don't see this client being used by a lot of nodes. Gocoin simply load all UTXO to RAM.Code: bitcoin@localhost:~> du -sh chainstate/ This would fit on my system in RAM by just setting -dbcache=5000 or something like that, even though I only have 8GB of system memory. Keep in mind I also have electrs and lightningd running. Code: bitcoin@localhost:~> free -h It could actually have something to do with the choice of programming language after all, as I mentioned before. Similar issues (high system resource utilization, hard to optimize / reduce) were reported by various people when using lnd (Go) instead of c-lightning (C). [...] But I guess it comes with the choice of programming language (Go). It would be great seeing a Rust implementation, since that should in theory give Title: Re: Exploring alternative implementations - Gocoin Post by: pooya87 on July 29, 2022, 03:41:23 AM It could actually have something to do with the choice of programming language after all, as I mentioned before. Similar issues (high system resource utilization, hard to optimize / reduce) were reported by various people when using lnd (Go) instead of c-lightning (C). I wish I knew more about GO to be able to give a technical answer but I still believe that optimizing code depends mostly on the developer not the language.For example if you implement UTXO like this (https://github.com/piotrnar/gocoin/blob/master/lib/utxo/unspent_rec.go#L21) in C# you'll get a lot of overhead because of the way the byte array is stored elsewhere in the memory then a reference to its position is stored inside the struct with a bunch of other overhead that increases the total memory consumption. To give you an idea about severity of this in C#, I was running an analysis on block headers and the program was consuming more than 200 MB. With some tweaks I brought it back down below 100 MB and closer to the file size itself which is about 60 MB. I use this example because people generally think C# is slow too which is not correct. P.S. FWIW the live UTXO size could be fetched from here: https://statoshi.info/d/000000009/unspent-transaction-output-set Title: Re: Exploring alternative implementations - Gocoin Post by: n0nce on July 29, 2022, 01:24:50 PM It could actually have something to do with the choice of programming language after all, as I mentioned before. Similar issues (high system resource utilization, hard to optimize / reduce) were reported by various people when using lnd (Go) instead of c-lightning (C). I wish I knew more about GO to be able to give a technical answer but I still believe that optimizing code depends mostly on the developer not the language.For example if you implement UTXO like this (https://github.com/piotrnar/gocoin/blob/master/lib/utxo/unspent_rec.go#L21) in C# you'll get a lot of overhead because of the way the byte array is stored elsewhere in the memory then a reference to its position is stored inside the struct with a bunch of other overhead that increases the total memory consumption. To give you an idea about severity of this in C#, I was running an analysis on block headers and the program was consuming more than 200 MB. With some tweaks I brought it back down below 100 MB and closer to the file size itself which is about 60 MB. I use this example because people generally think C# is slow too which is not correct. https://medium.com/safetycultureengineering/analyzing-and-improving-memory-usage-in-go-46be8c3be0a8 I can't say for sure, but it's possible that these Bitcoin Go projects that have higher memory usage than their C and Rust counterparts are simply not optimized like that. So in that case you would be right that it depends on the developers. Would love to see them get more optimized so they run on cheaper hardware, which is where people generally like to run their nodes (be it Raspberry Pis at MSRP or used x86 hardware). P.S. FWIW the live UTXO size could be fetched from here: https://statoshi.info/d/000000009/unspent-transaction-output-set That's handy! ;) Gocoin simply load all UTXO to RAM. What I don't understand is that the UTXO set is not this large. I am checking again, but it should be in the 4-5GB range; at least it was last I checked.Code: bitcoin@localhost:~> du -sh chainstate/ --snip-- FYI, UTXO file on Gocoin is 8.5GB (or 8.2GB uncompressed). I also expect Gocoin UTXO size on RAM is higher since it need to be deserialized. And why is it 8.5GB instead of <5.. Another multiple GB wasted - I think there's a lot of optimizing potential for this software before being widely used. Title: Re: Exploring alternative implementations - Gocoin Post by: ABCbits on July 30, 2022, 09:16:31 AM FYI, UTXO file on Gocoin is 8.5GB (or 8.2GB uncompressed). I also expect Gocoin UTXO size on RAM is higher since it need to be deserialized. You think it takes more than double the amount of memory to store it deserialized? So if I were to set my node's dbcache to something like 5GB in Bitcoin Core, it wouldn't fit the whole UTXO set?And why is it 8.5GB instead of <5.. Another multiple GB wasted - I think there's a lot of optimizing potential for this software before being widely used. It's not impossible if you ask me. Deserialized transaction on Bitcoin Core mempool is far higher than 2x. Looking at https://statoshi.info/d/000000020/memory-pool (https://statoshi.info/d/000000020/memory-pool) at 2022-07-30 09:10:00, totalTxBytes is 993KB but it's mempool memory usage is 4.94MB. Title: Re: Exploring alternative implementations - Gocoin Post by: NotATether on July 30, 2022, 02:50:39 PM FYI, UTXO file on Gocoin is 8.5GB (or 8.2GB uncompressed). I also expect Gocoin UTXO size on RAM is higher since it need to be deserialized. You think it takes more than double the amount of memory to store it deserialized? So if I were to set my node's dbcache to something like 5GB in Bitcoin Core, it wouldn't fit the whole UTXO set?And why is it 8.5GB instead of <5.. Another multiple GB wasted - I think there's a lot of optimizing potential for this software before being widely used. There is little that can be optimized in the UTXO data format, for example it certainly cannot be compressed using zlib or zstd because these only give paltry returns in space, but require at least linear time to decompress every time you need to check a UTXO set. At the end of the day, it really is down to the language you use. Even interpreters that are built on top of C++ tend to use much more resources for small programs than the equivalent codebase written directly in C++ (and even using a modest number of class objects). Title: Re: Exploring alternative implementations - Gocoin Post by: PowerGlove on August 08, 2022, 08:57:08 AM {...} Since Gocoin RAM usage is quite high, i tried to reduce it's RAM consumption {...} {...} Would it be possible for you to launch it with limited memory settings? {...} {...} What I don't understand is that the UTXO set is not this large. I am checking again, but it should be in the 4-5GB range; at least it was last I checked {...} Since a good amount of the discussion has been about excessive memory usage, I just wanted to leave this excerpt from the release notes [1] of Go 1.19 (released ~6 days ago): Quote from: Go 1.19 Release Notes {...} The runtime now includes support for a soft memory limit. This memory limit includes the Go heap and all other memory managed by the runtime, and excludes external memory sources such as mappings of the binary itself, memory managed in other languages, and memory held by the operating system on behalf of the Go program. This limit may be managed via runtime/debug.SetMemoryLimit or the equivalent GOMEMLIMIT environment variable {...} I haven't tried it, but maybe this will be of help to someone looking to get Gocoin's memory usage under control. [1] https://go.dev/doc/go1.19#runtime Title: Re: Exploring alternative implementations - Gocoin Post by: PowerGlove on August 08, 2022, 04:38:01 PM If i understood the release note correctly, it require modification Gocoin source code. You'll have to recompile with 1.19 to be on the new runtime, but I don't think changing Gocoin's source code is necessary. The soft memory limit can be configured via the new GOMEMLIMIT environment variable [1]. Like I said, I haven't experimented with this, it just caught my eye and I remembered this thread. [1] https://go.dev/doc/gc-guide#Memory_limit |