Mandacaru: A new Electrum server?

[TheButterZone]

https://github.com/jvsena42/mandacaru/

A lightweight Bitcoin validator node for Android, powered by Utreexo and Floresta.

Run a full Bitcoin node directly on your phone with minimal storage requirements thanks to Utreexo's compact accumulator design.

Saw it posted at https://x.com/bhbitdevs/status/2040815816950579233

Just submitted to https://walletscrutiny.com/asset/?sha256=4dd91c068c351e0de70a1cd0f414568e076db22ad8f2f0ab7afd28a3417b31b1

[nc50lc]

Do they have an official documentation other than links to "Utreexo" and "Floresta"?

Based from my initial review; it's more of a full node without a wallet (but can be loaded with desc)
with an integrated Electrum server to connect to the user's Electrum client that'll serve as the wallet.

And based from the "Powered by" links, it seems to be limited to pruned blockchain which is a huge limitation for an Electrum server.
But it'll work as a personal Electrum server.

[TheButterZone]

IDK, I'm about as far from an expert as you can get so that's why I wanted more eyes on discussing this... my understanding with pruned nodes was that it downloads everything, then prunes it down.

Let's say you create a new Electrum wallet & connect it to your Mandacaru node, on the Mandacaru side can you mark your client connection as "don't look back, it's new" then does your node take note of the Bloom filter but ultimately only accumulate UTXOs from peers that fit your filter after it connected? The peers see that your node is downloading all new UTXOs but they don't see that your node is insta-pruning them if they aren't relevant to your wallet addresses?

But if you're having the Mandacaru node filter for your imported Electrum wallet that was already filled with UTXOs, it has to process the whole UTXO set to find them all but also do it in stages so it doesn't have more than however little MB stored at a time vs gigabytes?

I'm reminded of https://walletscrutiny.com/android/de.schildbach.wallet/ which was all-in-one node & wallet, but I only watched it run & poked around changing settings without actually receiving to/spending from its wallet.

[nc50lc]

I'm reminded of https://walletscrutiny.com/android/de.schildbach.wallet/ which was all-in-one node & wallet, but I only watched it run & poked around changing settings without actually receiving to/spending from its wallet.

It's certainly similar, Mandacaru uses the improved version of that schildbach wallet's bloom filters which is "compact block filters".
I can't compare the two in detail yet though, since I've just stared reading Floresta's documentation.
(if you're interested to read, link: getfloresta.github.io/floresta-docs)

And I'm not even certain that Mandacaru completely follows its implementation since I'm not familiar with its language (Kotlin).
That's why I've asked for any specific documentation for Mandacaru.

Lastly, it's good that I found that library's docs since it contains a disclaimer for being "highly experimental".
We wouldn't want users to fully rely on Mandacaru where the library that it uses isn't even production-ready yet.
Ref:

⚠️ Keep in mind that Floresta is still highly experimental. We do not recommend using it for transactions involving meaningful amounts of satoshis.

[TheButterZone]

Let's say you create a new Electrum wallet & connect it to your Mandacaru node, on the Mandacaru side can you mark your client connection as "don't look back, it's new" then does your node take note of the Bloom filter but ultimately only accumulate UTXOs from peers that fit your filter after it connected? The peers see that your node is downloading all new UTXOs but they don't see that your node is insta-pruning them if they aren't relevant to your wallet addresses?

Indeed!

Pick your wallet's birthday year in Settings so the node only downloads BIP-158 filters from that point forward.

Wallet birthday year picker (in Settings) — choose the year your wallet activity starts, and the node only downloads compact filters from that point forward.
Initial storage drops from ~13 GB to under 2 GB — by skipping irrelevant historical filters, the on-device data footprint is roughly an order of magnitude smaller out of the box.
Changing the birthday automatically re-syncs the wallet, so transactions appear without manually re-loading the descriptor.

[nc50lc]

Let's say you create a new Electrum wallet & connect it to your Mandacaru node, on the Mandacaru side can you mark your client connection as "don't look back, it's new" then does your node take note of the Bloom filter but ultimately only accumulate UTXOs from peers that fit your filter after it connected? The peers see that your node is downloading all new UTXOs but they don't see that your node is insta-pruning them if they aren't relevant to your wallet addresses?

Indeed!

Pick your wallet's birthday year in Settings so the node only downloads BIP-158 filters from that point forward.

That's from yesterday's update, right?
It seems like they've just added a "birthday" feature to further decrease its resource requirement but "Compact Block Filters", BIP-158 is already being utilized even before that version.
This reminds me of light Monero clients.

Nice update BTW.

[TheButterZone]

My CanaKit RPi3B+ came with a SDC10G2/32GB Kingston microSDHC which is Class 10 but not A1?

Seems like it's not recommended to install Emteria Android on it, then?

Ultimately want to run Mandacaru on the Pi instead of on a mobile device which LOVES routing data through the mobile network for $$$ instead of the perfectly fine home WiFi 3 feet away...

[TheButterZone]

Alright, so I've updated through a few versions & it's up to v0.10.0 now.

https://github.com/jvsena42/mandacaru/issues/3 desperately needs resolution, it's been over a year... IBD & all intensive data needs a setting that forces them to WiFi & only allows mobile data failover for new blocks & light data. Had to disable mobile data in system settings once I realized that Mandacaru wasn't using my WiFi 3 feet away. Some Android apps include a network setting for WiFi-only, mobile-only, or both (WiFi if you're home/unmetered, mobile as failover, theoretically yet in practice...?!), but Mandacaru's would need to be more complex than general failover if we want to be absolutely certain the app will hold the intensive data phases for WiFi.
Fixed in 0.11.5! https://github.com/jvsena42/mandacaru/pull/109

The other issues I've had with it on a phone (which I put the card CanaKit included into, so now my RPi can't run concurrently unless I buy another card): https://github.com/jvsena42/mandacaru/issues?q=is%3Aissue%20author%3Athebutterzone

I hope in the near future https://github.com/jvsena42/mandacaru/issues/57 will be resolved so it's not just bound to localhost & Electrum desktop can connect to the Mandacaru phone/tablet/whatever Android device as well. Then, I would want to card the RPi again & see if I could Android+Mandacaru the RPi to at least be my LAN Electrum server without also having to install Electrum Android on it & run another monitor, monitor switcher, or remote into it to use the client. I didn't realize it was only localhost-bound until I got in too deep LOL

At this point, IMO it needs 2 major & 2 minor things to bare minimum work well with Electrum Android (presuming a fresh install of it, importing the pubkeys from Electrum desktop):

Major:
Only report 100% sync when 100% of the TXIDs have filtered in, with no need for the user to RPC-command multiple rescans over hours, days? because some transactions from the birthday year to well-before Mandacaru's IBD started, are missing.
Seems nearly-fixed (for Electrum, not Bluewallet, but that's probably BW's fault for not properly calculating what it's served) in 0.11.3
... I'm not going to call it totally fixed until there's a release that gets data storage down & I can safely nuke/not redownload all the way back up to >11GB, just to test the filter scanning

QR scanner in the app
Watch this space: https://github.com/jvsena42/mandacaru/issues/65
Added in 0.12.0 & fixed in 0.13.1

Minor:
De-duplication of wallet descriptors, so you can have a long list of unique descriptors instead of uniques mixed with duplicates
jaoleal pointed me to https://github.com/getfloresta/Floresta/pull/821
& fixed in https://github.com/jvsena42/mandacaru/releases/tag/v0.11.3

Have the error/notification box display on top of everything, instead of the box's sides peeking out from below a slightly skinnier opaque layer
Fixed in 0.10.1! https://github.com/jvsena42/mandacaru/pull/83

And about this...

Initial storage drops from ~13 GB to under 2 GB — by skipping irrelevant historical filters, the on-device data footprint is roughly an order of magnitude smaller out of the box.

11.26 GB total storage is used by Mandacaru v0.10.0, having run my current IBD, with 75 total transactions running from birthday year 2017->2025

So getting it to under 2 GB appears aspirational, not achieved yet? https://github.com/jvsena42/mandacaru/issues/53

[TheButterZone]

Edited previous with everything that was fixed through current, v0.13.1.

Now there are 2 major things left:

1) Storage of the filters still in double-digit GBs https://github.com/jvsena42/mandacaru/issues/53

2)

• There was an attempt to make the app update via downloading from GitHub to its own download directory & then pressing a button within the app to install.
• But that failed spectacularly.
• So it was replaced with an in-app update check only, which forced us back to downloading via browser & installing from that directory: what I'd call the nuclear option...
• So, in that nuclear PR's comments, I took a stab at trying to get the in-app download/install flow to work. Even forked & wrestled with ChatGPT to try to get working code in there. But I'm technologically limited from building it to see if it actually works, so I'm not going to submit a PR that could destroy the main repo.

[TheButterZone]

https://github.com/jvsena42/mandacaru/pull/115 made it so descriptors added to Mandacaru can be copied out to client apps.

But Mandacaru's descriptors have prefixes & suffixes added (even if they were scanned/copied in as simply xpub6a3434kas...), which makes them not-universally-compatible/importable: https://github.com/jvsena42/mandacaru/issues/113#issuecomment-4725300278

So... is this a good PR? It was so simple compared to the update fix attempt, I figured it was worth submitting.

[nc50lc]

https://github.com/jvsena42/mandacaru/pull/115 made it so descriptors added to Mandacaru can be copied out to client apps.

The issue is: descriptors doesn't use the prefix method used by Electrum (e.g.: zpub) in its extended public keys to tell which script type to use when restoring the extended key.
It sticks to the "xpub" format and include in the descriptor the necessary derivation paths and script type to derive the child keys from it.
In Electrum, xpub means that it's exported from a p2pkh or legacy P2SH MultiSig wallet, it'll do so when restoring the same key format in the correct restore menu.

And Electrum doesn't support descriptors yet (still in the works), so it can only import the extended key from it.
But if you import it to Electrum's "standard->use a master key" option by just removing everything except the extended public key,
Electrum will assume that you're trying to restore a p2pkh wallet by importing an xpub key as I mentioned above.

The next PR should include an additional step to convert the xpub based from script type indicated in the descriptor.
Better if there's an arg or GUI element to activate it since "descriptors" is now the widely-used standard.
e.g.:
If the descriptor is (script is "wpkh"):

Code:

wpkh([31337fff/84'/0'/0']xpub6BgZWwRHVTnaPoLsMstjkprpzaYMwG4p8ezHaFTJEbWnjk4R1xRn3FNakvDoPXBDHh8tvKGKYnh7i5pXxZ3eL9PD9V7zjuC3k85KmUaWqwb

The extended key that electrum will accept based from its xpub and script type is:

Code:

zpub6qM68Gm7npsY6Pj72bTzB13qLWqFpW3oxt2j93F4zcGYqwgsXGkuHNgroL8yPLV46yNWRGTSU7QDUf3fPwsfvckQtAWquiq2HaCcYgEezSk

Here's jlopp's xpub converter for reference: jlopp.github.io/xpub-converter
And Bitcoin Core's descriptors documentation: github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md

[TheButterZone]

FWIW, the way Mandacaru copies out its wpkh descriptors is (using your sample key)
wpkh(xpub6BgZWwRHVTnaPoLsMstjkprpzaYMwG4p8ezHaFTJEbWnjk4R1xRn3FNakvDoPXBDHh8tvKGKYnh 7i5pXxZ3eL9PD9V7zjuC3k85KmUaWqwb/<0;1>/*)

Not that I feel comfortable making a more advanced PR that does conversion... I'll quote/link to your reply though: https://github.com/jvsena42/mandacaru/pull/122#issuecomment-4776082680

There's a bigger problem for the past week between my Mandacaru & Electrum apps... "RequestCorrupted: inconsistent chunk hex and count" https://github.com/jvsena42/mandacaru/issues/121 - if it's going to require me wiping storage & redownloading all >11GB from scratch yet again, I feel like abandoning this project until they (everyone/anyone involved with Mandacaru/Floresta) get the storage finally compacted. It's hanging by a thread at this point, because BlueWallet only calculates the correct balance of my zpub wallet, not my xpub wallet (BW thinks I have xpub UTXOs I actually don't anymore - they were spent >6 months before I first installed Mandacaru, despite the Mandacaru-solely-connected Electrum app prior to a week ago correctly calculating & not having those UTXOs). And Blockstream Green won't even connect to Mandacaru at all.

[TheButterZone]

As I slept, the phone rebooted for whatever reason & grayed out Mandacaru, Electrum, and BlueWallet (all on adopted storage). Previously I was able to restore them to running again by rebooting, but no amount of rebooting worked. Some of my internal storage apps broke too (at least until I pulled the card out, then they worked again). I dumped bugreport, no bueno:

06-27 21:59:07.337 root 493 493 E vold : Failed to mount /dev/block/dm-11: I/O error
06-27 21:59:07.337 root 493 493 E vold : private:179,34 failed to mount: I/O error

Put the card in my computer & reformatted it for portable storage, then popped it back in the phone & another portable storage reformat by Android itself. But until https://github.com/jvsena42/mandacaru/issues/53 gets resolved the Floresta way, or Mandacaru has the ability added to run on internal storage but store its data on the portable storage, I think I probably shouldn't run Mandacaru anymore. Not sure how close to death the card is at this point, or what would really destroy it - it originally came in the CanaKit RPi 3B+. I don't have any other cards to burn.

P.S. This is a non-rooted phone & for personal reasons, I shouldn't root it even though I want to.

[nc50lc]

As I slept, the phone rebooted for whatever reason & grayed out Mandacaru, Electrum, and BlueWallet (all on adopted storage).

Maybe it performed a system update during that time?
But something terrible happened during the auto-update.

You can check your phone's settings for signs of update, it usually shows if the system is updated in "System->Software update" settings.

[TheButterZone]

As I slept, the phone rebooted for whatever reason & grayed out Mandacaru, Electrum, and BlueWallet (all on adopted storage).

Maybe it performed a system update during that time?
But something terrible happened during the auto-update.

You can check your phone's settings for signs of update, it usually shows if the system is updated in "System->Software update" settings.

Jun 27 was last successful check for update, with Android (12) security update: Mar 5, 2025

[nc50lc]

Jun 27 was last successful check for update, with Android (12) security update: Mar 5, 2025

Oh, it looks like the system wasn't updated at that time.
Otherwise, it'll actually tell that the system is updated recently.

Since it has similar symptoms before, perhaps it's a hardware issue all along and it's getting worse.
Anyways, that's something only a smartphone hardware technician can handle.

[TheButterZone]

So rather than using Mandacaru actively as a wallet, I'll be running my debug APK built from https://github.com/jvsena42/mandacaru/pull/131 to test it for realsies the next time I get a GitHub notification from Mandacaru of a new release. Passed every other test. Even got fancy & went back to GPG sign all my commits.

Was inspired to revisit my failed attempt at fixing updater detection after I succeeded in building an APK the first time ever, for https://github.com/TheButterZone/mandacaru/tree/user-selectable-datadir - but it's probably going to need more work than I can do in a single fork, to make portable storage accept being a datadir.