Is exporting->importing descriptors badly implemented? (workflow wise)

[takuma sato]

I think I have found something that isn't properly implemented in the following workflow. What I have been trying to do for the past month or so is figure out how to do a 2 laptop setup, one online and one offline, and be able to use the PSBT solution to quickly craft transactions using the GUI. I want to use Coin Control, pick what UTXO's I want to use for the transaction, and then generate this PSBT file, and then sign it in the offline node and then broadcast it into the online node.

But im using an old wallet, so first I need to migrate the wallet, so it gets converted, since to do the PSBT thing you need the descriptors of the addresses, this new wallet format that contains this information. So im assuming "File->Migrate Wallet" will convert the 2013 wallet im using (not sure exactly version was used to create this file, but around 2013) into the new format (and hopefully, it does not screw up your wallet.dat file during this migration, I would like to know how to check if the migration is successful btw) and then I can use PSBT files with this wallet, but I still need to create a watch-only wallet for this format.

Could you please check that this is correct?

Prepare the two laptops

Laptop 2 (offline signer):

    Physically remove WiFi and Bluetooth radios if possible. Physically disconnect any network cable. Once powered up into Debian, run: rfkill block all.
    Copy over Knots via USB after verifying the signatures.
    Create a private wallet directory (permissions 700), e.g. ~/knots_wallets/legacy/, and put a copy of wallet.dat inside as wallet.dat.

Laptop 1 (online full node):

    Run Knots as a full, non-pruned node (already done – just finish the IBD).
    You’ll import a watch-only wallet here later.

Safest way to load and migrate the old wallet (offline only)

Knots should be able to open old Berkeley DB wallets as “legacy” wallets. The modern standard is a descriptor wallet. Migrating is one-way, which should be fine here since we are ensuring that we have backups.

On Laptop 2 (offline):

    1. Start Knots with your legacy wallet (GUI: “Open wallet” and point at the folder, or CLI with -walletdir pointing to ~/knots_wallets).
    2. Take another fresh backup from Knots (“Backup wallet”) so you have a post-open snapshot (just being paranoid here).
    3. Upgrade to descriptor (this might be labeled as “Migrate wallet” in the GUI; via RPC it’s upgradewallet). This creates internal descriptors for your keys/addresses. Note that I have not done this myself, so I am relying on various search results (there is some discrepency on the label – maybe due to different Knots/Core versions). Let me know if you can’t find anything that seems relevant here, and I’ll do some more digging.
    4. After upgrade, export public descriptors (GUI export if present, or RPC):

    listdescriptors false > exports public (no private) descriptors, including active and internal (change) ones, with useful timestamps.

Save those descriptors to a text file on a clean USB (this file is safe to move – no secrets if you used false for listdescriptors).


Build the online watch-only wallet

On Laptop 1 (online):


    1. Create a new empty wallet (GUI: “Create wallet” > disable private keys / “watch-only”).
    2. Import the public descriptors you exported from offline:

    GUI import (if present), or RPC importdescriptors with the JSON from your file.
    Make sure the active, internal, range (for HD paths), and timestamp fields are preserved so Knots rescans correctly.

    3. Let it rescan the chain (can take a while). When done, balances and UTXOs should show as watch-only.



Sanity check:

    On the offline wallet, generate a receive address and write it down.
    On the online watch-only wallet, derive/show a receive address. It should match. (Matching here proves the descriptors line up.)


PSBT flow

Create (online):

    In the online watch-only wallet, enable Coin Control. Pick the UTXOs, set outputs, and choose “Create PSBT” (GUI) or walletcreatefundedpsbt (RPC). Save the .psbt to USB.

Sign (offline):

    On the offline wallet, open/sign the PSBT (GUI: “Load PSBT” > “Sign”; RPC: walletprocesspsbt). Save the partially or fully signed PSBT back to USB.

Broadcast (online):

    On the online node, Finalize & Broadcast (GUI) or finalizepsbt then sendrawtransaction with the hex.
    Verify the txid and watch it confirm.

These instructions I think were generated with the help of AI by someone. I have not tried this yet, but first of all, you are supposed to use  "File->Migrate Wallet". This gives you the new wallet.dat file (not sure if it overwrites it??) and then, you can proceed to export the descriptors data from the file, so you can import it into the watch-only wallet on your online laptop, which I just created.

The problem I see is... okay, why isn't there in the GUI, something to export this into a .json file, so you can import this file in the watch-only wallet, instead of having to copy-paste the output it gives on the console, and then apparently having to manually modify things because they don't meet the required format that it's expected with the import function?

Because apparently this is what is going on:

When you want to create the watch-only wallet, there is an option in the GUI to create it (by disabling the private keys, and enabling the descriptor wallet option to be able to use PSBT easier method) but the thing is.. okay great, they bothered to add this in the GUI, but then what, you've got this empty wallet, and there are no options in the GUI to import the stuff you need to actually use a watch-only wallet. So I had to look it up and see how it's done with the console. I found that guide above. This is the first thing I would complain about, add some export/import option on the GUI, because at least im not seeing anything.

But then there's this thing going on if you do with the console. Basically you have to go into your offline laptop, and generate a .json file, which you will be importing in the watch-only wallet on the online computer. Once again, no GUI option to do this, so I had to look it up and found out you have to use this:

listdescriptors false > exports public (no private) descriptors, including active and internal (change) ones, with useful timestamps.

So this lists the stuff you need, but you have to copy-paste this stuff in the right way. Why is there no way to export a .json file with the correct format so you can easily import this in the watch-only wallet? This would improve this workflow a lot. But what we have now, it is what it is. So my question is, what is exactly going on that needs to be modified so the format is correct to import it into the watch-only wallet? Because it appears the output from listdescriptors false will not work if you simply copy paste it into a .json file. See this:

From the offline box, you should have gotten something like:

Code:

{
  "wallet_name": "cold_migrated",
  "descriptors": [
    {
      "desc": "wpkh([abcd1234/84h/0h/0h]xpub.../0/*)#u5t4kl9d",
      "active": true,
      "timestamp": 1388534400,
      "range": [0, 999],
      "internal": false
    },
    {
      "desc": "wpkh([abcd1234/84h/0h/0h]xpub.../1/*)#p8d0yq56",
      "active": true,
      "timestamp": 1388534400,
      "range": [0, 999],
      "internal": true
    }
  ]
}

The importdescriptors RPC expects an array of objects with just the needed fields:

Code:

[
  {
    "desc": "wpkh([abcd1234/84h/0h/0h]xpub.../0/*)#u5t4kl9d",
    "active": true,
    "internal": false,
    "timestamp": 1388534400,
    "range": [0, 999],
    "label": "external"
  },
  {
    "desc": "wpkh([abcd1234/84h/0h/0h]xpub.../1/*)#p8d0yq56",
    "active": true,
    "internal": true,
    "timestamp": 1388534400,
    "range": [0, 999],
    "label": "change"
  }
]

So apparently, I would need to remove

Code:

"{
  "wallet_name": "cold_migrated",
  "descriptors": "

And copy-paste only what is cointained inside "[.....]"

But then, there's also a difference in

"internal": false / "internal": true

that is seen in the output, but that is not seen in the expected format to import it, which also has this

"label": "external" /     "label": "change"

which is not seen in the output... so what im supposed to do there? This would already worry the end user, which would think something may go wrong during this import/export workflow, let alone if you need to manually go address by address fixing things to match the expected import format, even tho im not sure this information is correct, so im asking here what's going on with this, and what would be the best way to finally get this done. I wanted to try this with testnet coins first, but I would need to download Bitcoin Core from 10 years ago, generate a new wallet, get some coins, open this wallet on the new Bitcoin version (im using Knots 28 right now) then proceed to Migrate the wallet, see if it works, and then do the stuff seen above, but I do not want to download and validate the blockchain again because it took forever, and it will probably be slower on an older version, so hopefully I can get some advice here. And im also reporting that this whole workflow needs to be polished, because unless im doing something wrong, this seems rather convoluted to do a simple import/export action, but for now, let's see if I could get this one with what we have.

[achow101]

I would like to know how to check if the migration is successful btw

You can use getaddressinfo on a bunch of your addresses and check that the wallet says "ismine": true for them. You can use listunspent before and after migration to verify that the wallet knows the same UTXOs. You can use listtransactions before and after migration to verify that the wallet knows the same transactions.

but first of all, you are supposed to use  "File->Migrate Wallet". This gives you the new wallet.dat file (not sure if it overwrites it??)

It overwrites the original wallet.dat file, but a backup of the original is created at the start of the migration process. That backup can be restored manually (by copying the file, or by using restorewallet or File > Restore Wallet), and is also restored automatically if the migration fails so that you don't have a wallet that is partially migrated.

The problem I see is... okay, why isn't there in the GUI, something to export this into a .json file, so you can import this file in the watch-only wallet, instead of having to copy-paste the output it gives on the console, and then apparently having to manually modify things because they don't meet the required format that it's expected with the import function?

I opened a PR a couple weeks ago to produce the watch-only wallet file directly instead of having to do the import stuff: https://github.com/bitcoin/bitcoin/pull/32489. There's a followup PR to add a menu action to the GUI to do that: https://github.com/bitcoin-core/gui/pull/872

This was slightly complicated to do because of the legacy wallet code. But with it removed, the implementation becomes much simpler.

and there are no options in the GUI to import the stuff you need to actually use a watch-only wallet.

There was a PR to add import things to the GUI but it was closed after a couple years of not getting any review.

So my question is, what is exactly going on that needs to be modified so the format is correct to import it into the watch-only wallet?

Everything in the "descriptors" field, i.e. everything after the opening square bracket, to the matching close square bracket, including the brackets themselves. The contents of the descriptors do not need to be modified and are intentionally such that the whole thing can be copy-pasted into importdescriptors.

But then, there's also a difference in

"internal": false / "internal": true

that is seen in the output, but that is not seen in the expected format to import it, which also has this

"label": "external" /     "label": "change"

which is not seen in the output... so what im supposed to do there?

Nothing, those are optional fields.

[takuma sato]

I will try to get it done and see what happens. The only thing im not convinced is deprecating the old wallet format. There's people that are paranoid of using HD-wallets. For instance with the quantum thing, if someone had one of these extended public keys, then who know was kind of derivation attacks could be performed. I generally don't like the idea of "this string of characters gives you access to all existing and future addresses on the wallet". If you have the old wallet, it may be more annoying to use, you need to do constant backups as you generate new keys, and you don't have ease of use features like PSBT, but you don't have that risk, they would need to know each of your public keys separately. This gives you more margin in trying to move funds in for instance, the quantum scenario. You could move your funds to whatever QC resistant solution is out there eventually, with a better peace of mind, since at least, it's one public address-one private key type of deal. So there should be ways to not lock people's funds for those that use older wallets, forcing them to run older software to be able to do anything with the money.

[achow101]

For instance with the quantum thing, if someone had one of these extended public keys, then who know was kind of derivation attacks could be performed.

Conversely, a proposed method for spending vulnerable coins post-quantum computers is to provide a zero knowledge proof of the master key and derivation path to the key. HD wallets have these other semi-private info that can make it easier to recover vulnerable funds in the event of a sudden quantum computer.

You could move your funds to whatever QC resistant solution is out there eventually, with a better peace of mind, since at least, it's one public address-one private key type of deal.

... You can do that too with HD wallets, it's not as if the people using HD wallets are not going to be able to migrate to a post quantum method. This concern makes no sense to me.

So there should be ways to not lock people's funds for those that use older wallets, forcing them to run older software to be able to do anything with the money.

Your existing funds aren't locked. You can migrate a non-HD wallet, and while that will change how new addresses are generated, already generated addresses and their associated addresses will still be spendable. Migration does not move any funds.

[takuma sato]

Your existing funds aren't locked. You can migrate a non-HD wallet, and while that will change how new addresses are generated, already generated addresses and their associated addresses will still be spendable. Migration does not move any funds.

What I meant is, if you choose to stay with the old wallet format, you are pretty much stuck using older versions forever now, at least if you want to use Core, im not sure what Knots will do with this thing.

Btw, what about labels? I always have transactions ordered with a label to know where coins are from, I also do this with sending addresses. Well my question is, when you use listdescriptors false to copypaste this into the .json file, will it include each label for each address? Because if in the watch-only wallet you just get a bunch addresses with no context, then im not sure what is going on there. I need the labels so in Coin Control you know exactly what coins you are moving.

[achow101]

Btw, what about labels? I always have transactions ordered with a label to know where coins are from, I also do this with sending addresses. Well my question is, when you use listdescriptors false to copypaste this into the .json file, will it include each label for each address? Because if in the watch-only wallet you just get a bunch addresses with no context, then im not sure what is going on there. I need the labels so in Coin Control you know exactly what coins you are moving.

Labels are not included in the listdescriptors output.

[takuma sato]

Btw, what about labels? I always have transactions ordered with a label to know where coins are from, I also do this with sending addresses. Well my question is, when you use listdescriptors false to copypaste this into the .json file, will it include each label for each address? Because if in the watch-only wallet you just get a bunch addresses with no context, then im not sure what is going on there. I need the labels so in Coin Control you know exactly what coins you are moving.

Labels are not included in the listdescriptors output.

Wait so what then? How im supposed to tell which coins im using in a transaction if I have hundreds of addresses there without the labels? I thought "label" on the .json would include the labels but apparently that's something else... so there's no way to do this? Without the labels this is completely useless, I just don't know what im even looking at, just a bunch of addresses with no context, this is so lame. There isn't even a way to do this manually? I'll need to go one by one, what a nightmare. Anyway, hopefully there is a workaround, otherwise I really don't see how im going to use this.

[achow101]

How im supposed to tell which coins im using in a transaction if I have hundreds of addresses there without the labels?

You can apply the labels manually.

I thought "label" on the .json would include the labels but apparently that's something else... so there's no way to do this?

Not at this time.

importdescriptors can take a label, but only for descriptors that map to a single address (i.e. descriptors which can derive multiple addresses because they use BIP 32 cannot take a label). Furthermore, labels are applied to addresses, not to labels, so exporting all of the descriptors will not result in any associated labels.

Anyway, hopefully there is a workaround, otherwise I really don't see how im going to use this.

The workaround is the PRs I mentioned earlier. Labels and other metadata are why I implemented a wallet file export rather than an easier way to import descriptors.

[takuma sato]

How im supposed to tell which coins im using in a transaction if I have hundreds of addresses there without the labels?

You can apply the labels manually.

Could you please explain what is the correct way to do this manually? Where is the context for each address located? What is at least in Coin Control showing as "Label" that is (not my pic btw because I cannot login to my node now, I was able to find this pic from 2015, hopefully it looks the same now)

https://i.sstatic.net/yUXEX.png

I don't remember right now what options the right-click dropdown menu gives you, but I reckon there was no way to edit the labels from the GUI in there, so I guess this has to be done before you import things, in this .json file from listdescriptors false I guess? But where does it go?

The workaround is the PRs I mentioned earlier. Labels and other metadata are why I implemented a wallet file export rather than an easier way to import descriptors.

How are is this from being merged? And do you think Knots would be ok with merging these PR's?

[achow101]

Where is the context for each address located?

Window > Receiving Addresses for the labels on your addresses.

I don't remember right now what options the right-click dropdown menu gives you, but I reckon there was no way to edit the labels from the GUI in there, so I guess this has to be done before you import things, in this .json file from listdescriptors false I guess? But where does it go?

No, labels can be edited after the fact, at any time. From the GUI, in the Receiving Addresses window, you can open the context menu (right click) and choose Edit, or double click the label to edit.

You can add labels using the setlabel RPC.

How are is this from being merged?

A few months probably, it is still waiting for more review. The soonest it can be in a release is v31 which is scheduled for 7 months from now.

And do you think Knots would be ok with merging these PR's?

Probably, but I would not expect knots to do a release sooner. Also, Luke generally merges PRs before their final state in Core, and long before they have had sufficient review so I would not necessarily expect everything to work as expected.