Customized export options

[takuma sato]

Bitcoin Core has a simple export option that gets you the complete track record of any movements, with "Sent to" and "Received with" as "Type". I reckon it would be interesting to be able to get some options such as, get a list of all unspent UTXOs, along with labels, and any other context. Or filter out any addresses that begin with 1. Or get a list of addresses with funds above 0.1 BTC, and so on. Is this possible? I guess with RPC and some clever commands involving "listunspent" and scripting with json files you could get it done, but it would be good to get a context GUI menu with filters, so you could get an exported file with exactly what you need, because right now like I said you just get a regular bank-statements type of track record file, but you may need to get some data out of it beyond that.

[hmbdofficial]

After studying this I think it would be useful improvement. Though  bitcoin core already has most of this information through RPC command like listunspent, so the data is much available. But rhe problem is that users need to use RPC or write a script to get and filter it.

Having this filter directly in the GUI would have made things much easier.  For example, user could choose to show only unspent UTXOs, addresses with more than 0.1BTC, specific address type, or address with certain labels and then go ahead to export the result. This would have been a useful wallet management feature without requiring any changes to bitcoin consensus rules.

[nc50lc]

Is this possible? I guess with RPC and some clever commands involving "listunspent" and scripting with json files you could get it done

Yeah, only through scripting, but listtransactions "*" 999999 is more suitable instead of listunspent.

If it's just the unspent outputs similar to listunspent's result:
You can utilize Core's "Coin Control" GUI element for the sorting and grouping.
Just enable "Settings->Options->Wallet->Enable Coin Control Features" and an Inputs button will show in the "Send" tab.
From there, you can see amounts, labels, dates, etc. and you can click each category to sort them out.
There's also "Tree Mode" to group them by address.

But for all transaction history's details, there's currently no workaround in the GUI.

[NotATether]

But for all transaction history's details, there's currently no workaround in the GUI.

Well technically, there is, but it involves using the Console inside Bitcoin Core's GUI. 

[takuma sato]

Is this possible? I guess with RPC and some clever commands involving "listunspent" and scripting with json files you could get it done

Yeah, only through scripting, but listtransactions "*" 999999 is more suitable instead of listunspent.

If it's just the unspent outputs similar to listunspent's result:
You can utilize Core's "Coin Control" GUI element for the sorting and grouping.
Just enable "Settings->Options->Wallet->Enable Coin Control Features" and an Inputs button will show in the "Send" tab.
From there, you can see amounts, labels, dates, etc. and you can click each category to sort them out.
There's also "Tree Mode" to group them by address.

But for all transaction history's details, there's currently no workaround in the GUI.

Yeah I know about Coin Control, it's great and I've been using it for years. What I need is to generate a csv file with a list of addresses with current balances, not having a bunch of transactions there, and so far I have only seen a way to export the regular csv that gives you the complete transaction history. So I have been looking at this file and see if I could manually work the data within Excel. However, I am interested in exporting what I want directly from the source, instead of having to use Excel features. I would like to generate a list of all and any addresses that contain funds, instead of having the same address showing up multiple times with each transation, just show each address with the current funds. Would this be possible somehow? It would be useful for some contexts, and im not seeing a clear way to export this information.

listtransaction isn't going to do this the way im reading the manual. The numbers just limit the amount of transactions within a range. I don't think listunspent also does this.

https://developer.bitcoin.org/reference/rpc/listtransactions.html
https://developer.bitcoin.org/reference/rpc/listunspent.html

To be clear, let us imagine you have a wallet with:


(Address1) = 0.2 BTC
(Address2) = 0.1 BTC
(Address3) = 0.5 BTC
(Address4) = 0.3 BTC
(Address5) = 0.5 BTC

And your transaction list contains a bunch of different payments for each address. Sometimes you have some transactions where you sent funds elsewhere. So if you export the csv, you will get a bunch of different transactions, not the clean list of the 5 addresses with the final, current funds for each. Some may have a bunch of different transactions going into Address1 to 5, and some going out in between.

If you have thousands of smaller transactions, it's really difficult trying to get a clean list of your addresses that actually contain funds. I've tried manually doing it in Excel by getting the entire list of addresses, removing any duplicates, removing any "Sent to" transactions, and then looking up each address in an explorer and copying the current amount for each. Somehow the sum does not match the current funds seen on the wallet, so I don't get it. If I sum all the transactions on the export csv file by Bitcoin Core, it matches the funds shown in the client, so im doing something wrong I guess. This all would be solved if we had more options when exporting data from within Bitcoin Core instead of having to play around with Excel which leads to mistakes.

[alegotardo]

Bitcoin Core has a simple export option that gets you the complete track record of any movements, with "Sent to" and "Received with" as "Type". I reckon it would be interesting to be able to get some options such as, get a list of all unspent UTXOs, along with labels, and any other context. Or filter out any addresses that begin with 1. Or get a list of addresses with funds above 0.1 BTC, and so on. Is this possible? I guess with RPC and some clever commands involving "listunspent" and scripting with json files you could get it done, but it would be good to get a context GUI menu with filters, so you could get an exported file with exactly what you need, because right now like I said you just get a regular bank-statements type of track record file, but you may need to get some data out of it beyond that.

As a programmer and Linux user, I like command lines and you can do a lot of things if you give up a graphical interface, but I confess.... it really could be easier.

But, if you have a little confidence to work with command line, try this:

- Unspent UTXOs with context: With 'listunspent' you can get the txid, vout, address (if it exists), value in BTC, number of confirmations, script and others indicators.
-If you only want addresses that starting with 1, you can filter the address field of the result by prefix.
- To get individual UTXOs of at least 0.1 BTC, then add the option 'minimumAmount'.
- UTXOs above 0.1? Group them by address, add up the values and only then apply limit.

Bitcoin Core has a lot of things that other wallets do not do, but you need to know how and where to dig around and you can experiment with the RPC workflow on a testnet, bearing in mind that its address prefixes differ from the mainnet.

[nc50lc]

I would like to generate a list of all and any addresses that contain funds, instead of having the same address showing up multiple times with each transation, just show each address with the current funds. Would this be possible somehow?

Currently nothing can output a CSV file in that exact format, in the GUI nor with RPC commands.

A script or something has to parse the command's result and format it to your desired output,
Try a good LLM if it can write you one.

Even BIP-0329 standard that's used by some famous wallets don't output in that format.
If ever Core implements it, it still wont apply to your use-case.

Well technically, there is, but it involves using the Console inside Bitcoin Core's GUI. 

That's listtransactions command that I've mentioned in that same post.

[Antidote47k]

~snip

I think the reason this probably gets awkward with the existing tooling is that core generally tracks spendable UTXOs rather than “address balances”.
If you’re reconstructing the current wallet state from the historical transaction CSV it’ll be inherently error prone, cause you’re essentially replaying past inputs and outputs to determine what remains unspent. The wallet’s current unspent outputs from listunspent is a better source for an export like this because the UTXOs can be grouped by script or address and their values aggregated.

If there ever was a GUI export feature, it'd be better to treat it as an UTXO based export with addresses and label metadata rather than an address based ledger. That will help maintain the output closer to what the wallet represents and tracks spendable state.

[NotATether]

Even BIP-0329 standard that's used by some famous wallets don't output in that format.
If ever Core implements it, it still wont apply to your use-case.

I know it's not applicable here, but I think it would have to take an industry initiative in order to get standardized wallet label export formats adopted in the major wallets.

I don't see that happening anytime soon, but as always, I would like to be proven wrong.

[nc50lc]

I don't see that happening anytime soon, but as always, I would like to be proven wrong.

It's quite widespread already, some wallets already implemented it while others have open discussion or PR.
Those that implemented it mostly aren't the major ones though.

Use this to track which wallets already support it: bip329.org

[takuma sato]

~snip

I think the reason this probably gets awkward with the existing tooling is that core generally tracks spendable UTXOs rather than “address balances”.
If you’re reconstructing the current wallet state from the historical transaction CSV it’ll be inherently error prone, cause you’re essentially replaying past inputs and outputs to determine what remains unspent. The wallet’s current unspent outputs from listunspent is a better source for an export like this because the UTXOs can be grouped by script or address and their values aggregated.

If there ever was a GUI export feature, it'd be better to treat it as an UTXO based export with addresses and label metadata rather than an address based ledger. That will help maintain the output closer to what the wallet represents and tracks spendable state.

I still don't understand why the amounts do not match. If you do the regular csv file export that gets you the list of transactions in order, and you go on Excel or any other similar software, and do a simple sum in a cell of the entire movements (the "Sent to" and "Received with", that is, the entire list), then the result is the same number of coins shown on the actual wallet. So this is correct.

However, if you remove duplicated addresses, and then just look up one by one in a block explorer to check the actual funds within them, type them next to each address in your spreadsheet, and do the sum, shouldn't this be the exact number of availible coins you have on the wallet? (or the sum of all the transactions from the regular exported file, which like I said, matches what you have in the wallet). I hope im being clear with this, but it makes no sense to me that the sum of the non-duplicated addresses with each current availible funds next to them as checked on a block explorer, does no match the funds you actually have on the wallet. It should be the exact same value, but im probably doing a mistake somewhere or something. If anyone could try this themselves, let me know if you get the same value.

[Antidote47k]

I still don't understand why the amounts do not match. If you do the regular csv file export that gets you the list of transactions in order, and you go on Excel or any other similar software, and do a simple sum in a cell of the entire movements (the "Sent to" and "Received with", that is, the entire list), then the result is the same number of coins shown on the actual wallet. So this is correct.

However, if you remove duplicated addresses, and then just look up one by one in a block explorer to check the actual funds within them, type them next to each address in your spreadsheet, and do the sum, shouldn't this be the exact number of availible coins you have on the wallet? (or the sum of all the transactions from the regular exported file, which like I said, matches what you have in the wallet). I hope im being clear with this, but it makes no sense to me that the sum of the non-duplicated addresses with each current availible funds next to them as checked on a block explorer, does no match the funds you actually have on the wallet. It should be the exact same value, but im probably doing a mistake somewhere or something. If anyone could try this themselves, let me know if you get the same value.

Yeah, I think the core issue stems from the fact that the total wallet balance isn’t an aggregation of known receiving addresses.
To put things into perspective, when you duplicate the addresses from the transaction history and query a block explorer, you may possibly miss the wallets internal change addresses. It’s possible for Core to route the remaining funds from a transaction to a fresh change output, which won’t necessarily appear among the receiving addresses you originally generated or labelled.

An address can also contain more than one UTXO and an unspent output can be on an internal change script that wasn't explicitly generated as a receiving address. That's why listunspent remains as my reliable first step. It queries the walkets actual live UTXO set directly which includes change outputs which can then be grouped by address or label where applicable.
Trying to reconstruct from historical trnsactions and then checking them on an explorer is bound to leave blind spots.

[takuma sato]

I still don't understand why the amounts do not match. If you do the regular csv file export that gets you the list of transactions in order, and you go on Excel or any other similar software, and do a simple sum in a cell of the entire movements (the "Sent to" and "Received with", that is, the entire list), then the result is the same number of coins shown on the actual wallet. So this is correct.

However, if you remove duplicated addresses, and then just look up one by one in a block explorer to check the actual funds within them, type them next to each address in your spreadsheet, and do the sum, shouldn't this be the exact number of availible coins you have on the wallet? (or the sum of all the transactions from the regular exported file, which like I said, matches what you have in the wallet). I hope im being clear with this, but it makes no sense to me that the sum of the non-duplicated addresses with each current availible funds next to them as checked on a block explorer, does no match the funds you actually have on the wallet. It should be the exact same value, but im probably doing a mistake somewhere or something. If anyone could try this themselves, let me know if you get the same value.

Yeah, I think the core issue stems from the fact that the total wallet balance isn’t an aggregation of known receiving addresses.
To put things into perspective, when you duplicate the addresses from the transaction history and query a block explorer, you may possibly miss the wallets internal change addresses. It’s possible for Core to route the remaining funds from a transaction to a fresh change output, which won’t necessarily appear among the receiving addresses you originally generated or labelled.

An address can also contain more than one UTXO and an unspent output can be on an internal change script that wasn't explicitly generated as a receiving address. That's why listunspent remains as my reliable first step. It queries the walkets actual live UTXO set directly which includes change outputs which can then be grouped by address or label where applicable.
Trying to reconstruct from historical trnsactions and then checking them on an explorer is bound to leave blind spots.

Frankly, I still don't get it. The final result of doing the sum of all "Sent to" and "Recieved with" transactions from the regular exported csv file by Bitcoin Core, does not include any "change" addresses. Every address I have is a recognizable by me transaction, with a label, and it's clearly stated what is what, and I don't see any change movements in there. So im not really understanding how if I get a non-duplicated list from there, and check it on the block explorers, it shouldn't match, unless, from what im understaning here:

"It queries the walkets actual live UTXO set directly which includes change outputs which can then be grouped by address or label where applicable. "

Then this may mean that some of these have change addresses summed to them, even if I always used the default option of generating a new address for change addresses, so I found this confusing, how one address would be the sum of another addres and show up there. It would be clearer if Bitcoin Core included change address movements in the export too, or at least have an option to do so.

So I guess im going to need to run listunspent and then from what I see there, try to find what change addresses are missing to add them to the list and see if I can get a full list of unique addresses with availible funds from a wallet. This should all be able to be done with a single click because it's important information many users may need for a variety of reasons. I guess this is shown in Coin Control window, so a csv with a readable format ordered by date could be generated. For users that use Bitcoin Core as a wallet, you would want to have many more export options. Hopefully someone is able to do some improvements there and propose them on github.

[Forsyth Jones]

.....
So I guess im going to need to run listunspent and then from what I see there, try to find what change addresses are missing to add them to the list and see if I can get a full list of unique addresses with availible funds from a wallet. This should all be able to be done with a single click because it's important information many users may need for a variety of reasons. I guess this is shown in Coin Control window, so a csv with a readable format ordered by date could be generated. For users that use Bitcoin Core as a wallet, you would want to have many more export options. Hopefully someone is able to do some improvements there and propose them on github.

I see that you and I are frequent Bitcoin Core users. I don't usually export my transactions list and I almost never used the feature to label transactions, I started adopting this practice this year. Whenever I want to know all my current UTXOs, I always search with listunspent. I haven't tested it yet (I'm on Windows by now), but you can follow alegotardo's advice, it seems that call arguments like minimumAmount, maximumAmount and etc exist.

I agree that it would be useful if such a feature were available in the GUI. I would even like there to be an organization in the list of receiving and sending addresses tabs, separated by address script type (I think there's already an issue about this on github). Anyway, it would be a good idea for you to open an issue on github requesting it as a feature request, describing exactly your pain point, as it could be useful to many other users.

It's usually more likely, the devs will close your issue and not implement it, but it doesn't cost anything to try.

Edit: Based on my experience, I've opened issues on Bitcoin Core, Electrum and I think even Sparrow, and the developers ask you to explain the use case or objective of your proposal (even if you make it explicit in your text). They wonder if they think it's useful from a non-technical point of view, if users will know how to use it, they make a thousand assumptions... time passes and then close your issue, and you've wasted all your time writing the damn text...

From what I understand, the devs have their priority list, they have many accumulated issues and PRs, and the chances of closing your issue are very high (no matter how useful it may be), but they don't have time to evaluate and do the necessary testing. The chances of it being submitted as a PR might increase, but will it be worth the effort to have your PR refected in the end?

It would be more useful if we united several users with the same interest and fueled the discussion of a particular issue. That way, they would see that there is a demand for it, and perhaps the chances would increase...