Organic UTXO consolidation (in the course of regular payments)

[simplexum]

For UTXO-based cryptocurrencies, it is important to have the set of managed Unspent Transaction Outputs in optimal shape, because if you have too much small UTXO, you will have to create bigger transactions to send the same amount of funds, and hence pay bigger network fee, or even need to send several transactions to transfer the needed amount.

Typical approach to handle this is to do periodical sweep of UTXO, bundling them all together, at the time when network fee is low.

We implemented another approach, that does not require manual UTXO consolidation, and at the same time, conserves network fee - when fee in the network is high, the algorithm tries to use less UTXO per transaction, and when network fee is low, it will create bigger transaction, effectively performing UTXO consolidation along with regular payments. This organic UTXO consolidation allows to maintain healthy UTXO set automatically, without manual consolidation runs.

We implemented it this way:

Define a rate that you are OK to pay in terms of monetary percentage: we call it "fee per full coin". Basically, you say, "to send 1 btc, I am willing to pay 0.001 btc"

When building a transaction, we calculate optimal tx size like this:

    optimal_tx_vsize = utxo_sum * fee_per_full_coin / fee_per_kb

The closer candidate transaction to the optimal size, the better score it will get.

Of course we cannot always find the optimal transaction, but we will try to choose the one with best score, depending on available UTXO set.

The resulting effect is that when fee_per_kb is high, optimal_tx_vsize will be low (may be even zero), and we will try to find the smallest tx possible with current UTXO set. When fee_per_kb is low, optimal_tx_vsize will be high, and we will choose transactions with more UTXO, performing organic UTXO consolidation.

---

If more wallets implement this type of optimizations, it will result in more healthy global UTXO set.

[simplexum]

Maybe you could have it generate a sweep dust transaction automatically when the fees drop below a threshold.

You can do special sweeping tx without this formula. The idea here is that if you do payments regularly, the fee in the network varies, and you set your fee_per_full_coin reasonably high - you will get gradual UTXO consolidation, without the need for separate sweeping transaction.

It may not be very useful for a single user wallet that spends occasionally, but for busy service that receives and sends a lot of payments - might mean that when they suddenly need to send a big payment, and the network fee is high - they won't need to send a huge-sized tx and pay a lot in fees, because their UTXO set will be in good shape.

[simplexum]

I've done a lot of work in this space

Same. (see: https://simplexum.com), but it is not focused solely on coin selection, so probably you've done more work in this direction.

This seems like a reasonable approach for consumer wallets to use, but I don't think it's going to scale very far.

What would prevent this from scaling ?

the more utxo's you have the easier it is.

Yes, but if you have too much small utxo, and you need to send large payment at inconvenient time (when fee is high) - you will pay a premium, as delaying payment is often not an option. Also you might hit transaction size limitations, and will need to send several transactions to satisfy one payment.

Holding both fair amount of big utxo and a lot of small utxo in hot wallet might mean your hot wallet balance is higher than necessary, and your risks are also higher.

If it takes a week to confirm, it's not a big deal.

But this means you are freezing this funds for a week, and need to get funds for operation from somewhere else, like your cold wallet.

Also, big sweeping transaction is more visible.

[simplexum]

You can spend unconfirmed UTXO/input and thus make both transaction could be confirmed faster (aka CPFP).

CPFP means you just paying more fee in the child tx - if you wanted to use the result of the sweep soon, you could just have added more fee at the tx creation

However, there are multiple privacy concern when you receive Bitcoin from multiple party and you don't want any party how much Bitcoin you have or with whom you made Bitcoin transaction.

If you do not conflate utxo from different wallets (or sub-wallets), there's no problem - user should have used different wallets/sub-wallets for different purposes, if this is a concern

[simplexum]

a concept of a "min balance" and will never go under it.

under normal circumstances, you're only risking 30 BTC (Say your service gets hacked). If things go catastrophically wrong (e.g. your locked down wallet machine gets hacked) you're risking the full 100 BTC.

An interesting concept. One of the imperatives that was set when we created our system was that the risks should be eliminated or mitigated to the maximum extent possible, so this was not considered. But this is quite easy to add, as an option for those who can tolerate the risk, so we might think about implementing it.

Another important requirement was that if there is a spendable balance on wallet, and a payment order comes to spend this balance, it should be sent - the developer using an API should not bother about if there enough big utxo available, or if the transaction limitation might be hit, or whatever - if there's enough funds for the order amount (and network fee, if the fee is taken from the wallet, and not from recipient)  - send it, use every utxo available, even if it means sending several txs.

[simplexum]

Say fees are high, so your algorithm is going to try avoid consolidation for as long as possible.

fee_per_full_coin should be adjusted if the median fee, averaged over a prolonged period, becomes significantly higher (or lower). And even if you lag a bit with this adjustment, your utxo set will not degrade instantly. You can monitor the recent median fee and tell if your fee_per_full_coin setting is too low or too high - you basically want it so that with median fee, you will be mostly choosing averagely small txs (assuming your utxo set is already in good shape). 2 or 3 inputs seem like a good choice, but as the size of inputs varies depending on the properties of addresses that the wallet contains, the actual setting will be specific to the concrete wallet setup.

keep a bit extra money in their wallet

70 out of 100, more than 2x of usable amount, is quite a big extra, IMO.