Shower thoughts about the growth of the utxo set (with stats inside)

[laurentmt]

FWIW, here are a few observations related to the growth of the utxo set.

EDIT: For a TL/DR, you can jump to the conclusion

The growth rate of the utxo set has increased since late 2013 / early 2014.


(please note this chart counts the cumulated number of utxos in the blockchain (different from the utxo set of bitcoin core) but this observation remains valid for the utxo set in bitcoin core (can be checked with statoshi tool)).


More interestingly, a repeated pattern has appeared since early 2014, showing steps every sunday (around 100k utxos added to the set every sunday).




This pattern is interesting because activity in the bitcoin network is low on sunday in term of number of transactions




or in term of volume




We can also observe huge peaks every sunday in the average number of utxos created per transaction


(max value is the average number of utxos created per transaction at blocks level and is the max value among all blocks created on a given day)


A few months ago, my intuition was that these peaks are "caused" by a small number of actors (bitcoin services ?) reorganizing their utxos or doing some weekly payment. A very quick manual check in the blockchain led me to think that faucets doing weekly payments may play a role here but others actors may also have a role. A more formal analysis would be welcome.

So, please, don't get me wrong. I don't call to a new witch hunt targeting some actors from the ecosystem. Even if these observations are correct, there's no hint of a malicious intent. Actually, it's quite the opposite, if we consider that these transactions occur during low activity periods in the network.

Here's my point:
If the current growth of the utxo set is caused by a small number of actors, it's possible that these actors may change a few things in their model without disrupting their business. For example, if it appears that faucets doing weekly payments are the main cause of the actual growth, may be they could replace weekly payments by monthly or quaterly payments. A payment done every 2 or 3 months may decrease the growth by one order of magnitude. It's by no means a solution to scalability issues but it may be an "easy" way to buy more time and that sounds like a good news.

My 2 satoshis

[virtualx]

This is very interesting. Any idea why the transactions rise so much on those days? 

If the current growth of the utxo set is caused by a small number of actors, it's possible that these actors may change a few things in their model without disrupting their business. For example, if it appears that faucets doing weekly payments are the main cause of the actual growth, may be they could replace weekly payments by monthly or quaterly payments. A payment done every 2 or 3 months may decrease the growth by one order of magnitude. It's by no means a solution to scalability issues but it may be an "easy" way to buy more time and that sounds like a good news.

The growth is from something else,  I don't know what but perhaps salaries or automated payments.

[laurentmt]

This is very interesting. Any idea why the transactions rise so much on those days? 

To be honest the answer is no because my tool needs some more work before allowing to identify the source of these steps.

A few months ago, I've done a quick manual check in the blockchain and I was able to spot transactions which obviously participate in the growth of the utxo set. These transactions seem associated to faucets doing weekly payments but nothing says that these faucets are the unique cause.

Actually, it's likely this growth is caused by a small number of services and not by a unique service. If it's the case, it's not easy for these services to detect that their individual usage of the blockchain might cause a problem. Especially if you think that public discussion about the growth of the utxo set is very recent.

On the positive side:
- For now, nothing proves that we face malicious intent. That might be just another good example of the tragedy of the commons.
- I'm convinced that startups like coinalytics or chainalysis might help here to get more detailed information. It's up to them to decide if they want to help the community on that matter but I think they have a financial incentive to do it, since the success of their business is linked to the success of bitcoin.
- If these steps are caused by a small number of services, I want to think that they have an incentive to improve things since their success is linked too to the success of bitcoin. It might happen that some of these actors cannot change their behavior without breaking their business and it's by no means my intent to ask them to sacrify their business for the sake of bitcoin.

It's far to give us a long term solution but, as I wrote in the OP, that might be an easy way to buy more time.

[laurentmt]

The growth is from something else,  I don't know what but perhaps salaries or automated payments.

Well, I've just done again a quick manual check for 2 sunday (in July & November 2014). Here are the results.

Disclaimer:
- I didn't check all these txs but some services seem to have a very specific signature identified by the number of utxos created.
- services identified are just the easier to spot manually because they produce a significant part of the utxos and they don't try to hide their activity.
  I guess we may find a few others services with an automated analysis.

06/07/2014: Around 50k utxos created with 28k created by 2 services

Bitcoin zebra (3txs with 2501 outputs)
https://blockchain.info/tx/afc134b433dcaeeaa69605e37aa4e2c8b328a2cd63c4f27e28ab94950eabcbad
https://blockchain.info/tx/afbc3ff8afa6c8707921b86df680333a3b84abecf6840ad0f5f1791b9c5ae4a0
https://blockchain.info/tx/2eed90a71e51e8e54683ab4b9f49f3ea1600cdbd05c81bd0f2a2b2bc033c324a

FreeBitco.in (likely around 106 txs with 200 outputs)
https://blockchain.info/tx/f64def0f342cf722c1bf45f364a379df027623f9113b4be521ed73e0a0947e69
https://blockchain.info/tx/90de40e6624bcac782e8f656ae4877388e9808eda9cc0ee17946dfb626689246
https://blockchain.info/tx/a52bdbd79c57c43b17f1946656c9dc233e5e811c628297c66e47b51683687594
https://blockchain.info/tx/e840defa14c8a44028cf032ee946c7b9a1b479feb55d6267b647ba380847593a
https://blockchain.info/tx/1afda13cf53654346ed02e41653f967b0d88fc27e002996f6cbbf4034f11968a
...

Discus fish (1 tx with 2935 outputs)
https://blockchain.info/tx/e3eef4a5ca594b1e96cdd5bae50e60b7f690ee97b060585375a4700e44a759cc


16/11/2014 : Around 114k utxos created with aound 82k created by 2 services

Bitcoin zebra (26 txs with 2501 outputs)
https://blockchain.info/tx/038ad19e17c3418cf3204c482696561b018fec804ae2a66caf617ca0fa0bcb09
https://blockchain.info/tx/48fffd7ec8f5208fc4bf80e6724f106c66eb201547159ed8731a14f1bec18d72

FreeBitco.in (likely around 84 txs with 200 utxos)
https://blockchain.info/tx/11edc3c06f927008289825eeb603f3a5be8c1c3a865d721c6f2495d5524f3b59
https://blockchain.info/tx/b29d2a48e676089c00f30d927ebe7e356d576de0f7b39e35ca7321a6b2cac078
https://blockchain.info/tx/bc5e27271d424b625c077c05e519a1f78f271a43bdbab1b12a2e3a691beb21b4
https://blockchain.info/tx/7a2b94f6d11e44dc24f0e91a57b59a77a578cd54fd8aa2993a6cfa8b6cadd625
...

Pbmining.com - 1Payday... (3 txs with around 2000 utxos)
https://blockchain.info/tx/96f57534d3fc84be9fa6e1b39f2df6a74817f866b5f9788a4a78112af906965e
https://blockchain.info/tx/ef19fee6af7bb739c1c8743cca620b6f5725f4f35fd28cda993cd5ca5c59ce72
https://blockchain.info/tx/6a3274d0eb161d878aeeec13daffd38e48bc8054741eff8fe939417681dd6435

Unknown (1 tx with 4865 utxos)
https://blockchain.info/tx/e53dcd980d36e74f4ed9c3dd7a4964915f1164613d39992de2c2523b6209af6d


EDIT: Updated 1Payday... address with associated service Pbmining.com (reported in PM by another bitcoiner)

[solex]

This is really good research, as it is clear that these types of tx are "underpriced" and there is a clear mismatch with cost in storage.
Watching the thread with interest.

[laurentmt]

This is really good research, as it is clear that these types of tx are "underpriced" and there is a clear mismatch with cost in storage.
Watching the thread with interest.

I'm far to be an expert on these matters but my feeling is the subject is hard.

For example, these services could send a lot of txs with 2 or 3 utxos instead of a single tx with so many utxos. That may raise the fees received by the miners, but on the other side, that would increase the number of tx inputs (which embed signatures).

As a result, we would have the same growth for the utxo set but with a bigger blockchain. That may be a good compromise for miners and pruned nodes. It's not so good for the services and for full nodes which don't want to prune the blockchain and don't receive fees.

[laurentmt]

Here are a few additional stats. Later this week, I'll check if I can produce new ones.

Number of utxos consumed & produced per day





While the values remain close during the week, a gap can be observed on sunday: #utxos consumed decreases while we have peaks for #utxos produced.
Nothing new here, just another way to state previous observations.


Address reuse

Address reuse is the ratio (#utxos created which have an address already seen in the blockchain) / (#utxos created)



Whatever the services, this chart may confirm the intuition that the growth of the utxo set on sunday is related to services doing periodic automated payments and reusing the same address for the payee.

[laurentmt]

A few additional charts which aggregate data for 2014. That should help to confirm/refute the first intuitions.
Comments/suggestions are welcome ! Don't hesitate to send me a PM if you're interested by the raw data used for this charts (csv files).

Disclaimer:
"#UTXOs created" is the number of utxos created in the blockchain (including op_return...) which is different from the concept of UTXO set in bitcoin core.


Weekly cycle

The two following charts confirm the existence of a weekly cycle since Q2 2014, with peaks occuring every 7 days.

Ratio (#UTXOs created / #UTXOs consumed) per day




#UTXOs added per day (i.e. #UTXOs created - #UTXOs consumed)




Sundays VS others days of the week

#UTXOs added per day of the week



#Transactions per day of the week



Fees per day of the week




Transactions fingerprints

#UTXOs added per #UTXOs created per transaction (sundays only)



Transactions with 200 utxos created per tx have added 1,635,768 utxos (45.6% of utxos added on sundays)
Transactions with 2501 utxos created per tx have added 1,112,202 utxos (31% of utxos added on sundays)

These two patterns of transactions represent around 76% of all utxos added on sundays.


#UTXOs added per #UTXOs created per transaction (all days)



Transactions with 200 utxos created per tx have added 2,120,703 utxos (around 26.96% of utxos added in 2014)
Transactions with 2501 utxos created per tx have added 1,561,468 utxos (around 19.85% of utxos added in 2014)
Transactions with 138 utxos created per tx have added 834,686 utxos (around 10.6% of utxos added in 2014)
Transactions with 2935 utxos created per tx have added 651,348 utxos (around 8.28% of utxos added in 2014)
Transactions with 101 utxos created per tx have added 555,311 utxos (around 7% of utxos added in 2014)

These five patterns of transactions represent around 72% of all utxos added in 2014.


Fees & incentives

These charts are for Solex.

Notes:
- each point stands for a day
- fees are expressed in satoshis

Fees vs #Transactions



Fees vs #UTXOs consumed



Fees vs #UTXOs created



Fees vs ratio (#UTXOs created / #UTXOs consumed)

[laurentmt]

Out of curiosity, I've tried to find which entities have sent transactions with 138 and 101 outputs.

It's likely that transactions with 138 outputs are related to a faucet called Microwallet (see 1MiCRoEgje2a8U1FthQGDAwaWabDbcBsTe)

Transactions with 101 outputs are a funny case that I'll call 'Crazy10073'.
They might be related to long peeling chains, splitting an utxo in 100 outputs of 10073 satoshis + 1 output used at the next step (example 112LJ7ZjaHwhsXWcq44tgG6KX6qmjdKrhY)
And if you try to find the origin of these coins, you'll find coins coming from...SatoshiDice.
I guess that Crazy10073 is pure spam or a kind of postmodern experiment...

[laurentmt]

I guess it's time to draw a few "conclusions" from this journey into the blockchain

A few numbers for 2014:
- 78,669,554 utxos were created
- 70,806,194 utxos were consumed
- as a direct consequence 7,863,360 utxos were added into the blockchain (about 10% of #utxos created)

Looking for patterns in the number of utxos created per transaction, we can detect 5 specific fingerprints (transactions with 101, 135, 200, 2501, 2935)

Considered together, transactions with these 5 patterns:
- are about 0.09% of #txs created in 2014
- have created 5,753,098 utxos (about 7,3% of #utxos created in 2014)
- have consumed 29,582 utxos (about 0,04% of #utxos consumed in 2014)
- have added 5,723,516 utxos into the blockchain (about 99,4% of #utxos created by themselves and about 72,8% of #utxos added in 2014)

While a more detailed analysis would be necessary to validate the point, it's likely that all these transactions are created by a small number of legit services doing recurring micropayments (faucets, mining pools, ...)

Charts of fees vs #txos and #txs seem to confirm that current model used for fees doesn't help to "regulate" the growth of the utxo set.
(Actually, nothing new here, it's straightforward to reach this conclusion if you know the model used to compute fees)


My very personal conclusions about all this stuff:

While the memes "blockchain" & "micropayments" often walk together (especially in media and discourses of some startups), it seems obvious that the current architecture of bitcoin isn't very adequate for this use case. I guess it's something all bitcoin experts already know for long but I fear the public perception is very different...

While many services (exchanges, tipping services, ...) have already took a alternative path with centralized off-chain transactions, this little journey into the blockchain also tells me that even a small number of services doing micropayments may have a "big" impact on bitcoin (see how the growth of the utxo set has come up into the discussions about the blocksize).

I like the fact that some core devs, like Jeff Garzik, are more and more vocal about the need for a decentralized Layer2 built upon bitcoin. Except if we consider that services doing micropayments are spammers (which I don't), I think we (developers) should be more supportive of Layer2 projects (payment channels, hub & spoke, lightning, ...) and that we should try to better explain to the public the importance of these projects.

My 2 satoshis.