Storing data on the bitcoin blockchain

[unsigned_long_long]

Hi,

Not sure if this is the correct forum for this question, mods please move it to the correct forum if not.

I'm interested in the idea of storing arbitrary data on the bitcoin blockchain in a standardized way. I know this is not a new idea, but AFAIK there is no standardized way to do it. I'd like to find like minded people with a view to:

1. Working on a standard for storing key+value data on the blockchain which includes a facility for indicating mime types and possibly encryption, possibly other facilities too

2. Submitting the idea as a BIP

3. Making a fork of bitcoin core to add this functionality. This would entail scanning new blocks for onchain data and adding them to an index, and adding APIs for retrieving data.

That's about it. It may be a good idea to attempt this on a similar blockchain first instead (I'm thinking Litecoin). The good thing about this idea is it could be implemented prior to writing the BIP, or even without it being accepted by the community.

I'm eager to hear from anybody who is interested in this idea. I know there is a contingent of people who don't like the idea of storing data on the bitcoin blockchain. I think people are going to put data on the blockchain anyway, so we may as well come up with a standard for doing it.

Looking forward to replies!

[1715107889]

1715107889

[1715107889]

1715107889

[1715107889]

1715107889

[1715107889]

1715107889

[gmaxwell]

I expect you'll find that a large portion of the community will be fairly hostile to your proposal due to the externalities generated by that sort of misuse of the system.

Transaction fees will also make it astronomically expensive, so no one is likely to want to use it in any case, even if you were to create it. To the extent that Bitcoin provides something at all of value for 'data' that isn't available much cheaper elsewhere, it would be timestamping-- which is accomplished by opentimestamps at essentially no cost to the Bitcoin system. So I think your assumption that people will anyways is not true as is demonstrated by current practices.

Separately, Bitcoin software generally goes out of its way to avoid extracting arbitrary data from the chain because it can be a vector for malware, code injection, and social engineering attacks.

[MixMAx123]

I would suggest block: 230009 (Tx: 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713) as a standard.
The whitepaper.pdf
The way it's encoded is pretty smart.

[pooya87]

since bitcoin is a payment system and its blockchain is a place to store history of these payments, storing arbitrary data in it will never make any sense even though it is possible.
how about you look into side-chains if you want to use bitcoin?

I would suggest block: 230009 (Tx: 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713) as a standard.
The whitepaper.pdf
The way it's encoded is pretty smart.

smart? i don't think so. it is just interesting as a puzzle, otherwise it is the worst possible way of doing what they did.
for starters it is practically burning 1 satoshi for no reason.
secondly it is creating 946 garbage UTXOs which are now in every node's UTXO set as a burden which they have to store and load every time.
and finally it is using a script, and for some weird reason an OP_CHECKMULTISIG one (OP_1 <push65><push65><push65>OP_3 OP_CHECKMULTISIG) which makes no sense to me at all. why multisig? why 1of3? why 65? (max standard script length is 10k, and max push is standard 520 a bare multisig could have had more and remain standard)

a much smarter way would have been not to limit themselves to standardness and to simply put the data in a single output with amount=0 which would have solved all the weirdness above. and also reconstructing the PDF wouldn't have been such a pain in the ass!
P.S. it was 2013 so no OP_RETURN, if it were more recent an even better solution would have been starting the single output with an OP_RETURN to eliminate that 1 UTXO burden too.

[ABCbits]

There are 3 existing options
1. Use OP_RETURN as much as you need.
2. Store the hash of the data with OP_RETURN and store the data elsewhere. IPFS is common option.
3. Use side-chain or off-chain which offer such feature.

With technical difficulty and high cost, i think only extremely important data (which usually hunted by government) will be stored on Bitcoin these days.

[unsigned_long_long]

Transaction fees will also make it astronomically expensive, so no one is likely to want to use it in any case, even if you were to create it.

People will not store data if it's astronomically expensive even if we make it easier for them to do so. If people don't want to use it, it will not create any blockchain bloat. But I think people will want to use it.

I think storing small files with important information is an interesting and important use-case. Sometimes people want to get some data "out there" in a way that it can not be deleted or censored and is accessible to everybody. I don't expect people to put their photo albums or mp3 files on the blockchain, but small text files containing important information.

I also think associating the data with easy-to-remember keys rather than difficult-to-remember tx hashes is also an  important advancement. That way, people can memorise or scribble down these keys, which effectively gives them a mnemonic.

I call these mnemonics "bitcodes". Think of a bitcode as a cross between a hashtag and a domain name. It's like a hashtag insofar as it conveys an idea and can be prepended with a symbol such as $ and parsed, then linked to the blockchain so the underlying data can be viewed. They can be transmitted easily by word of mouth, writing, or digitally with little chance of error, and can be memorised. Bitcodes are similar to domain names insofar as they point to some other, bigger data, but are not free to "register" - you must pay for blockspace, once registered, they are owned by one person, who can update the underlying content when they wish, or make it final, so that not even they can erase or modify it. "Ownership" of a bitcode would be 100% based on cryptography - no centralised authority involved.

I think this is a powerful idea and it's going to happen anyway, and I feel it's the natural evolution from hashtags. I'm interested in hearing from anybody who'd like to discuss the technical details of how to implement such a system. I've got some ideas of my own.

[joniboini]

I think storing small files with important information is an interesting and important use-case. Sometimes people want to get some data "out there" in a way that it can not be deleted or censored and is accessible to everybody. I don't expect people to put their photo albums or mp3 files on the blockchain, but small text files containing important information.

I also think associating the data with easy-to-remember keys rather than difficult-to-remember tx hashes is also an  important advancement. That way, people can memorise or scribble down these keys, which effectively gives them a mnemonic.

If your data is important you should not upload it to a public blockchain where anyone can view it. Sure, it might be encrypted but there's always that possibility that someone with enough free time collects all 'weird' hash and cracks the message.

Mnemonic is good but I don't think making easy-to-remember keys is enough to protect you from brute-force. Especially because you publish your data on the internet. It's like putting your safe in the street where everyone can crack it up.

[pooya87]

I don't expect people to put their photo albums or mp3 files on the blockchain, but small text files containing important information.

this is not something you can work with only based on expectations. if something is possible then people are going to do it. for instance Satoshi probably never expected someone would inject a virus signature into bitcoin blockchain but they did and caused a lot of problem for regular users who were confused why their block data files are being "cleaned" by an AntiVirus!

[TheArchaeologist]

There are 3 existing options
1. Use OP_RETURN as much as you need.
2. Store the hash of the data with OP_RETURN and store the data elsewhere. IPFS is common option.
3. Use side-chain or off-chain which offer such feature.

With technical difficulty and high cost, i think only extremely important data (which usually hunted by government) will be stored on Bitcoin these days.

Also:

 - Add a custom message to the coinbase-field. This is only possible when you mine a new block so it might be a bit tricky
- Use specific addresses which have a special crafted RIPEMD-160 which can be converted to readable characters. Sending funds to these addresses will be lost though since the Private Key for these addresses is unknown. This also means you will be polluting the UTXO with unspendable transactions;
-  Use OP_RETURN in a regular transaction as an extra output with a value of 0.00. This way you can add 40 bytes of custom data without polluting the UTXO-database and without creating unspendable addresses. Since the transaction itself will become bigger this way (maximum of 43 bytes) the fees for such a transaction will go up.

[unsigned_long_long]

Few people with enough money to spare will put information which most likely won't be deleted/censored on blockchain.

I think you're missing the point. I'm not trying to sell the idea of putting data on the blockchain. I'm saying - people (and organizations) are doing it anyway, so why not come up with a standard?

For example Microsoft recently announced they are building their new decentralized identity system on top of Bitcoin, meaning they will be putting data on the blockchain. What data? I don't know, maybe it's only hashes, but they are putting data on the blockchain. That's a fact.

So imagine we're back in the 1980s and we are a bunch of computer enthusiasts who have just discovered that you can save images to disk, but everybody is using different formats. Pretty confusing right? But not if we all agree on a standard like .bmp or .tiff.

My idea is nothing but to come up with a standard for storing files on the blockchain, so Alice can put a file on the blockchain with a certain filename, can give that filename to Bob and Bob can read the file without too much pain.

In short - I want to make it easier for users to do something they are already doing.

[TheArchaeologist]

My idea is nothing but to come up with a standard for storing files on the blockchain, so Alice can put a file on the blockchain with a certain filename, can give that filename to Bob and Bob can read the file without too much pain.

The blockchain is not intended to store files, it never has been and it never will be. You are describing a way to exchange files between Bob and Alice in a way it will be stored on every full node forever. Either the size of the blockchain would become enormous or regular transactions (the intended use case of the bitcoin blockchain) would not fit in anymore. Furthermore at current fees it would become extremely expensive to do so. In short: you are trying to find a "standard" for something that is not intended for the bitcoin blockchain and never will.

In case Alice wants to put up a reference to a file for Bob she can use the standard OP_RETURN.

[ABCbits]

Few people with enough money to spare will put information which most likely won't be deleted/censored on blockchain.

I think you're missing the point. I'm not trying to sell the idea of putting data on the blockchain. I'm saying - people (and organizations) are doing it anyway, so why not come up with a standard?

I'm sure i got the point and i never said you're trying to sell your idea.

What i'm opposed is using the idea for trivial case such as transfer file between 2 users (which you mentioned), there are many ways to do it such as IPFS or Tor.
I don't mind if it's used for something really important such as activist / journalist share important information to the world.

[Wind_FURY]

OP, Bitcoin CASH SV store weather data on "the blockchain", try your idea there, I believe your proposal would be welcome by their community.

[unsigned_long_long]

The blockchain is not intended to store files, it never has been and it never will be. You are describing a way to exchange files between Bob and Alice in a way it will be stored on every full node forever. Either the size of the blockchain would become enormous or regular transactions (the intended use case of the bitcoin blockchain) would not fit in anymore. Furthermore at current fees it would become extremely expensive to do so. In short: you are trying to find a "standard" for something that is not intended for the bitcoin blockchain and never will.

In case Alice wants to put up a reference to a file for Bob she can use the standard OP_RETURN.

I totally get that.

As an engineer, using the blockchain as a way to store files is overkill even if you want censorship resistance and for the file to be there forever. But tell me - what other options are out there? I'm currently looking into IPFS, but from what I hear there are some risks of your data being deleted or disappearing.

For the record, I'm not limiting myself to the Bitcoin blockchain - I think this should be an option on Litecoin, BSV, BCH, and all the Bitcoin forks - and any blockchain.

[NotFuzzyWarm]

For example Microsoft recently announced they are building their new decentralized identity system on top of Bitcoin, meaning they will be putting data on the blockchain. What data? I don't know, maybe it's only hashes, but they are putting data on the blockchain. That's a fact.

Odds are they are not building on the BTC blockchain but rather their own blockchain. Nothing new there - private blockchains are already being used to track farm produce from field to store, cities are using it to track SSL streetlight and surveillance camera installations again, from manufacture right up to where/when/who (contractor used) they are installed onto poles, etc.

[stompix]

For example Microsoft recently announced they are building their new decentralized identity system on top of Bitcoin, meaning they will be putting data on the blockchain. What data? I don't know, maybe it's only hashes, but they are putting data on the blockchain. That's a fact.

Odds are they are not building on the BTC blockchain but rather their own blockchain.

He is probably talking about ION
https://github.com/decentralized-identity/ion
It's a second layer solution on top of BTC blockchain

[unsigned_long_long]

For example Microsoft recently announced they are building their new decentralized identity system on top of Bitcoin, meaning they will be putting data on the blockchain. What data? I don't know, maybe it's only hashes, but they are putting data on the blockchain. That's a fact.

Odds are they are not building on the BTC blockchain but rather their own blockchain.

He is probably talking about ION
https://github.com/decentralized-identity/ion
It's a second layer solution on top of BTC blockchain

here is the official MS blog post on it.

It's built on Bitcoin core and IPFS and MongoDB. I think Bitcoin is mainly used as a timestamping mechanism, but there you go... the BTC chain being used to store data by one of the largest tech companies in the world.

[Wind_FURY]

For example Microsoft recently announced they are building their new decentralized identity system on top of Bitcoin, meaning they will be putting data on the blockchain. What data? I don't know, maybe it's only hashes, but they are putting data on the blockchain. That's a fact.

Odds are they are not building on the BTC blockchain but rather their own blockchain.

He is probably talking about ION
https://github.com/decentralized-identity/ion
It's a second layer solution on top of BTC blockchain

here is the official MS blog post on it.

It's built on Bitcoin core and IPFS and MongoDB. I think Bitcoin is mainly used as a timestamping mechanism, but there you go... the BTC chain being used to store data by one of the largest tech companies in the world.

Then Peter Todd's OpenTimestamps is for you, https://opentimestamps.org/#how-it-works

Thank Bitcoin's POW/high hashing power for making it very secure, and possible.

[TheArchaeologist]

It's built on Bitcoin core and IPFS and MongoDB. I think Bitcoin is mainly used as a timestamping mechanism, but there you go... the BTC chain being used to store data by one of the largest tech companies in the world.

Data != Files

[HeRetiK]

It's built on Bitcoin core and IPFS and MongoDB. I think Bitcoin is mainly used as a timestamping mechanism, but there you go... the BTC chain being used to store data by one of the largest tech companies in the world.

Data != Files

I mean, technically the distinction between a file and data (and anything digital really) is rather arbitrary ¯\_(ツ)_/¯

But yeah, storing small amounts of data / small files is not the same as storing documents or even media files. I do think Bitcoin and IPFS can complement each other rather well though and using one to verify the data of the other is IMO as close as it gets to what I assume OP had in mind. Whether that's something that should be turned into a BIP is a different question of course.