Bitcoin Forum

Hi,

Not sure if this is the correct forum for this question, mods please move it to the correct forum if not.

I'm interested in the idea of storing arbitrary data on the bitcoin blockchain in a standardized way. I know this is not a new idea, but AFAIK there is no standardized way to do it. I'd like to find like minded people with a view to:

1. Working on a standard for storing key+value data on the blockchain which includes a facility for indicating mime types and possibly encryption, possibly other facilities too

2. Submitting the idea as a BIP

3. Making a fork of bitcoin core to add this functionality. This would entail scanning new blocks for onchain data and adding them to an index, and adding APIs for retrieving data.

That's about it. It may be a good idea to attempt this on a similar blockchain first instead (I'm thinking Litecoin). The good thing about this idea is it could be implemented prior to writing the BIP, or even without it being accepted by the community.

I'm eager to hear from anybody who is interested in this idea. I know there is a contingent of people who don't like the idea of storing data on the bitcoin blockchain. I think people are going to put data on the blockchain anyway, so we may as well come up with a standard for doing it.

Looking forward to replies!

I expect you'll find that a large portion of the community will be fairly hostile to your proposal due to the externalities generated by that sort of misuse of the system.

Transaction fees will also make it astronomically expensive, so no one is likely to want to use it in any case, even if you were to create it. To the extent that Bitcoin provides something at all of value for 'data' that isn't available much cheaper elsewhere, it would be timestamping-- which is accomplished by opentimestamps at essentially no cost to the Bitcoin system. So I think your assumption that people will anyways is not true as is demonstrated by current practices.

Separately, Bitcoin software generally goes out of its way to avoid extracting arbitrary data from the chain because it can be a vector for malware, code injection, and social engineering attacks.

I would suggest block: 230009 (Tx: 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713) as a standard.
The whitepaper.pdf
The way it's encoded is pretty smart.

since bitcoin is a payment system and its blockchain is a place to store history of these payments, storing arbitrary data in it will never make any sense even though it is possible.
how about you look into side-chains if you want to use bitcoin?

smart? i don't think so. it is just interesting as a puzzle, otherwise it is the worst possible way of doing what they did.
for starters it is practically burning 1 satoshi for no reason.
secondly it is creating 946 garbage UTXOs which are now in every node's UTXO set as a burden which they have to store and load every time.
and finally it is using a script, and for some weird reason an OP_CHECKMULTISIG one (OP_1 <push65><push65><push65>OP_3 OP_CHECKMULTISIG) which makes no sense to me at all. why multisig? why 1of3? why 65? (max standard script length is 10k, and max push is standard 520 a bare multisig could have had more and remain standard)

a much smarter way would have been not to limit themselves to standardness and to simply put the data in a single output with amount=0 which would have solved all the weirdness above. and also reconstructing the PDF wouldn't have been such a pain in the ass!
P.S. it was 2013 so no OP_RETURN, if it were more recent an even better solution would have been starting the single output with an OP_RETURN to eliminate that 1 UTXO burden too.

There are 3 existing options
1. Use OP_RETURN as much as you need.
2. Store the hash of the data with OP_RETURN and store the data elsewhere. IPFS is common option.
3. Use side-chain or off-chain which offer such feature.

With technical difficulty and high cost, i think only extremely important data (which usually hunted by government) will be stored on Bitcoin these days.

People will not store data if it's astronomically expensive even if we make it easier for them to do so. If people don't want to use it, it will not create any blockchain bloat. But I think people will want to use it.

I think storing small files with important information is an interesting and important use-case. Sometimes people want to get some data "out there" in a way that it can not be deleted or censored and is accessible to everybody. I don't expect people to put their photo albums or mp3 files on the blockchain, but small text files containing important information.

I also think associating the data with easy-to-remember keys rather than difficult-to-remember tx hashes is also an  important advancement. That way, people can memorise or scribble down these keys, which effectively gives them a mnemonic.

I call these mnemonics "bitcodes". Think of a bitcode as a cross between a hashtag and a domain name. It's like a hashtag insofar as it conveys an idea and can be prepended with a symbol such as $ and parsed, then linked to the blockchain so the underlying data can be viewed. They can be transmitted easily by word of mouth, writing, or digitally with little chance of error, and can be memorised. Bitcodes are similar to domain names insofar as they point to some other, bigger data, but are not free to "register" - you must pay for blockspace, once registered, they are owned by one person, who can update the underlying content when they wish, or make it final, so that not even they can erase or modify it. "Ownership" of a bitcode would be 100% based on cryptography - no centralised authority involved.

I think this is a powerful idea and it's going to happen anyway, and I feel it's the natural evolution from hashtags. I'm interested in hearing from anybody who'd like to discuss the technical details of how to implement such a system. I've got some ideas of my own.

If your data is important you should not upload it to a public blockchain where anyone can view it. Sure, it might be encrypted but there's always that possibility that someone with enough free time collects all 'weird' hash and cracks the message.

Mnemonic is good but I don't think making easy-to-remember keys is enough to protect you from brute-force. Especially because you publish your data on the internet. It's like putting your safe in the street where everyone can crack it up.

this is not something you can work with only based on expectations. if something is possible then people are going to do it. for instance Satoshi probably never expected someone would inject a virus signature into bitcoin blockchain but they did and caused a lot of problem for regular users who were confused why their block data files are being "cleaned" by an AntiVirus!

Also:

 - Add a custom message to the coinbase-field. This is only possible when you mine a new block so it might be a bit tricky ;)
- Use specific addresses which have a special crafted RIPEMD-160 which can be converted to readable characters. Sending funds to these addresses will be lost though since the Private Key for these addresses is unknown. This also means you will be polluting the UTXO with unspendable transactions;
-  Use OP_RETURN in a regular transaction as an extra output with a value of 0.00. This way you can add 40 bytes of custom data without polluting the UTXO-database and without creating unspendable addresses. Since the transaction itself will become bigger this way (maximum of 43 bytes) the fees for such a transaction will go up.

I think you're missing the point. I'm not trying to sell the idea of putting data on the blockchain. I'm saying - people (and organizations) are doing it anyway, so why not come up with a standard?

For example Microsoft recently announced they are building their new decentralized identity system on top of Bitcoin, meaning they will be putting data on the blockchain. What data? I don't know, maybe it's only hashes, but they are putting data on the blockchain. That's a fact.

So imagine we're back in the 1980s and we are a bunch of computer enthusiasts who have just discovered that you can save images to disk, but everybody is using different formats. Pretty confusing right? But not if we all agree on a standard like .bmp or .tiff.

My idea is nothing but to come up with a standard for storing files on the blockchain, so Alice can put a file on the blockchain with a certain filename, can give that filename to Bob and Bob can read the file without too much pain.

In short - I want to make it easier for users to do something they are already doing.

The blockchain is not intended to store files, it never has been and it never will be. You are describing a way to exchange files between Bob and Alice in a way it will be stored on every full node forever. Either the size of the blockchain would become enormous or regular transactions (the intended use case of the bitcoin blockchain) would not fit in anymore. Furthermore at current fees it would become extremely expensive to do so. In short: you are trying to find a "standard" for something that is not intended for the bitcoin blockchain and never will.

In case Alice wants to put up a reference to a file for Bob she can use the standard OP_RETURN.

I'm sure i got the point and i never said you're trying to sell your idea.

What i'm opposed is using the idea for trivial case such as transfer file between 2 users (which you mentioned), there are many ways to do it such as IPFS or Tor.
I don't mind if it's used for something really important such as activist / journalist share important information to the world.

OP, Bitcoin CASH SV store weather data on "the blockchain", try your idea there, I believe your proposal would be welcome by their community. 8)

I totally get that.

As an engineer, using the blockchain as a way to store files is overkill even if you want censorship resistance and for the file to be there forever. But tell me - what other options are out there? I'm currently looking into IPFS, but from what I hear there are some risks of your data being deleted or disappearing.

For the record, I'm not limiting myself to the Bitcoin blockchain - I think this should be an option on Litecoin, BSV, BCH, and all the Bitcoin forks - and any blockchain.

Odds are they are not building on the BTC blockchain but rather their own blockchain. Nothing new there - private blockchains are already being used to track farm produce from field to store, cities are using it to track SSL streetlight and surveillance camera installations again, from manufacture right up to where/when/who (contractor used) they are installed onto poles, etc.

He is probably talking about ION
https://github.com/decentralized-identity/ion
It's a second layer solution on top of BTC blockchain

here (https://techcommunity.microsoft.com/t5/identity-standards-blog/ion-booting-up-the-network/ba-p/1441552) is the official MS blog post on it.

It's built on Bitcoin core and IPFS and MongoDB. I think Bitcoin is mainly used as a timestamping mechanism, but there you go... the BTC chain being used to store data by one of the largest tech companies in the world.

Then Peter Todd's OpenTimestamps is for you, https://opentimestamps.org/#how-it-works

Thank Bitcoin's POW/high hashing power for making it very secure, and possible.

Data != Files

I mean, technically the distinction between a file and data (and anything digital really) is rather arbitrary ¯\_(ツ)_/¯

But yeah, storing small amounts of data / small files is not the same as storing documents or even media files. I do think Bitcoin and IPFS can complement each other rather well though and using one to verify the data of the other is IMO as close as it gets to what I assume OP had in mind. Whether that's something that should be turned into a BIP is a different question of course.

Agreed! :)

However I was mainly responding to OP who said: "My idea is nothing but to come up with a standard for storing files on the blockchain, so Alice can put a file on the blockchain with a certain filename, can give that filename to Bob and Bob can read the file without too much pain." and then saying: "See, even Microsoft does this now".  In that context: storing a small amount of data using the standard OP_RETURN is not the same as Microsoft now storing files in the bitcoin blockchain.

Data ~= Files  ;)

A file is nothing but some data with an associated key. What I'm talking about is storing data, with a key so it is easily retrieved. Files are traditionally identified by an extension such as ".jpeg" but this is quite wasteful so I propose replacing it with a number (maybe 3 or 4 byte or a VarInt). So you have:

1. Raw data
2. Key (filename)
3. Data type (1=jpg, 2=png, ...)

FYI, Bitcoin SV is already doing something like this, and they already have a system devised for splitting up large files into several OP_RETURN transactions, but I'm not sure if they have a keying mechanism. That is the icing on the cake. - but the idea could be taken further with features such as splitting up large files, versioning, and whatever else.

So, to re-phrase, my idea is to come up with a *standard* store data on the blockchain (not a new idea, already being done my MS and others), together with a key (possibly a new idea, but I won't be so arrogant as to think somebody else has not already thought of it).

And this would be a standard - it would not be limited to BTC core, it would be a standard that could be implemented on any blockchain.

that is all.

Do you have any idea what it would cost to store even a simple .jpg picture on the bitcoin blockchain? Also, what if that picture would show unwanted/illegal stuff? It would mean all full nodes in the entire world would now store this unwanted/illegal stuff.

Just my two cents: You are having an idea for a standard for something that is not wanted in any way on the bitcoin blockchain. The moment it would become a standard I would for sure turn off my node. I can't imagine any other public blockchain would ever try to accomplish this. You want to store/exhcange files use usenet/torrent or whatever.

Typically I would expect people to store small text files, not jpegs, because they cost too much. If somebody wants to put a small image of their new born baby or a recently lost loved one, I would expect it. But such an image could cost $10 or more so market dynamics will decide


There already is illegal images on the bitcoin BTC blockchain, so you're hosting them on your node. I don't expect this to become the de-facto method by which sickos distribute their sick images. It makes no sense economically and the blockchain makes it easy for them to get caught, so I don't expect this to be a problem.


Not wanted by you, maybe. Do you speak for the entire community? I'm sure many people would welcome a way to store small, critical pieces of data, safe in the knowledge that the data can never be lost and they can access it any time in the future.

This is not just about private individuals storing their favourite cat pictures though, you need to think bigger. How about law firms storing important legal information? Think bigger.


I don't give a flying fuck


Bitcoin SV are very bullish on this idea and are already doing it.


Usenet is 90's tech. Torrents are 00's tech and also outdated. Torrents are great for sharing transient data but if you want persistence torrents are useless. Your response is not just condecending, but shows up your mental laziness.

 This is not the use case I'm discussing.

Ok that's my last post in this thread. Mod, can you please lock/delete this thread ? I'm clearly wasting my time.

what happens when the encryption is inevitably broken and all those sensitive files are now out in the open?

EDIT: as the thread starter you can lock this thread yourself i believe. but an answer to my question 1st would be cool.

it is because these altcoins have no use cases as a cryptocurrency or anything else. so their owners have to come up with different "ideas" that could invent some use case and create spam so that the coin doesn't look abandoned when you look at how many transactions are being made every day.

i find it very strange that they are splitting the data into multiple OP_Return outputs instead of one since each output will have at least 10 extra repeated bytes! i already explained why it is the worst in my previous comment.

it won't. and you don't need to shut down your node, you can easily reject anything that you think is non-standard as your node is already doing it.

I don't agree that encryption will inevitably be broken. That's a straw-man argument. If you're really worried about encryption, nobody said you had to use a standard encryption algorithm like AES-256. You could use a 512-bit or 1024 bit AES if you like. If you're worried about quantum attacks you can use a quantum-resistant encryption scheme. Heck, you can use 10 different quantum algorithms and 10 classical algorithms strung together.

I never said anything about encrypting anyway - the data could be non-sensitive data. It could be hashes or timestamps. If it is encrypted data, then it's really no more risky than storing it in the cloud. In fact it's a trade-off. If you store your data in the cloud you run the risk of them losing your data or confiscating it. When it comes to encryption you have 2 choices with the cloud: they manage the keys, or you manage the keys. Of course you can do both and double-encrypt your data.

The bottom line is: you need to take care of your own keys. If they get compromised, either through a physical attack or cyber attack, the attacker will be able to access your data whether you store it on a private cloud or on a public blockchain. So the issue comes down to key management. There are several ways to do this, most notably key distribution so there are multiple keys in different physical locations.

The bottom line is: If you are a law firm, and you have sensitive data, any time you need to access that data, you need to grab the encrypted data from somewhere, decrypt it, then view it, or do whatever you need to do. Whether that somewhere is a public blockchain or private cloud is irrelevant, so long as you're using a decent encryption scheme. It's just transferring bytes which are absolutely useless to anybody not in possession of the keys and with knowledge of the encryption scheme.

but bcash -> bcashsv is a hard fork, in fact there has been at least 4 or 5 hard forks there which means the exchange/users have had to update their clients multiple times already. being centralized and having very small number of nodes, users and miners also helps a lot.

Then that's why your "BIP"/"proposal"? No would give a flying fuck about that too. The Core developers have made sure they made design-decisions to regulate the block size, to maintain decentralization, and maintain the network's security.

Not so: People have falsely claimed there were on a number of occasions but to the best of my knowledge that has always just been bamboozled media people talking about simple URLs to long dead hidden services, not "illegal images" themselves.  Also, data buried in an effectively inaccessible location that requires special tools and knowledge to access isn't the same as a file serving service practically or legally.

Doing so would be somewhere between entirely pointless and professional misconduct.

Ultimately the Bitcoin system doesn't exist to shove your data retention costs onto others-- Piling every proof-of-work quorum system in the world into one dataset doesn't scale. (https://bitcointalk.org/index.php?topic=1790.msg28917#msg28917) Existing nodes don't provide random access to the blockchain and if anything you should expect them to provide less in the future rather than more.


When you originally created this thread "I don't give a flying fuck" was about what I thought of your proposal. Yet I kept my response (https://bitcointalk.org/index.php?topic=5255218.msg54609543#msg54609543) matter of fact.  Too bad you didn't extend similar courtesy to other participants here. You were even warned that what you were asking for wasn't going to get a loving response, so it seems silly to be offended by the predictable outcome...


As you wish.