Download and Upload big file to blockchain program

[Maktubeer]

Hello...

I'm create cross platform Qt Desktop program... that upload and download big (unlimited size) file to cryptocurrency blockchain (bitcoin and altcoin like litecoin, dogecoin e.t.c).
you can find c++ source in https://github.com/Maketobeer/QtMonolite

when i test it on litecoin testnet i upload *.mp4 file (hd, 1:34 min ~26 MB) in txid: 657338821f809c030e2053b81af683c99dab05f8e4c272efcb513027ad31cd15 and it take >24 hours because i set pause between transaction (if not i have too-long-mempool-chain error)....

now i create my own cryptocurrency(file hosting) and i need disable or reset too-long-mempool-chain protection in source and make it more faster... where i can find this parameters? and why this parameter set in bitcoin?

thanks!

[1715538492]

1715538492

[1715538492]

1715538492

[joniboini]

Why not use IPFS or torrent?

Blockchain is not suitable to store a huge file. The cost outweighs the benefit. That being said, you might be interested in Filecoin.

[Maktubeer]

Why not use IPFS or torrent?

ipfs and torrent can't give guarantees saved data... if you load file in blockchain file can't be deleted and you can find it anytime and anyway.

Blockchain is not suitable to store a huge file. The cost outweighs the benefit.

Yes, it true if we took about bitcoin... but if using existing coin with low fee, like dogecoin cost can be low..
it is more environmentally friendly because miner need buy ssd instead ASIC and it spends less electricity.

That being said, you might be interested in Filecoin.

Thank you. But i find it troubles because it seems like ipfs and data can be removed after time. or I'm wrong?
however i think cryptocurrency like dogecoin and cash can give cheap recording space... but need less time between blocks

[HeRetiK]

Blockchain is not suitable to store a huge file. The cost outweighs the benefit.

Yes, it true if we took about bitcoin... but if using existing coin with low fee, like dogecoin cost can be low..
it is more environmentally friendly because miner need buy ssd instead ASIC and it spends less electricity.

Be aware that the moment you start storing huge files on low-fee cryptocurrencies you'll see either (a) the fee increasing significantly, due to lack of available block-space or (b) the blockchain facing serious scalability issues due to each node having to validate and propagate huge amounts of data. It sounds like a fun project nonetheless though.

now i create my own cryptocurrency(file hosting) and i need disable or reset too-long-mempool-chain protection in source and make it more faster... where i can find this parameters? and why this parameter set in bitcoin?

I think this thread should answer you which settings to add to the config file:
https://bitcointalk.org/index.php?topic=3791899.0

Apart from that you should be able to find the default constants being set here (starting at line 54):
https://github.com/bitcoin/bitcoin/blob/b688b859dbb2b5af2e9d19cae9dce3e3e14bd2c1/src/validation.h

[Dabs]

Torrents will survive as long as there are seeders. Files stored on IPFS, as long as at least one node is keeping it. There are entire movies still available that were uploaded 3 to 5 years ago.

If you need files accessible for a shorter time period and smaller file sizes, there are plenty of file hosting sites that will keep your file for 30 days or 3 months or something like that.

I use a few of those when travelling internationally and don't want any data stored on the laptop or computer as devices can be searched at airports or worse copied. You could also just encrypt the files with a secure and long password.

If you really want to store large files on a blockchain, you should try BCH and BSV, they have really large blocks.

[joniboini]

Yes, it true if we took about bitcoin... but if using existing coin with low fee, like dogecoin cost can be low..
it is more environmentally friendly because miner need buy ssd instead ASIC and it spends less electricity.

Won't this give another problem? Those 'low-cost coins' hash is not as high as Bitcoin, so if there's an attack, your file might be affected too. Not to mention storing huge data is not an easy task, and considering the block space, it might take weeks to save a single file.

[Maktubeer]

I think this thread should answer you which settings to add to the config file:
https://bitcointalk.org/index.php?topic=3791899.0

Apart from that you should be able to find the default constants being set here (starting at line 54):
https://github.com/bitcoin/bitcoin/blob/b688b859dbb2b5af2e9d19cae9dce3e3e14bd2c1/src/validation.h

thank you very much!

i try modified *.conf file with

limitancestorsize=99999999
limitdescendantsize=99999999

limitancestorcount=10000
imitdescedantcount=10000

maxmempool=9999999999

walletrejectlongchains=0

but this did not solve the problem. i try edit validation.h file and recompile... i write later

Won't this give another problem? Those 'low-cost coins' hash is not as high as Bitcoin, so if there's an attack, your file might be affected too.

Maybe. 51% attack might be easier in this case. but i think in bitcoin and litcoin difficulty is redundant and miner can transfer part of the cost from creating 00000000... in hash to file storage without risk.
Of course if set max_block_size very bigger we will have problem with low hash rate. Just need find a balance(compromise) between security and system bandwidth.

Not to mention storing huge data is not an easy task, and considering the block space, it might take weeks to save a single file.

but bitcoin blockchain size > 120 Gb and it not problem for miners. if blockchain size will be > 2 TB miners can use ZFS + ssd raid + Broadband network.
Users don't need have full node. they can use thin client for download/upload files.. or not?

[NeuroticFish]

now i create my own cryptocurrency(file hosting) and i need disable or reset too-long-mempool-chain protection in source and make it more faster... where i can find this parameters? and why this parameter set in bitcoin?

Keep in mind that you'll need to think on some good incentive for the full nodes, else nobody would store extremely big blockchain.
You may also think on some sort of sharding for your blockchain to keep the sizes at a reasonable level, just then you have to fine tune it to avoid data loss when nodes leave the network.

Maybe you discuss with the guys from DxBox project, they also store files in the blockchain https://bitcointalk.org/index.php?topic=5189356

[nc50lc]

Don't you think POS (Proof of Stake) is a better solution than POW for this kind of projects?
Since your coin is all about storing files (big ones), then the "mining" should be done by those who's willing to store the whole blockchain.

Plus the coin will eventually become centralized to limited number of nodes in the future (for obvious reasons), so it's best to create it as a POS.
Unless there's a limit on how much a user can store (since you've mentioned that it can store unlimited data),
the blockchain will grow from 1GB to 10TB in just weeks/a month, and no one will be able to run a node unless he owns a data center.

[LoyceV]

ipfs and torrent can't give guarantees saved data... if you load file in blockchain file can't be deleted and you can find it anytime and anyway.

This is true, unless:

now i create my own cryptocurrency(file hosting)

A blockchain is only as "forever" as the people who use it. Bitcoin's blockchain has been running stable for 10 years thanks to a financial reward for miners.

If you want to prove a certain file existed on a certain date, all you have to do is store a sha256 hash on the blockchain, and keep the file itself on several mirrors. Mirroring data has worked very well for decades, and is much easier to use than getting a file that's scattered in tiny pieces on a blockchain.

[kzv]

If you want to prove a certain file existed on a certain date, all you have to do is store a sha256 hash on the blockchain, and keep the file itself on several mirrors.

Hashing it that way is only may by a proof that file not changed. But saving data in blockchain is a guarantee that the data will never lost. These are slightly different things.
How many mirrors are needed to reliably protect data from loss?
Where and how should information about these mirrors be stored? May be on several other mirors?

Why such difficulties if there is a method of reliable storage in just one place?

[LoyceV]

Why such difficulties if there is a method of reliable storage in just one place?

I think we'll have to disagree on which method is more reliable:
On the one hand, there's mirrors that have stored huge amounts of data for decades.
On the other hand, there's a new blockchain without any proven record of being able to store even moderate amounts of data.
Needless to say, I pick the first one

[kzv]

On the other hand, there's a new blockchain without any proven record of being able to store even moderate amounts of data.
Needless to say, I pick the first one

Anyone can store data in Bitcoin blockchain. It will be expensive but as reliable as possible. This is for most critical data.
If data is not too critical, anyone can store data in Litecoin, Dogecoin or other compatible forks.

Custom fork is excessive for this task IMHO

[BrewMaster]

in my opinion you shouldn't think about the cost (fees) or size of blockchain and scalability, and things like that. the only argument that matters is the purpose of the blockchain you are trying to insert the file into.

the "bitcoin" blockchain is a place to only store [money] transactions not files. the fact that it offers a special feature with an OP code shouldn't lead you to attempt doing that.

now i create my own cryptocurrency(file hosting)

the question you should be thinking about is, why do you need an immutable distributed ledger to store files? and what would be the advantage of that.

for example the current existing systems have the advantage of being able to remove a file from existence. something that is not possible in an immutable ledger. so what is your plan for when (note that it is not a matter of if only when) someone uploads a malware to this ledger? or worse, things that are illegal?

[Dabs]

So, I go on the pirate bay, click on movies, and find one that was first uploaded in 2012, it still has 600 seeders today, 7 years later. They have no financial incentive to seed it, it's just a popular movie, ... about 1.5 gigs only.

So, what do you need stored on a blockchain that can be stored anywhere else anyway? At one time (don't know if true today) Blizzard let you download their latest games or updates using some sort of torrent or peer to peer tech, that included Diablo, Warcraft and Starcraft.

If you need a relatively small file (less than 1 GB would be small enough) to last longer than a year, you can pay for a few services that will keep your file backed up or available for download.

Otherwise, you can look up filecoin, storj as others have mentioned or a few of those other storage coins. As well as trying to save larger files on BSV as they increased their block sizes.

There are or were hybrid attempts where the files are stored on IPFS and a blockchain was used to index certain files, keeping only a hash, or the hash of a hash or something small. They were called video coins or something similar.

[kzv]

If you need a relatively small file (less than 1 GB would be small enough) to last longer than a year, you can pay for a few services that will keep your file backed up or available for download.

Not all people live in countries where there is freedom of speech.
Imagine that there are countries in which Internet access is strictly regulated.
A decentralized repository of information will be a real breath of fresh air for people from these countries.
Cloud storages of commercial companies are forced to cooperate with corrupt authoritarian leaders in order to be able to conduct business in their countries.

in my opinion you shouldn't think about the cost (fees) or size of blockchain and scalability, and things like that.

in my opinion no one has the right to tell others what and how they should think

the "bitcoin" blockchain is a place to only store [money] transactions not files.

No way!
This is your opinion, but not a mandatory rule.
Anyone can manage their own bitcoins whatever they want. So if someone want to store files in Bitcoin blockchain then he can pay fee and do that. No one should try to dictate terms here!

[ABCbits]

Not to mention storing huge data is not an easy task, and considering the block space, it might take weeks to save a single file.

but bitcoin blockchain size > 120 Gb and it not problem for miners. if blockchain size will be > 2 TB miners can use ZFS + ssd raid + Broadband network.
Users don't need have full node. they can use thin client for download/upload files.. or not?

That's because miner don't need to run full node to mine Bitcoin, only pool or miners who use P2Pool who need to run full nodes.

Besides, while most user won't run full nodes, most Bitcoiner agree that running full nodes should be cheap.

the "bitcoin" blockchain is a place to only store [money] transactions not files. the fact that it offers a special feature with an OP code shouldn't lead you to attempt doing that.

As long as the consensus allows it, people will try to do it. Even without OP_RETURN, there are many attempt to store files on Bitcoin's blockchain.
See http://www.righto.com/2014/02/ascii-bernanke-wikileaks-photographs.html

[Maktubeer]

Apart from that you should be able to find the default constants being set here (starting at line 54):
https://github.com/bitcoin/bitcoin/blob/b688b859dbb2b5af2e9d19cae9dce3e3e14bd2c1/src/validation.h

its work! i edit: DEFAULT_ANCESTOR_LIMIT  DEFAULT_ANCESTOR_SIZE_LIMIT DEFAULT_DESCENDANT_LIMIT  DEFAULT_DESCENDANT_SIZE_LIMIT and now i can make up loading speed... Thank!!!

Don't you think POS (Proof of Stake) is a better solution than POW for this kind of projects?

I think about it but i afraid create super-centralized system because if will not competition miner set censure data... I think we can sacrifice the stability of the system due to greater decentralization... But you can create analog with POS if you want.. the more options we create, the sooner we will find the best.

A blockchain is only as "forever" as the people who use it.

I understand it. if people want pay for "forever" options, this is what they need. if not they can use torrent or ipfs or youtube or ..... e.t.c

Custom fork is excessive for this task IMHO

Yes. i just want find a optimal parameters allowing to store max number bit without big problem for miners...

the "bitcoin" blockchain is a place to only store [money] transactions not files. the fact that it offers a special feature with an OP code shouldn't lead you to attempt doing that.

but firs block in bitcoin have text message (array of bits). what does that mean in you opinion? may by we should use bitcoin like twitter? why not?

That's because miner don't need to run full node to mine Bitcoin, only pool or miners who use P2Pool who need to run full nodes.

I took about solo miners... i think it not be great problem because i can buy for example this EC2 it's not very expensive especially if i will have explorer with advertising or something like this

[nc50lc]

Don't you think POS (Proof of Stake) is a better solution than POW for this kind of projects?

I think about it but i afraid create super-centralized system because if will not competition miner set censure data... I think we can sacrifice the stability of the system due to greater decentralization... But you can create analog with POS if you want.. the more options we create, the sooner we will find the best.

If the problem is centralization; then whether it's POW or POS, in the end the number of full nodes will be limited to a number or users due to an inevitably huge blockchain size.

But for POS, the incentive to run a full node works "natively" (like I explained above) and I'm sure that you'll likely encounter problems with POW with a file-sharing blockchain.
It's just a suggestion, you can try it anytime later after some tests with your POW coin.

[BrewMaster]

...

of course you can't prevent people from using a "feature" that bitcoin has. but that doesn't make uploading an arbitrary file to bitcoin blockchain less of an abuse. and this is not just my opinion! most others are sharing the same views.
https://en.bitcoin.it/wiki/OP_RETURN

Many members of the Bitcoin community believe that use of OP_RETURN is irresponsible in part because Bitcoin was intended to provide a record for financial transactions, not a record for arbitrary data.

since i think continuing to discuss an ancient subject is pointless i redirect you to here: https://github.com/bitcoin/bitcoin/pull/3737