Bitcoin block data (1013 GB): inputs, outputs and transactions

[LoyceV]

Background
I've used Blockchair Database Dumps for several projects, but when I need it I don't want to wait weeks months to download all files. So I created a mirror for Bitcoin data

The data
See blockdata.loyce.club
blockdata.loyce.club/inputs/ (714 GB)
blockdata.loyce.club/outputs/ (221 GB)
blockdata.loyce.club/transactions/ (79 GB)
block_hash_version_versionHex_merkleroot_time_mediantime_nonce_bits_difficulty_ chainwork_nTx_strippedsize_size_weight.tsv.gz (94 MB)

Updates
New files are added daily.

Samples
inputs:

Code:

block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable spending_block_id spending_transaction_hash spending_index spending_time spending_value_usd spending_sequence spending_signature_hex spending_witness lifespan cdd
141719 3c86cc88726f38d2f1e861850b143fe8d710e171a91291628ffe6418d8b60190 1 2011-08-20 06:12:23 990322200 112.8967 1L2wqJ4z4pjXpqP4EFudnn7TKE4sjksQQj pubkeyhash 76a914d0c893ee2fa4278a2542691e2f569247d833805988ac 0 -1 152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 0 2011-11-11 00:05:55 28.4222 4294967295 4830450220015c796fc6ddf0040c70115539d476426f5c69e1537bb8b7da90d4d671c564d302210094fc145108b617352db33880118a69c059055538af7472ae60a5f4b37d558a060141048396aebc72f8a7b26ea0f676c7c8b9379b0a93b487058787f78b2d256d2a4a6e37d7df1ac8314ee09c1f9a57d48f8c07fd1e4e3056bb17c0075ec703599cada9 7149212 819.4471476975
152763 5733d4a453dcea2690bdc8e8866f5788adadca82ba4639a3dada3d26f738ae9a 0 2011-11-10 23:53:08 1167280 0.035 1JFVo6cGL3cpyXZctC5Tw3HHvmAzuwkjSj pubkeyhash 76a914bd37ff9aa142d5ed87b1f9385c66b856b6c60e6088ac 0 -1 152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 1 2011-11-11 00:05:55 0.0335 4294967295 493046022100ba9fa562476163f5eaac9a6bbee123a0c3f1421377c35208af4b9606816603d1022100c1a797af43d8297f08011248c16f01ee9e95d03ff5d527c2ace2adb545eb88de0141047d949c30d6ffffe8e5ecd2ee774745bda2563e4842b7a9836bd0be8461786a2cceb1d844b0789afde927eace479955f9c27ff98e0e5afe71d4dde54e621aaa1f 767 0.0001036231
141326 5c51c88388d3452c0da455ad03ee50ca121ab4812d8790402527df74cf327db1 0 2011-08-17 11:03:42 700000000 77.35 17AqNueHmeFbk6rxnpvQM3QGFJM9oAs1XY pubkeyhash 76a91443ace1cda1b0aede06ca68bc0f60f4c11ecbfdc288ac 0 -1 152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 0 2011-11-11 00:05:55 20.09 4294967295 493046022100b848925df3d18baa10f26b169bd917927c0787aa5719d3e2811176ec66e8297f022100abf4688ed1b744fd3d64e2263137feb2b057ff93d35d8e8e8ddb13fee49c40780141049042d124b779c07a9c825fd572a5002c7a50efc1fea451ad390308196f75ea7b0747c86dbfa369400b0b8d7c1965999d3fe0ad2a4f87afd96c90a5d3ed2c5ecb 7390933 598.8024421296
152754 6b5cd836c52d17a1390e1a9533303941ad23e497f0b508922f1dc55c00d293f0 0 2011-11-10 21:46:08 59000000 1.77 1KGBGWUVbVpvW6Sw7GBVsjS2NS8uWbMsgM pubkeyhash 76a914c85113063d2c642f0566d605572581075dd7bd7188ac 0 -1 152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 1 2011-11-11 00:05:55 1.6933 4294967295 483045022100a4a8ce69ccbee08899a0c5693c1c43ddac5e4c187093fd7bd1efdcea70a92c3502200fe64020f94229976febf8427d4b1510187f500f95a004a4a7be498b0439c565014104175747a43f4f51bcb9f1753398157bf5b02cd9b239ea11b544d95b3f7c81881c46c0a86b57e445f0e803c630dc1f51e7b6f7d59f36b408509cd1b3cdda918625 8387 0.057272338
145520 f49804f14a44a7a478a870635662df8b6b089317c2e878b701c6ab4c1955cf2d 1 2011-09-15 22:35:54 600000000 33.3 1HxmXQjcU51CGuQuugjFBL4Aa2VvroqpsF pubkeyhash 76a914ba0e0eb50dce94d5979626f15745ab12e89369eb88ac 0 -1 152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 0 2011-11-11 00:05:55 17.22 4294967295 483045022038b72c52176158ddd3de61eb7b86f905e8983a6c2c7fce35426cba359420649f022100a1fa8f55a281bc2f8440d561814a59be10b10193c2243c949de9574bed992a85014104d2777e3eb09b35a223ec2a5a0b0254282741046976f8765ab07bbee3715c46b8572e7b3143f0de5c615b3de6e24cd491bb52ec84bac50658bcc879a3d8e86bd5 4843801 336.3750694444
142760 1bcd0fa9ce0b1953364cb52015729e913c91f77774d43aa431667c32ae005e03 1 2011-08-27 05:29:51 400000000 34.6 1J5Z4mFHXQbLqehEkzcoa2Qgw2SGWyn6dw pubkeyhash 76a914bb56918ea3dc5b96343ac5b67843a09a1870c4a388ac 0 -1 152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 0 2011-11-11 00:05:55 11.48 4294967295 4830450221009877ac9a00d2182878aa86555e1675f9a939e4cbff68d98edb9f89698d41f4ae02201832f05f5dd2d26e67a7444c85959d53fa23bd761a9caa6e6d4c556f61a05f3b0141046f13594b5b3275d5acc8fc97ba642eda76595e1e891227560ddcd617cc12a7b3cff8081ea4f5c4625f12f20690c6df2594480822162e5fd7dc0606ff6d5747c0 6546964 303.1001851852
152758 4b28cbbfb472316b3ca32772cc2a1d05a87a37db91d951aab72b09a4d3811816 0 2011-11-10 22:58:57 761730212732 22851.9062 141utQD6sbepcubqvm7Xk8hWS71mojJg1Y pubkeyhash 76a91421148647f9b6aa60ab8161fa02cf8600e46b793e88ac 0 -1 152764 d81a2bad4b0dcec1e7fc79adb9143c85ab35c9347aa4b2c5660f0e1432c8e3dd 0 2011-11-11 00:05:55 21861.6562 4294967295 48304502207712beff0dc14506dd154a465f1560522e4577d96b4c16915c134cef4eeeed66022100c69776719e12208d0a33ff71f45bfe0d323ca3a6a7b7decb5ba61647fde41405014104b0807a7e5ab5d740f45c6f95421cfb8db0c5f1548f2bc0d10a15074a2e3691888cf90258257e6274c9a9e6033dfb001709035a52d0a7679c50a5276ade272fe2 4018 354.239814208
152754 b119c44a9c9bee8dec87492d5ce0d96a9784de1e14b26f83eaacbfccecafaaf5 0 2011-11-10 21:46:08 89089420 2.6727 18j82gzKFWJJWEiXx1RyGWUY69rYxCPTxr pubkeyhash 76a91454c037efc87d749eb4d30234c2e0dd73717ee7a588ac 0 -1 152764 d81a2bad4b0dcec1e7fc79adb9143c85ab35c9347aa4b2c5660f0e1432c8e3dd 1 2011-11-11 00:05:55 2.5569 4294967295 48304502206a86cbe79575735c709e303cab2380b6344c702b3b6d47ecb06be0f6ec840f78022100bd9c0510780afd4f14ea540eadf30f84e1897f623da628593ebd9cb61a88504601410490123b018833a24b864517617762eaef3cc80739a901120ec97f4ea02646debfd0b963bdcfb4a378e9ae1820a30cf2322e8d4ee172e76ea567b326897d8b7e93 8387 0.0864806673
152731 575b6a2694542ddc66e14d1c827aaa8924bdc95636869c94f09e70520d8ad4f7 1 2011-11-10 18:00:06 772000000 23.16 1QKaXk2hUDv61aRy5uQViq1Nr41CcttWCU pubkeyhash 76a914ffce5eab41b4213ebe134108331071337984959588ac 0 -1 152764 3815ece30a32cd879bf11fd4a4f6644695b63cf1521e1a1b1f7a9908bf0e8b54 0 2011-11-11 00:05:55 22.1564 4294967295 47304402203ee62439de32dd165c013302159772130792428135a9b4733e32bafafc32ca91022044c736b7883ed1b8c0af94c8393070effd75ee0fa8e62b61887dd99a29fe9336014104d9cd7d986ce7cbf9056ad41cc3bc4f56b6f93fb3a3eb07be16dffc80e521849631f69c8cf5e886682808d35059e27e4eca77fde73b8ebbcc0e33240e09a467eb 21949 1.9611837963

outputs:

Code:

block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable
152764 498ec7f88857d23bd19370252da439876960dad296a64534987ad54081a9cc39 0 2011-11-11 00:05:55 5001400010 143.5402 17aA19GvhzMHsq8xPwSXAPZutyr6kuzLEB pubkey 4104a8dd3f118a6122c2f8c6be3261670f7af76568cac8ff0ed95e5ef63238018e69b223dbdef9f9d5e94a053ff1afc390e230844c0b71f3648405807cd668979958ac 1 1
152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 0 2011-11-11 00:05:55 1199480 0.0344 1KPxwAbFVoDimPrVECF2zgiyfX9jGW9TCy pubkeyhash 76a914c9ca1b452087cdc6b89754c1090928d7a67ef23988ac 0 -1
152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 1 2011-11-11 00:05:55 990290000 28.4213 1AUzdKPJtPyFgrg88nK6fBH7jKkgPyphUj pubkeyhash 76a91468047787a9dce43acf61af61eb1867a9cb0724b688ac 0 -1
152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 0 2011-11-11 00:05:55 9940000 0.2853 1LVcc51n4Cdd8oKLc8GWEpu2wZzfg82Bpb pubkeyhash 76a914d5d3c21182c74b3c306e9ca9746b40d29971223388ac 0 -1
152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 1 2011-11-11 00:05:55 749060000 21.498 1ByQHoyjyMNn5NznRbSHwYcRf7DHXNL3ih pubkeyhash 76a914785bfe399c2cca278e45d76e34317dfa5941fe9f88ac 0 -1
152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 0 2011-11-11 00:05:55 74916000 2.1501 1JwtyTt2w7xZJgrgPg4rCyaV7DKEa4n5uH pubkeyhash 76a914c4dbfc4dac46dd1ca2cab8af4d431bd422a6d7d688ac 0 -1
152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 1 2011-11-11 00:05:55 525084000 15.0699 14rKLXhpX6Sukznqm5cykcgZMt4qZXamQB pubkeyhash 76a9142a3c10f240c9266f43c110242cbe877868f12ea088ac 0 -1
152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 0 2011-11-11 00:05:55 2824000 0.081 1AVntqwcaMtYVqYuH9f8zXYM1Gnz8QHVWe pubkeyhash 76a914682b16d8f9bad9fcb6993950233fac67f75c55c588ac 0 -1
152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 1 2011-11-11 00:05:55 397176000 11.399 1J9xrxAKgaRR5ddYtot559r7yDwQiS9dk1 pubkeyhash 76a914bc2c18bc245d5a4bc64567bca525741c0a8069d188ac 0 -1

transactions:

Code:

block_id hash time size weight version lock_time is_coinbase has_witness input_count output_count input_total input_total_usd output_total output_total_usd fee fee_usd fee_per_kb fee_per_kb_usd fee_per_kwu fee_per_kwu_usd cdd_total
152764 498ec7f88857d23bd19370252da439876960dad296a64534987ad54081a9cc39 2011-11-11 00:05:55 135 540 1 0 1 0 1 1 0 0 5001400010 143.5402 0 0 0 0 0 0 0
152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 2011-11-11 00:05:55 439 1756 1 0 0 0 2 2 991489480 28.4557 991489480 28.4557 0 0 0 0 0 0 819.4471476975
152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 2011-11-11 00:05:55 439 1756 1 0 0 0 2 2 759000000 21.7833 759000000 21.7833 0 0 0 0 0 0 598.80244212963
152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 2011-11-11 00:05:55 258 1032 1 0 0 0 1 2 600000000 17.22 600000000 17.22 0 0 0 0 0 0 336.37506944444
152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 2011-11-11 00:05:55 258 1032 1 0 0 0 1 2 400000000 11.48 400000000 11.48 0 0 0 0 0 0 303.10018518519
152764 d81a2bad4b0dcec1e7fc79adb9143c85ab35c9347aa4b2c5660f0e1432c8e3dd 2011-11-11 00:05:55 438 1752 1 0 0 0 2 2 761819302152 21864.2129 761819302152 21864.2129 0 0 0 0 0 0 0
152764 3815ece30a32cd879bf11fd4a4f6644695b63cf1521e1a1b1f7a9908bf0e8b54 2011-11-11 00:05:55 436 1744 1 0 0 0 2 2 10771000000 309.1277 10771000000 309.1277 0 0 0 0 0 0 140.63292604167
152764 09a4a4d5706d7123c13a8128040a50e00a79d4d369b3618f7b6140fe60e9c89c 2011-11-11 00:05:55 1519 6076 1 0 0 0 8 2 78001010386 2238.6289 78001010386 2238.6289 0 0 0 0 0 0 245.16466314468
152764 1b80bdad1902800146ef1a48665c4f13e409442feda0f9f442627aa6c3479297 2011-11-11 00:05:55 257 1028 1 0 0 0 1 2 43999811790 1262.7946 43999811780 1262.7946 10 0 38.9105 0 9.72763 0 0

block_hash_version_versionHex_merkleroot_time_mediantime_nonce_bits_difficulty_ chainwork_nTx_strippedsize_size_weight.tsv.gz:

Code:

block hash version versionHex merkleroot time mediantime nonce bits difficulty chainwork nTx strippedsize size weight
0 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f 1 00000001 4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b 1231006505 1231006505 2083236893 1d00ffff 1 0000000000000000000000000000000000000000000000000000000100010001 1 285 285 1140
1 00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 1 00000001 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098 1231469665 1231469665 2573394689 1d00ffff 1 0000000000000000000000000000000000000000000000000000000200020002 1 215 215 860
2 000000006a625f06636b8bb6ac7b960a8d03705d1ace08b1a19da3fdcc99ddbd 1 00000001 9b0fc92260312ce44e74ef369f5c66bbb85848f2eddd5a7a1cde251e54ccfdd5 1231469744 1231469665 1639830024 1d00ffff 1 0000000000000000000000000000000000000000000000000000000300030003 1 215 215 860
3 0000000082b5015589a3fdf2d4baff403e6f0be035a5d9742c1cae6295464449 1 00000001 999e1c837c76a1b7fbb7e57baf87b309960f5ffefbf2a9b95dd890602272f644 1231470173 1231469744 1844305925 1d00ffff 1 0000000000000000000000000000000000000000000000000000000400040004 1 215 215 860
4 000000004ebadb55ee9096c9a2f8880e09da59c0d68b1c228da88e48844a1485 1 00000001 df2b060fa2e5e9c8ed5eaf6a45c13753ec8c63282b2688322eba40cd98ea067a 1231470988 1231469744 2850094635 1d00ffff 1 0000000000000000000000000000000000000000000000000000000500050005 1 215 215 860
......
757434 000000000000000000029d11a2ff7d9446a7de7e575e50a79b0c83de37783f0b 805298176 2fffe000 c418d5a0e6322ed776887941f110cbf8b7a44cd2831b369413d193a76c3dfdf4 1665098543 1665096510 73455270 1708f9ae 31360548173144.85 000000000000000000000000000000000000000036907e07c3f7165e0e06d7a8 1464 386366 699282 1858380
757435 0000000000000000000136d2e367c81f36bd4c1e320231ed5d501b4e654151a1 1073733636 3fffe004 1fa72363a7ad010bcf30e56ce33ba558f23526d00117ab0793503eac2424418f 1665099208 1665097058 861948933 1708f9ae 31360548173144.85 000000000000000000000000000000000000000036909a8d92d25a922c67bb40 1680 529683 957930 2546979
757436 000000000000000000049c9f9176c2544b2d429d9ddc4d5461aeaa379fafcf8d 671080448 27ffe000 cd28bf0a6fca605d917ce3fee5dc0d97bcd44eeef8897e61b4dfe840d183cb8e 1665100292 1665097881 2098456665 1708f9ae 31360548173144.85 00000000000000000000000000000000000000003690b71361ad9ec64ac89ed8 2142 757447 1721126 3993467
757437 00000000000000000007d55451ffad86e079bf9df2db6314fcfa8eed609df8eb 541065216 20400000 091b86fb517f368ee8b1c4060b3908e5ff185db863aae9915c1c235584e7e7a3 1665100530 1665097963 426079596 1708f9ae 31360548173144.85 00000000000000000000000000000000000000003690d3993088e2fa69298270 1133 342043 658547 1684676

Bandwidth
This server is allowed unlimited bandwidth, at 250 Mbit/s. I choose this instead of 10 Gbit limited at 32 TB/month.

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

No spam please.
Self-moderated against spam. Discussion and questions are welcome.

Related topics
Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used
[~500 GB] Bitcoin block data: inputs, outputs and transactions
[800 GB] Ethereum data (see next post)

[LoyceV]

Ethereum data

I've added ethereumdata.loyce.club Abandoned! I'm out of disk space. I have a local backup, if you really need this: ask!
ethereumdata.loyce.club/calls/ (345 GB)
ethereumdata.loyce.club/erc-20_transactions/ (378 GB)
ethereumdata.loyce.club/transactions/ (71 GB)

All funded Ethereum addresses: ethereumdata.loyce.club/blockchair_ethereum_addresses_latest.tsv.gz (5.1 GB).  Blockchair doesn't updated this file regularly, so I don't do automatic updates.

Disclaimer
I don't like Ethereum, which is a centralized shitcoin that abandoned it's one unique selling point ("code is law") the moment it was convenient for the creator. Don't waste your money on it!
And since it's a shitcoin, I don't think it deserves it's own topic.

Missing files
Those 2 files are missing from my mirror because they're corrupted:
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20200113.tsv.gz
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20211110.tsv.gz
If anyone has the tools to rebuild those 2 files, I'd love to add them!

[LoyceV]

October 7, 2022: I've consolidated a few posts here, so that I can remove Vod's post (sorry Vod) and have the second post in this topic for myself.

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

Spam discussion/question.    The page mentions 10 kb/s - is there any cost or requirement for the 100?

The page mentions 10 kb/s - is there any cost or requirement for the 100?

At some point they increased the speed limit, but didn't update their Readme.
The fool speed is a tad above my budget for this:

I just contacted Blockchair for an API key, which enables people to download at the fast rate, and a support rep told me they cost $500/month.

Bump:

blockdata.loyce.club/inputs/ (159 GB - currently incomplete, growing 8 GB per day)

This directory is now 349 GB in size. It's almost complete

[~DefaultTrust]

I've used Blockchair Database Dumps for several projects, but when I need it I don't want to wait weeks to download all files. So I created a mirror for Bitcoin data

What is the problem in to make your own blockchain parser?

[LoyceV]

What is the problem in to make your own blockchain parser?

This is easier
And I don't mind the fact that each file can be verified at an independent third party.

[bitmover]

The data
See blockdata.loyce.club
blockdata.loyce.club/inputs/ (159 GB - currently incomplete, growing 8 GB per day)
blockdata.loyce.club/outputs/ (117 GB)
blockdata.loyce.club/transactions/ (43 GB)

Hello LoyceV.

I am curious this file.

Can you create a sample of each file, with just a few MB, so people can see the that and how it is organized before downloading 500GB?

What is the problem in to make your own blockchain parser?

This is easier

I never saw the blockchain in CSV format, which is a easy to use format for data analysis. Is there anywhere else where you can download the blockchain in csv format other than in loyce.club and blockchair?

[~DefaultTrust]

I never saw the blockchain in CSV format, which is a easy to use format for data analysis.

Will you analize 500 GB data by-hand? 
I think the best format is a relational database. SQLite for example. But it is not exactly 

[LoyceV]

Can you create a sample of each file, with just a few MB, so people can see the that and how it is organized before downloading 500GB?

Lol You didn't click the links, now did you? There are thousands of files (one for each day), so pick a sample

Is there anywhere else where you can download the blockchain in csv format other than in loyce.club and blockchair?

Not to my knowledge.

I think the best format is a relational database. SQLite for example.

I'll leave that to the reader. You're free to download and convert to whatever format best fits your needs.

[~DefaultTrust]

I'll leave that to the reader. You're free to download and convert to whatever format best fits your needs.

Do not see any reasons to download blockchain from half-trusted third-party side. This is contrary to blockchain ideology.
If I will need a blockchain in text format then I will easy to convert it from JSON format provided by native bitcoin RPC-API And I will be 100% sure of the security of this data

[LoyceV]

If I will need a blockchain in text format then I will easy to convert it from JSON format provided by native bitcoin RPC-API And I will be 100% sure of the security of this data

Can I challenge you to do it, and verify Blockchair's data?

[pooya87]

Do not see any reasons to download blockchain from half-trusted third-party side. This is contrary to blockchain ideology.
If I will need a blockchain in text format then I will easy to convert it from JSON format provided by native bitcoin RPC-API And I will be 100% sure of the security of this data

That's assuming you already have the >320 GB blockchain downloaded by your own running node. Not everyone runs a full node and data like the one provided by blockchair.com and mirrored by OP is not supposed to be used by nodes or for anything security sensitive. It is just data useful for data analysis.
For example the ~800 MB file containing addresses with balance is useful in a lot of recovery cases as a database to search against where the user doesn't remember their address or not all of it.

[~DefaultTrust]

If I will need a blockchain in text format then I will easy to convert it from JSON format provided by native bitcoin RPC-API And I will be 100% sure of the security of this data

Can I challenge you to do it, and verify Blockchair's data?

Hmm. Just for fun?
Not. Only if I will see some benefit in it

[LoyceV]

I just notice the dump have both inputs and outputs, but what's the difference? Which one is UTXO in this case?

I didn't create this format I'll show some samples:
inputs:

Code:

block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable spending_block_id spending_transaction_hash spending_index spending_time spending_value_usd spending_sequence spending_signature_hex spending_witness lifespan cdd
141719 3c86cc88726f38d2f1e861850b143fe8d710e171a91291628ffe6418d8b60190 1 2011-08-20 06:12:23 990322200 112.8967 1L2wqJ4z4pjXpqP4EFudnn7TKE4sjksQQj pubkeyhash 76a914d0c893ee2fa4278a2542691e2f569247d833805988ac 0 -1 152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 0 2011-11-11 00:05:55 28.4222 4294967295 4830450220015c796fc6ddf0040c70115539d476426f5c69e1537bb8b7da90d4d671c564d302210094fc145108b617352db33880118a69c059055538af7472ae60a5f4b37d558a060141048396aebc72f8a7b26ea0f676c7c8b9379b0a93b487058787f78b2d256d2a4a6e37d7df1ac8314ee09c1f9a57d48f8c07fd1e4e3056bb17c0075ec703599cada9 7149212 819.4471476975
152763 5733d4a453dcea2690bdc8e8866f5788adadca82ba4639a3dada3d26f738ae9a 0 2011-11-10 23:53:08 1167280 0.035 1JFVo6cGL3cpyXZctC5Tw3HHvmAzuwkjSj pubkeyhash 76a914bd37ff9aa142d5ed87b1f9385c66b856b6c60e6088ac 0 -1 152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 1 2011-11-11 00:05:55 0.0335 4294967295 493046022100ba9fa562476163f5eaac9a6bbee123a0c3f1421377c35208af4b9606816603d1022100c1a797af43d8297f08011248c16f01ee9e95d03ff5d527c2ace2adb545eb88de0141047d949c30d6ffffe8e5ecd2ee774745bda2563e4842b7a9836bd0be8461786a2cceb1d844b0789afde927eace479955f9c27ff98e0e5afe71d4dde54e621aaa1f 767 0.0001036231
141326 5c51c88388d3452c0da455ad03ee50ca121ab4812d8790402527df74cf327db1 0 2011-08-17 11:03:42 700000000 77.35 17AqNueHmeFbk6rxnpvQM3QGFJM9oAs1XY pubkeyhash 76a91443ace1cda1b0aede06ca68bc0f60f4c11ecbfdc288ac 0 -1 152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 0 2011-11-11 00:05:55 20.09 4294967295 493046022100b848925df3d18baa10f26b169bd917927c0787aa5719d3e2811176ec66e8297f022100abf4688ed1b744fd3d64e2263137feb2b057ff93d35d8e8e8ddb13fee49c40780141049042d124b779c07a9c825fd572a5002c7a50efc1fea451ad390308196f75ea7b0747c86dbfa369400b0b8d7c1965999d3fe0ad2a4f87afd96c90a5d3ed2c5ecb 7390933 598.8024421296
152754 6b5cd836c52d17a1390e1a9533303941ad23e497f0b508922f1dc55c00d293f0 0 2011-11-10 21:46:08 59000000 1.77 1KGBGWUVbVpvW6Sw7GBVsjS2NS8uWbMsgM pubkeyhash 76a914c85113063d2c642f0566d605572581075dd7bd7188ac 0 -1 152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 1 2011-11-11 00:05:55 1.6933 4294967295 483045022100a4a8ce69ccbee08899a0c5693c1c43ddac5e4c187093fd7bd1efdcea70a92c3502200fe64020f94229976febf8427d4b1510187f500f95a004a4a7be498b0439c565014104175747a43f4f51bcb9f1753398157bf5b02cd9b239ea11b544d95b3f7c81881c46c0a86b57e445f0e803c630dc1f51e7b6f7d59f36b408509cd1b3cdda918625 8387 0.057272338
145520 f49804f14a44a7a478a870635662df8b6b089317c2e878b701c6ab4c1955cf2d 1 2011-09-15 22:35:54 600000000 33.3 1HxmXQjcU51CGuQuugjFBL4Aa2VvroqpsF pubkeyhash 76a914ba0e0eb50dce94d5979626f15745ab12e89369eb88ac 0 -1 152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 0 2011-11-11 00:05:55 17.22 4294967295 483045022038b72c52176158ddd3de61eb7b86f905e8983a6c2c7fce35426cba359420649f022100a1fa8f55a281bc2f8440d561814a59be10b10193c2243c949de9574bed992a85014104d2777e3eb09b35a223ec2a5a0b0254282741046976f8765ab07bbee3715c46b8572e7b3143f0de5c615b3de6e24cd491bb52ec84bac50658bcc879a3d8e86bd5 4843801 336.3750694444
142760 1bcd0fa9ce0b1953364cb52015729e913c91f77774d43aa431667c32ae005e03 1 2011-08-27 05:29:51 400000000 34.6 1J5Z4mFHXQbLqehEkzcoa2Qgw2SGWyn6dw pubkeyhash 76a914bb56918ea3dc5b96343ac5b67843a09a1870c4a388ac 0 -1 152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 0 2011-11-11 00:05:55 11.48 4294967295 4830450221009877ac9a00d2182878aa86555e1675f9a939e4cbff68d98edb9f89698d41f4ae02201832f05f5dd2d26e67a7444c85959d53fa23bd761a9caa6e6d4c556f61a05f3b0141046f13594b5b3275d5acc8fc97ba642eda76595e1e891227560ddcd617cc12a7b3cff8081ea4f5c4625f12f20690c6df2594480822162e5fd7dc0606ff6d5747c0 6546964 303.1001851852
152758 4b28cbbfb472316b3ca32772cc2a1d05a87a37db91d951aab72b09a4d3811816 0 2011-11-10 22:58:57 761730212732 22851.9062 141utQD6sbepcubqvm7Xk8hWS71mojJg1Y pubkeyhash 76a91421148647f9b6aa60ab8161fa02cf8600e46b793e88ac 0 -1 152764 d81a2bad4b0dcec1e7fc79adb9143c85ab35c9347aa4b2c5660f0e1432c8e3dd 0 2011-11-11 00:05:55 21861.6562 4294967295 48304502207712beff0dc14506dd154a465f1560522e4577d96b4c16915c134cef4eeeed66022100c69776719e12208d0a33ff71f45bfe0d323ca3a6a7b7decb5ba61647fde41405014104b0807a7e5ab5d740f45c6f95421cfb8db0c5f1548f2bc0d10a15074a2e3691888cf90258257e6274c9a9e6033dfb001709035a52d0a7679c50a5276ade272fe2 4018 354.239814208
152754 b119c44a9c9bee8dec87492d5ce0d96a9784de1e14b26f83eaacbfccecafaaf5 0 2011-11-10 21:46:08 89089420 2.6727 18j82gzKFWJJWEiXx1RyGWUY69rYxCPTxr pubkeyhash 76a91454c037efc87d749eb4d30234c2e0dd73717ee7a588ac 0 -1 152764 d81a2bad4b0dcec1e7fc79adb9143c85ab35c9347aa4b2c5660f0e1432c8e3dd 1 2011-11-11 00:05:55 2.5569 4294967295 48304502206a86cbe79575735c709e303cab2380b6344c702b3b6d47ecb06be0f6ec840f78022100bd9c0510780afd4f14ea540eadf30f84e1897f623da628593ebd9cb61a88504601410490123b018833a24b864517617762eaef3cc80739a901120ec97f4ea02646debfd0b963bdcfb4a378e9ae1820a30cf2322e8d4ee172e76ea567b326897d8b7e93 8387 0.0864806673
152731 575b6a2694542ddc66e14d1c827aaa8924bdc95636869c94f09e70520d8ad4f7 1 2011-11-10 18:00:06 772000000 23.16 1QKaXk2hUDv61aRy5uQViq1Nr41CcttWCU pubkeyhash 76a914ffce5eab41b4213ebe134108331071337984959588ac 0 -1 152764 3815ece30a32cd879bf11fd4a4f6644695b63cf1521e1a1b1f7a9908bf0e8b54 0 2011-11-11 00:05:55 22.1564 4294967295 47304402203ee62439de32dd165c013302159772130792428135a9b4733e32bafafc32ca91022044c736b7883ed1b8c0af94c8393070effd75ee0fa8e62b61887dd99a29fe9336014104d9cd7d986ce7cbf9056ad41cc3bc4f56b6f93fb3a3eb07be16dffc80e521849631f69c8cf5e886682808d35059e27e4eca77fde73b8ebbcc0e33240e09a467eb 21949 1.9611837963

outputs:

Code:

block_id transaction_hash index time value value_usd recipient type script_hex is_from_coinbase is_spendable
152764 498ec7f88857d23bd19370252da439876960dad296a64534987ad54081a9cc39 0 2011-11-11 00:05:55 5001400010 143.5402 17aA19GvhzMHsq8xPwSXAPZutyr6kuzLEB pubkey 4104a8dd3f118a6122c2f8c6be3261670f7af76568cac8ff0ed95e5ef63238018e69b223dbdef9f9d5e94a053ff1afc390e230844c0b71f3648405807cd668979958ac 1 1
152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 0 2011-11-11 00:05:55 1199480 0.0344 1KPxwAbFVoDimPrVECF2zgiyfX9jGW9TCy pubkeyhash 76a914c9ca1b452087cdc6b89754c1090928d7a67ef23988ac 0 -1
152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 1 2011-11-11 00:05:55 990290000 28.4213 1AUzdKPJtPyFgrg88nK6fBH7jKkgPyphUj pubkeyhash 76a91468047787a9dce43acf61af61eb1867a9cb0724b688ac 0 -1
152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 0 2011-11-11 00:05:55 9940000 0.2853 1LVcc51n4Cdd8oKLc8GWEpu2wZzfg82Bpb pubkeyhash 76a914d5d3c21182c74b3c306e9ca9746b40d29971223388ac 0 -1
152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 1 2011-11-11 00:05:55 749060000 21.498 1ByQHoyjyMNn5NznRbSHwYcRf7DHXNL3ih pubkeyhash 76a914785bfe399c2cca278e45d76e34317dfa5941fe9f88ac 0 -1
152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 0 2011-11-11 00:05:55 74916000 2.1501 1JwtyTt2w7xZJgrgPg4rCyaV7DKEa4n5uH pubkeyhash 76a914c4dbfc4dac46dd1ca2cab8af4d431bd422a6d7d688ac 0 -1
152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 1 2011-11-11 00:05:55 525084000 15.0699 14rKLXhpX6Sukznqm5cykcgZMt4qZXamQB pubkeyhash 76a9142a3c10f240c9266f43c110242cbe877868f12ea088ac 0 -1
152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 0 2011-11-11 00:05:55 2824000 0.081 1AVntqwcaMtYVqYuH9f8zXYM1Gnz8QHVWe pubkeyhash 76a914682b16d8f9bad9fcb6993950233fac67f75c55c588ac 0 -1
152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 1 2011-11-11 00:05:55 397176000 11.399 1J9xrxAKgaRR5ddYtot559r7yDwQiS9dk1 pubkeyhash 76a914bc2c18bc245d5a4bc64567bca525741c0a8069d188ac 0 -1

transactions:

Code:

block_id hash time size weight version lock_time is_coinbase has_witness input_count output_count input_total input_total_usd output_total output_total_usd fee fee_usd fee_per_kb fee_per_kb_usd fee_per_kwu fee_per_kwu_usd cdd_total
152764 498ec7f88857d23bd19370252da439876960dad296a64534987ad54081a9cc39 2011-11-11 00:05:55 135 540 1 0 1 0 1 1 0 0 5001400010 143.5402 0 0 0 0 0 0 0
152764 5cc203b9389f0f7eb50669eba04ac32d666892fc95f07c25603da8a6ed9316ae 2011-11-11 00:05:55 439 1756 1 0 0 0 2 2 991489480 28.4557 991489480 28.4557 0 0 0 0 0 0 819.4471476975
152764 e0105a83ce04598fe208f1910643a4f75951e9f95f40a0e88ac697d0ec72679d 2011-11-11 00:05:55 439 1756 1 0 0 0 2 2 759000000 21.7833 759000000 21.7833 0 0 0 0 0 0 598.80244212963
152764 6f2ddb0352071a02e89f9ed9bfaed33f0c1dc2fadd97b6f3a34e76bfe3ea6aed 2011-11-11 00:05:55 258 1032 1 0 0 0 1 2 600000000 17.22 600000000 17.22 0 0 0 0 0 0 336.37506944444
152764 0a3c1eb28c3003405ad745a5483ff25c583ac51f93bdbcd761e444ac8639f919 2011-11-11 00:05:55 258 1032 1 0 0 0 1 2 400000000 11.48 400000000 11.48 0 0 0 0 0 0 303.10018518519
152764 d81a2bad4b0dcec1e7fc79adb9143c85ab35c9347aa4b2c5660f0e1432c8e3dd 2011-11-11 00:05:55 438 1752 1 0 0 0 2 2 761819302152 21864.2129 761819302152 21864.2129 0 0 0 0 0 0 0
152764 3815ece30a32cd879bf11fd4a4f6644695b63cf1521e1a1b1f7a9908bf0e8b54 2011-11-11 00:05:55 436 1744 1 0 0 0 2 2 10771000000 309.1277 10771000000 309.1277 0 0 0 0 0 0 140.63292604167
152764 09a4a4d5706d7123c13a8128040a50e00a79d4d369b3618f7b6140fe60e9c89c 2011-11-11 00:05:55 1519 6076 1 0 0 0 8 2 78001010386 2238.6289 78001010386 2238.6289 0 0 0 0 0 0 245.16466314468
152764 1b80bdad1902800146ef1a48665c4f13e409442feda0f9f442627aa6c3479297 2011-11-11 00:05:55 257 1028 1 0 0 0 1 2 43999811790 1262.7946 43999811780 1262.7946 10 0 38.9105 0 9.72763 0 0

P.S. There's similar discussion at Sync your blockchain easily!.

That's different, the data in my topic isn't meant to be used by a wallet for transactions.

[bitmover]

That's assuming you already have the >320 GB blockchain downloaded by your own running node. Not everyone runs a full node and data like the one provided by blockchair.com and mirrored by OP is not supposed to be used by nodes or for anything security sensitive. It is just data useful for data analysis.

I was taking a look at input files to try to make some analysis. But the files are huge, which is a problem for my current set up (HD size and bandwidth).

My first attempt is to group inputs by transaction ID, then group addresses which were spend together. Creating wallets... Just like walletexplorer.com does (but it would be amazing if I could do this myself).

For example, I know that these 2 addresses belongs to the same wallet:


But this is a much more complicated job than I expected, as files are very large.

This analysis would expose privacy weakness. This kind of analysis is very easy to do for few addresses (if someone is being investigated), and people are not aware of that.

[LoyceV]

I was taking a look at input files to try to make some analysis. But the files are huge, which is a problem for my current set up

You can instantly remove the data you don't need, that makes it a bit smaller.

Update:
Files are complete now:

The data
See blockdata.loyce.club
blockdata.loyce.club/inputs/ (386 GB)
blockdata.loyce.club/outputs/ (121 GB)
blockdata.loyce.club/transactions/ (44 GB)

I've activated daily updates.

[NotATether]

Be kind
Please only download with one thread at a time. I don't want to be kicked off another server.

Just curious, do you have any disk space or bandwidth/s quota on the server hosting this, and is it shared with your other projects?

[LoyceV]

do you have any disk space or bandwidth/s quota on the server hosting this, and is it shared with your other projects?

I get 1 TB disk space and 10 TB bandwidth per month. It's not shared with my other projects (but it is the only server I have that can handle this much data).
So far, it's perfect for this project

[LoyceV]

Loyce as I see the server for this is still unavaible?

Unfortunately, yes

It looks like Chia mining killed cheap storage VPS servers.

[LoyceV]

Loyce do you know is any other webpage than blockchair.com for this?

Nope
What data are you looking for exactly?

[LoyceV]

number of transaction + bitcoin addres sender (not pubkey)

For each Bitcoin address ever used?

[LoyceV]

number of transaction + bitcoin addres sender (not pubkey)

I don't think this data is in here.
I've checked blockchair_bitcoin_inputs_20101104.tsv.gz and blockchair_bitcoin_outputs_20101104.tsv.gz, and both show the recipient address.
I think the sending addresses can be extracted from one of the hashes in there, but I'm not sure how. If you tell me how to get what you need, I wouldn't mind going through the data on a rainy Sunday afternoon.

[bolovo]

Hi, any news about finding a host?

Great contributions!

[LoyceV]

Hi, any news about finding a host?

Nope. Large storage space turns out to be either expensive or annoying. I found 2 affordable hosts, but the first one doesn't accept Bitcoin, the other asks for my ID.

[bolovo]

Any thoughts in making an torrent with an http web seed?

Oh boy, I'm desperate, downloading from blockchair at 100k   

[LoyceV]

Any thoughts in making an torrent with an http web seed?

That won't work without having a large VPS. I don't want to be seeding it from my own PC.

Oh boy, I'm desperate, downloading from blockchair at 100k    

That takes a few months. Been there, done that
I could spin up a pay by the hour VPS, but I'm not really looking forward to upload 600 GB just for a one time download.

[PawGo]

Why do not you host it on your private/local NAS + dyndns solution? I understand that your internet provider (bandwidth or transfer limit if you have) could be a problem, but you may limit download speed or just do not run it 24/7.

[LoyceV]

I understand that your internet provider (bandwidth or transfer limit if you have) could be a problem

That's why
And privacy of course, I'm not hosting things at home.

I found a good offer for my other project, with 15 TB monthly bandwidth. It would be great if that same host offers storage servers in the future.
These used to be my little side projects, but I already have 2 (virtual) servers running for it now. Things tend to get out of hand quickly

[maa@slm]

This data is unavailable until I find an affordable VPS with enough disk space and cheap bandwidth.

Background
I've used Blockchair Database Dumps for several projects, but when I need it I don't want to wait weeks months to download all files. So I created a mirror for Bitcoin data

The data
See blockdata.loyce.club
blockdata.loyce.club/inputs/ (386 GB)
blockdata.loyce.club/outputs/ (121 GB)
blockdata.loyce.club/transactions/ (44 GB)

Updates
New files are added daily.

Be kind
Please only download with one thread at a time. I don't want to be kicked off another server.

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

No spam please.
Self-moderated against spam. Discussion and questions are welcome.

Related topics
Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used
[~500 GB] Bitcoin block data: inputs, outputs and transactions

Hello Sir,
please can you update Transaction link
any new link please send

Thank you in advance

maa

[LoyceV]

please can you update Transaction link
any new link please send

I'm not sure what you mean by "transaction link". If you're looking for the current location of this data, it's on my external USB drive. Unfortunately:

It looks like Chia mining killed cheap storage VPS servers.

[maa@slm]

please can you update Transaction link
any new link please send

I'm not sure what you mean by "transaction link". If you're looking for the current location of this data, it's on my external USB drive. Unfortunately:

It looks like Chia mining killed cheap storage VPS servers.

sir
i need blockdata.loyce.club/transactions/ (44 GB) link
i am newbie sir , just learning sir......

Thank you sir

[LoyceV]

i need blockdata.loyce.club/transactions/ (44 GB) link

Sorry, as I wrote earlier, I currently don't have hosting for it.

[maa@slm]

i need blockdata.loyce.club/transactions/ (44 GB) link

Sorry, as I wrote earlier, I currently don't have hosting for it.

Thank you for your reply

please inform me after solving hosting

[LoyceV]

please inform me after solving hosting

You can click "notify" at the bottom of any topic

[LoyceV]

The best (temporary) option for hosting I found so far is this:

$10/mo ($0.015/hr)
2 CPU, 2GB RAM, 650GB HDD

This offers 7.5TB/month bandwidth, and charges $4/TB for additional bandwidth.
Would anyone be interested in this data at this price?

[LoyceV]

For $10, that VPS is very cheap. Do you mind share the name of the VPS provider?

RamNode.com. This is the "MASSIVE KVM", which means it's HDD and not meant for heavy computing tasks.

[LoyceV]

I have bit concern about payment processor they use though (Bitpay and Payssion), let's hope one of them won't ask for KYC verification or strict blacklist.

Last time I funded my account was last May 2021, using a low-fee Forkcoin through Bitpay. I funded it with $10 at a time, just in case they'd hit me with KYC. They didn't. But as far as I know that's no longer possible.
I've never used Payssion.

[LoyceV]

New hosting
On my quest for new hosting, I ended up on a seedbox. It's not big enough for all data, but to start, I've uploaded outputs and transactions. See: blockdata. Unfortunately, there's no nicer URL for now.

Torrents!
outputs.torrent
transactions.torrent
I think the problem is with torrent trackers, but I'm not sure how to fix it. It works now

[LoyceV]

Torrents!
outputs.torrent
transactions.torrent

Has anyone tried these torrents? No data was uploaded, so if you tried and it's not working, please let me know.

I got a new server today: 1 TB, so enough to add the "inputs" data too. I'm currently trying to Torrent the files from the old to the new server, but so far it's not doing anything. Hence my question if anyone else tried

[PawGo]

Torrents!
outputs.torrent
transactions.torrent

Has anyone tried these torrents? No data was uploaded, so if you tried and it's not working, please let me know.

I got a new server today: 1 TB, so enough to add the "inputs" data too. I'm currently trying to Torrent the files from the old to the new server, but so far it's not doing anything. Hence my question if anyone else tried

I have just started download, but I have only one seed, for Transactions.
I will leave it and see if it finish.

[LoyceV]

I have just started download, but I have only one seed, for Transactions.
I will leave it and see if it finish.

Thanks for testing. I see it's indeed uploading now. My own download (from the new server) hasn't started yet.
My guess is there might be a problem with the Trackers I choose.

Update:
My own download started and completed overnight. Judging by the amount of sent data, your download stopped after about a third.
I'd appreciate some pointers from someone with experience creating Torrents.

[LoyceV]

how do you create the torrent?

I have a Seedbox at PulsedMedia.com, which by default uses ruTorrent. I clicked "Create Torrent", selected the data, and added some trackers that I found in other torrents. Then, I set the Piece size as large as possible (16 MB, because it's 50+ GB of data). After a couple of minutes, it was ready and I could save the .torrent file.

In past i used mktorrent (to create torrent) and qbittorrent (on both server and client to upload/download) to transfer data due to unstable connection on my end. I don't use any tracker, but i disable private flag so i could use DHT. However, my data size was smaller.

When creating the torrent, ruTorrent doesn't have the option to use DHT.

[ABCbits]

how do you create the torrent?

I have a Seedbox at PulsedMedia.com, which by default uses ruTorrent. I clicked "Create Torrent", selected the data, and added some trackers that I found in other torrents. Then, I set the Piece size as large as possible (16 MB, because it's 50+ GB of data). After a couple of minutes, it was ready and I could save the .torrent file.

1. ruTorrent (which use rTorrent as backend) is reliable (according to discussion on different forum), although it's development has slowed down in past 3 years. However it should able to create torrent without problem.
2. You might want to check this thread about torrent piece size, https://torrentinvites.org/f29/piece-size-guide-167985/.
3. Take note not all tracker accept any kind of torrent.

In past i used mktorrent (to create torrent) and qbittorrent (on both server and client to upload/download) to transfer data due to unstable connection on my end. I don't use any tracker, but i disable private flag so i could use DHT. However, my data size was smaller.

When creating the torrent, ruTorrent doesn't have the option to use DHT.

Really? It's weird since rTorrent (the backend) support DHT. Can you use different tool (which support DHT) on your Seedbox?

[LoyceV]

Really? It's weird since rTorrent (the backend) support DHT.

These are the only options it gives:

Can you use different tool (which support DHT) on your Seedbox?

I can start the following:

Code:

Deluge
Rclone
qBittorrent

I haven't tried those yet.

2. You might want to check this thread about torrent piece size, https://torrentinvites.org/f29/piece-size-guide-167985/.

I read about that already, it's a compromise. I could select a smaller piece size, but that means it has many pieces to deal with. I'll try 4MB next time.

3. Take note not all tracker accept any kind of torrent.

That's the tricky part: I couldn't find much about it, only a list of recommended trackers. That's what makes me think the slow download start might be related to the trackers.

[seoincorporation]

Hello LoyceV.

After reading the thread I see you are having problems with the amount of data and the way to share it, and I was thinking about a solution or how I would do it.

You need a VPS with a huge disk and unlimited band bandwidth, and those aren't cheap at all, but I know a cheap way to handle this.

You can buy a laptop (workstation) with a 1TB SSD and configure it as a server with NO-IP. This way your bandwidth will be limited only by upload and download speed, but not for the amount of data. And you can run the server in your house.

If you like the idea and want some help with the server configuration just let me know. 

[LoyceV]

You need a VPS with a huge disk and unlimited band bandwidth, and those aren't cheap at all

The Seedbox actually isn't that bad: It has 6TB monthly bandwidth, and when it reaches the limit it continues at 100 Mbit/s.

You can buy a laptop (workstation) with a 1TB SSD and configure it as a server with NO-IP. This way your bandwidth will be limited only by upload and download speed, but not for the amount of data. And you can run the server in your house.

There are several problems with that: First, I don't want to expose my home IP address for this. And I don't want my upload bandwidth to be maxed out for several days in a row when someone downloads 600 GB.
Third problem is that I don't think my internet provider would appreciate it if I upload a few TB each month, so they'll probably start throttling my bandwidth. And on top of that, given current electricity prices, running a system 24/7 is more expensive than the Seedbox.

If I can't get the Torrents to work, the files can still be downloaded directly through http, but I haven't even finished uploading everything yet. And if barely anyone is going to use this, I might just have to give up on this project entirely at some point.

[LoyceV]

Since it include option "Private Torrent", usually it means DHT is used by default. And when i tried it on my VPS (where i remove all tracker before start the torrent), the DHT is working and i can start the download.

To test this, I tried to create a new one with "Private torrent" enabled, and 4MB piece size. It didn't show up, so I deleted the existing one first. It still didn't show up. I'm now testing without "Private" again: this works.

Yeah, usually i just rely on DHT and share my IP:Port (if needed) so other user can it directly to list of peers on their torrent software.

How would that IP/port-sharing work? Usually I just click the Torrent or Magnet link, and don't need to manually add IPs.

My VPS have spare bandwidth, do you want me to try to see whether i can complete the download?

Feel free to test it

[LoyceV]

But it's not practical option.

Indeed. I don't want users to manually add IPs.

Testing it now.

Only one out of 2 Seedboxes is currently uploading (at a very decent speed).

I'm uploading the rest of "inputs", after that I'll fix the file dates and see if I can get a more permanent solution for hosting those files.

[LoyceV]

1. I remove tracker on both torrent before i start download process.

Why? And how? When I open a torrent in Transmission, it just starts and doesn't ask to remove trackers.

2. "output" downloaded started some time (about 15 - 30 minutes) after i add it.

I'm used to only downloading Torrents with many seeders, and those usually start within seconds. I have no idea why this takes so long. The Seedbox doesn't seem to lack capacity.

3. Average speed was 24-30MB/s. Although "transaction" have 2x speed after it connected to 2 seeder.

I think "outputs" shifted seeding to the other Seedbox at some point. I expected both to seed simultaneously.

4. I use qbittorent-nox with default configuration.

CLI? Even though I use CLI for most things, I've never used it for Torrent.

I think i can conclude that you made the torrent files correctly and there's no problem with your seedbox.

Thanks! I have about 100 GB upload to go, I expect it to be done tomorrow. After that, I'll create 3 brand new Torrent files with the most complete data (including correct file dates). I'll also schedule daily downloads to update the latest files again, update OP, and add the new download link.
I won't update the Torrent file that often, which means it'll miss the latest days (or weeks, months....).

[ABCbits]

Why?

Because i wanted to test whether DHT works.

And how? When I open a torrent in Transmission, it just starts and doesn't ask to remove trackers.

On qbittorrent-nox, you can choose not to start torrent after adding it. It has tracker tab where you can remove each tracker manually.

I'm used to only downloading Torrents with many seeders, and those usually start within seconds. I have no idea why this takes so long. The Seedbox doesn't seem to lack capacity.

I have no idea either, but the weird thing is "transaction" torrent start within few seconds.

CLI? Even though I use CLI for most things, I've never used it for Torrent.

I only use CLI to start the program, qbittorrent-nox offer web UI.

[LoyceV]

I've finalized my update, and after 10 months, I'm happy to announce This project is back online!

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

Feel free to test (and/or seed) the torrents if you have some bandwidth and disk space to waste

I haven't tested the daily updates yet. Let me know if anything needs my attention.

[PawGo]

I've finalized my update, and after 10 months, I'm happy to announce This project is back online!

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

Torrents works well now, download is fast. Is it normal that only one tracker returns 'success' and peers data?

[LoyceV]

Torrents works well now, download is fast.

I still don't get it: my own test (from another Seedbox) hasn't downloaded anything after an hour.

Is is normal that only one tracker returns 'success' and peers data?

That's probably not good, but I'm not sure how to change to fix it.
I (arbitrarily) selected those trackers:

Code:

udp://tracker.opentrackr.org:1337/announce
udp://denis.stalker.h3q.com:6969/announce
udp://opentracker.i2p.rocks:6969/announce
udp://tracker.internetwarriors.net:1337/announce
udp://tracker.coppersurfer.tk:6969/announce

And indeed, only the first one seems to recognize the torrent.

[PawGo]

Torrents works well now, download is fast.

I still don't get it: my own test (from another Seedbox) hasn't downloaded anything after an hour.

For me it works instantly.

Is is normal that only one tracker returns 'success' and peers data?

That's probably not good, but I'm not sure how to change to fix it.
I (arbitrarily) selected those trackers:

Code:

udp://tracker.opentrackr.org:1337/announce
udp://denis.stalker.h3q.com:6969/announce
udp://opentracker.i2p.rocks:6969/announce
udp://tracker.internetwarriors.net:1337/announce
udp://tracker.coppersurfer.tk:6969/announce

And indeed, only the first one seems to recognize the torrent.

[LoyceV]

For me it works instantly.

That's something Overnight, "transactions" completed downloading on the other Seedbox, and later "outputs" did the same. I didn't try "inputs" due to disk space limitations. So I think I can conclude it works, and it may or may not start instantly.
I'll stop fiddling and let it run for now, anyone who's in a rush can always go for the http download.

[LoyceV]

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

Barely 3 days later, I received an email:

There is a backend software glitch which can cause all data on /home to be removed.

I still have "outputs" and "transactions" on the other server, but this means I'll have to upload 469 GB from my local backup again. It also means they've moved me to another server, which means the above links are no longer valid.
I'm not amused.

[yogg]

I'm not amused.

I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ?


More seriously, I can help hosting "balances" if that interests you.
I will need this archive for a personal project anyway, it's not a big deal to make it available on port 80.

I have some extra storage space avail on my setup.

Let me know, I can provide an FTP access to you.

[LoyceV]

I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ?

Lol. I'll have to stand in the hallway to count the blinks.

I'm still uploading data, but I'm not sure if I'll extend this hosts contract. It's now spitting out hardware errors:

Code:

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.842800] [Hardware Error]: Corrected error, no action required.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.846023] [Hardware Error]: CPU:8 (15:1:2) MC4_STATUS[-|CE|MiscV|-|AddrV|-|CECC]: 0x9c44c00001080a13

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.849212] [Hardware Error]: Error Addr: 0x0000001f8d366350

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.852463] [Hardware Error]: MC4 Error (node 1): DRAM ECC error detected on the NB.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.855864] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: RES (no timeout)

More seriously, I can help hosting "balances" if that interests you.

Which "project" do you mean exactly? List of all Bitcoin addresses with a balance? That one is covered, it only takes 20 GB to host. Although by now the 2 TB monthly bandwidth isn't enough anymore, at day 15 it already used 1.2 TB so it'll go offline at the end of the month.
If you happen to have 700 GB available, by all means, share "inputs, outputs and transactions"

Let me know, I can provide an FTP access to you.

FTP, scp, rsync, anything works for me

[LoyceV]

People still use FTP these days? I remember using FTP with FileZilla long time ago.

I still use FTP on my local network, where I don't mind the lack of encryption.

I would demand partial/full refund.

It's paid per month, and really not worth the effort. I'd much rather have them fix the problems so I can finally keep this data online. I've uploaded it too many times already, and the whole point of having it on a server is so I can access it when needed.
If anyone finds a good offer on 1 TB storage, feel free to post it.

[PawGo]

If anyone finds a good offer on 1 TB storage, feel free to post it.

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.
https://www.hetzner.com/storage/storage-box

[LoyceV]

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.

I need a VPS, this looks a bit like "shared hosting". I guess I wasn't specific enough. It could work, but having to upload updates instead of just processing them on the server is far from ideal.
A bigger deal breaker is the fact that Hetzner doesn't accept Bitcoin. I don't want to link my credit card to anything crypto-related. From what I've read, I'd expect Hetzner to offer good quality hosting.

[yogg]

Let me know, I can provide an FTP access to you.

People still use FTP these days? I remember using FTP with FileZilla long time ago.

Nothing better than the good old foundations of IT.
Like, emacs is more pure and "C-itonic" as it's been delivered to us mere mortals by the founder of GNU himself. (f*ck vi "<esc> :wq!" wtf)

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

Which "project" do you mean exactly? List of all Bitcoin addresses with a balance? That one is covered, it only takes 20 GB to host. Although by now the 2 TB monthly bandwidth isn't enough anymore, at day 15 it already used 1.2 TB so it'll go offline at the end of the month.
If you happen to have 700 GB available, by all means, share "inputs, outputs and transactions"

Yes, this is the one I meant to use for my project. (Bitcoin addresses with a balance)

I can find the 700 GB and host "inputs, outputs and transactions"
Luckily the bandwidth is unmetered so I can also provide a mirror for some of the other dumps.

It could work, but having to upload updates instead of just processing them on the server is far from ideal.

How is the data processed ? Running on Bitcoin Core ?

[LoyceV]

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

I actually prefer rsync, because scp changes the original file date, which is undesirable.

Yes, this is the one I meant to use for my project. (Bitcoin addresses with a balance)

In that case, just get the latest version No need for me to upload that one.

I can find the 700 GB and host "inputs, outputs and transactions"
Luckily the bandwidth is unmetered so I can also provide a mirror for some of the other dumps.

It would be great to have a mirror with decent bandwidth. If I don't have to upload 665 GB again when my host fails, it's already worth it.

How is the data processed ? Running on Bitcoin Core ?

If only I had the skills for that! I use this:

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

(f*ck vi "<esc> :wq!" wtf)

What's wrong with vi? I literally use it every day

[yogg]

However we'll go with SCP for this one. LoyceV I'm PMing you the credentials soon.

I actually prefer rsync, because scp changes the original file date, which is undesirable.

Ugh ... that leaves me with no other choice than use touch.



(Actually I'd prefer to do that .. can't do rsync straight to that appliance and I would rather avoid setting up a buffer VM just for rsync)

No need for me to upload that one.

Spot on.

It would be great to have a mirror with decent bandwidth. If I don't have to upload 665 GB again when my host fails, it's already worth it.

How do you download this dump ?
I'm asking, since yeah the data needs to be updated every once in a while.

Are you downloading every update at 100kB/s ?
If yes, we'd "only" save some time if you upload it to my end.

Giving you the ability to blink my hardware's light emitting diodes might be pointless.

How is the data processed ? Running on Bitcoin Core ?

If only I had the skills for that! I use this:

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

Ha. Makes sense.
Some time ago I started coding a script to get something similar to the all addresses balance dump.
Unfortunately the initial processing of the blockchain was too resource expensive. The process never completed.

However ... This got me thinking..
If the "input outputs transaction" dump is something like a csv file, just compiling the "missing" data with Bitcoin Core and append it to the dump could do the trick.

(f*ck vi "<esc> :wq!" wtf)

What's wrong with vi? I literally use it every day

The people who showed me the path to emacs warned me about vi users.
Everytime you use the dd p shortcut, the init entity turns off the numlock on boomer's keyboard so they can no longer login to their windows session. And can't figure out why.
You monster !  

[PawGo]

(f*ck vi "<esc> :wq!" wtf)

What's wrong with vi? I literally use it every day

Is that true that if you tell average user to leave vi, you will get a natural RNG or at least a good example of password (compliant with any strong password policy)?

[LoyceV]

Ugh ... that leaves me with no other choice than use touch.

I used wget instead, this should preserve file dates.

How do you download this dump ?

Also using wget.

Are you downloading every update at 100kB/s ?

Yes.

If yes, we'd "only" save some time if you upload it to my end.

Using rsync would be so easy

Giving you the ability to blink my hardware's light emitting diodes might be pointless.

It should be going wild now.

If the "input outputs transaction" dump is something like a csv file, just compiling the "missing" data with Bitcoin Core and append it to the dump could do the trick.

Until I can get the full data from my own Bitcoin Core installation, I'll keep using the data dumps. And even if I can get the data from Bitcoin Core, it would further increase the VPS requirements.

The people who showed me the path to emacs warned me about vi users.

What's emacs?

I kid you not

Is that true that if you tell average user to leave vi

It can't be: the average user doesn't use vi.

[LoyceV]

I've updated all links to the new host. Torrents should work again too:

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

You could run Bitcoin Core and processing script on local device, then upload the result to your VPS/seedbox.

Apart from the fact that I wouldn't know how to do this, I don't really want to add more load to my local PC.

[PawGo]

You could run Bitcoin Core and processing script on local device, then upload the result to your VPS/seedbox.

Apart from the fact that I wouldn't know how to do this, I don't really want to add more load to my local PC.

Wait, I am lost. How do you currently extract the data you publish?
For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.

[LoyceV]

How do you currently extract the data you publish?

See:

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

(nobody ever reads the OP)

For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.

Memory usage to get balances: ~18GB. That would take a strong VPS.

[PawGo]

See:

Credits
Blockchair Database Dumps has a staggering amount of data, easily accessible (at 10 100 kB/s) with daily updates. All data in this topic comes from Blockchair.

(nobody ever reads the OP)

;-) Your point.
The same applies to the privacy policy and the washing machine manual.

For operations on local blocks I have used that parser: https://github.com/gcarq/rusty-blockparser
Configuration was really easy, processing of course takes some time.

Memory usage to get balances: ~18GB. That would take big VPS.

Ops. I did not use it, only transactions dump. It is up to you if you want to use it, even partially.

Yesterday I was looking for some hosting services for BTC and honestly speaking I haven't found anything I would trust. Once I tried one hosting just to compare with my main one, but service and reliability was terrible. I had impression all that companies which offer servers for BTC are running their "datacenters" somewhere in the garage.
Have you seen https://www.sync.com/pricing-individual/ ?

[LoyceV]

Yesterday I was looking for some hosting services for BTC and honestly speaking I haven't found anything I would trust.

For projects like this topic I don't really need AWS-level uptime, so I gladly go for the budget hosts. I run this project at Racknerd (good deals via Lowendtalk.com), and (so far) I'm quite happy with it. If I look at prices on their own site, it's much higher.
You could also go for RamNode, pay by the hour and less cheap, but from what I've seen RamNode has very solid performance too.
I don't think any of those are run from their garage I run this project at Gullo's Hosting (again: see Lowendtalk.com for deals), which has the unique feature that it's run by one guy. Servers are international, so not from his garage, but you'll always deal with the same guy. He'll try his best, which for this project is everything I need.

I also found out that several of the higher range VPS providers don't accept Bitcoin, or demand a copy of my passport (yeah, right!) first.

Have you seen https://www.sync.com/pricing-individual/ ?

That's not a VPS, so won't help me much.

[LoyceV]

Big update

First: the Torrents won't last. I got an anonymous sponsor for a dedicated server. The Seedbox expires on April 6, so the Torrens won't work much longer:

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

The data is back at it's original location:

Back online! The data
See blockdata.loyce.club
blockdata.loyce.club/inputs/ (469 GB)
blockdata.loyce.club/outputs/ (145 GB)
blockdata.loyce.club/transactions/ (52 GB)

This server is has a 50 TB/month bandwidth limit. So far, at most a few people per month downloaded this (crazy amount of) data, so it should be sufficient.

Note: Some files are missing, those are still being updated now. By tomorrow automated daily updates should be on track again.

[PrimeNumber7]

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

My VPS doesn't have enough storage for "inputs" torrent, so i didn't try until download is finished. But it starts almost immediately with similar speed on previous speed.

It is ~never appropriate to store that much data on a VPS. You are much better off storing these files in a storage bucket. If you try any other solution, you will either quickly hit your transfer limitations, or your files will eventually get taken down because you are taking up too many resources.

Hosing your files in a storage bucket will mean that others can ~instantly download your files (limited only by their own bandwidth and computer equipment). Allowing people to download your files from the internet will be expensive, however, this can be addressed by configuring your bucket such that the requestor (the person downloading your file) will pay for the egress bandwidth.

All major cloud providers offer storage buckets. Many smaller cloud providers do as well.

[LoyceV]

Hosing your files in a storage bucket

We discussed this already. But I just got a very nice Xeon-powered dedicated server (no more VPS!) from an anonymous donation, so I'm covered for now.

All major cloud providers offer storage buckets. Many smaller cloud providers do as well.

I'm curious: what would it cost to store a TB in a storage bucket?

[PrimeNumber7]

Hosing your files in a storage bucket

We discussed this already. But I just got a very nice Xeon-powered dedicated server (no more VPS!) from an anonymous donation, so I'm covered for now.

Ahh, yes, there it is. I thought I remembered giving you this advice, but I couldn't find the discussion in this thread.

All major cloud providers offer storage buckets. Many smaller cloud providers do as well.

I'm curious: what would it cost to store a TB in a storage bucket?

AWS and GCS have the same pricing structure, generally speaking. If you hosted the files on either of those platforms, it would cost approximately US$20 per month. This is not the same as hosting your files on a VPS. Accessing and transferring the files would be much quicker compared to using a VPS.

I believe you have previously stated that you don't want to use AWS because you don't want to have to use a credit card to pay, nor associate your IRL identity with the service. My argument would be that this is really your only option, as using a VPS with a smaller provider (as you have been doing) is eventually going to result in your account being shut down, or your bandwidth being exhausted.

There might be other cloud storage providers that can offer storage buckets at a lower price that might not be as reliable or have as high of throughput that offers their services for less. Your project is not one that is critical to always have access to all your data at a moment's notice, so this may be okay. It is possible you can find one that will be willing to accept crypto for their services.

If you use a storage bucket on AWS or GCS, someone will need to pay $0.09/gb to transfer the file to the internet, or $0.01 to transfer your file to another data center run by the same cloud provider located on the same continent where your data is housed (or $0.00/gb -- free - within the same datacenter location). You can configure the storage bucket so that the person downloading the file will need to pay the transfer charges.

[DaveF]

...I believe you have previously stated that you don't want to use AWS because you don't want to have to use a credit card to pay, nor associate your IRL identity with the service. My argument would be that this is really your only option, as using a VPS with a smaller provider (as you have been doing) is eventually going to result in your account being shut down, or your bandwidth being exhausted...

Not unless he is working with a really small provider. More and more 1GB un-metered for co-location is the standard.^* Or if it is metered it's in the multi TB range.
Bandwidth has gotten so cheap at data centers that it's pointless to rate it anymore. Case in point, Cogent just made us an offer for a 1GB fiber loop to our rack for under $400 a month all in. 3 years ago that same circuit was well over $1500. Hurricane is under $2500 for a 10GB circuit. And we are a very small buyer of bandwidth. "Real" companies that are buying multi 100GB circuits are paying very very little.

-Dave

^* Most places are going for a 10 to 1 over subscription so you probably may not get the 1GB all the time but the point is the same.

[PrimeNumber7]

...I believe you have previously stated that you don't want to use AWS because you don't want to have to use a credit card to pay, nor associate your IRL identity with the service. My argument would be that this is really your only option, as using a VPS with a smaller provider (as you have been doing) is eventually going to result in your account being shut down, or your bandwidth being exhausted...

Not unless he is working with a really small provider. More and more 1GB un-metered for co-location is the standard.^* Or if it is metered it's in the multi TB range.
Bandwidth has gotten so cheap at data centers that it's pointless to rate it anymore. Case in point, Cogent just made us an offer for a 1GB fiber loop to our rack for under $400 a month all in. 3 years ago that same circuit was well over $1500. Hurricane is under $2500 for a 10GB circuit. And we are a very small buyer of bandwidth. "Real" companies that are buying multi 100GB circuits are paying very very little.

-Dave

^* Most places are going for a 10 to 1 over subscription so you probably may not get the 1GB all the time but the point is the same.

It is unlikely that Loyce would be dealing with Cogent directly, but rather would be dealing with one of Cogent's customers.

Even if someone's bandwidth is "unmetered", I can assure you that usage is still "monitored". As you note, Cogent is going to oversell their capacity, and most likely, Cogent's customer who sells VPS services will also oversell their capacity. If Loyce is constantly sending hundreds of GB's worth of data to the internet, there will be less capacity for other customers to send their own data to the internet, which will degrade service for others.

The files that Loyce is hosting total over 660 GB. It would not take much for Loyce to run into hitting the multi TB range with files of that size, especially considering that it is trivial for someone to request those files multiple times.

[LoyceV]

AWS and GCS have the same pricing structure, generally speaking. If you hosted the files on either of those platforms, it would cost approximately US$20 per month. This is not the same as hosting your files on a VPS. Accessing and transferring the files would be much quicker compared to using a VPS.

There are several problems with that: first, I can find a much better deal for that price. The Seedbox for instance costs much less for 6 TB bandwidth (and unlimited at 100 Mbit after that). And I don't really mind if someone has to wait a day for a 665 GB download. That's a small inconvenience, and I prefer that over having to buy their own storage bucket and pay for bandwidth before they can download the files.

I believe you have previously stated that you don't want to use AWS because you don't want to have to use a credit card to pay, nor associate your IRL identity with the service. My argument would be that this is really your only option, as using a VPS with a smaller provider (as you have been doing) is eventually going to result in your account being shut down, or your bandwidth being exhausted.

For this project, I don't really mind if it runs out of bandwidth. That just means some people have to wait until the next month. Torrent was a nice workaround, but more work to update.
As I wrote earlier: at most a few people per month have downloaded this data, which means my current limit of 50 TB should be sufficient. To quote my sponsor about hitting the bandwidth limit:

Then we can upgrade the server or limitate the download speed
Or sell apikeys for downloading 😛

If you use a storage bucket on AWS or GCS, someone will need to pay $0.09/gb to transfer the file to the internet, or $0.01 to transfer your file to another data center run by the same cloud provider located on the same continent where your data is housed (or $0.00/gb -- free - within the same datacenter location). You can configure the storage bucket so that the person downloading the file will need to pay the transfer charges.

Up to $60 to download a few files. No wonder Bezos is rich

The files that Loyce is hosting total over 660 GB. It would not take much for Loyce to run into hitting the multi TB range with files of that size, especially considering that it is trivial for someone to request those files multiple times.

It still takes several people crazy enough to download this much data per day to reach 50 TB. My other project currently uses over 2 TB per month, so for now I'm good

[DaveF]

...I believe you have previously stated that you don't want to use AWS because you don't want to have to use a credit card to pay, nor associate your IRL identity with the service. My argument would be that this is really your only option, as using a VPS with a smaller provider (as you have been doing) is eventually going to result in your account being shut down, or your bandwidth being exhausted...

Not unless he is working with a really small provider. More and more 1GB un-metered for co-location is the standard.^* Or if it is metered it's in the multi TB range.
Bandwidth has gotten so cheap at data centers that it's pointless to rate it anymore. Case in point, Cogent just made us an offer for a 1GB fiber loop to our rack for under $400 a month all in. 3 years ago that same circuit was well over $1500. Hurricane is under $2500 for a 10GB circuit. And we are a very small buyer of bandwidth. "Real" companies that are buying multi 100GB circuits are paying very very little.

-Dave

^* Most places are going for a 10 to 1 over subscription so you probably may not get the 1GB all the time but the point is the same.

It is unlikely that Loyce would be dealing with Cogent directly, but rather would be dealing with one of Cogent's customers.

Even if someone's bandwidth is "unmetered", I can assure you that usage is still "monitored". As you note, Cogent is going to oversell their capacity, and most likely, Cogent's customer who sells VPS services will also oversell their capacity. If Loyce is constantly sending hundreds of GB's worth of data to the internet, there will be less capacity for other customers to send their own data to the internet, which will degrade service for others.

The files that Loyce is hosting total over 660 GB. It would not take much for Loyce to run into hitting the multi TB range with files of that size, especially considering that it is trivial for someone to request those files multiple times.

Cogent and most providers like them as a rule do not oversubscribe, if anything they under subscribe.

They used to do it, but stopped around the GFC. No I don't think it's related, but around 2007 - 8 - 9 we just stopped seeing it, at least in the major DCs. I think there are just too many competitors. I can call Monday AM and have an agreement in a couple of days with another provider have the fiber connected a couple of days after that and be running a couple of days after that. So if I don't get my 1GB you're gone. Heck when Sandy took out people in NY loops were being turned up in hours as we all ran around the DCs getting stuff wired.

The people they sell to oversubscribe what they bought.

If you use a storage bucket on AWS or GCS, someone will need to pay $0.09/gb to transfer the file to the internet, or $0.01 to transfer your file to another data center run by the same cloud provider located on the same continent where your data is housed (or $0.00/gb -- free - within the same datacenter location). You can configure the storage bucket so that the person downloading the file will need to pay the transfer charges.

Up to $60 to download a few files. No wonder Bezos is rich

Till we got out of that business we used to joke Amazon is our best salesperson.
Their product is good, don't get me wrong, but it's super expensive and most people don't need it and a 2nd or 3rd tier provider is usually fine.
Not to mention, if your provender is not peered fast enough with Amazon they tend to be super slow since they prioritize amazon.com traffic over the aws traffic. In most major markets it's not a big deal. But in the middle of nowhere when your local ISP only has a 10GB connection to the peering network come Christmas time it can get bad.

Either way this is drifting from the main point of the thread, so I'm going to leave it alone for now.

Side note, do you know how many people are actually using this data, it's neat and all but outside of the geek-base on this forum I don't see a lot of people caring.

-Dave

[LoyceV]

I noticed "outputs" wasn't updating because the cronjob started when "transactions" was still running (and Blockchair.com only allows one connection at a time). It's updated now and daily updates should work again.

I used this data to check for myself if the Bitcoin blockchain has duplicate transaction hashes:

Code:

for day in `ls /var/www/blockdata.loyce.club/public_html/outputs/*gz`; do echo $day; cat $day | gunzip | cut -f 1-2 | grep -v transaction_hash > outputs.tmp; for block in `cat outputs.tmp | cut -f1 | sort -nu`; do grep -P "^$block\t" outputs.tmp | cut -f2 | awk '!a[$0]++' >> all_transaction_hashes_outputs.txt; done; rm outputs.tmp; done
# The awk-part removes duplicates within each block
cat all_transaction_hashes_outputs.txt | sort -S69% | uniq -d
d5d27987d2a3dfc724e359870c6644b40e497bdc0589a033220fe15429d88599
e3bf3d07d4b0375638d5f1db5255fe07ba2c4cb067cd81b84ee974b6585fb468

This confirmed 2 transactions have the same hash.

I tried to do the same for "inputs", but unfortunately I don't have the disk space to sort this large amount of data.

[PawGo]

Just a question - for the list of founded addresses, do you take in to account unconfirmed balance at the moment of snapshot or not? And opposite - if address with unconfirmed spending is still included or already not?

[LoyceV]

Just a question - for the list of founded addresses, do you take in to account unconfirmed balance at the moment of snapshot or not? And opposite - if address with unconfirmed spending is still included or already not?

The datadump comes from the blockchain only. Anything in mempool is ignored.

[LoyceV]

Would anyone be interested in block data for Ethereum? Apart from the disk space it takes (it's even larger than Bitcoin's block data), I wouldn't mind adding it to my collection.
I'm currently downloading it to compile a list of Eth addresses and their balance from this data. Amazingly, I couldn't find a complete list anywhere.

[LoyceV]

Would anyone be interested in block data for Ethereum?

Update: I have downloaded "calls", "erc-20" and "transactions". The total size is 712 GB, and I don't have the server space to store it on. I have it on my local storage (so I can upload it in about 2 days).

Unfortunately, 2 files are corrupted:

• blockchair_ethereum_calls_20200113.tsv.gz
• blockchair_ethereum_calls_20211110.tsv.gz

I contacted Blockchair, but they "haven't detected the issue".

I'm currently downloading it to compile a list of Eth addresses and their balance from this data. Amazingly, I couldn't find a complete list anywhere.

By now Blockchair has the complete Ethereum address list online: blockchair_ethereum_addresses_latest.tsv.gz. That ends my efforts to compile this list myself.

[PawGo]

In the list of founded addresses there is unnecessary line "address"  (kind of column header) between addresses "3.." and "bc..", line 31808561. Like you had separated lists and you concatenated them.

[LoyceV]

In the list of founded addresses there is unnecessary line "address"  (kind of column header) between addresses "3.." and "bc..", line 31808561. Like you had separated lists and you concatenated them.

Thanks! I'll take this to my other topic.

[PawGo]

Hi Loyce,

Would it be possible to prepare a single file with:
- all block hashes
- all transaction IDs

Or maybe you have such a script I may execute on full node. Otherwise I would have to download packs like from http://blockdata.loyce.club/ , decompress and parse tsv (which seems to be the simplest solution
Do you plan to backup /blocks/ folder from blockchair?

[LoyceV]

Would it be possible to prepare a single file with:
- all block hashes
- all transaction IDs

Or maybe you have such a script I may execute on full node.

What format are you looking for? Just a long list of hashes, or do you need to know which txid belong to which block?
I think the easiest way is to get the data from /transactions/ (55 GB) and /blocks/.

I can save you a 55 GB download if you get me the format you want.
To test, this is running now in /transactions/:

Code:

for file in `ls`; do echo "Now processing $file"; gunzip -c $file | grep -v 'block_id' | cut -f1-2 >> ../PawGo.tsv; done; gzip ../PawGo.tsv

See: blockdata.loyce.club/PawGo.tsv or blockdata.loyce.club/PawGo.tsv.gz. It's 27 GB now, hashes don't compress very well. It's scheduled to be deleted in 7 days.

Otherwise I would have to download packs like from http://blockdata.loyce.club/ , decompress and parse tsv (which seems to be the simplest solution

That's what I would do

Do you plan to backup /blocks/ folder from blockchair?

No need: these files are tiny, so downloading them from Blockchair directly shouldn't take too long anyway.

[PawGo]

See: blockdata.loyce.club/PawGo.tsv or blockdata.loyce.club/PawGo.tsv.gz. It's 27 GB now, hashes don't compress very well. It's scheduled to be deleted in 7 days.

Otherwise I would have to download packs like from http://blockdata.loyce.club/ , decompress and parse tsv (which seems to be the simplest solution

That's what I would do

Perfect, great, fantastic, thank you!
That's exactly what I was thinking about. I owe you beer & frites.

[LoyceV]

Perfect, great, fantastic, thank you!
That's exactly what I was thinking about.

Is it? It doesn't have the block hashes you asked for, only txids and block numbers.

I owe you beer & frites.

No worries, I've had my fair share today already

[PawGo]

That's exactly what I was thinking about.

Is it? It doesn't have the block hashes you asked for, only txids and block numbers.

Yes, I was just stunned by amount of data to download. I need only txids.
With block hashes it is indeed simpler.
(you may delete the file)

[LoyceV]

I just got a very nice Xeon-powered dedicated server (no more VPS!) from an anonymous donation, so I'm covered for now.

Update: the server is currenly offline for an upgrade With more disk space, I can add Ethereum data from Blockchair (which I have locally already), and later Dogecoin data too.

I've been playing with an idea: I want to make a graph of funded (potential) ChipMixer chips over the years, with daily data.

Assumptions:
I'll start by looking for addresses that received 0.512BTC. I'll exclude all addresses that received more than one transaction (ever), and I'll count the chips from the day they were funded until they day they're emptied. In Bitcoin's early years, before ChipMixer even existed, many potential chips were emptied the same day again. Those won't be counted.

Problem:
There's a lot of data, and I don't do databases. I was actually running out of disk space to sort the data, so this upgrade came at the right time.

[PawGo]

Update: the server is currenly offline for an upgrade With more disk space, I can add Ethereum data from Blockchair (which I have locally already), and later Dogecoin data too.

Hi, what's the state of ETH addresses dump? Is it uploaded already?

Problem:
There's a lot of data, and I don't do databases. I was actually running out of disk space to sort the data, so this upgrade came at the right time.

Do you want to talk about doing it in DB?

[LoyceMobile]

Hi, what's the state of ETH addresses dump? Is it uploaded already?

I have it, updated until June only, but not online yet. And I'm currently sailing so can't access it.
The data I meant is the full Ethereum transaction data, about 800 GB. That will take my home internet a while to upload.

Do you want to talk about doing it in DB?

I'm still a total noob, but would be good to learn.

[LoyceV]

Hi, what's the state of ETH addresses dump? Is it uploaded already?

All funded Ethereum addresses: ethereumdata.loyce.club/blockchair_ethereum_addresses_latest.tsv.gz (2.1 GB). I downloaded this from gz.blockchair.com/ethereum/addresses/ on July 21, 2022. There's currently no update available.

[LoyceV]

I stumbled upon something peculiar. Take blockchair_bitcoin_outputs_20220203.tsv.gz for example:

Code:

721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        11      2022-02-03 01:43:54     51200000        18970.623       bc1qt2kc82kr0wdyyyqns7qyvl377dap69ygzkpwmc      witness_v0_scripthash   00145aad83aac37b9a4210138780467e3ef37a1d1488    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        11      2022-02-03 01:43:54     51200000        18970.623       bc1qt2kc82kr0wdyyyqns7qyvl377dap69ygzkpwmc      witness_v0_scripthash   00145aad83aac37b9a4210138780467e3ef37a1d1488    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        11      2022-02-03 01:43:54     51200000        18970.623       bc1qt2kc82kr0wdyyyqns7qyvl377dap69ygzkpwmc      witness_v0_scripthash   00145aad83aac37b9a4210138780467e3ef37a1d1488    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        12      2022-02-03 01:43:54     51200000        18970.623       bc1qx9jm85e08jasw75g0drr7y2x9xx45xck6xvxhe      witness_v0_scripthash   00143165b3d32f3cbb077a887b463f1146298d5a1b16    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        13      2022-02-03 01:43:54     51200000        18970.623       bc1qmx0eraunssvj9ukel8m40cpt8ez4wxj4t2jn4q      witness_v0_scripthash   0014d99f91f793841922f2d9f9f757e02b3e45571a55    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        13      2022-02-03 01:43:54     51200000        18970.623       bc1qmx0eraunssvj9ukel8m40cpt8ez4wxj4t2jn4q      witness_v0_scripthash   0014d99f91f793841922f2d9f9f757e02b3e45571a55    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        13      2022-02-03 01:43:54     51200000        18970.623       bc1qmx0eraunssvj9ukel8m40cpt8ez4wxj4t2jn4q      witness_v0_scripthash   0014d99f91f793841922f2d9f9f757e02b3e45571a55    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        14      2022-02-03 01:43:54     51200000        18970.623       bc1qr4jgu3t5fnjrcux646kssfmavsw5zftmj4tsc6      witness_v0_scripthash   00141d648e45744ce43c70daaead08277d641d41257b    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        15      2022-02-03 01:43:54     51200000        18970.623       bc1qw6dnjhw8qjyxtszn950l5zh57x60tm5lcsdtkr      witness_v0_scripthash   0014769b395dc7048865c0532d1ffa0af4f1b4f5ee9f    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        15      2022-02-03 01:43:54     51200000        18970.623       bc1qw6dnjhw8qjyxtszn950l5zh57x60tm5lcsdtkr      witness_v0_scripthash   0014769b395dc7048865c0532d1ffa0af4f1b4f5ee9f    0       -1
721577  f2ec8c7f07725959014613a5cf04dde4cf3079c8948bc011298479e751935fc3        15      2022-02-03 01:43:54     51200000        18970.623       bc1qw6dnjhw8qjyxtszn950l5zh57x60tm5lcsdtkr      witness_v0_scripthash   0014769b395dc7048865c0532d1ffa0af4f1b4f5ee9f    0       -1

There are many duplicated lines! For the compressed filesize it doesn't matter much, but if I remove them, the number of lines drops by 1,046,856-879,157=167,699!
I checked more archives: the older ones have only a few duplicate lines, the newer archives have tens or hundreds of thousands of duplicates.

What could be the reason? And worse: it also makes me wonder if other entries could be missing.

[LoyceV]

Ethereum data

I've added ethereumdata.loyce.club:
ethereumdata.loyce.club/calls/ (329 GB)
ethereumdata.loyce.club/erc-20_transactions/ (363 GB)
ethereumdata.loyce.club/transactions/ (69 GB)

I needed this data from gz.blockchair.com/ethereum/ for a project (which I abandoned before my download completed). Since I now have the data, I'll share it (downloading 762 GB at 100 kB/s took forever 3 months).

Updates
New files are added daily.

Disclaimer
I don't like Ethereum, which is a centralized shitcoin that abandoned it's one unique selling point ("code is law") the moment it was convenient for the creator. Don't waste your money on it!
And since it's a shitcoin, I don't think it deserves it's own topic.

Missing files
Those 2 files are missing from my mirror because they're corrupted:
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20200113.tsv.gz
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20211110.tsv.gz

I sent Blockchair an email about it 3 months ago, but the response I got was this:

We've checked and haven't detected the issue with the file.

If anyone has the tools to rebuild those 2 files, I'd love to add them!

[LoyceV]

From how to dump or parse information from all available blocks on a full-node, I realized there's more block data missing from these data dumps.

Example:

Code:

bitcoin-cli getblock $(./bitcoin-cli getblockhash 700000) | head -n25
{
  "hash": "0000000000000000000590fc0f3eba193a278534220b2b37e9849e1a770ca959",
  "confirmations": 57334,
  "height": 700000,
  "version": 1073733636,
  "versionHex": "3fffe004",
  "merkleroot": "1f8d213c864bfe9fb0098cecc3165cce407de88413741b0300d56ea0f4ec9c65",
  "time": 1631333672,
  "mediantime": 1631331088,
  "nonce": 2881644503,
  "bits": "170f48e4",
  "difficulty": 18415156832118.24,
  "chainwork": "0000000000000000000000000000000000000000216dd8dc61fdffabb624feeb",
  "nTx": 1276,
  "previousblockhash": "0000000000000000000aa3ce000eb559f4143be419108134e0ce71042fc636eb",
  "nextblockhash": "00000000000000000002f39baabb00ffeb47dbdb425d5077baa62c47482b7e92",
  "strippedsize": 907224,
  "size": 1276422,
  "weight": 3998094,
  "tx": [
    "1d8149eb8d8475b98113b5011cf70e0b7a4dccff71286d28b8b4b641f94f1e46",
    "ed25927576988e38e4cc8e4b19d1272c480f113fb605271b190df05aa983714e",
    "744556a5586736471d496c928ccca8fd58dadac6071394eca846c180b0dec6fe",
    "adfcbcbd4f87a725337ab0b4eb657f97123806d30ccd50fa0c107b5124692e1d",
    "afe5de49b7a84bb5d79d114601d81645264ebb4fcb8e1b45c280f6d788a8a7ba"
~

If anyone wants a dump of this data, just ask. I don't mind adding it.

[citb0in]

Hi Loyce and thanks for adding "merkleroot" to your data. Would you mind to add also the rest ?

Code:

 "hash": "0000000000000000000590fc0f3eba193a278534220b2b37e9849e1a770ca959",
  "version": 1073733636,
  "versionHex": "3fffe004",
  "merkleroot": "1f8d213c864bfe9fb0098cecc3165cce407de88413741b0300d56ea0f4ec9c65",
  "time": 1631333672,
  "mediantime": 1631331088,
  "nonce": 2881644503,
  "bits": "170f48e4",
  "difficulty": 18415156832118.24,
  "chainwork": "0000000000000000000000000000000000000000216dd8dc61fdffabb624feeb",
  "nTx": 1276,
  "previousblockhash": "0000000000000000000aa3ce000eb559f4143be419108134e0ce71042fc636eb",
  "nextblockhash": "00000000000000000002f39baabb00ffeb47dbdb425d5077baa62c47482b7e92",
  "strippedsize": 907224,
  "size": 1276422,
  "weight": 3998094,

[LoyceV]

Hi Loyce and thanks for adding "merkleroot" to your data. Would you mind to add also the rest ?

I'll create a TSV with this:

Code:

block   hash    version versionHex      merkleroot      time    mediantime      nonce   bits    difficulty      chainwork       nTx     strippedsize    size    weight

I left out the previous and next block hash to reduce the size.

It should be done in about 11 hours, which means I'll share the output tomorrow. I expect the file to be around 150 MB, so I didn't bother making smaller daily files.
Once done, this will get daily updates, up to the same block as Blockchair's data dumps (so not the latest blocks from my own Bitcoin Core).

Update
block_hash_version_versionHex_merkleroot_time_mediantime_nonce_bits_difficulty_ chainwork_nTx_strippedsize_size_weight.tsv.gz (80 MB).

I've updated the OP

[LoyceV]

I've added ethereumdata.loyce.club Abandoned! I'm out of disk space. I have a local backup, if you really need this: ask!
ethereumdata.loyce.club/calls/ (345 GB)
ethereumdata.loyce.club/erc-20_transactions/ (378 GB)
ethereumdata.loyce.club/transactions/ (71 GB)

I only added this centralized shitcoin to utilize the 2 TB storage disk, but adding several gigabytes per week filled it. This gives enough space for a long time again.

[Zensi]

pls add all bch addresses

[LoyceV]

pls add all bch addresses

You can find the data at gz.blockchair.com/bitcoin-cash/. I don't really want to add hundreds of GB of Forkcoin data to my server. If you need help downloading or processing the data, feel free to make me an offer.

[LoyceV]

Bump!

Zensi never made me an offer. Blockchair also has BCH, Dash, Doge, LTC or Zcash data dumps available (for a very slow download).

[COBRAS]

Bump!

Zensi never made me an offer. Blockchair also has BCH, Dash, Doge, LTC or Zcash data dumps available (for a very slow download).

Bro, in what files on your server  r,s,z data of BTC ?  In files with transactions, outupts or

I looking for bulk data , not json etc....


Thank.You.

[LoyceV]

Bro, in what files on your server  r,s,z data of BTC ?  In files with transactions, outupts or

See the samples in the OP. Or download one file from each directory. If you find it, post it here.

[COBRAS]

Bro, in what files on your server  r,s,z data of BTC ?  In files with transactions, outupts or

See the samples in the OP. Or download one file from each directory. If you find it, post it here.

This data from input transactions data, file name  

blockchair_bitcoin_inputs_20100304.tsv.gz

This file with r,s,z ?


block_id   transaction_hash   index   time   value   value_usd   recipient   type   script_hex   is_from_coinbase   is_spendable   spending_block_id   spending_transaction_hash   spending_index   spending_time   spending_value_usd   spending_sequence   spending_signature_hex   spending_witness   lifespan   cdd      
42495   44f339ccc78c6395cca0163a5a3da7ff4b76bff329ec8e1f45996859717385ac   0   2010-02-25 9:22:30   5000000000   0.5   1K5ULJgsJR1xQsSDmPsFCFsFgEmKkMtNQP   pubkey   4104127c91de8d72c7e1cb21cbf784b074e3a4d556c8ca8f17b046a25dbbbb4e7b05a3881cefcfe 52475877ea654fcebe7901be53a0bef042050905c7aa21293b137ac   1   1   43736   0aaad829c52080b76292ebb168f684f264c5d4a092643063afaddacfb83e6553   0   2010-03-04 14:25:07   0.5   4294967295   493046022100a738f0a95a5ffdcfd688e33a05d2d118c0cba2bb994a5cad94ab0c59027c5542022 100d81a0d56454abd9ea240ce4c02c1fcda29c68216d4558ab96149fa65379a26e901      622957   360.5075231      
42365   687f509be527da767f8d44332111221a92e951c1b2d6f64e5d9539e2af4efa29   0   2010-02-24 12:26:00   5000000000   0.5   13izg2rAiGaFFpUCqBKeUm8BSohqfJpoS4   pubkey   41041e99b07917e497b0e98215fa05ffb926aa1b3ec507b4d621a584c4bf093828433858d9e63b7 19ec7e838f539ac5a9fecad99bdf099393d1b670270cc46733350ac   1   1   43759   15293ee1b14f9a2c5041b8952e89aa6eca455c7a08a9f9048fbadbd5bae1c60b   0   2010-03-04 17:36:55   0.5   4294967295   473044022015e67872d04797f7ac31dd59750c670954522bea767ab37bd3c1a710dcfcbbab02205 cd8bc98e6070cf0afed8968f8ef5f3ada1f51b9a4d0bb817b3a70e035d4d4c201      709855   410.7957176      
42017   3304b670a5e5a864e2ad416b52300cf355038c67789c199775cb4bfa4ad8ddc2   0   2010-02-22 23:32:50   5000000000   0.5   1PiuHkfNomTJRnMbywnNjhrgbHDJNPWkEc   pubkey   4104b205906f0304ae29c994125279f4be7b2de3ab2bddbf8607ad24f4d627bbbeb9fc076159d2d 5a87f4bbf418bc2bcfe8ba06dda1512061903bcb00842a2a627fcac   1   1   43759   15293ee1b14f9a2c5041b8952e89aa6eca455c7a08a9f9048fbadbd5bae1c60b   1   2010-03-04 17:36:55   0.5   4294967295   49304602210083ffc2fd4853517c9a51c3744b0aa8bf3700e2822d74813ae6b75053d6372bfd022 1009466c70a0180688e459da015357bbc2ab9aeb0de3beac0cb1efaba623eca30fe01      842645   487.6417824      
43034   3c34970dba0eeb2960001e77d667d3e9df00da0ba0c0351bf4af10ef9b67e4ce   0   2010-02-28 14:51:06   5000000000   0.5   14hJYeFAjckhuzpJkUj6mont2yZheizUXt   pubkey   4104bc6235e6ad560ab0f1f09120454948755a055ba591db3019302b11732637a53155315057a15 b7de9afddd5ce4e722f59c6fc46ac87bffd827a84e7340985ac68ac   1   1   43759   15293ee1b14f9a2c5041b8952e89aa6eca455c7a08a9f9048fbadbd5bae1c60b   2   2010-03-04 17:36:55   0.5   4294967295   483045022100c683817cc7010389e185144a7fe411af7b8564ca8a13dbf9c27dcd1986685afc022 01fae568aa4be674f993957f4030119569cd6e06ae87894a67a5b9bf44d17833101      355549   205.7575231      
43567   eafb39f3340676d900190badabb898756e2862594c280927061d8ded5d3c1343   0   2010-03-03 14:45:17   5000000000   0.5   17FdnuFaT8wuuBSyaJDBPZ3cLLgr8bpVzn   pubkey   4104780f11944bbabef7753de80994abd0fb01f739cd461c317c805ccaaac82908955a49bc5e1b9 c1c2a10c0713c972b531870f6e2d20bb9565316b9c0dbbba1af9aac   1   1   43759   15293ee1b14f9a2c5041b8952e89aa6eca455c7a08a9f9048fbadbd5bae1c60b   3   2010-03-04 17:36:55   0.5   4294967295   47304402204b19320ece79b7a3b4a70eacffbd22828468fb682822aa5693fa5084e05990aa02207 02899a2ea9781f38a89ac5d56ce3824bc19cd15ca9a3b4e5611600cb10d165701      96698   55.95949074      
43256   529e358c5a5bab4f8aedad3e1c304bc49dfa0b69b439b49f15589567e3dbb0c3   0   2010-03-01 20:01:46   5000000000   0.5   1MKDNdVs2zRY75LJh5DYaF954AgwtYAwy4   pubkey   4104615a801f9da7e4365768009b1606fd13ae8df9187ce79759d7e23a021a3e73e950555df6213 9d4f252e9d01a60990f07c78cb1ecab1ae241c7308b79e6948f26ac   1   1   43759   15293ee1b14f9a2c5041b8952e89aa6eca455c7a08a9f9048fbadbd5bae1c60b   4   2010-03-04 17:36:55   0.5   4294967295   483045022100d57a8cc989301c0c5c3b3c47dc38bd3c11d5eef69d8f83d72c540a659760c210022 017a4b4b6e4ec0ee37a6655186dce4bb564b2ed285c317d7b9a87666469b5f8cd01      250509   144.9704861      
43569   43ccf58532ba0d51c2b3ad71066417c7473c6b6f4e70e4f8ed0c665b758eb2d8   0   2010-03-03 15:06:12   5000000000   0.5   18ePkq5kbYnHMBTr6VMEcsPm6DzxXAnfuR   pubkey   410443bd83a3c18343736b26511d7225bd63b5ba3a5ef032c097bddaa1426d31318ed0c1c5b6168 236a967acf94d3621011bf09e638af913e6dab00098b719425d04ac   1   1   43759   15293ee1b14f9a2c5041b8952e89aa6eca455c7a08a9f9048fbadbd5bae1c60b   5   2010-03-04 17:36:55   0.5   4294967295   483045022100ae71a3f8516280088d441ac20ac8d9c1f33bd6a05385c96d809599965b8e66ba022 07ae75074af73903209a7c1d9cc9cefcccb4f374b7cb6fc6020bd818d0adf3ff201      95443   55.23321759      
38596   cb3d1c126178ab5d560f12a966f3b592e705773e08c3c1177005f16a17e4d875   0   2010-02-06 6:02:51   5000000000   0.5   1PmVA4Hnf3a8h8NzPtkXpWS4xvxa2o4Kiz   pubkey   4104c78455c3267666e8e7a9e2c6ac4f5d1225c2556a50d9b79d6f6d97bd4413f7c463ce6b31863 2a40f4004cc36cc0f319c2e0c05de3bf54d4c1c8365699bcc65f6ac   1   1   43789   4681ae540cf953bf69ac890af6eba3647dcd3469b5f20052723a3ec3414ec2a6   0   2010-03-04 20:34:45   0.5   4294967295   493046022100ee23a1d3cad9ac03b5aa6845ebe68615b52d4a5830d40dd838015f67690291f6022 100aa702b9d4490ed255b6a641f00e08a612ff242e379fc20b6112d32c7305a0e8b01      2298714   1330.274306      
31271   e3077f5229f41ec6db52ec5c3308666ec1b64e4e5e3f303a8873ac84122fe275   0   2009-12-24 1:05:46   5000000000   0.5   16ECYkGoqCw4SeXvGbbPmQzjqCggjobige   pubkey   41048205e75a1fb24902b71c0c1f49b993c1e0ff31dcd8bd636873eecd1a9837287f70a2fe13ece c492ab8541340f8a19558bb1dbb6752d984ffc6f29533ea83ba5dac   1   1   43789   4681ae540cf953bf69ac890af6eba3647dcd3469b5f20052723a3ec3414ec2a6   1   2010-03-04 20:34:45   0.5   4294967295   48304502205853639a66e15d0dd805de9841a5a974892b9869fd2fcf894f89c3721efdc8e602210 0ab2d66be1b4c183587ae481e81e6143a88e66b2ef0453c65fc0e282329fec93701      6118139   3540.589699   

Bro, in what files on your server  r,s,z data of BTC ?  In files with transactions, outupts or

See the samples in the OP. Or download one file from each directory. If you find it, post it here.

I find, r,s,z data in input tx files, not in output.


Thank you   

[Vod]

Bro, in what files on your server  r,s,z data of BTC ?  In files with transactions, outupts or

See the samples in the OP. Or download one file from each directory. If you find it, post it here.

Bro, you have infinite patience.   

[LoyceV]

If anyone wants this data, speak up NOW:

Ethereum data

I've added ethereumdata.loyce.club Abandoned! I'm out of disk space. I have a local backup, if you really need this: ask!
ethereumdata.loyce.club/calls/ (345 GB)
ethereumdata.loyce.club/erc-20_transactions/ (378 GB)
ethereumdata.loyce.club/transactions/ (71 GB)

All funded Ethereum addresses: ethereumdata.loyce.club/blockchair_ethereum_addresses_latest.tsv.gz (5.1 GB).  Blockchair doesn't updated this file regularly, so I don't do automatic updates.

Disclaimer
I don't like Ethereum, which is a centralized shitcoin that abandoned it's one unique selling point ("code is law") the moment it was convenient for the creator. Don't waste your money on it!
And since it's a shitcoin, I don't think it deserves it's own topic.

Missing files
Those 2 files are missing from my mirror because they're corrupted:
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20200113.tsv.gz
https://gz.blockchair.com/ethereum/calls/blockchair_ethereum_calls_20211110.tsv.gz
If anyone has the tools to rebuild those 2 files, I'd love to add them!

I want my disk space back, so I'll wipe this 855 GB soon. It took a lot of time to download, so if anyone wants a copy for some reason, better speak up now

[LoyceV]

This project wasn't updated for over a week. I think my other download got so big, it was still running at the time this cronjob started. And Blockchair only allows one connection at a time, so it was rejected every day.
I'm now doing a manual update, and from now on updates will happen 4 hours later. That should give me many years before this same problem will occur.

[LoyceV]

Update!

After my (sponsored) dedicated server disappeared last week, I had to rush to get a new server. I'm still in the process of restoring data. From all my large data projects, this is the first one to be back online. This data reached 1 TB now. I've updated the sizes in this quote:

See blockdata.loyce.club
blockdata.loyce.club/inputs/ (714 GB)
blockdata.loyce.club/outputs/ (221 GB)
blockdata.loyce.club/transactions/ (79 GB)
NEW! Bandwidth
This server is allowed unlimited bandwidth, at 250 Mbit/s. I choose this instead of 10 Gbit limited at 32 TB/month.

VPS server specs:
8 cores AMD EPYC 7551P
32 GB RAM
1 TB NVMe
2 TB SAN (Storage Area Network)
I paid $220 for a year.

Let me know if anything fails.

block_hash_version_versionHex_merkleroot_time_mediantime_nonce_bits_difficulty_ chainwork_nTx_strippedsize_size_weight.tsv.gz is still updating.

[LoyceV]

Update:

block_hash_version_versionHex_merkleroot_time_mediantime_nonce_bits_difficulty_ chainwork_nTx_strippedsize_size_weight.tsv.gz (94 MB)

It took 2 days to get this data from Bitcoin Core, but it's ready. Daily updates should be okay again.

[ABCbits]

VPS server specs:
8 cores AMD EPYC 7551P
32 GB RAM
1 TB NVMe
2 TB SAN (Storage Area Network)
I paid $220 for a year.

Out of curiosity, what VPS do you use? While i can easily find VPS which offer 8 core/32GB RAM at slightly higher price, usually it comes with much smaller storage.

[Vod]

Out of curiosity, what VPS do you use? While i can easily find VPS which offer 8 core/32GB RAM at slightly higher price, usually it comes with much smaller storage.

Use storage from a cloud provider - you'll get better results than a shared disk, unless your VPS is doing heavy computations on the data.    For a VPS, you should buy the best processor/RAM you can afford.

[LoyceV]

Out of curiosity, what VPS do you use?

I used this Black Friday offer, or more specifically: 8 Slices of this one + 2 TB extra SAN storage.
It says "Dedicated Core", but I had my doubts about that, considering the price. Support told me I can use the CPUs to the max, but after many hours of Trust data processing, I saw 60% CPU steal this morning. That's unfortunate, but doesn't really matter for what I need. I'll adjust my usage a bit so it takes a bit longer at lower CPU load. I don't mind if the oldest pages have to wait a bit longer to update their navigation bar, but I'd like other tasks to not be delayed too much.
According to Lowendtalk, the host has a good reputation. All in all I think it's a good deal.

While i can easily find VPS which offer 8 core/32GB RAM at slightly higher price, usually it comes with much smaller storage.

That is what convinced me to get this one. 2 TB slower storage and 1 TB fast storage is a good combination for my intended use.

Use storage from a cloud provider - you'll get better results than a shared disk, unless your VPS is doing heavy computations on the data.

As far as I know, they get you on bandwidth charges. On my old (dedicated) server, I used about 10-20 TB per month. On this server, I'm at 4 TB in 5 days. That's why I choose 250 Mbit unlimited instead of 10 Gbit with a 32 TB bandwidth limit.

[ABCbits]

Out of curiosity, what VPS do you use?

I used this Black Friday offer, or more specifically: 8 Slices of this one + 2 TB extra SAN storage.
It says "Dedicated Core", but I had my doubts about that, considering the price. Support told me I can use the CPUs to the max, but after many hours of Trust data processing, I saw 60% CPU steal this morning. That's unfortunate, but doesn't really matter for what I need. I'll adjust my usage a bit so it takes a bit longer at lower CPU load. I don't mind if the oldest pages have to wait a bit longer to update their navigation bar, but I'd like other tasks to not be delayed too much.
According to Lowendtalk, the host has a good reputation. All in all I think it's a good deal.

While i can easily find VPS which offer 8 core/32GB RAM at slightly higher price, usually it comes with much smaller storage.

That is what convinced me to get this one. 2 TB slower storage and 1 TB fast storage is a good combination for my intended use.

It's really cheap. Although 60% CPU steal (means you get 40% of CPU computing power ?) and very few online discussion about serverica is somewhat concerning. As for about "Dedicated Core", lowendtalk website claim it only applies to VDS (not VPS) while serverica website claim it also applies to their VPS.

[LoyceV]

It's really cheap. Although 60% CPU steal (means you get 40% of CPU computing power ?) and very few online discussion about serverica is somewhat concerning. As for about "Dedicated Core", lowendtalk website claim it only applies to VDS (not VPS) while serverica website claim it also applies to their VPS.

I also suspect to not get 8 full cores, but 8 half or maybe even 8 quarter cores. They're quite a lot slower than my (older) Xeon server, while these are AMD EPYC 7551P cores.
The CPU steal was kinda expected, at least they're not suspending me. I used to process all my Trust updates a lot faster, now it takes over a day. But that's okay, as there are 6 more days left in the week and I'm not in a rush (for this).
My Merit update also takes longer, it used to finish on Sunday, but it's still running. It only uses one core, and the CPU steal ended. So if hammering all 8 cores for many hours in a row leads to 60% steal, I can live with that.
I've been screwed by disappearing servers in the past. In this case, I could have paid per month to reduce the risk, but took the risk out of convenience. As for the lack of online discussion: servarica_hani joined Lowendtalk in 2013.
For what it's worth: I've improved my backup strategy. I now make backups of everything.

[LoyceV]

Blockchair missed a day:

Code:

blockchair_bitcoin_outputs_20250105.tsv.gz         05-Jan-2025 23:59     66M
blockchair_bitcoin_outputs_20250106.tsv.gz         06-Jan-2025 23:59     69M
blockchair_bitcoin_outputs_20250107.tsv.gz         07-Jan-2025 23:59     67M
blockchair_bitcoin_outputs_20250108.tsv.gz         08-Jan-2025 23:59     73M
blockchair_bitcoin_outputs_20250109.tsv.gz         09-Jan-2025 23:59     80M
blockchair_bitcoin_outputs_20250110.tsv.gz         10-Jan-2025 23:59     73M
blockchair_bitcoin_outputs_20250111.tsv.gz         11-Jan-2025 23:59     63M
blockchair_bitcoin_outputs_20250113.tsv.gz         13-Jan-2025 23:59     69M

I've never seen that before. If they don't correct this, it leaves a permanent hole in this data, which also means my list of all Bitcoin addresses ever used will be incomplete forever.

Update: it's okay again:

Code:

blockchair_bitcoin_outputs_20250105.tsv.gz         05-Jan-2025 23:59     66M
blockchair_bitcoin_outputs_20250106.tsv.gz         06-Jan-2025 23:59     69M
blockchair_bitcoin_outputs_20250107.tsv.gz         07-Jan-2025 23:59     67M
blockchair_bitcoin_outputs_20250108.tsv.gz         08-Jan-2025 23:59     73M
blockchair_bitcoin_outputs_20250109.tsv.gz         09-Jan-2025 23:59     80M
blockchair_bitcoin_outputs_20250110.tsv.gz         10-Jan-2025 23:59     73M
blockchair_bitcoin_outputs_20250111.tsv.gz         11-Jan-2025 23:59     63M
blockchair_bitcoin_outputs_20250112.tsv.gz         12-Jan-2025 23:59     63M
blockchair_bitcoin_outputs_20250113.tsv.gz         13-Jan-2025 23:59     69M
blockchair_bitcoin_outputs_20250114.tsv.gz         14-Jan-2025 23:59     69M
blockchair_bitcoin_outputs_20250115.tsv.gz         15-Jan-2025 23:59     69M