Bitcoin block data (728 GB): inputs, outputs and transactions

[LoyceV]

how do you create the torrent?

I have a Seedbox at PulsedMedia.com, which by default uses ruTorrent. I clicked "Create Torrent", selected the data, and added some trackers that I found in other torrents. Then, I set the Piece size as large as possible (16 MB, because it's 50+ GB of data). After a couple of minutes, it was ready and I could save the .torrent file.

In past i used mktorrent (to create torrent) and qbittorrent (on both server and client to upload/download) to transfer data due to unstable connection on my end. I don't use any tracker, but i disable private flag so i could use DHT. However, my data size was smaller.

When creating the torrent, ruTorrent doesn't have the option to use DHT.

[1714956843]

1714956843

[1714956843]

1714956843

[1714956843]

1714956843

[1714956843]

1714956843

[1714956843]

1714956843

[ABCbits]

how do you create the torrent?

I have a Seedbox at PulsedMedia.com, which by default uses ruTorrent. I clicked "Create Torrent", selected the data, and added some trackers that I found in other torrents. Then, I set the Piece size as large as possible (16 MB, because it's 50+ GB of data). After a couple of minutes, it was ready and I could save the .torrent file.

1. ruTorrent (which use rTorrent as backend) is reliable (according to discussion on different forum), although it's development has slowed down in past 3 years. However it should able to create torrent without problem.
2. You might want to check this thread about torrent piece size, https://torrentinvites.org/f29/piece-size-guide-167985/.
3. Take note not all tracker accept any kind of torrent.

In past i used mktorrent (to create torrent) and qbittorrent (on both server and client to upload/download) to transfer data due to unstable connection on my end. I don't use any tracker, but i disable private flag so i could use DHT. However, my data size was smaller.

When creating the torrent, ruTorrent doesn't have the option to use DHT.

Really? It's weird since rTorrent (the backend) support DHT. Can you use different tool (which support DHT) on your Seedbox?

[LoyceV]

Really? It's weird since rTorrent (the backend) support DHT.

These are the only options it gives:

Can you use different tool (which support DHT) on your Seedbox?

I can start the following:

Code:

Deluge
Rclone
qBittorrent

I haven't tried those yet.

2. You might want to check this thread about torrent piece size, https://torrentinvites.org/f29/piece-size-guide-167985/.

I read about that already, it's a compromise. I could select a smaller piece size, but that means it has many pieces to deal with. I'll try 4MB next time.

3. Take note not all tracker accept any kind of torrent.

That's the tricky part: I couldn't find much about it, only a list of recommended trackers. That's what makes me think the slow download start might be related to the trackers.

[seoincorporation]

Hello LoyceV.

After reading the thread I see you are having problems with the amount of data and the way to share it, and I was thinking about a solution or how I would do it.

You need a VPS with a huge disk and unlimited band bandwidth, and those aren't cheap at all, but I know a cheap way to handle this.

You can buy a laptop (workstation) with a 1TB SSD and configure it as a server with NO-IP. This way your bandwidth will be limited only by upload and download speed, but not for the amount of data. And you can run the server in your house.

If you like the idea and want some help with the server configuration just let me know. 

[LoyceV]

You need a VPS with a huge disk and unlimited band bandwidth, and those aren't cheap at all

The Seedbox actually isn't that bad: It has 6TB monthly bandwidth, and when it reaches the limit it continues at 100 Mbit/s.

You can buy a laptop (workstation) with a 1TB SSD and configure it as a server with NO-IP. This way your bandwidth will be limited only by upload and download speed, but not for the amount of data. And you can run the server in your house.

There are several problems with that: First, I don't want to expose my home IP address for this. And I don't want my upload bandwidth to be maxed out for several days in a row when someone downloads 600 GB.
Third problem is that I don't think my internet provider would appreciate it if I upload a few TB each month, so they'll probably start throttling my bandwidth. And on top of that, given current electricity prices, running a system 24/7 is more expensive than the Seedbox.

If I can't get the Torrents to work, the files can still be downloaded directly through http, but I haven't even finished uploading everything yet. And if barely anyone is going to use this, I might just have to give up on this project entirely at some point.

[LoyceV]

Since it include option "Private Torrent", usually it means DHT is used by default. And when i tried it on my VPS (where i remove all tracker before start the torrent), the DHT is working and i can start the download.

To test this, I tried to create a new one with "Private torrent" enabled, and 4MB piece size. It didn't show up, so I deleted the existing one first. It still didn't show up. I'm now testing without "Private" again: this works.

Yeah, usually i just rely on DHT and share my IP:Port (if needed) so other user can it directly to list of peers on their torrent software.

How would that IP/port-sharing work? Usually I just click the Torrent or Magnet link, and don't need to manually add IPs.

My VPS have spare bandwidth, do you want me to try to see whether i can complete the download?

Feel free to test it

[LoyceV]

But it's not practical option.

Indeed. I don't want users to manually add IPs.

Testing it now.

Only one out of 2 Seedboxes is currently uploading (at a very decent speed).

I'm uploading the rest of "inputs", after that I'll fix the file dates and see if I can get a more permanent solution for hosting those files.

[LoyceV]

1. I remove tracker on both torrent before i start download process.

Why? And how? When I open a torrent in Transmission, it just starts and doesn't ask to remove trackers.

2. "output" downloaded started some time (about 15 - 30 minutes) after i add it.

I'm used to only downloading Torrents with many seeders, and those usually start within seconds. I have no idea why this takes so long. The Seedbox doesn't seem to lack capacity.

3. Average speed was 24-30MB/s. Although "transaction" have 2x speed after it connected to 2 seeder.

I think "outputs" shifted seeding to the other Seedbox at some point. I expected both to seed simultaneously.

4. I use qbittorent-nox with default configuration.

CLI? Even though I use CLI for most things, I've never used it for Torrent.

I think i can conclude that you made the torrent files correctly and there's no problem with your seedbox.

Thanks! I have about 100 GB upload to go, I expect it to be done tomorrow. After that, I'll create 3 brand new Torrent files with the most complete data (including correct file dates). I'll also schedule daily downloads to update the latest files again, update OP, and add the new download link.
I won't update the Torrent file that often, which means it'll miss the latest days (or weeks, months....).

[ABCbits]

Why?

Because i wanted to test whether DHT works.

And how? When I open a torrent in Transmission, it just starts and doesn't ask to remove trackers.

On qbittorrent-nox, you can choose not to start torrent after adding it. It has tracker tab where you can remove each tracker manually.

I'm used to only downloading Torrents with many seeders, and those usually start within seconds. I have no idea why this takes so long. The Seedbox doesn't seem to lack capacity.

I have no idea either, but the weird thing is "transaction" torrent start within few seconds.

CLI? Even though I use CLI for most things, I've never used it for Torrent.

I only use CLI to start the program, qbittorrent-nox offer web UI.

[LoyceV]

I've finalized my update, and after 10 months, I'm happy to announce This project is back online!

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

Feel free to test (and/or seed) the torrents if you have some bandwidth and disk space to waste

I haven't tested the daily updates yet. Let me know if anything needs my attention.

[PawGo]

I've finalized my update, and after 10 months, I'm happy to announce This project is back online!

New! Torrents!
inputs.torrent
outputs.torrent
transactions.torrent
For privacy, you may want to consider using a VPN so other users can't see your IP address.

Torrents works well now, download is fast. Is it normal that only one tracker returns 'success' and peers data?

[LoyceV]

Torrents works well now, download is fast.

I still don't get it: my own test (from another Seedbox) hasn't downloaded anything after an hour.

Is is normal that only one tracker returns 'success' and peers data?

That's probably not good, but I'm not sure how to change to fix it.
I (arbitrarily) selected those trackers:

Code:

udp://tracker.opentrackr.org:1337/announce
udp://denis.stalker.h3q.com:6969/announce
udp://opentracker.i2p.rocks:6969/announce
udp://tracker.internetwarriors.net:1337/announce
udp://tracker.coppersurfer.tk:6969/announce

And indeed, only the first one seems to recognize the torrent.

[PawGo]

Torrents works well now, download is fast.

I still don't get it: my own test (from another Seedbox) hasn't downloaded anything after an hour.

For me it works instantly.

Is is normal that only one tracker returns 'success' and peers data?

That's probably not good, but I'm not sure how to change to fix it.
I (arbitrarily) selected those trackers:

Code:

udp://tracker.opentrackr.org:1337/announce
udp://denis.stalker.h3q.com:6969/announce
udp://opentracker.i2p.rocks:6969/announce
udp://tracker.internetwarriors.net:1337/announce
udp://tracker.coppersurfer.tk:6969/announce

And indeed, only the first one seems to recognize the torrent.

[LoyceV]

For me it works instantly.

That's something Overnight, "transactions" completed downloading on the other Seedbox, and later "outputs" did the same. I didn't try "inputs" due to disk space limitations. So I think I can conclude it works, and it may or may not start instantly.
I'll stop fiddling and let it run for now, anyone who's in a rush can always go for the http download.

[LoyceV]

Back online! The data
See this new location instead of blockdata.loyce.club
le6-1-81flash.pulsedmedia.com/public-loyceclu/inputs/ (469 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/outputs/ (145 GB)
le6-1-81flash.pulsedmedia.com/public-loyceclu/transactions/ (52 GB)

Barely 3 days later, I received an email:

There is a backend software glitch which can cause all data on /home to be removed.

I still have "outputs" and "transactions" on the other server, but this means I'll have to upload 469 GB from my local backup again. It also means they've moved me to another server, which means the above links are no longer valid.
I'm not amused.

[yogg]

I'm not amused.

I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ?


More seriously, I can help hosting "balances" if that interests you.
I will need this archive for a personal project anyway, it's not a big deal to make it available on port 80.

I have some extra storage space avail on my setup.

Let me know, I can provide an FTP access to you.

[LoyceV]

I don't see why you're upset.
When uploading, I usually like to count the blinks on my switch and compare with the amount of packets really circulating on the wires.

Much fun. Don't you do that ?

Lol. I'll have to stand in the hallway to count the blinks.

I'm still uploading data, but I'm not sure if I'll extend this hosts contract. It's now spitting out hardware errors:

Code:

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.842800] [Hardware Error]: Corrected error, no action required.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.846023] [Hardware Error]: CPU:8 (15:1:2) MC4_STATUS[-|CE|MiscV|-|AddrV|-|CECC]: 0x9c44c00001080a13

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.849212] [Hardware Error]: Error Addr: 0x0000001f8d366350

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.852463] [Hardware Error]: MC4 Error (node 1): DRAM ECC error detected on the NB.

Message from syslogd@le6-1-93ghostman at Mar 15 08:42:07 ...
 kernel:[220101.855864] [Hardware Error]: cache level: L3/GEN, mem/io: MEM, mem-tx: RD, part-proc: RES (no timeout)

More seriously, I can help hosting "balances" if that interests you.

Which "project" do you mean exactly? List of all Bitcoin addresses with a balance? That one is covered, it only takes 20 GB to host. Although by now the 2 TB monthly bandwidth isn't enough anymore, at day 15 it already used 1.2 TB so it'll go offline at the end of the month.
If you happen to have 700 GB available, by all means, share "inputs, outputs and transactions"

Let me know, I can provide an FTP access to you.

FTP, scp, rsync, anything works for me

[LoyceV]

People still use FTP these days? I remember using FTP with FileZilla long time ago.

I still use FTP on my local network, where I don't mind the lack of encryption.

I would demand partial/full refund.

It's paid per month, and really not worth the effort. I'd much rather have them fix the problems so I can finally keep this data online. I've uploaded it too many times already, and the whole point of having it on a server is so I can access it when needed.
If anyone finds a good offer on 1 TB storage, feel free to post it.

[PawGo]

If anyone finds a good offer on 1 TB storage, feel free to post it.

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.
https://www.hetzner.com/storage/storage-box

[LoyceV]

Did you consider Hetzner? From 3.45 for 1T, unlimited traffic.

I need a VPS, this looks a bit like "shared hosting". I guess I wasn't specific enough. It could work, but having to upload updates instead of just processing them on the server is far from ideal.
A bigger deal breaker is the fact that Hetzner doesn't accept Bitcoin. I don't want to link my credit card to anything crypto-related. From what I've read, I'd expect Hetzner to offer good quality hosting.