Bitcoin Forum
September 24, 2021, 10:25:53 PM *
News: Latest Bitcoin Core release: 22.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 [3]  All
  Print  
Author Topic: Bitcoin block data available in CSV format  (Read 1597 times)
This is a self-moderated topic. If you do not want to be moderated by the person who started this topic, create a new topic. (2 posts by 2+ users deleted.)
bitmover
Legendary
*
Online Online

Activity: 1344
Merit: 2765



View Profile WWW
January 04, 2021, 12:53:57 PM
 #41

Since it's your own server blocking you: is there another way to work around this?

Quote
Can you enable https for your address.loyce.club?
I now use 3 different (cheap) servers for 3 different subdomains (addresses.loyce.club, alladdresses.loyce.club and work in progress blockdata.loyce.club), and I've already replaced several older (also cheap) servers. I'm a bit hesitant to enable https on all of them, as it will make it more complicated to migrate to a different server again.
So I can enable it, but I'd rather not Tongue

LoyceV, I have tried many things. I made all types of requests I know and all of them failed.

I made a brief research about this topic, and it is not my server which is blocking: it is the browser. All modern browsers will not allow mixed content of HTTPS and HTTP

Quote
What is mixed content?

Mixed content occurs when initial HTML is loaded over a secure HTTPS connection, but other resources (such as images, videos, stylesheets, scripts) are loaded over an insecure HTTP connection. This is called mixed content because both HTTP and HTTPS content are being loaded to display the same page, and the initial request was secure over HTTPS.

Requesting subresources using the insecure HTTP protocol weakens the security of the entire page, as these requests are vulnerable to on-path attacks, where an attacker eavesdrops on a network connection and views or modifies the communication between two parties. Using these resources, attackers can track users and replace content on a website, and in the case of active mixed content, take complete control over the page, not just the insecure resources.

Although many browsers report mixed content warnings to the user, by the time this happens, it is too late: the insecure requests have already been performed and the security of the page is compromised.

This is why browsers are increasingly blocking mixed content. If you have mixed content on your site, then fixing it will ensure the content continues to load as browsers become more strict.
https://web.dev/what-is-mixed-content/

1632522353
Hero Member
*
Offline Offline

Posts: 1632522353

View Profile Personal Message (Offline)

Ignore
1632522353
Reply with quote  #2

1632522353
Report to moderator
1632522353
Hero Member
*
Offline Offline

Posts: 1632522353

View Profile Personal Message (Offline)

Ignore
1632522353
Reply with quote  #2

1632522353
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
Vod
Legendary
*
Offline Offline

Activity: 3444
Merit: 2929


Licking my boob since 1970


View Profile WWW
January 04, 2021, 08:54:46 PM
Merited by bitmover (1)
 #42

All modern browsers will not allow mixed content of HTTPS and HTTP

With all due respect Mr. President, this is not true. 

vod.fan - coming soon!  Free image hosting and URL shortening
clubcrypto.live coming soon!  Work and play at home - earn crypto and video game NFTs
bitmover
Legendary
*
Online Online

Activity: 1344
Merit: 2765



View Profile WWW
January 04, 2021, 10:04:42 PM
 #43

All modern browsers will not allow mixed content of HTTPS and HTTP

With all due respect Mr. President, this is not true. 

Hello. I am new to this, I might have made a mistake.
Do you know how to work around this problem?

Vod
Legendary
*
Offline Offline

Activity: 3444
Merit: 2929


Licking my boob since 1970


View Profile WWW
January 04, 2021, 11:44:00 PM
 #44

All modern browsers will not allow mixed content of HTTPS and HTTP

With all due respect Mr. President, this is not true. 

Hello. I am new to this, I might have made a mistake.
Do you know how to work around this problem?

I'll help you, but as to not hijack the thread, send me a PM with your OS, browser and what you are trying to do.

vod.fan - coming soon!  Free image hosting and URL shortening
clubcrypto.live coming soon!  Work and play at home - earn crypto and video game NFTs
LoyceV
Legendary
*
Offline Offline

Activity: 2352
Merit: 8926


Thick-Skinned Gang Leader


View Profile WWW
January 05, 2021, 12:22:39 PM
Merited by bitmover (1)
 #45

LoyceV, I have tried many things. I made all types of requests I know and all of them failed.
Can I just upload total_number_of_funded_addresses.json to your server after each update? All I need is an FTP-account (which isn't encrypted by default but I don't really worry about this data being compromised) or SSH-access so I can scp or rsync the file.
Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.

bitmover
Legendary
*
Online Online

Activity: 1344
Merit: 2765



View Profile WWW
January 05, 2021, 01:32:32 PM
 #46

I'm just using a standard Linux server.
Running this once a day works:
Code:
wget https://loyce.club/blockdata/blockdata.lastday.txt.gz
gunzip blockdata.lastday.txt.gz
mv blockdata.lastday.txt $(date -r blockdata.lastday.txt "+blockdata_%m-%d-%Y.txt")
Result: a file named blockdata_12-23-2020.txt.
I could store this myself somewhere, but I kinda don't want to keep adding cronjobs. I've added it, see loyce.club/bitmover/. Starting today, it will keep daily snapshots. File blockdata.lastweek.txt will have data for the last 7 days.
Note that it currently only holds data for 1 day, adding one day per day.
Also note that this is untested, we'll know tomorrow if it works.

I just saw this now.

I will add a last week tab in my dashboard. Thanks Smiley

Can I just upload total_number_of_funded_addresses.json to your server after each update? All I need is an FTP-account (which isn't encrypted by default but I don't really worry about this data being compromised) or SSH-access so I can scp or rsync the file.
Alternatively, you can just wget it yourself from a cronjob, but the exact timing for my update varies.

Let me try to add this cronojob. My first attempt.

I used this command:

Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json

Let's see if it works.

LoyceV
Legendary
*
Offline Offline

Activity: 2352
Merit: 8926


Thick-Skinned Gang Leader


View Profile WWW
January 05, 2021, 05:07:51 PM
Merited by bitmover (1)
 #47

I just saw this now.
By now it actually has a full week worth of data.

Quote
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
If it doesn't work: add the full path for wget. You can run it a bit more often, to catch updates quicker. For instance by replacing the second zero ("hours") by 7,8,10,14 (Amsterdam time).
You may also want to add "-q" so you don't receive an email every time it runs.

bitmover
Legendary
*
Online Online

Activity: 1344
Merit: 2765



View Profile WWW
January 06, 2021, 07:45:24 PM
 #48

I just saw this now.
By now it actually has a full week worth of data.

Quote
Code:
0 0 * * * wget http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json
If it doesn't work: add the full path for wget. You can run it a bit more often, to catch updates quicker. For instance by replacing the second zero ("hours") by 7,8,10,14 (Amsterdam time).
You may also want to add "-q" so you don't receive an email every time it runs.

Thanks for suggestions. It is working, and I am also downloading it 3 times a day only if the file is newer (-N, i hope it works). Smiley

Code:
0 0,8,16 * * * wget -N -q http://addresses.loyce.club/total_number_of_funded_addresses.json -O /home2/bitmover/public_html/json/total_number_of_funded_addresses.json

Loyce, I found an error in your file:

Code:
"06/01/2021":[
{"total": 33069791,
 "1": 24175084,
 "3": 6562107,
 "bc1q": 2332583}]
}

24175084+6562107+2332583 = 33069774

But your "total" is 33069791.

I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.

I am  ignoring the total column for now...

Another possibility is that there are other address types with balance that do not start with neither 1, 3 or bc1q? That's new to me, but it could be. Maybe some people mistakenly generated address format?

You can see the first version of this job here:



LoyceV
Legendary
*
Offline Offline

Activity: 2352
Merit: 8926


Thick-Skinned Gang Leader


View Profile WWW
January 06, 2021, 09:16:15 PM
 #49

Loyce, I found an error in your file:

Code:
"06/01/2021":[
{"total": 33069791,
 "1": 24175084,
 "3": 6562107,
 "bc1q": 2332583}]
}

24175084+6562107+2332583 = 33069774

But your "total" is 33069791.

I checked a few days, all of them have a total of 17 more than the sum of all 1,3 and bc1q.
This whole discussion would fit more in List of all Bitcoin addresses with a balance than in this topic. That's where I posted this:

If you add up the addresses starting with 1, 3 and bc1q, you'll notice 16 addresses are missing. Those are:
Code:
bc1p23jk6urvv96x2gp3yqszqgpqyqszqgqa6qtuj
bc1p8qsysgrgypgjqufqtgs85gpcyqjzqsqfrw0l9
bc1p8ysyjgrfypfzqu3q9usrqgpeyqnzqfgexpv74
bc1pmfr3p9j00pfxjh0zmgp99y8zftmd3s5pmedqhyptwy6lm87hf5ss52r5n8
bc1pq2kqvpm76ewe20lcacq740p054at9sv7vxs0jn2u0r90af0k633322m7s8v
bc1pqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqszqgpqyqs3wf0qm
bc1pv22mcnt30gwvk8g72szz700n4tkkx2qur2adj6pt8hl37hcf9dascxyf42
bc1px5sy2gr9yp8zqm3q2us8wgp4yq4jq0guggdp8
bc1pxcsyvgrxyp8jqmeqtqs8sgpkyq7zq0snaecz5
bc1pxgsyygrzyp9jq6eq2ss8ggpjyq5zq2gqvjed5
bc1pxqsrzgpjyqejqdpqx5srvgphyquzqwgdd7yg9
bc1pxssyggryypxjqmfq2cs8vgp5yqsjq0c760r6g
bc1pxusywgr8ypgzqupqtys8jgphyq4zqgcwqe32u
bc1pxvsyxgrrypxzqmpq25s82gpnypajqlgtqkfun
bc1pxysyzgrpyp9zq63q2vs8xgp3ypdjqhguvkagn
bc1zqyqsywvzqe
I don't know the story behind them, someone has been creating non-standard outputs. See txid 8bb2ce18914cfcb68e21686362b879396c2c27b51f1ec4be25c064f48f848f2d for most of them.

LoyceV
Legendary
*
Offline Offline

Activity: 2352
Merit: 8926


Thick-Skinned Gang Leader


View Profile WWW
July 13, 2021, 10:56:11 AM
 #50

4 month bump

Pages: « 1 2 [3]  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!