Bitcoin Forum
December 12, 2018, 02:16:20 AM *
News: Latest Bitcoin Core release: 0.17.0 [Torrent].
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Data history for mined blocks  (Read 197 times)
bomberb17
Hero Member
*****
Offline Offline

Activity: 612
Merit: 500


View Profile
September 07, 2018, 02:46:10 AM
 #1

Hello
I am looking for historic data for mining pools to perform a study.
e.g. for Antpool, how many blocks it mined in January 2018, and for each block what was the Bitcoin network hashrate and what was the pool's hashrate.
I would also like to export them in readable format e.g. JSON.
I only found this in btc.com
https://btc.com/stats/pool/AntPool
which the data is not exportable (you have to click the button on the bottom a million times)
Blockchain.info only has history for 4 days..
Any suggestions?
Thanks!

(P.S. To moderators: Please do not delete my post again for no reason. If I'm posting on the wrong section please let me know which forum section would be applicable and move it there.)
1544580980
Hero Member
*
Offline Offline

Posts: 1544580980

View Profile Personal Message (Offline)

Ignore
1544580980
Reply with quote  #2

1544580980
Report to moderator
1544580980
Hero Member
*
Offline Offline

Posts: 1544580980

View Profile Personal Message (Offline)

Ignore
1544580980
Reply with quote  #2

1544580980
Report to moderator
1544580980
Hero Member
*
Offline Offline

Posts: 1544580980

View Profile Personal Message (Offline)

Ignore
1544580980
Reply with quote  #2

1544580980
Report to moderator
PLAY NOW
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
bomberb17
Hero Member
*****
Offline Offline

Activity: 612
Merit: 500


View Profile
September 07, 2018, 04:07:44 AM
 #2

Upd:
I was looking at blockchain.info API
https://www.blockchain.com/en/api/blockchain_api
and in particular to the bottom: Blocks for specific pool: https://blockchain.info/blocks/$pool_name?format=json
I substitute pool_name with a pool but it doesn't work..
pooya87
Legendary
*
Offline Offline

Activity: 1484
Merit: 1283


Buy bitcoin they said... who listened?


View Profile
September 07, 2018, 04:18:46 AM
Merited by bomberb17 (1)
 #3

I substitute pool_name with a pool but it doesn't work..

you should replace $pool_name with the name:
eg: https://blockchain.info/blocks/AntPool?format=json
i believe it gives you the past 24 hours blocks found by that pool. in case you needed more you either have to seek other APIs to see whether they can help or just take the first transaction of each block (the coinbase tx) and decode their OP_Return output if they have any to see which pool that belongs to (similar to what explorers do)

for example block #540291 has this:
Code:
6a 24aa21a9eda1df4536a0b991727e4caf58f1aae977786a13708f015a589b8e0d02d67e9306
which contains the string "Mined by AntPool60" in it.

bomberb17
Hero Member
*****
Offline Offline

Activity: 612
Merit: 500


View Profile
September 07, 2018, 04:24:17 AM
 #4

I substitute pool_name with a pool but it doesn't work..

you should replace $pool_name with the name:
eg: https://blockchain.info/blocks/AntPool?format=json



Thanks a lot.
Still however this is not enough data, I tried to manually enter the timespan as follows:
https://blockchain.info/blocks/AntPool?format=json&timespan=10days
But if I enter a larger than 10 days timespan, it doesn't return the data (must be a blockchain.info limitation)
Any ideas where I can find such data?
pooya87
Legendary
*
Offline Offline

Activity: 1484
Merit: 1283


Buy bitcoin they said... who listened?


View Profile
September 07, 2018, 04:34:33 AM
Merited by bomberb17 (1)
 #5

as i said you should check out other explorers. there is a lot of them Tongue
like this one:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool
and it supports pagination so you can get as many blocks as you want, you just have to figure out how many pages to go back (check the time field it is a unix timestamp):
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?sort=height&next=1
you can get a list of all valid pools from this call:
https://api.smartbit.com.au/v1/blockchain/pools

since you also want hashrates you can use this API call which gives you total difficulty and hashrate and share of each pool:
https://api.smartbit.com.au/v1/blockchain/stats?to=2018-01-01T14:53:11Z


here is the link to the documentation:
https://www.smartbit.com.au/api

bomberb17
Hero Member
*****
Offline Offline

Activity: 612
Merit: 500


View Profile
September 07, 2018, 05:00:15 AM
 #6

Hmm, yes that looks like a much more useful API.
I tried something like this
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?sort=height&limit=1000
and it gives me up to July 20.
However if I set a limit higher than 1000 then it only returns 10 blocks...
Thanks for helping btw!
pooya87
Legendary
*
Offline Offline

Activity: 1484
Merit: 1283


Buy bitcoin they said... who listened?


View Profile
September 07, 2018, 05:10:15 AM
Merited by bomberb17 (1)
 #7

However if I set a limit higher than 1000 then it only returns 10 blocks...

limit is the number of results (blocks mined) that the call returns. if you check out the documentation at the bottom of my previous post you can see that it only accepts a number between 1 and 1000 so if you give it something outside of that range it defaults to 10 hence the 10 returned blocks.

if you need more you need to go to next page using "next" like this:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=1
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=2
...

bomberb17
Hero Member
*****
Offline Offline

Activity: 612
Merit: 500


View Profile
September 07, 2018, 05:24:45 AM
 #8

Still however it looks like https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=1
and
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=2
return the same data..
nc50lc
Full Member
***
Offline Offline

Activity: 434
Merit: 149


∙Self-proclaimed-Genius ㊙️


View Profile WWW
September 07, 2018, 05:49:18 AM
 #9

You can try https://bitcoinstrings.com/, but it's totally converted to text and the blocks' data aren't displayed.
OP_return data were translated too, just ignore them or highlight all the "Mined by" lines then search from other source about which blocks are included in that particular blkXXXXX.dat number.
Sort it out, and you might find what you need.

However, the site's updates were not that frequent (about daily to four times a week)

(っ◕‿◕)っ Newbies and Newbies at heart! Remember to Lock your Thread(s) after receiving enough replies/sufficient answers. 
FEELING GENEROUS?: 39EKeFj43inkH6Ctkosh9E7oskx3tvhSXi ∙ Do not buy non-mainstream ASICs at second-batch and onwards, you know the risk!
pooya87
Legendary
*
Offline Offline

Activity: 1484
Merit: 1283


Buy bitcoin they said... who listened?


View Profile
September 07, 2018, 06:00:17 AM
Merited by bomberb17 (1)
 #10


sorry, that was my bad. i always make the same mistake with pages in API calls. Roll Eyes
basically you make the first call:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=20&sort=height (used 20 so it opens up fast)
then you look for this key in the JSON:
[pool][block_paging][next_link]
which contains the link to next page:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?next=NTQwMjA2&limit=20

in case you are wondering what the heck "NTQwMjA2" is, it is base64 encoding of the last block height in previous page. in this case the block at index 19 of the array result of the first call which is "540206"

bomberb17
Hero Member
*****
Offline Offline

Activity: 612
Merit: 500


View Profile
September 07, 2018, 06:09:18 AM
 #11

You can try https://bitcoinstrings.com/, but it's totally converted to text and the blocks' data aren't displayed.
OP_return data were translated too, just ignore them or highlight all the "Mined by" lines then search from other source about which blocks are included in that particular blkXXXXX.dat number.
Sort it out, and you might find what you need.

However, the site's updates were not that frequent (about daily to four times a week)

Thanks, however I would need to devote time to convert text to usable data...


sorry, that was my bad. i always make the same mistake with pages in API calls. Roll Eyes
basically you make the first call:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=20&sort=height (used 20 so it opens up fast)
then you look for this key in the JSON:
[pool][block_paging][next_link]
which contains the link to next page:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?next=NTQwMjA2&limit=20

in case you are wondering what the heck "NTQwMjA2" is, it is base64 encoding of the last block height in previous page. in this case the block at index 19 of the array result of the first call which is "540206"

Great thanks! Looks like I will have to find a way to "glue" all these JSONs into a big one..
ETFbitcoin
Legendary
*
Offline Offline

Activity: 1526
Merit: 1335

Use SegWit and enjoy lower fees


View Profile WWW
September 07, 2018, 01:04:30 PM
Merited by bomberb17 (1)
 #12

This might be crazy idea, but you could try using Bitcoin API JSON-RPC and your own script to get coinbase text which pool usually use to mark it's mined block.
getblockhash, getblock true and getrawtransaction {true/false} with script should able to do the trick.

You can try https://bitcoinstrings.com/, but it's totally converted to text and the blocks' data aren't displayed.
OP_return data were translated too, just ignore them or highlight all the "Mined by" lines then search from other source about which blocks are included in that particular blkXXXXX.dat number.
Sort it out, and you might find what you need.

However, the site's updates were not that frequent (about daily to four times a week)

I was going to mention that website, but there's no info of the block height and it's useless for OP unless he know about how Bitcoin Core manages it's blocks/leveldb and have full copy of blockchain/run full nodes.

spirali
Member
**
Offline Offline

Activity: 66
Merit: 60


View Profile
September 09, 2018, 10:38:15 AM
Merited by bomberb17 (1)
 #13

Great thanks! Looks like I will have to find a way to "glue" all these JSONs into a big one..

"jq" (https://stedolan.github.io/jq/) is a very good tool to achieve that (and any JSON processing in general). Learning the syntax is a bit cumbersome at the beginning but it's definitely worth it because it's very powerful.

1 BTC = 1 BTC
bomberb17
Hero Member
*****
Offline Offline

Activity: 612
Merit: 500


View Profile
September 10, 2018, 03:43:43 PM
 #14

I eventually used the JSON data from api.smartbit.com.au, saved me a lot of time of putting the data together myself.

Thank you all for your inputs!
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!