Data history for mined blocks

[bomberb17]

Hello
I am looking for historic data for mining pools to perform a study.
e.g. for Antpool, how many blocks it mined in January 2018, and for each block what was the Bitcoin network hashrate and what was the pool's hashrate.
I would also like to export them in readable format e.g. JSON.
I only found this in btc.com
https://btc.com/stats/pool/AntPool
which the data is not exportable (you have to click the button on the bottom a million times)
Blockchain.info only has history for 4 days..
Any suggestions?
Thanks!

(P.S. To moderators: Please do not delete my post again for no reason. If I'm posting on the wrong section please let me know which forum section would be applicable and move it there.)

[bomberb17]

Upd:
I was looking at blockchain.info API
https://www.blockchain.com/en/api/blockchain_api
and in particular to the bottom: Blocks for specific pool: https://blockchain.info/blocks/$pool_name?format=json
I substitute pool_name with a pool but it doesn't work..

[pooya87]

I substitute pool_name with a pool but it doesn't work..

you should replace $pool_name with the name:
eg: https://blockchain.info/blocks/AntPool?format=json
i believe it gives you the past 24 hours blocks found by that pool. in case you needed more you either have to seek other APIs to see whether they can help or just take the first transaction of each block (the coinbase tx) and decode their OP_Return output if they have any to see which pool that belongs to (similar to what explorers do)

for example block #540291 has this:

Code:

6a 24aa21a9eda1df4536a0b991727e4caf58f1aae977786a13708f015a589b8e0d02d67e9306

which contains the string "Mined by AntPool60" in it.

[bomberb17]

I substitute pool_name with a pool but it doesn't work..

you should replace $pool_name with the name:
eg: https://blockchain.info/blocks/AntPool?format=json

Thanks a lot.
Still however this is not enough data, I tried to manually enter the timespan as follows:
https://blockchain.info/blocks/AntPool?format=json&timespan=10days
But if I enter a larger than 10 days timespan, it doesn't return the data (must be a blockchain.info limitation)
Any ideas where I can find such data?

[pooya87]

as i said you should check out other explorers. there is a lot of them
like this one:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool
and it supports pagination so you can get as many blocks as you want, you just have to figure out how many pages to go back (check the time field it is a unix timestamp):
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?sort=height&next=1
you can get a list of all valid pools from this call:
https://api.smartbit.com.au/v1/blockchain/pools

since you also want hashrates you can use this API call which gives you total difficulty and hashrate and share of each pool:
https://api.smartbit.com.au/v1/blockchain/stats?to=2018-01-01T14:53:11Z


here is the link to the documentation:
https://www.smartbit.com.au/api

[bomberb17]

Hmm, yes that looks like a much more useful API.
I tried something like this
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?sort=height&limit=1000
and it gives me up to July 20.
However if I set a limit higher than 1000 then it only returns 10 blocks...
Thanks for helping btw!

[pooya87]

However if I set a limit higher than 1000 then it only returns 10 blocks...

limit is the number of results (blocks mined) that the call returns. if you check out the documentation at the bottom of my previous post you can see that it only accepts a number between 1 and 1000 so if you give it something outside of that range it defaults to 10 hence the 10 returned blocks.

if you need more you need to go to next page using "next" like this:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=1
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=2
...

[bomberb17]

Still however it looks like https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=1
and
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=2
return the same data..

[nc50lc]

You can try https://bitcoinstrings.com/, but it's totally converted to text and the blocks' data aren't displayed.
OP_return data were translated too, just ignore them or highlight all the "Mined by" lines then search from other source about which blocks are included in that particular blkXXXXX.dat number.
Sort it out, and you might find what you need.

However, the site's updates were not that frequent (about daily to four times a week)

[pooya87]

Still however it looks like https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=1
and
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=2
return the same data..

sorry, that was my bad. i always make the same mistake with pages in API calls.
basically you make the first call:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=20&sort=height (used 20 so it opens up fast)
then you look for this key in the JSON:
[pool][block_paging][next_link]
which contains the link to next page:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?next=NTQwMjA2&limit=20

in case you are wondering what the heck "NTQwMjA2" is, it is base64 encoding of the last block height in previous page. in this case the block at index 19 of the array result of the first call which is "540206"

[bomberb17]

You can try https://bitcoinstrings.com/, but it's totally converted to text and the blocks' data aren't displayed.
OP_return data were translated too, just ignore them or highlight all the "Mined by" lines then search from other source about which blocks are included in that particular blkXXXXX.dat number.
Sort it out, and you might find what you need.

However, the site's updates were not that frequent (about daily to four times a week)

Thanks, however I would need to devote time to convert text to usable data...

Still however it looks like https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=1
and
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=1000&sort=height&next=2
return the same data..

sorry, that was my bad. i always make the same mistake with pages in API calls.
basically you make the first call:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?limit=20&sort=height (used 20 so it opens up fast)
then you look for this key in the JSON:
[pool][block_paging][next_link]
which contains the link to next page:
https://api.smartbit.com.au/v1/blockchain/pool/AntPool?next=NTQwMjA2&limit=20

in case you are wondering what the heck "NTQwMjA2" is, it is base64 encoding of the last block height in previous page. in this case the block at index 19 of the array result of the first call which is "540206"

Great thanks! Looks like I will have to find a way to "glue" all these JSONs into a big one..

[ABCbits]

This might be crazy idea, but you could try using Bitcoin API JSON-RPC and your own script to get coinbase text which pool usually use to mark it's mined block.
getblockhash, getblock true and getrawtransaction {true/false} with script should able to do the trick.

You can try https://bitcoinstrings.com/, but it's totally converted to text and the blocks' data aren't displayed.
OP_return data were translated too, just ignore them or highlight all the "Mined by" lines then search from other source about which blocks are included in that particular blkXXXXX.dat number.
Sort it out, and you might find what you need.

However, the site's updates were not that frequent (about daily to four times a week)

I was going to mention that website, but there's no info of the block height and it's useless for OP unless he know about how Bitcoin Core manages it's blocks/leveldb and have full copy of blockchain/run full nodes.

[spirali]

Great thanks! Looks like I will have to find a way to "glue" all these JSONs into a big one..

"jq" (https://stedolan.github.io/jq/) is a very good tool to achieve that (and any JSON processing in general). Learning the syntax is a bit cumbersome at the beginning but it's definitely worth it because it's very powerful.

[bomberb17]

I eventually used the JSON data from api.smartbit.com.au, saved me a lot of time of putting the data together myself.

Thank you all for your inputs!