Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: mahurovihamilo on May 21, 2024, 02:54:52 AM



Title: Best way or API to fetch UTXOS
Post by: mahurovihamilo on May 21, 2024, 02:54:52 AM
Hi All

Does anyone know the best way to fetch UTXOs from specific addresses? via API calls or any other way?

Thank you !


Title: Re: Best way or API to fetch UTXOS
Post by: nc50lc on May 21, 2024, 06:55:05 AM
Does anyone know the best way to fetch UTXOs from specific addresses? via API calls or any other way?
Have you tried any of the famous blockexplorers' APIs?

For example, Blockstream's public API:
Code:
blockstream.info/api/address/1BitcoinEaterAddressDontSendf59kuE/txs/chain
Returns with the latest 25 transaction of the specified address.

To continue to the next set of transactions, add the last txid from the result above. (check if it's the transaction's txid, not its input)
e.g.:
Code:
https://blockstream.info/api/address/1BitcoinEaterAddressDontSendf59kuE/txs/chain/fab012bec43a909d787e95a1afcbced1a66fb1a6a62d0d22cad9578c560627cc


Title: Re: Best way or API to fetch UTXOS
Post by: RickDeckard on May 21, 2024, 07:59:25 AM
In addition to nc50lc reply, you can also try using the API provided by mempool.space[1] to access that information:
Code:
curl -sSL "https://mempool.space/api/address/1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY/utxo"
Returns:
Code:
[
  {
    txid: "12f96289f8f9cd51ccfe390879a46d7eeb0435d9e0af9297776e6bdf249414ff",
    vout: 0,
    status: {
      confirmed: true,
      block_height: 698642,
      block_hash: "00000000000000000007839f42e0e86fd53c797b64b7135fcad385158c9cafb8",
      block_time: 1630561459
    },
    value: 644951084
  },
  ...
]
mempool.space doesn't publish what are their API limits so once you start getting HTTP 429 error you will know that you've reached them. If you intend to make several calls perhaps it would be wise to balance them between some blockexplorers' APIs to avoid being blocked/rejected by their APIs.

[1]https://mempool.space/docs/api/rest#get-address-utxo (https://mempool.space/docs/api/rest#get-address-utxo)


Title: Re: Best way or API to fetch UTXOS
Post by: mahurovihamilo on May 21, 2024, 01:34:09 PM
Thank you for your replies gentlemen. That is very helpful.

I had tried some other API frameworks before, like:

wget -qO- https://api.blockcypher.com/v1/btc/main/addrs/{line.rstrip()[:]} > index.data'

But this is getting me a lot of transaction information but not specifically the UTXOs so I was having to parse through a lot of needlessly useless data.

Thank you again.


Title: Re: Best way or API to fetch UTXOS
Post by: mahurovihamilo on May 21, 2024, 05:05:25 PM
Soooo if you don't mind I have a further question about formatting:

I did: wget -qO- https://mempool.space/api/address/1Lpu1LJAhYTWLveYW5mwi9T2kF3NKWQcJP/utxo  > data.json

using the mempool API, however the file is not quite formatted as usual json files are  and that makes parsing a lot harder with bash and python. However if from that page I manually click on raw date and then click on pretty print and manually save,  the saved file (through graphical interface) is actually formatted as usual json files.

I tried adding &type=json at the end of the wget command, but that messes up and the end point is not found.

So my question is, do you know in mempool how to insure the downloaded UTXO file (via wget) is an actual formatted json?

This is necessary to parse for batches of UTXO calls from script...

Thank you !!


Title: Re: Best way or API to fetch UTXOS
Post by: mahurovihamilo on May 21, 2024, 05:36:31 PM
In addition to nc50lc reply, you can also try using the API provided by mempool.space[1] to access that information:
Code:
curl -sSL "https://mempool.space/api/address/1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY/utxo"
Returns:
Code:
[
  {
    txid: "12f96289f8f9cd51ccfe390879a46d7eeb0435d9e0af9297776e6bdf249414ff",
    vout: 0,
    status: {
      confirmed: true,
      block_height: 698642,
      block_hash: "00000000000000000007839f42e0e86fd53c797b64b7135fcad385158c9cafb8",
      block_time: 1630561459
    },
    value: 644951084
  },
  ...
]
mempool.space doesn't publish what are their API limits so once you start getting HTTP 429 error you will know that you've reached them. If you intend to make several calls perhaps it would be wise to balance them between some blockexplorers' APIs to avoid being blocked/rejected by their APIs.

[1]https://mempool.space/docs/api/rest#get-address-utxo (https://mempool.space/docs/api/rest#get-address-utxo)

I can't get an output formatted like that. When I do the call: curl -sSL "https://mempool.space/api/address/1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY/utxo" > data.json

The ooutput comes formated like this, which is a pain to parse:

[{"txid":"33717a68856ecf397455b12b2a21321bf64eb6b7062939b46f630365c79d5604","vout":22,"status":{"confirmed":true,"block_height":838458,"block_hash":"0000000000000000000269654a31dbdfad3f0b7589433b1977a511e5641897c6","block_time":1712672676},"value":8021},{"txid":"5f3dafcdd142358b332e0939d37174b76472735df8df29325901d2c7d18a152c","vout":476,"status":{"confirmed":true,"block_height":834708,"block_hash":"00000000000000000000292964b4b00d1d414583bd18d1ebea7a8041d4409c6d","block_time":1710447063},"value":546},{"txid":"5557fbd7f4504d3b1886c4540db80fe5e56e326bbf419bcdfeecddf232bd31d1","vout":87,"status":{"confirmed":true,"block_height":838624,"block_hash":"000000000000000000002ffad6c02ed22351bd4df615b0fad3e1fbbb4bdc24dc","block_time":1712764942},"value":6040},{"txid":"988774c19446c5d2b174588f9839b43e86334b59aa5b0a8f05255bd29be53807","vout":2,"status":{"confirmed":true,"block_height":842495,"block_hash":"00000000000000000001a6c7110b453f369db51cbcb2d108f6909de4460fa6c8","block_time":1715122220},"value":546},{"txid":"e9abd6cf3b26236ef8b7f819a8fbd4a001379a74f86a1fff43b000382858abdb","vout":670,"status":{"confirmed":true,"block_height":840255,"block_hash":"000000000000000000020cf2611ecdda237904fcf1aa2edb532e1571b80614a9","block_time":1713735799},"value":888},{"txid":"b694b0fe40acde2855a1097350c68ce2c8015c364d8945a357b903ae9bbad4ec","vout":6,"status":{"confirmed":true,"block_height":841064,"block_hash":"0000000000000000000017f1db3dd49d05d4eea6761daaa25ad345a9985bcb18","block_time":1714201794},"value":15120},{"txid":"9cbf451cb05ac019c0cac3d7eb0d175c99ebd587fc4fe3e6b0662f724adf92be","vout":454,"status":{"confirmed":true,"block_height":840885,"block_hash":"000000000000000000023f33c91331450913710361a6f34426ebbe853efc18ee","block_time":1714085239},"value":546},{"txid":"9fd0b6e5c96f0537f0a61a4e1c32167c91511cd1ccfc34a92801842701c4df95","vout":0,"status":{"confirmed":true,"block_height":841305,"block_hash":"0000000000000000000077f28728313f4085275acb1745a438b71f197fee2a00","block_time":1714353023},"value":4287},{"txid":"0c2d34ad6c724b6d2554cd577e0db66d2ed4786edeab9d2246b5698fe842e215","vout":565,"status":{"confirmed":true,"block_height":840253,"block_hash":"0000000000000000000069440489b457550d1df7da1ec2ec52475d6353245948","block_time":1713735717},"value":888},{"txid":"9587acf268975b518733bfa432a1a2153a06e3de39ed118005d6b645120aaff7","vout":1,"status":{"confirmed":true,"block_height":841305,"block_hash":"0000000000000000000077f28728313f4085275acb1745a438b71f197fee2a00","block_time":1714353023},"value":52500},{"txid":"826f8191cc6ece6a2f895fbed111d80022a2efd4ee5482a25ae8398b4687bfff","vout":59,"status":{"confirmed":true,"block_height":841064,"block_hash":"0000000000000000000017f1db3dd49d05d4eea6761daaa25ad345a9985bcb18","block_time":1714201794},"value":19216},{"txid":"8a006d51d3173a36d257b357f55bc5155331e0c5f21cab754fd73957a250fc34","vout":378,"status":{"confirmed":true,"block_height":840657,"block_hash":"00000000000000000002d4a9bbd7a2437fe32df0d442148635dfae522514b5af","block_time":1713962541},"value":546},{"txid":"d0eb2d477784a892f012943dab7f11dc47fcf1d0fd236ac483237e3048e87ff2","vout":490,"status":{"confirmed":true,"block_height":836194,"block_hash":"00000000000000000000361121795cda72a2494ef23648f19685977610c1d55a","block_time":1711337405},"value":546}]

In priciple should be the same but my SEd and Python scripts that work on regular json, dont work on this output. I started customising Sed commands to reformat but it takes forever.

How do you get it to download in that json format?

Thanks !



Title: Re: Best way or API to fetch UTXOS
Post by: RickDeckard on May 21, 2024, 06:02:42 PM
I can't get an output formatted like that. When I do the call: curl -sSL "https://mempool.space/api/address/1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY/utxo" > data.json
What if you try the following code:
Code:
curl -sSL "https://mempool.space/api/address/1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY/utxo" | jq '.' > data.json
All I did was adding | jq '.' to your initial input. jq is a JSON processing tool[1] - and since API usually return JSON data when called - it should get the job done by formatting the data before saving it to your named file (in this case data.json).

[1]https://www.digitalocean.com/community/tutorials/how-to-transform-json-data-with-jq (https://www.digitalocean.com/community/tutorials/how-to-transform-json-data-with-jq)


Title: Re: Best way or API to fetch UTXOS
Post by: mahurovihamilo on May 21, 2024, 06:20:15 PM
 Thank you @RickDeckard

That is perfect! it does download in the right format. Thanks.


Title: Re: Best way or API to fetch UTXOS
Post by: mahurovihamilo on May 22, 2024, 03:47:39 PM
Good day gentlemen...

I did the mempool call, but I got a message back on my shell telling the the request for UTXOs for that address  exceeded  limit of outputs > 500

curl -sSL "https://mempool.space/api/address/XXXXXXXXXXXXXXXXXXX/utxo"  > index.data

The mempool documentation is a little scarce in detail. Do you know of a flag I can add to the CURL call to limit it to 500 or less? I cant really afford a payed plan, not at those prices....

I tried: curl -sSL "https://mempool.space/api/address/XXXXXXXXXXXXXXXXXXX/utxo&limit=450 , as this works somewhere else, but not here.....


Thank you!


Title: Re: Best way or API to fetch UTXOS
Post by: RickDeckard on May 22, 2024, 05:35:55 PM
I think you have two options that you can try:
  • Run your local instance of mempool.space (by using Docker[1] for example) + Local Node. I would highly recommend doing this considering the amount of freedom and privacy that you'll get;
  • mempool.space doesn't seem to have a capability to filter UTXO's. You either look for another blockexplorer API or perhaps you can try to add more arguments to jq. Considering the documentation[2] you could try filtering the results? For example, if you use [0:15] as an additional argument, it supposedly only prints the first 15 UTXO's. It should be something like this:
Code:
curl -sSL "https://mempool.space/api/address/AddressGoesHere/utxo" | jq '.[0:15]' > data.json

    [1]https://github.com/mempool/mempool/tree/master/docker (https://github.com/mempool/mempool/tree/master/docker)
    [2]https://jqlang.github.io/jq/manual/ (https://jqlang.github.io/jq/manual/)


    Title: Re: Best way or API to fetch UTXOS
    Post by: mahurovihamilo on May 22, 2024, 06:57:16 PM
    Thank you @RickDeckard

    I am going to explore running my own instance of mempool! Cheers!


    Title: Re: Best way or API to fetch UTXOS
    Post by: mahurovihamilo on May 24, 2024, 02:11:06 AM
    Hello RickDeckard,

    So I went ahead and installed Docker and mempool, is up and running with a few glitches, but for now my question is, for this to work, do I also need to be running a full BTC node as well?

    Thank you.


    Title: Re: Best way or API to fetch UTXOS
    Post by: RickDeckard on May 24, 2024, 09:03:16 PM
    I advise you to follow a guide[1] that covers the full installation of mempool.space correctly. As far as  know you need both a Bitcoin Core + Electrum Server in order to fully benefit from mempool.space capabilities.

    [1]https://www.youtube.com/watch?v=I2SzBqcsXaE (https://www.youtube.com/watch?v=I2SzBqcsXaE)


    Title: Re: Best way or API to fetch UTXOS
    Post by: mahurovihamilo on May 24, 2024, 10:49:35 PM
    Ok, thank you!