Bitcoin Forum
May 04, 2024, 08:32:16 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Balances for accounts at a specific block  (Read 237 times)
devonneburger7 (OP)
Newbie
*
Offline Offline

Activity: 19
Merit: 0


View Profile
May 03, 2018, 08:19:59 AM
 #1

Its possible to migrate a copy of the main ledger to a testnet and then reverse the data to a certain block for knowing what account balances were at that specific block?

Or would it be better to aggregate from the genesis block to the required block?
1714854736
Hero Member
*
Offline Offline

Posts: 1714854736

View Profile Personal Message (Offline)

Ignore
1714854736
Reply with quote  #2

1714854736
Report to moderator
The trust scores you see are subjective; they will change depending on who you have in your trust list.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1714854736
Hero Member
*
Offline Offline

Posts: 1714854736

View Profile Personal Message (Offline)

Ignore
1714854736
Reply with quote  #2

1714854736
Report to moderator
mocacinno
Legendary
*
Offline Offline

Activity: 3388
Merit: 4919


https://merel.mobi => buy facemasks with BTC/LTC


View Profile WWW
May 03, 2018, 08:21:25 AM
 #2

Its possible to migrate a copy of the main ledger to a testnet and then reverse the data to a certain block for knowing what account balances were at that specific block?

Or would it be better to aggregate from the genesis block to the required block?

You always aggregate from the genesis block to the required block... The latest block does not specify all unspent outputs (which you can use to calculate the total value funding each of your addresses). Basically, a block contains a bunch of transactions. Each transaction just tells you which unspent output(s) from the UTXO set are used, and which new (unspent) outputs are generated. Offcourse, there is also some other data in a block (like the signature , the block header).

In order to generate the UTXO set (the set of unspent outputs that you can use to calculate your balance), you start from the genesis block and follow each unspent output. As soon as a transaction uses an unspent output from your current utxo set, it must be removed from the utxo set, and the output of the transaction will be added to the utxo set... The coinbase transaction also generates a new unspent output. If you do this upto height x, you automatically know all unspent outputs at height x, so you can use them to calculate the balances.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
devonneburger7 (OP)
Newbie
*
Offline Offline

Activity: 19
Merit: 0


View Profile
May 03, 2018, 10:33:05 AM
 #3

That makes sense.

If it is always aggregated, how are balance lookups so fast?
mocacinno
Legendary
*
Offline Offline

Activity: 3388
Merit: 4919


https://merel.mobi => buy facemasks with BTC/LTC


View Profile WWW
May 03, 2018, 10:53:37 AM
 #4

That makes sense.

If it is always aggregated, how are balance lookups so fast?

Because when you install a node, it syncs the complete blockchain. During this syncing, the blocks are parsed and verified, and a node builds the UTXO set while syncing (it aggregates from block #1 up untill the current height). This syncing process takes hours, sometimes even days (depending on your node's version, the IO capacity, the memory speed, the cpu,...). After the initial sync, the UTXO set up untill a certain height is built, so as long as you keep your node running 24/7, or at least start it a couple times a week, it can keep up with all the new blocks pretty fast, your node just parses a new block when it receives it, and adapts its utxo set accordingly.

So, when you use a full client, your PC/server already did all these aggregations in the past, so you can now see your balance instantly. If you use an SPV client (like electrum, or most of the hardware wallets), this SPV client is connected to a full node, this node built the UTXO set in the past when it was syncing, so it can also reply instantly when an SPV client querys the unspent outputs funding a certain address.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
devonneburger7 (OP)
Newbie
*
Offline Offline

Activity: 19
Merit: 0


View Profile
May 03, 2018, 11:32:47 AM
 #5

Got it.

You said earlier I would have to aggregate it by following the blocks but would it not be easier to just modify the daemon to stop syncing at a specific block height and then use existing RPC api's to query the balances from that node?
starmyc
Full Member
***
Offline Offline

Activity: 198
Merit: 130

Some random software engineer


View Profile
May 03, 2018, 11:49:23 AM
 #6

Got it.

You said earlier I would have to aggregate it by following the blocks but would it not be easier to just modify the daemon to stop syncing at a specific block height and then use existing RPC api's to query the balances from that node?

Having already done this once, yes, it is way more easy to do this way.

Hi, I'm just some random software engineer.
You can check my projects: Bitcoin & altcoin balances/addresses listing dumps: https://balances.crypto-nerdz.org/
DannyHamilton
Legendary
*
Offline Offline

Activity: 3388
Merit: 4616



View Profile
May 03, 2018, 12:00:48 PM
 #7

Got it.

You said earlier I would have to aggregate it by following the blocks but would it not be easier to just modify the daemon to stop syncing at a specific block height and then use existing RPC api's to query the balances from that node?

Or, even easier...

Start at the beginning aggregating balances. As you encounter each new block, record the block height and current balances in a database.  Then when you want to know any historical balance, your can just query your database for the block height in question.  Meanwhile, your node can continue to remain synchronized and can continue to update your database as new blocks are received.
mocacinno
Legendary
*
Offline Offline

Activity: 3388
Merit: 4919


https://merel.mobi => buy facemasks with BTC/LTC


View Profile WWW
May 03, 2018, 12:19:53 PM
Merited by suchmoon (1)
 #8

Or, even easier...

Start at the beginning aggregating balances. As you encounter each new block, record the block height and current balances in a database.  Then when you want to know any historical balance, your can just query your database for the block height in question.  Meanwhile, your node can continue to remain synchronized and can continue to update your database as new blocks are received.

Wouldn't that require a huge database?
I think it wouldn't be practical to save all balances for all addresses that ever existed at each blockheight, but you could probably save balance of each address whose balance was changed by a transaction in a block at each height... That way you should be able to query the value of the sum of the unspent outputs funding address x at height y where the blockheight = the max blockheight for an entry for address x.

Something like this
Code:
CREATE TABLE balances (
    id int NOT NULL AUTO_INCREMENT,
    address varchar(45),
    balance_at_height int(15),
    height int(10)
    PRIMARY KEY (id)
);

For example, if my address was funded for the very first time at height 40000, then was funded for a second time at 401000 and i spent both outputs at 402000, only 3 inserts would be needed for my total history...

Code:
insert into (address, balace_at_height, height) values ("1MocACiWLM8bYn8pCrYjy6uHq4U3CkxLaa", 100000, 400000);
insert into (address, balace_at_height, height) values ("1MocACiWLM8bYn8pCrYjy6uHq4U3CkxLaa", 200000, 401000);
insert into (address, balace_at_height, height) values ("1MocACiWLM8bYn8pCrYjy6uHq4U3CkxLaa", 0, 402000);

Each new block you'd have to parse 1500? transactions, so if you'd only have to save the balances of addresses whose balance changed at each blockheight, you'd have to insert at least ~1500 changes/block * ~144 blocks/day = 216000 changes each day... That seems doable

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
devonneburger7 (OP)
Newbie
*
Offline Offline

Activity: 19
Merit: 0


View Profile
May 03, 2018, 12:37:23 PM
 #9

Thanks for the advice. You guys have pointed me in the right redirection.

Are there any stats anywhere on how many addresses without zero balances are in existence?
mocacinno
Legendary
*
Offline Offline

Activity: 3388
Merit: 4919


https://merel.mobi => buy facemasks with BTC/LTC


View Profile WWW
May 03, 2018, 12:41:17 PM
 #10

Thanks for the advice. You guys have pointed me in the right redirection.

Are there any stats anywhere on how many addresses without zero balances are in existence?

I've generated a list of all funded addresses for somebody on this forum a while ago... I used this script: https://github.com/cryptah/utxo-dump

IIRC, at that time, there were ~40.000.000 unspent outputs in the UTXO set, don't remember how many addresses were funded (multiple unspent outputs can fund the same address).
I can only estimate this number to have grown since then (about a month ago iirc)

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
starmyc
Full Member
***
Offline Offline

Activity: 198
Merit: 130

Some random software engineer


View Profile
May 03, 2018, 02:08:25 PM
 #11

Thanks for the advice. You guys have pointed me in the right redirection.

Are there any stats anywhere on how many addresses without zero balances are in existence?

I've exported those last week: http://bit.ly/BtcBalances20180427
I'm counting ~21.8 millions of addresses with balance currently.

Hi, I'm just some random software engineer.
You can check my projects: Bitcoin & altcoin balances/addresses listing dumps: https://balances.crypto-nerdz.org/
DannyHamilton
Legendary
*
Offline Offline

Activity: 3388
Merit: 4616



View Profile
May 03, 2018, 02:16:54 PM
 #12

you could probably save balance of each address whose balance was changed by a transaction in a block at each height.

While I may not have spelled out the exact process, this is effectively what I intended.

If a balance hasn't changed, there is no need to store another record for it since you already have the balance and the block height where that balance was established.
hatuey
Newbie
*
Offline Offline

Activity: 54
Merit: 0


View Profile
May 13, 2018, 03:56:15 AM
 #13

Thanks for the advice. You guys have pointed me in the right redirection.

Are there any stats anywhere on how many addresses without zero balances are in existence?

I've exported those last week: http://bit.ly/BtcBalances20180427
I'm counting ~21.8 millions of addresses with balance currently.

would you mind re-sharing the file? your link is down. Thanks
hatuey
Newbie
*
Offline Offline

Activity: 54
Merit: 0


View Profile
May 13, 2018, 06:45:09 AM
 #14

Thanks for the advice. You guys have pointed me in the right redirection.

Are there any stats anywhere on how many addresses without zero balances are in existence?

I've exported those last week: http://bit.ly/BtcBalances20180427
I'm counting ~21.8 millions of addresses with balance currently.

This guy asks for money to share the dump.  Undecided

My curiosity about btc whales and monitoring the increase of active wallets doesn't go that far.
starmyc
Full Member
***
Offline Offline

Activity: 198
Merit: 130

Some random software engineer


View Profile
May 13, 2018, 03:47:18 PM
 #15

Thanks for the advice. You guys have pointed me in the right redirection.

Are there any stats anywhere on how many addresses without zero balances are in existence?

I've exported those last week: http://bit.ly/BtcBalances20180427
I'm counting ~21.8 millions of addresses with balance currently.

This guy asks for money to share the dump.  Undecided

My curiosity about btc whales and monitoring the increase of active wallets doesn't go that far.

Bad evil guy who is asking money to pay bandwidth Sad

The tools I wrote to generate those are opensourced. You're free to use them to generate it and make it available for everyone!

Hi, I'm just some random software engineer.
You can check my projects: Bitcoin & altcoin balances/addresses listing dumps: https://balances.crypto-nerdz.org/
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!