Tracking large amount of receive addresses for web service

[o0o0]

I want to create a service around using bitcoin/altcoins so have started researching into it. I've seen a lot of mentioning of the wallet notify action in the conf file directing it to a fast processing php / python whatever script. Other topics i've seen around the internet and in reddit mention that after generating large amounts of addresses the daemon can take quite some time to start up (think i saw references of 2 hours start up time).

So i started thinking and wondering a few things and thought i'd post here to get opinions from people well versed in this. Lets assume a service increases from 100 users -> 10,000 users -> 1 million users

1. How do the likes of shapeshift, bittrex, poloniex and other high amount clients handle such large amounts of receiving addresses? Do they have a modified client or do they just use wallet notify with processing scripts?

2. Do they spread their wallets across multiple server instances and say instead of having 1 wallet with 1 million receive addresses have 2 with 500k each as a sort of load balancer?

3. Could it work better if you had a limited pool of receive addresses say 100 and spread them over multiple people and determine the depositer by the address it comes from? I thought that would be a good solution so people have to send from an address but i suppose with 1 time use addresses etc this might not be a good idea? It seemed it would work well for proof of withdrawal with signing something from the sending address private key.

[1714811926]

1714811926

[1714811926]

1714811926

[1714811926]

1714811926

[1714811926]

1714811926

[1714811926]

1714811926

[Dabs]

Bitcoin Core should be able to handle a million addresses fine. (I've only tried about a hundred thousand.)

3. Could it work better if you had a limited pool of receive addresses say 100 and spread them over multiple people and determine the depositer by the address it comes from?

You don't want to do that. People will use web wallets or shared addresses, how do you determine who owns what. Use at least one address per person. Better if you give them a new deposit address after the first one has been sent any amount of coin.

[o0o0]

Bitcoin Core should be able to handle a million addresses fine. (I've only tried about a hundred thousand.)

3. Could it work better if you had a limited pool of receive addresses say 100 and spread them over multiple people and determine the depositer by the address it comes from?

You don't want to do that. People will use web wallets or shared addresses, how do you determine who owns what. Use at least one address per person. Better if you give them a new deposit address after the first one has been sent any amount of coin.

Doesn't this increase the load time though on the wallet with large amounts of addresses? Is there any way to keep a daemon load time at under 10 mins when there is site related wallet issues etc?

As to the new address each send i understand this is for more privacy and incase original address gets compromised etc why is it that exchanges leave the same address then unless the user requests a different address?

With the 100 thousand addresses you tried what was the daemon boot up time?

[Dabs]

Is there any way to keep a daemon load time at under 10 mins when there is site related wallet issues etc?
...
With the 100 thousand addresses you tried what was the daemon boot up time?

The time became a factor when the oldest address is very old (used in 2012 or 2013), or if the wallet has plenty of transactions. Took anywhere between 20 to 30 minutes, maybe 40. But I am using a HDD on a laptop. I'm on a spinner, not solid state.

If you create a new wallet today, the load time even with a rescan is quite short.

That's the key, it's the load or rescan time that takes time. When the wallet is running, it works fine.

A way to keep load time low is to use SSDs, or as I have stated in another thread, to store the whole thing in RAM. Get 128 GB of RAM and create a RAM drive. Use an old server (5 year old off-lease servers with that much RAM can be bought for $600 USD).

Or use a bunch of SSDs (maybe 8 to 12, or more) and run them all RAID 0 (zero).

[o0o0]

Is there any way to keep a daemon load time at under 10 mins when there is site related wallet issues etc?
...
With the 100 thousand addresses you tried what was the daemon boot up time?

The time became a factor when the oldest address is very old (used in 2012 or 2013), or if the wallet has plenty of transactions. Took anywhere between 20 to 30 minutes, maybe 40. But I am using a HDD on a laptop. I'm on a spinner, not solid state.

If you create a new wallet today, the load time even with a rescan is quite short.

That's the key, it's the load or rescan time that takes time. When the wallet is running, it works fine.

A way to keep load time low is to use SSDs, or as I have stated in another thread, to store the whole thing in RAM. Get 128 GB of RAM and create a RAM drive. Use an old server (5 year old off-lease servers with that much RAM can be bought for $600 USD).

Or use a bunch of SSDs (maybe 8 to 12, or more) and run them all RAID 0 (zero).

and wallet notify is still the best option? is this how exchanges and big services likely do it? or is there a better way?

[Abdussamad]

exchanges probably use custom implementations. back in the day we learned that one of the reasons mt. gox lost so many bitcoins is because their custom implementation didn't keep up with patches to the satoshi client. I don't think the need for a custom version has changed so i think exchanges are still probably using a custom implementation because otherwise they couldn't keep up with the load.