Technical question regarding POS coins and CPU time

[allcrypt]

We are experimenting with a possible feature of having coins that generate POS actually credit the coins created to users who hold those coins in their AllCrypt.com balance.

A problem we're having in the preliminary testing of this is that SOME, not all, of the POS coins we host chew up 100% CPU time when they are, presumably, staking the POS coins.

I am not a coin dev, but I am a coder at heart. We have patch code that modifies every coin we run at AllCrypt.com (adding in missing features, like walletnotify on those coins that use it, or patching out 'dumb' restrictions like not being able to use the 'move' RPC command on amounts too small to send across the blockchain, etc), so I am personally familiar with a lot of how the code works. I was under the impression that there should not be much, if any, CPU usage for POS mining? You're not running random numbers against a target, it's based on coins you already have, and their age.

Am I misunderstanding how POS minting works? I'm trying to better understand it so that we can mitigate the CPU usage of these coins. In addition, not all coins are doing this. If it were all of them, I'd just throw in the towel and say "POS minting takes too much CPU to leave it turned on for the exchange's wallets" - but coins like JackpotCoin and LockCoin bounce between 5% and 35% CPU time when minting, while coins like LiteCoinPlus and CapitalCoin are consistently chewing up 100% CPU. CoolCoin and DistroBlitz are coins we've had to turn off, those never drop below 100%.

It seems the "polite" coins use about 25-30% CPU and the others take all they can.

We'd love to have the feature, but cannot afford to have wallets dominating the server because they are running wild with CPU time minting POS blocks.

If anyone has any insight on what exactly happens with POS minting, links (I've tried searching, all I find is links to options for bitcoin for POS, and not anything technical about how it works), insight on why some coins are nice and others go nuts, we'd love to hear it.

Thanks,
Ed
AllCrypt.com lead developer

[1714112798]

1714112798

[1714112798]

1714112798

[1714112798]

1714112798

[clemahieu]

Have you profiled the application?  Where's the CPU time being spent?  It's entirely possible the algorithm to calculate PoS is broken in a way that still consumes all the CPU time.

It's true PoS should result in lower CPU overhead but any algorithm anywhere, if improperly implemented, can give horrible efficiency.

[sx100]

I had some cool coins and the wallet gave my computer a BSOD each time I exited it. The block chain was growing incredibly quickly because the coin stake age required seemed like it was 5 minutes or some incredibly low value. I cannot remember what the CPU usage was, but the coins never seemed to stop staking.

[bee7]

Although the stake search thread is set the low priority it will eat up the CPU time if the number of transactions in the wallet is significant. For each input tx that is met staking age conditions it tries to find a so called PoS kernel for each second in the past since it completed the last pass through the set of transactions. The way the "standard" most advanced implementation does the search is sub-optimal IMO.

[allcrypt]

Have you profiled the application?  Where's the CPU time being spent?  It's entirely possible the algorithm to calculate PoS is broken in a way that still consumes all the CPU time.

It's true PoS should result in lower CPU overhead but any algorithm anywhere, if improperly implemented, can give horrible efficiency.

We actually recompiled one wallet with profiling and analyzed it with gprof, and for some reason all the time %'s were listed as 0. 0% time for everything. Tried it with another tool and it ran so slow, after an hour it still hadn't finished scanning/validating the blockchain. Figured we'd ask here and see if we could find what the issue was in the code itself.

[clemahieu]

Have you profiled the application?  Where's the CPU time being spent?  It's entirely possible the algorithm to calculate PoS is broken in a way that still consumes all the CPU time.

It's true PoS should result in lower CPU overhead but any algorithm anywhere, if improperly implemented, can give horrible efficiency.

We actually recompiled one wallet with profiling and analyzed it with gprof, and for some reason all the time %'s were listed as 0. 0% time for everything. Tried it with another tool and it ran so slow, after an hour it still hadn't finished scanning/validating the blockchain. Figured we'd ask here and see if we could find what the issue was in the code itself.

0% time for everything?  Are you running the program inside a virtual machine?  Virtual machines don't have profiling hardware so they'll ruin gprof results, otherwise I haven't seen that issue.

[allcrypt]

Although the stake search thread is set the low priority it will eat up the CPU time if the number of transactions in the wallet is significant. For each input tx that is met staking age conditions it tries to find a so called PoS kernel for each second in the past since it completed the last pass through the set of transactions. The way the "standard" most advanced implementation does the search is sub-optimal IMO.

So, if that is the case, it might behoove us to collect all those small transactions and group them into one huge send back to the wallet to clean them up? I can see how that would get exponentially worse... 300 small transactions lead to 300 small POS rewards which then leads to 600 small transactions leading to 600 smaller POS rewards, which turn into 1200 small POS rewards, etc...

Given we are an exchange with tons of small deposits (from mining pools) we have a much larger than usual number of transactions/addresses in the wallet than 'normal' wallets do. Tomorrow I'll take one of the offending coins and try consolidating all of the transactions into one address and see what that does to the CPU time for the wallet.

Thanks for the tip.

That being said, JPC has been one of our most active coins. It's still one of the most polite ones as far as CPU time and staking.

[bee7]

Have you profiled the application?  Where's the CPU time being spent?  It's entirely possible the algorithm to calculate PoS is broken in a way that still consumes all the CPU time.

It's true PoS should result in lower CPU overhead but any algorithm anywhere, if improperly implemented, can give horrible efficiency.

We actually recompiled one wallet with profiling and analyzed it with gprof, and for some reason all the time %'s were listed as 0. 0% time for everything. Tried it with another tool and it ran so slow, after an hour it still hadn't finished scanning/validating the blockchain. Figured we'd ask here and see if we could find what the issue was in the code itself.

What coin wallet you are talking about?
How many unspent inputs do you have in that wallet?

[bee7]

Although the stake search thread is set the low priority it will eat up the CPU time if the number of transactions in the wallet is significant. For each input tx that is met staking age conditions it tries to find a so called PoS kernel for each second in the past since it completed the last pass through the set of transactions. The way the "standard" most advanced implementation does the search is sub-optimal IMO.

So, if that is the case, it might behoove us to collect all those small transactions and group them into one huge send back to the wallet to clean them up? I can see how that would get exponentially worse... 300 small transactions lead to 300 small POS rewards which then leads to 600 small transactions leading to 600 smaller POS rewards, which turn into 1200 small POS rewards, etc...

Given we are an exchange with tons of small deposits (from mining pools) we have a much larger than usual number of transactions/addresses in the wallet than 'normal' wallets do. Tomorrow I'll take one of the offending coins and try consolidating all of the transactions into one address and see what that does to the CPU time for the wallet.

Thanks for the tip.

That being said, JPC has been one of our most active coins. It's still one of the most polite ones as far as CPU time and staking.

You could of course merge small inputs into bigger ones. Just keep in mind that some coins, e.g. Nova, has maximum reward limit per PoS stake transaction for the sake of maintaining good PoS difficulty. Otherwise everybody would glue all their inputs and switch on the wallet once a month for 5 minutes.

Edit:

provided presently you do not do stake minting at all it will not hurt the coin if you glue your inputs on your discretion and start to perform staking.

[allcrypt]

Have you profiled the application?  Where's the CPU time being spent?  It's entirely possible the algorithm to calculate PoS is broken in a way that still consumes all the CPU time.

It's true PoS should result in lower CPU overhead but any algorithm anywhere, if improperly implemented, can give horrible efficiency.

We actually recompiled one wallet with profiling and analyzed it with gprof, and for some reason all the time %'s were listed as 0. 0% time for everything. Tried it with another tool and it ran so slow, after an hour it still hadn't finished scanning/validating the blockchain. Figured we'd ask here and see if we could find what the issue was in the code itself.

0% time for everything?  Are you running the program inside a virtual machine?  Virtual machines don't have profiling hardware so they'll ruin gprof results, otherwise I haven't seen that issue.

Nope. Linux box. No VMS. Here's a snippet of the gprof output we got after running it against the gmon.out:
Flat profile:

Each sample counts as 0.01 seconds.
 no time accumulated

  %   cumulative   self              self     total          
 time   seconds   seconds    calls  Ts/call  Ts/call  name    
  0.00      0.00     0.00 2733398524     0.00     0.00  xor_salsa8(unsigned int
  0.00      0.00     0.00 354052679     0.00     0.00  operator<(uint256 const&
  0.00      0.00     0.00 41734266     0.00     0.00  GetStakeModifierSelection
  0.00      0.00     0.00 28480211     0.00     0.00  void WriteCompactSize<CHa
  0.00      0.00     0.00 19838644     0.00     0.00  Shutdown(void*)
  0.00      0.00     0.00 13992220     0.00     0.00  std::_Rb_tree<uint256, st
  0.00      0.00     0.00 13990945     0.00     0.00  CMerkleTx::GetDepthInMain
  0.00      0.00     0.00 13990786     0.00     0.00  CMerkleTx::GetDepthInMain
  0.00      0.00     0.00 12933773     0.00     0.00  CScript::GetOp2(__gnu_cxx
  0.00      0.00     0.00 11657139     0.00     0.00  CDataStream::read(char*,
  0.00      0.00     0.00 10263946     0.00     0.00  std::vector<unsigned char
  0.00      0.00     0.00  9092240     0.00     0.00  CWalletTx::IsSpent(unsign
  0.00      0.00     0.00  8314437     0.00     0.00  void std::vector<char, ze
  0.00      0.00     0.00  7166960     0.00     0.00  operator<(uint160 const&,
  0.00      0.00     0.00  5848572     0.00     0.00  GetTime()
  0.00      0.00     0.00  4991290     0.00     0.00  CTransaction::IsFinal(int

It's all 0% (And we let the wallet run for 30 mins before shutting it down and analyzing the results)

Very odd. Spent an hour googling why gmon would give 0% on everything.

[allcrypt]

Have you profiled the application?  Where's the CPU time being spent?  It's entirely possible the algorithm to calculate PoS is broken in a way that still consumes all the CPU time.

It's true PoS should result in lower CPU overhead but any algorithm anywhere, if improperly implemented, can give horrible efficiency.

We actually recompiled one wallet with profiling and analyzed it with gprof, and for some reason all the time %'s were listed as 0. 0% time for everything. Tried it with another tool and it ran so slow, after an hour it still hadn't finished scanning/validating the blockchain. Figured we'd ask here and see if we could find what the issue was in the code itself.

What coin wallet you are talking about?
How many unspent inputs do you have in that wallet?

It's a bunch of them. Some we turned POS on work nicely (Lock, JPC, some others) but many just kill the CPU.

Havent checked the unspent outputs, and its 11:22pm here, so, not at the servers to check easily.

[bee7]

Have you profiled the application?  Where's the CPU time being spent?  It's entirely possible the algorithm to calculate PoS is broken in a way that still consumes all the CPU time.

It's true PoS should result in lower CPU overhead but any algorithm anywhere, if improperly implemented, can give horrible efficiency.

We actually recompiled one wallet with profiling and analyzed it with gprof, and for some reason all the time %'s were listed as 0. 0% time for everything. Tried it with another tool and it ran so slow, after an hour it still hadn't finished scanning/validating the blockchain. Figured we'd ask here and see if we could find what the issue was in the code itself.

What coin wallet you are talking about?
How many unspent inputs do you have in that wallet?

It's a bunch of them. Some we turned POS on work nicely (Lock, JPC, some others) but many just kill the CPU.

Havent checked the unspent outputs, and its 11:22pm here, so, not at the servers to check easily.

Ok, does no matter. Just take that for each input for each second of time it has to perform sha256d over 28 bytes. Then you could estimate how many sha256d it has to calculate during the day apart from other necessary calculations to find a stake.

See the edit to my previous message.

[gatra]

Ok, does no matter. Just take that for each input for each second of time it has to perform sha256d over 28 bytes. Then you could estimate how many sha256d it has to calculate during the day apart from other necessary calculations to find a stake.

only profiling will tell, but I'm afraid that the "other necessary calculations to find a stake" part may be more cpu intensive than the sha256d
It has to access many blocks to get the "stake modifier" for each tx.

[allcrypt]

Going to take one of the offenders and try consolidating all the unspent outputs. One of the coins, litecoinplus, is using 99.8% CPU right now, and is one of our low volume coins. It has 5609 unspent outputs. Not sure if thats an absurdly high number - we've never bothered to look at things like that.

That being said, Jackpotcoin, one of our most popular coins, is on the same server as litecoinplus, is using 30% CPU, and has... wow. 2411 unspent. I would have sworn it was higher, given it's a higher traffic coin. But, I guess with more withdrawals those unspent outputs are being cleaned up a lot more often.

Going to clean up XLC and post the results. If it works, maybe the key is to just run some cleanup maintenance whenever the unspent inputs reach a certain threshold.

[othe]

Most pos coins use scrypt for staking which is of course utterly stupid, that comes from the fact that they are forked from Novacoin.
If they were clever they would have killed scrypt and used Sha256 as the staking PoW...

  0.00      0.00     0.00 2733398524     0.00     0.00  xor_salsa8(unsigned int

[bee7]

Most pos coins use scrypt for staking which is of course utterly stupid, that comes from the fact that they are forked from Novacoin.
If they were clever they would have killed scrypt and used Sha256 as the staking PoW...

  0.00      0.00     0.00 2733398524     0.00     0.00  xor_salsa8(unsigned int

This is not true! All PoS coins cloned from PPC and eventually form NVC are using sha256d to find a stake solution. Read the sources carefully: CBlock::GetHash() and ::Hash() have completely different implementations

[othe]

https://github.com/rat4/blackcoin/commit/850c9f683f925d12225e634fbb9e4f757daf8a35#diff-d41d8cd98f00b204e9800998ecf8427e

Not too sure but i think bc is forked from Novacoin?
However they use or used scrypt for the blockheader hash which causes orphans on fast blocktimes.

and Gethash returned scrypt_blockhash(CVOIDBEGIN(nVersion));

GetHash is called in CheckStakeKernelHash etc:

https://github.com/rat4/blackcoin/blob/850c9f683f925d12225e634fbb9e4f757daf8a35/src/kernel.cpp#L268

So while it might not be used for staking directly theres definately a lot of useless overhead especially if its no longer mineable.


Peercoin based forks should be fine, they use sha for everything.

PS: enough PoS for me for a next months

[bee7]

https://github.com/rat4/blackcoin/commit/850c9f683f925d12225e634fbb9e4f757daf8a35#diff-d41d8cd98f00b204e9800998ecf8427e

Not too sure but i think bc is forked from Novacoin?
However they use or used scrypt for the blockheader hash which causes orphans on fast blocktimes.

and Gethash returned scrypt_blockhash(CVOIDBEGIN(nVersion));

GetHash is called in CheckStakeKernelHash etc:

https://github.com/rat4/blackcoin/blob/850c9f683f925d12225e634fbb9e4f757daf8a35/src/kernel.cpp#L268

So while it might not be used for staking directly theres definately a lot of useless overhead especially if its no longer mineable.


Peercoin based forks should be fine, they use sha for everything.

PS: enough PoS for me for a next months

Ah, yes, I am sorry, you are correct about the call to GetHash inside CheckStakeKernelHash. I have been talking about the seeking a proof-of-stake hash itself. So, on top of that sha256d we also have a call to input's block hash calculation that is not cached anyhow (compare to optimizations done for CBlockIndex). So, the situation is even worse than I described.

[allcrypt]

Prelim findings: Success (sort of)

So I took one of our worst offenders, CoolCoin. I spent all afternoon learning a hell of a lot about creating raw transactions, signing them, transaction size, required fees, etc, I wrote an admit tool that checks a coin and groups up all the tiny unspent inputs, batches them into huge transactions, and sends them back to a new address in the wallet.

I reduced the # of unspent inputs on CoolCoin from 7667 to 85. The CPU usage instantly dropped from 99.8% to 32.9%.

I consider it a partial victory though, because as it continues to stake and we get deposits, the inputs will grow again. In addition, JackpotCoin, a coin that currently has 2432 unspent inputs, is using 34.9% CPU. 30x the number of inputs to check for staking, and using the same amount of CPU time. LOCK has twice the number of unspent inputs, and is almost always below JPC's percentage by at least 5-10%.

So there is absolutely something "wrong" with some of these coins. Going to keep searching. Collecting all the inputs together isn't the best solution. Going to read through the links posted here when I have some more time. Didn't expect to spend my afternoon fixing this mess (but, I learned a hell of a lot of cool new stuff. Might even help AllCrypt actually calculate real transaction fees if we implement internal transaction creation and batching - Not in a hurry to get your withdrawal? Let us batch it up with others, save on fees! Or something similar).

But I'd love to hear further ideas of the causes (and where to look in the code) of where the staking is happening. Just from my own coding experience, it seems like a runaway thread. Like JPC is well coded and only stakes when it needs to, and CoolCoin is a spaztic kid running around screaming "GIMMIE POS COINS WOOWOO!!!1!1!!one!eleven!1!"

[bee7]

As othe correctly mentioned the blockhash function is calculated per each try as well as sha256d. All these coins have different block hash algo's, so they require different time to get a block hash. I told that I see the implementation far from optimal. I already did some efforts to speed it up (my raspberry pi won't mint all my PoS stakes fast enough), but these changes are still raw and not well tested to make them publicly available.

Edit:
JPC is also persistently chews the list of inputs, but its developer merged in some optimizations/performance fixes from NVC.