[ANN][CLAM] CLAMs, Proof-Of-Chain, Proof-Of-Working-Stake, a.k.a. "Clamcoin"

[SuperClam]

     Today is my 1 year Clammaversary.  One your ago today I got my first Clam stake.    This was pre-lottery and pre-fixed reward.  It was for a whopping 0.00657097 clams. 
      Thanks to all for make the past year an enjoyable one, and wishing all the best for the next year.   

Awesome work with the stake data chilly2k - we're happy to have you as part of the family

[presstab]

The JD staking wallet checks something like 30k outputs in 4 seconds.

I believe this means that you are missing 3 variations of hashes for each output per every 4 seconds?

[dooglus]

The JD staking wallet checks something like 30k outputs in 4 seconds.

I believe this means that you are missing 3 variations of hashes for each output per every 4 seconds?

I don't understand what you're asking, sorry.

The CLAM protocol has a time granularity of 16 seconds. Every 16 seconds it checks each of its unspent outputs to see if it can stake, then sleeps until then next 16-second "tick" of the clock.

If it takes you more than 16 seconds to check all your unspent outputs then you'll be missing out on staking opportunities, because you'll be falling further and further behind. JD is able to do all its staking work in 4 seconds and have a 12 second snooze before the next opportunity comes up.

I don't understand the bit about "missing 3 variations of hashes". There's only one hash per output per 16 seconds.

Now if JD was able to check all of its outputs in 0.001 seconds then it would probably have a lower orphan rate, since it is kind of a race. Maybe that's what you're referring to?

[dooglus]

Awesome work with the stake data chilly2k - we're happy to have you as part of the family

Deb calls it "the clamily".

[dooglus]

4 Seconds? Is that needed for finding a block each time?

That's how long it takes to search for staking opportunities each 16 seconds. Most times you check you find that there isn't any such opportunity. There's only one per minute globally, so even with 100% of the staking weight you'd have around a 1 in 4 chance of any particular search being successful. When we do find a block, it will be at a random point through that 4 second search, and so on average a successful search takes 2 seconds ((0 + 4) / 2).

Given a difficulty leading to 16 seconds these 4 seconds would be huge. I mean the difference to orphan blocks or get block orphaned. Or isnt it needed to calculate a block?

I don't think you're understanding still. "Difficulty leading to 16 seconds" isn't what's happening. The difficulty adjusts how hard it is to stake a block. It adjusts such that we find around one block per minute. But blocks can only be found when the time (in seconds since some date in 1970) is a multiple of 16. That only happens every 16 seconds. That's fixed by the protocol (until the developers change the protocol again, of course), and isn't related to the difficulty.

So the 4 seconds are real and you meant it goes through each of that outputs (shouldnt it be inputs as long as they arent sent out?) and checks if it finds a hash? Does the output amount matter here? I mean you described rounding the amount of clams down to an integer. Does this apply to the address these outputs are on, so a big amount of clams or only to the single output? If the latter then one could get an advantage by sending the clams in amounts of 1 to a new address. The chance to find a block would be maximized?

It's a real 4 seconds. 4 seconds out of every 16 seconds the CPU on one core of JD's staking wallet server is pegged at 100%. They're outputs of the transactions that created them. They're not the inputs of any transactions yet, or they wouldn't be unspent. They're potential inputs, if you like, but actual outputs. When they stake they become inputs of the staking transaction.

The rounding down to an integer was related to how the age of an output affected its staking power in an older version of CLAM. I think it used to multiply the value by the age and round down to an integer. I don't think it does that rounding any more, or the multiplication by the age. These days the staking power (called the "weight") is just the same as the value in CLAMs. Each output is considered separately. It doesn't matter if you have lots of 1 CLAMs outputs on a single address, or in lots of different addresses. They each get their own individual chance of staking, with a probability proportional to their own individual value in CLAMs.

There is a benefit to splitting your outputs up into several smaller outputs. Suppose you have 1000 CLAMs. It will stake very quickly, and become 1001 CLAMs. But then it will take 8 hours to mature before it can stake again. The best you could hope for it that it will stake 3 times per day (since that's how many 8 hour maturation periods you can fit into a day).

If instead you split it into 1000 outputs of size 1, each one tries to stake independently. Each one has a 1000 times lower chance of staking than the 1000 CLAM output did, but there are 1000 of them, so it takes roughly the same time for one of them to stake, and turn from 1 CLAM to 2 CLAMs. Then, however, only the 2 CLAM output is frozen for 8 hours while it matures. The other 999 CLAMs continue trying to stake. So you have saved yourself an 8 hour wait for 99.9% of your value.

If you split your value up into *too* many outputs, you'll have too much hashing to do every 16 seconds that you won't be able to get through it all. And if you ever want to spent your outputs, having them split up into millions of tiny pieces makes the transaction which spends them very big (and so very expensive in tx fees).

So there's a tradeoff - split enough, but not too much.

[presstab]

The JD staking wallet checks something like 30k outputs in 4 seconds.

I believe this means that you are missing 3 variations of hashes for each output per every 4 seconds?

I don't understand what you're asking, sorry.

The CLAM protocol has a time granularity of 16 seconds. Every 16 seconds it checks each of its unspent outputs to see if it can stake, then sleeps until then next 16-second "tick" of the clock.

If it takes you more than 16 seconds to check all your unspent outputs then you'll be missing out on staking opportunities, because you'll be falling further and further behind. JD is able to do all its staking work in 4 seconds and have a 12 second snooze before the next opportunity comes up.

I don't understand the bit about "missing 3 variations of hashes". There's only one hash per output per 16 seconds.

Now if JD was able to check all of its outputs in 0.001 seconds then it would probably have a lower orphan rate, since it is kind of a race. Maybe that's what you're referring to?

Again, I may be picturing this incorrectly in my head, so let me explain my reading of the code and see if it lines up with yours.

15 seconds is the future drift limit (which is I think what you are referring to when you say time granularity?).

https://github.com/nochowderforyou/clams/blob/master/src/main.h#L74

Code:

inline int64_t FutureDriftV2(int64_t nTime) { return nTime + 15; }

and the search interval is 1 second.

https://github.com/nochowderforyou/clams/blob/master/src/main.cpp#L2603

Code:

int64_t nSearchInterval = IsProtocolV2(nBestHeight+1) ? 1 : nSearchTime - nLastCoinStakeSearchTime;

The stake hashing is iterated by the search interval not the future drift.
https://github.com/nochowderforyou/clams/blob/master/src/wallet.cpp#L2108

Code:

for (unsigned int n=0; n<min(nSearchInterval,(int64_t)nMaxStakeSearchInterval) && !fKernelFound && pindexPrev == pindexBest; n++)

The search interval appears to be one second after protocol 2, so that is why I am lead to the conclusion that if it takes 4 seconds to hash a set of outputs for one timestamp, then you have have skipped 3 timestamps that you could have tested. And of course this would only apply to very very large groups of outputs, for normal stakers this is a non issue.

I also can't find the relevant part of the code that tells it to pause looking for 16 seconds. I may be wrong, but I think there is no pause (a simple monitoring of the CPU consumption could give us a clue too). All I see is the normal minersleep parameter that is set at 500 milliseconds.

Again I could be totally wrong, and completely missing some of those parts of the code that you mention. The staking code is so all over the place that its hard to track down everything.



 

[dooglus]

Again, I may be picturing this incorrectly in my head, so let me explain my reading of the code and see if it lines up with yours.

Good idea. When you're specific like that it's easier to pinpoint where we're talking at cross purposes.

15 seconds is the future drift limit (which is I think what you are referring to when you say time granularity?).

No, I'm talking about kernel.h:

// To decrease granularity of timestamp
// Supposed to be 2^n-1
static const int STAKE_TIMESTAMP_MASK = 15;

and main.cpp:

    if (IsProtocolV2(nBestHeight+1))
        txCoinStake.nTime &= ~STAKE_TIMESTAMP_MASK;
    int64_t nSearchTime = txCoinStake.nTime; // search to current time

That is 'AND'ing the time with binary ~1111, ie. zeroing the last 4 bits, ie. rounding down to a multiple of 16.

and the search interval is 1 second.

https://github.com/nochowderforyou/clams/blob/master/src/main.cpp#L2603

Code:

int64_t nSearchInterval = IsProtocolV2(nBestHeight+1) ? 1 : nSearchTime - nLastCoinStakeSearchTime;

No, because look at the lines before that:

   if (nSearchTime > nLastCoinStakeSearchTime)
    {
        int64_t nSearchInterval = IsProtocolV2(nBestHeight+1) ? 1 : nSearchTime - nLastCoinStakeSearchTime;

It only gets into that block if nSearchTime is greater than the last search time. And nSearchTime has already been rounded down to a multiple of 16 seconds. So it will only get into this block at most once per 16 seconds.

I also can't find the relevant part of the code that tells it to pause looking for 16 seconds.

... and now you can!

[presstab]

Interesting about the mask, that is the part of the code I was not processing correctly. Ok so I think we have the wait 16 seconds part of it cleared.

No, because look at the lines before that:

    if (nSearchTime > nLastCoinStakeSearchTime)
    {
        int64_t nSearchInterval = IsProtocolV2(nBestHeight+1) ? 1 : nSearchTime - nLastCoinStakeSearchTime;

It only gets into that block if nSearchTime is greater than the last search time. And nSearchTime has already been rounded down to a multiple of 16 seconds. So it will only get into this block at most once per 16 seconds.

After 16 seconds have passed this code continues, and nSearchInterval becomes 1. I still can't follow exactly where it iterates 16 different timestamps in one hashing session? I might have to add a couple of log prints with timestamps into the code to visualize this better.

[SuperClam]

Awesome work with the stake data chilly2k - we're happy to have you as part of the family

Deb calls it "the clamily".

Interesting about the mask, that is the part of the code I was not processing correctly. Ok so I think we have the wait 16 seconds part of it cleared.

No, because look at the lines before that:

   if (nSearchTime > nLastCoinStakeSearchTime)
    {
        int64_t nSearchInterval = IsProtocolV2(nBestHeight+1) ? 1 : nSearchTime - nLastCoinStakeSearchTime;

It only gets into that block if nSearchTime is greater than the last search time. And nSearchTime has already been rounded down to a multiple of 16 seconds. So it will only get into this block at most once per 16 seconds.

After 16 seconds have passed this code continues, and nSearchInterval becomes 1. I still can't follow exactly where it iterates 16 different timestamps in one hashing session? I might have to add a couple of log prints with timestamps into the code to visualize this better.

dooglus can verify and expound on this; but, the concept is that it doesn't iterate all 16 One second increment timestamps.  In fact, if you look at the timestamps of on-chain blocks (approved by network consensus) - all of them have 16 second interval timestamps.

CheckCoinStakeTimestamp() mandates that block timestamps must follow the 16 second mask.
You could hash all 16 seconds if you pleased, but peers would reject the block as a timestamp violation.

[presstab]

Awesome work with the stake data chilly2k - we're happy to have you as part of the family

Deb calls it "the clamily".

Interesting about the mask, that is the part of the code I was not processing correctly. Ok so I think we have the wait 16 seconds part of it cleared.

No, because look at the lines before that:

   if (nSearchTime > nLastCoinStakeSearchTime)
    {
        int64_t nSearchInterval = IsProtocolV2(nBestHeight+1) ? 1 : nSearchTime - nLastCoinStakeSearchTime;

It only gets into that block if nSearchTime is greater than the last search time. And nSearchTime has already been rounded down to a multiple of 16 seconds. So it will only get into this block at most once per 16 seconds.

After 16 seconds have passed this code continues, and nSearchInterval becomes 1. I still can't follow exactly where it iterates 16 different timestamps in one hashing session? I might have to add a couple of log prints with timestamps into the code to visualize this better.

dooglus can verify and expound on this; but, the concept is that it doesn't iterate all 16 One second increment timestamps.  In fact, if you look at the timestamps of on-chain blocks (approved by network consensus) - all of them have 16 second interval timestamps.

CheckCoinStakeTimestamp() mandates that block timestamps must follow the 16 second mask.
You could hash all 16 seconds if you pleased, but peers would reject the block as a timestamp violation.

Wow! Ok now that you say that I see the same 16 second mask in the check stake code. Now this all makes sense in the context dooglus put it in. This is a pretty cool way to hash for proof of stake. Now I know why I was so lost :/

[dooglus]

Wow! Ok now that you say that I see the same 16 second mask in the check stake code. Now this all makes sense in the context dooglus put it in. This is a pretty cool way to hash for proof of stake. Now I know why I was so lost :/

The nSearchInterval variable used to (before v2 of the protocol) be set to how many seconds since the last time we tried hashing, and the hashing function would then iterate through all those missed seconds.

In v2 we don't want it doing that, so we set the variable to 1 so we get a single iteration of the loop.

The 16 second mask thing makes it 16 times easier to find a block than it was before, since we get to try 16 times less often, but still want the same frequency of blocks being solved. This makes it 16 times more likely that two peers will find a block at the same time, and so we end up with 16 times as many orphans as before. Other than that it's cool though.

Oh, and as for SuperClam's "dooglus can verify and expound on this; but", I don't need to. She pretty much nailed it.

[presstab]

Its a very clever concept. Essentially you were able to reduce the amount of work involved in the proof of stake hashing, while doing it in a manner that does not compromise solving blocks because the difficulty will adjust to make the statistics approximately correct.

I have always wondered why the CLAM difficulty chart moves in such smooth waves compared to other coins, I suppose this could have something to do with that. Although now I wouldn't be surprised if you clammers redesigned the difficulty adjustment code as well.

[dooglus]

Its a very clever concept. Essentially you were able to reduce the amount of work involved in the proof of stake hashing, while doing it in a manner that does not compromise solving blocks because the difficulty will adjust to make the statistics approximately correct.

I have always wondered why the CLAM difficulty chart moves in such smooth waves compared to other coins, I suppose this could have something to do with that. Although now I wouldn't be surprised if you clammers redesigned the difficulty adjustment code as well.

Three things:

1) When the time granularity changed from 1 second to 16 seconds, another accidental change went in at the same time which missed out a factor of 10^8 in the difficulty calculation. It made blocks 100 million times easier to solve, but gave us 16 times less chances to try, so the two factors combined made it 6.25 million times easier than before to stake. As a result everyone was staking every 16 seconds, everyone was orphaning everyone else for a few hours, the network difficulty adjusted upwards very quickly, and after 3 or 4 hours everything was back to normal. Is was kind of amazing to watch the difficulty adjustment code do its thing and deal with the accidental 10^8 error. Do you have long-term difficulty charts? If so you'll easily see the time I'm talking about. You need a log scale, or it will look like the difficulty was 0 before the steep rise.

Edit: I don't see how to adjust the date range on your chart, but http://blocktree.io/charts/CLAM shows this:



The difficulty wasn't really 0 before November 2014, it just looks that way when you don't use a log y-axis.

2) I don't think the CLAM project is responsible for the 16 second granularity idea or code. I think it is from one of the upstream projects. I've never paid attention to any of them, but I think it was dark coin, black coin, or something like that.

3) The difficulty adjustment code was written by me specifically for CLAM, and went live at the same time as the v2 protocol and the 10^8 error. It's not ideal how it waves up and down several times per day, but it's better than the previous system of difficulty adjustment, which was far too reactive to how long the last 1 block took to stake.

[SuperClam]

...
3) The difficulty adjustment code was written by me specifically for CLAM, and went live at the same time as the v2 protocol and the 10^8 error. It's not ideal how it waves up and down several times per day, but it's better than the previous system of difficulty adjustment, which was far too reactive to how long the last 1 block took to stake.

The network has grown a great deal and achieved a certain measure of stability.

If we ever fork again, what would be the downside of increasing the window and further smoothing difficulty?

[dooglus]

If we ever fork again, what would be the downside of increasing the window and further smoothing difficulty?

I just wonder if there isn't a better way of doing it that doesn't cause this constant swinging up and down.

Is there a chart of the DOGE difficulty over time I wonder. They use some difficulty adjustment system with a fancy name that I forgot. Turbo lightning. Black vortex. You know, something silly like that...

[presstab]

Interesting, thanks for filling me in a bit on the history. I might do some searching into which coins also have that granularity in it. I have never seen it or heard about it before. But then again I tend to stay in my own little corner, so why would I know

My long range chart for diff is here. My charts aren't anything fantastic at the moment, hope to make them better in the future. My focus has been more behind the scenes this last two weeks. I just launched a new indexing method that should keep the addresses fairly precise, as it will not DB any transactions until they are deep enough in the chain.

Also, preparing to stake clams I think I may add my 'splitblock' feature to my own build of the GUI that allows splitting one output into multiple outputs in one easy transaction, unless this is something clams has built in somewehre? I think the best strategy is more outputs = more chances to stake, instead of the theory that higher value outputs will stake easier?

[SuperClam]

Interesting, thanks for filling me in a bit on the history. I might do some searching into which coins also have that granularity in it. I have never seen it or heard about it before. But then again I tend to stay in my own little corner, so why would I know
My long range chart for diff is here. My charts aren't anything fantastic at the moment, hope to make them better in the future. My focus has been more behind the scenes this last two weeks. I just launched a new indexing method that should keep the addresses fairly precise, as it will not DB any transactions until they are deep enough in the chain.
Also, preparing to stake clams I think I may add my 'splitblock' feature to my own build of the GUI that allows splitting one output into multiple outputs in one easy transaction, unless this is something clams has built in somewehre? I think the best strategy is more outputs = more chances to stake, instead of the theory that higher value outputs will stake easier?

We have quite a bit of additional functionality around the stake system, such as:

Flags:
-maxstakevalue, -splitsize, -combinelimit

Commands "sendtoaddress", "sendfrom", and "rawtransaction" allow JSON style specification, like {"amount":aaa,"count":ccc} to cause the send amount to be split into ccc outputs each of size aaa CLAMs.

[SebastianJu]

Given a difficulty leading to 16 seconds these 4 seconds would be huge. I mean the difference to orphan blocks or get block orphaned. Or isnt it needed to calculate a block?

I don't think you're understanding still. "Difficulty leading to 16 seconds" isn't what's happening. The difficulty adjusts how hard it is to stake a block. It adjusts such that we find around one block per minute. But blocks can only be found when the time (in seconds since some date in 1970) is a multiple of 16. That only happens every 16 seconds. That's fixed by the protocol (until the developers change the protocol again, of course), and isn't related to the difficulty.

Yes, but what i wanted to ask, just dice has the chance to find a block each 16 seconds. When jd really needs 4 seconds to go through all outputs then this is huge. Most other staking wallets dont have so many outputs so that they most probably need way less time to search for hashes. This sounds, at least, like the chance of find blocks that get orphaned are high. Not taken the time into consideration that other miners need to propage their block to jd.

When you think about it... clam can be lucky that you are the owner of jd. You easily could orphan all other blocks, where jd finds a block too, and only find blocks yourself, or raise that chance only so slight that its not found out. Nobody could proof how long jd needs to proof a block another miner found.

So the 4 seconds are real and you meant it goes through each of that outputs (shouldnt it be inputs as long as they arent sent out?) and checks if it finds a hash? Does the output amount matter here? I mean you described rounding the amount of clams down to an integer. Does this apply to the address these outputs are on, so a big amount of clams or only to the single output? If the latter then one could get an advantage by sending the clams in amounts of 1 to a new address. The chance to find a block would be maximized?

It's a real 4 seconds. 4 seconds out of every 16 seconds the CPU on one core of JD's staking wallet server is pegged at 100%. They're outputs of the transactions that created them. They're not the inputs of any transactions yet, or they wouldn't be unspent. They're potential inputs, if you like, but actual outputs. When they stake they become inputs of the staking transaction.

I get now how you come to name them outputs. You see them from a protocol perspective. I saw them from the wallet perspective. They are inputs into the wallet. Though outputs transactionwise.

The rounding down to an integer was related to how the age of an output affected its staking power in an older version of CLAM. I think it used to multiply the value by the age and round down to an integer. I don't think it does that rounding any more, or the multiplication by the age. These days the staking power (called the "weight") is just the same as the value in CLAMs. Each output is considered separately. It doesn't matter if you have lots of 1 CLAMs outputs on a single address, or in lots of different addresses. They each get their own individual chance of staking, with a probability proportional to their own individual value in CLAMs.
There is a benefit to splitting your outputs up into several smaller outputs. Suppose you have 1000 CLAMs. It will stake very quickly, and become 1001 CLAMs. But then it will take 8 hours to mature before it can stake again. The best you could hope for it that it will stake 3 times per day (since that's how many 8 hour maturation periods you can fit into a day).

If instead you split it into 1000 outputs of size 1, each one tries to stake independently. Each one has a 1000 times lower chance of staking than the 1000 CLAM output did, but there are 1000 of them, so it takes roughly the same time for one of them to stake, and turn from 1 CLAM to 2 CLAMs. Then, however, only the 2 CLAM output is frozen for 8 hours while it matures. The other 999 CLAMs continue trying to stake. So you have saved yourself an 8 hour wait for 99.9% of your value.

If you split your value up into *too* many outputs, you'll have too much hashing to do every 16 seconds that you won't be able to get through it all. And if you ever want to spent your outputs, having them split up into millions of tiny pieces makes the transaction which spends them very big (and so very expensive in tx fees).

So there's a tradeoff - split enough, but not too much.

Ah ok. I thought about this. So i guess you could calculate the optimum amount of clams in an output in order to get the most stakes and consider the 8 h wait time. Then add the average withdraw amount over time with the staked clams and calculate the additional fees (for taking many outputs as inputs) into consideration too. Then use a server that brings down calculation time to < 1 second and you would have the optimum setup. Right?

[Bitcoiner2015]

If we ever fork again, what would be the downside of increasing the window and further smoothing difficulty?

I just wonder if there isn't a better way of doing it that doesn't cause this constant swinging up and down.

Is there a chart of the DOGE difficulty over time I wonder. They use some difficulty adjustment system with a fancy name that I forgot. Turbo lightning. Black vortex. You know, something silly like that...

Do you mean kimoto gravity well?

[tspacepilot]

Interesting, thanks for filling me in a bit on the history. I might do some searching into which coins also have that granularity in it. I have never seen it or heard about it before. But then again I tend to stay in my own little corner, so why would I know
My long range chart for diff is here. My charts aren't anything fantastic at the moment, hope to make them better in the future. My focus has been more behind the scenes this last two weeks. I just launched a new indexing method that should keep the addresses fairly precise, as it will not DB any transactions until they are deep enough in the chain.
Also, preparing to stake clams I think I may add my 'splitblock' feature to my own build of the GUI that allows splitting one output into multiple outputs in one easy transaction, unless this is something clams has built in somewehre? I think the best strategy is more outputs = more chances to stake, instead of the theory that higher value outputs will stake easier?

We have quite a bit of additional functionality around the stake system, such as:

Flags:
-maxstakevalue, -splitsize, -combinelimit

Commands "sendtoaddress", "sendfrom", and "rawtransaction" allow JSON style specification, like {"amount":aaa,"count":ccc} to cause the send amount to be split into ccc outputs each of size aaa CLAMs.

If I have 1 clam and I split it as far as possible down to 100,000,000 "clamtoshis" in order to have the best chance to stake is this really the right thing to do?  And if everyone else does it wouldn't it make your UTXO set astronomically huge?  And wouldn't it make spending any reasonable amount of clams extremely expensive (because of all the nano-outputs you'd have to reference)?

I'm sure I'm missing something.  Thanks in advance!