bitcoin generation broken in 0.3.8? (64-bit)

[lfm]

I was starting to wonder when my systems seemed to quit generating coins if there was something going on. They went from about 1 block / day to none in a week.

ArtForz in irc suggested I run a test isolated net with two nodes only connected to each other with empty wallet dir. I took a couple of quad core systems and set them up. they have produced no blocks in about 90 minutes now while hashing at a combined rate of over 8000 khash/sec. Is this evidence of a problem yet or is it more bad luck?

The systems are Linux 64 bit one Intel quad q6600 and one AMD quad phenom II 940.

[1714129356]

1714129356

[1714129356]

1714129356

[1714129356]

1714129356

[1714129356]

1714129356

[1714129356]

1714129356

[FreeMoney]

The difficulty went up again just after 3.8 was out. I have not gotten any with 4000khash in about 15 days, so 90 minutes of 8000k doesn't mean much. I'm sure we'd know if 3.8 wasn't generating any blocks.

[lfm]

The difficulty went up again just after 3.8 was out. I have not gotten any with 4000khash in about 15 days, so 90 minutes of 8000k doesn't mean much. I'm sure we'd know if 3.8 wasn't generating any blocks.

This is on an isolated test net with just two machines, connections = 1, difficulty = 1

[lfm]

The difficulty went up again just after 3.8 was out. I have not gotten any with 4000khash in about 15 days, so 90 minutes of 8000k doesn't mean much. I'm sure we'd know if 3.8 wasn't generating any blocks.

maybe only a problem with 64 bit linux or something

[knightmb]

I haven't had any problems with 0.3.8 release, generated some coin on windows and Linux clients (32/64)bit just fine this week.

[FreeMoney]

The difficulty went up again just after 3.8 was out. I have not gotten any with 4000khash in about 15 days, so 90 minutes of 8000k doesn't mean much. I'm sure we'd know if 3.8 wasn't generating any blocks.

This is on an isolated test net with just two machines, connections = 1, difficulty = 1

Oh, that is odd then, that would be like not finding one in over 500 hours, possible I guess, but getting out there

[lfm]

I haven't had any problems with 0.3.8 release, generated some coin on windows and Linux clients (32/64)bit just fine this week.

I guess I want to know how long I need to run 8000 khash/s at difficulty 1.0 to have any reasonable evidence of a problem. ArtForz said it should tell me in an hour or so.

[ArtForz]

At ~ 8Mhash/s combined not generating a block at 1.0 diff in over 1h is pretty unlikely.
As in, under 0.1% probability unlikely (50% is ~6 min, 1% ~42 min)
<edit>
Forgot to mention, he's running a custom compile of stock 0.3.8 sources.

[lfm]

I haven't had any problems with 0.3.8 release, generated some coin on windows and Linux clients (32/64)bit just fine this week.

I am wondering if its something odd in the way I have my systems set up or just various plain ubuntu installs so far as I know.

[lachesis]

I'm having a similar problem with my own compile of the SVN trunk. I just reverted to the stock builds to test and see what happens. I should try setting up my own network of two nodes with my builds.

[lfm]

Ok chatting with lachesis in irc he tried this and I get the same result: We added some prints in main.cpp at the SHA calls like so :

Code:

        loop
        {
            SHA256Transform(&tmp.hash1, (char*)&tmp.block + 64, &midstate);
            printf("mid hash =\n");
            for (int i = 0; i < 8; i++)
              printf(" %08x", ((unsigned *)&tmp.hash1)[i]);
            printf("\n");
            SHA256Transform(&hash, &tmp.hash1, pSHA256InitState);
            printf("full hash =\n");
            for (int i = 0; i < 8; i++)
              printf(" %08x", ((unsigned *)&hash)[i]);
            printf("\n");

            if (((unsigned short*)&hash)[14] == 0)

and then in the log we get:

mid hash =
 6a09e667 bb67ae85 3c6ef372 a54ff53a 510e527f 9b05688c 1f83d9ab 5be0cd19
full hash =
 6a09e667 bb67ae85 3c6ef372 a54ff53a 510e527f 9b05688c 1f83d9ab 5be0cd19
mid hash =
 6a09e667 bb67ae85 3c6ef372 a54ff53a 510e527f 9b05688c 1f83d9ab 5be0cd19
full hash =
 6a09e667 bb67ae85 3c6ef372 a54ff53a 510e527f 9b05688c 1f83d9ab 5be0cd19

repeating! The hash call isn't doing anything!
(he maybe got a different repeating value, I don't know)

[knightmb]

There is one behavior that I've noticed.

Block sync stalling.

When the client (be it Windows or Linux) has been running for a few days, *sometimes* they get stuck in block limbo. What happens is your client keeps trying to solve the current block and loses track of the entire block system. So as time goes by, other blocks are solved and your PC is still stuck on the same block. The problem is, if you stop and restart the client, it just picks up where it left off on the same stuck block. The only way I've seen to fix this is to delete the block chain so that the client will re-download it.

I've had to clear out a few block chains for a few servers because of this, I'll notice they are stuck on block 70,000 for example, while the rest of the network is working on block 70,839 for example.

That might explain why you go weeks without winning a block with a 24/7 PC going.

The server farm I had running before were spitting out blocks all day, but now that I'm just down to a few dozen servers and the difficulty is way up, I barely see about 3 or 4 blocks a day spread across all of them. But I know the 0.3.8 version is working like it should, except for the occasional block freeze.

[lfm]

This seems to be a different problem. The blocks do not seem to be "stuck" on my systems. The getinfo shows them up to date

It seems the sha256 code is not getting built right for linux 64. Not sure if/how it could work on some and not on others.

There is one behavior that I've noticed.

Block sync stalling.

When the client (be it Windows or Linux) has been running for a few days, *sometimes* they get stuck in block limbo. What happens is your client keeps trying to solve the current block and loses track of the entire block system. So as time goes by, other blocks are solved and your PC is still stuck on the same block.

...

[tcatm]

Did you compile it yourself? There's a define in makefile.unix. Something like -DCRYPTOPP_DISABLE_SSE2. Remove that. Else the SHA256_Transform will return the initstate. I had the same problem when switchting to 0.3.6.

[lfm]

Did you compile it yourself? There's a define in makefile.unix. Something like -DCRYPTOPP_DISABLE_SSE2. Remove that. Else the SHA256_Transform will return the initstate. I had the same problem when switchting to 0.3.6.

That seems to be the root of the problem. I think even the bundled binary for Linux 64 in 0.3.8 was compiled wrong then.

[lachesis]

That seems to be the root of the problem. I think even the bundled binary for Linux 64 in 0.3.8 was compiled wrong then.

That appears to fix it for me too.

EDIT:
I've uploaded corrected Linux builds to http://www.alloscomp.com/bitcoin/binaries/release-r123-2010-08-08/. Enjoy!

[knightmb]

Did you compile it yourself? There's a define in makefile.unix. Something like -DCRYPTOPP_DISABLE_SSE2. Remove that. Else the SHA256_Transform will return the initstate. I had the same problem when switchting to 0.3.6.

That seems to be the root of the problem. I think even the bundled binary for Linux 64 in 0.3.8 was compiled wrong then.

I just tested it out, I get the same results. Repeating values with stock, more random values with the flag modification. Probably needs to be fixed for the next release, though I'm not sure how mine are generating anything then?

[nimnul]

@lachesis: 404 not found

@knightmb: What about a fixed CentOS build?

[knightmb]

@lachesis: 404 not found

@knightmb: What about a fixed CentOS build?

Should be simple enough for CentOS, I was going to test it out first to see if those suffered the same issue, thought it seems like all the builds would suffer this issue if the compile flag is what makes the difference.

[knightmb]

This seems to be a different problem. The blocks do not seem to be "stuck" on my systems. The getinfo shows them up to date

It seems the sha256 code is not getting built right for linux 64. Not sure if/how it could work on some and not on others.

There is one behavior that I've noticed.

Block sync stalling.

When the client (be it Windows or Linux) has been running for a few days, *sometimes* they get stuck in block limbo. What happens is your client keeps trying to solve the current block and loses track of the entire block system. So as time goes by, other blocks are solved and your PC is still stuck on the same block.

...

I'm starting to wonder if the two are connected now since the hashing seems to get stuck, maybe that's why it gets stuck on a block.