BFGMiner --expiry value for solo mining (new block detection lag)

[jurASIC]

Hello all,

I started mining solo few days ago and I noticed a significant lag in new network block detection by both my Bitcoin Core (v0.9.0-beta 64-bit) and further by BFGMiner (3.10.0).  When I see a new block on blockchain.info, from that point it takes about 60 seconds for it to appear in my Bitcoin-Qt's debug.log and then it further takes about 120 seconds for my BFGMiner to report "New block detected on network".  That's about 3 minutes of lost time, since I am pretty sure that my ASIC miners won't start mining the new block until both my Bitcoin Core and BFGMiner know about it.

I started playing with --expiry option in BFGMiner and found out that if I make it less than the default 120 seconds, then BFGMiner will recognize there's a new block on the network faster.  For example:

./bfgminer --http-port 8330 -o http://127.0.0.1:8332 -u user -p password --coinbase-addr 1F2qbpkfApQm2gM63YvjTauyjaHxq2oekS --coinbase-sig "Cube102" --expiry 60

I would like to make the --expiry value even less, let's say 10 seconds, so I start working on a new block as soon as possible.  Really, 3 minutes is not acceptable since my miners on average lose 30% of chance to find a new block.  That's 30% of lost electricity.  Additionally I see quite a few "inputs already spent" and "already have block X" in my Bitcoind debug.log, which may be connected to the fact that I'm still working on the block that has been already solved.

My question therefore is, can I safely use --expiry 10 (or maybe even 5) or will that cause BFGminer to consider some shares stale, while they are still valid. I've read somewhere that while mining solo, you don't submit any shares, but I may be wrong.  Can I use --expiry 10 with my BFGMiner and what else can be done to make both Bitcoind and BFGMiner recognize faster the fact that there's a new block on the network while mining solo?  Thanks.

https://dl.dropboxusercontent.com/u/77333437/2014-03-30_13-07-58.jpg

[1714773616]

1714773616

[1714773616]

1714773616

[1714773616]

1714773616

[os2sam]

Hello all,

I started mining solo few days ago and I noticed a significant lag in new network block detection by both my Bitcoin Core (v0.9.0-beta 64-bit) and further by BFGMiner (3.10.0).  When I see a new block on blockchain.info, from that point it takes about 60 seconds for it to appear in my Bitcoin-Qt's debug.log and then it further takes about 120 seconds for my BFGMiner to report "New block detected on network".  That's about 3 minutes of lost time, since I am pretty sure that my ASIC miners won't start mining the new block until both my Bitcoin Core and BFGMiner know about it.

That's bizarre.  Do you have regular pools in your failover list?  If not try adding some and see when they detect a new block versus your Bitcoin-QT Client.  Maybe that will help with getting new work sooner too??

[-ck]

I do recall a push by the bfgminer author to put longpoll into the bitcoind client to inform mining software of block changes but no such thing exists, so perhaps he does nothing special to make sure you're working on a new block. Cgminer checks every 0.5 second when mining solo to bitcoind.

[jurASIC]

Thanks, I will install cgminer and see if the block changes as soon as I see a new one accepted in bitcoind debug log.  Another issue is that my installation of Bitcoin-Qt (on a Mac) doesn't report a new block for about 60 to 120 seconds after I see it in blockchain.info.  I run it without port 8333 open, so it has only 8 active connections.  Do you think that if I run it as a full node, with a lot more connections, that would ensure that it gets new blocks basically as soon as they appear on the blockchain?

And, as previous commenter suggested, I will add a regular pool to the failover list, because right now I don't have any.  That's probably a good idea anyway.

[-ck]

The more outgoing connections your bitcoind has, the faster it will pick up the block (provided you don't exhaust your bandwidth), and even better if you can find a supernode with ultra fast connection and consciously try and make your bitcoind detect it. Most of the bigger pools do that to minimise their risk of creating orphans.

[Luke-Jr]

The option you want is probably --scan-time
Alternatively, you could setup a pool for failover, and BFGMiner will use it to pick up on new blocks faster.
If you prefer to just use bitcoind itself, you can merge the longpolling branch.
Finally, many solo miners choose to run their own pool (but this is not for the faint of heart!).

As far as bitcoind connections, the p2p networking code is pretty bad, so it's more important to have good peers, than many peers.
Eligius's public node is at relay.eligius.st; I don't think other pools publish theirs, but you can try to find and add those too.
Use the bitcoind option -addnode=relay.eligius.st to use this.

[jurASIC]

I tried --scan-time with various values, but that doesn't seem to do anything for me, as far as getting new block information faster from my bitcoind.  The only option that did it was --expiry, and I set it to 10.  That causes BFGMiner to pick up new block information from my bitcoind in less than 10 seconds.  Since solo mining doesn't use shares, I guess it is perfectly OK to set it to 10, maybe even 5 (which was my original question )

As far as setting up a failover pool, I've tried that, and this is what I get:

  [2014-03-31 13:01:27] Block change for http://127.0.0.1:8332 detection via http://stratum.mining.eligius.st:3334 stratum
  [2014-03-31 13:01:27] Pool 0 http://127.0.0.1:8332 alive
  [2014-03-31 13:01:28] Pool 1 http://stratum.mining.eligius.st:3334 alive
  [2014-03-31 13:01:28] Pool 1 is hiding block contents from us

But then the current block in BFGMiner never gets updated.  I think this is because, as the above says, block changes are now handled by Eligius pool, but further down, it seems to be hiding block content.  So, it doesn't work at all, but I bet I must be doing something wrong....

Regarding Bitcoind, yes, adding addnode=relay.eligius.st helped a lot, thank you for the tip. It now picks up new blocks on the network in less than 20 seconds!  I also opened port 8333 on the router to run it as full node and used maxconnections=22 to try to limit traffic, although I've read somewhere that this alone won't guarantee that bitcoind won't kill my upload bandwidth.

So for now I'll stick to --expiry 10 for BFGMiner and keep getting new block info from my Bitcoind.  I may try to install my own pool software, thanks for the link.  BTW, do you know what I may be doing wrong with failover pool, since it seems to be hiding block info from me?

./bfgminer --http-port 8330 -o http://127.0.0.1:8332 -u user -p password -o http://stratum.mining.eligius.st:3334 -u user -p password --coinbase-addr 1F2qbpkfApQm2gM63YvjTauyjaHxq2oekS --coinbase-sig "Cube102"

[os2sam]

Regarding Bitcoind, yes, adding addnode=relay.eligius.st helped a lot, thank you for the tip. It now picks up new blocks on the network in less than a minute.  I also opened port 8333 on the router to run it as full node and used maxconnections=22 to try to limit traffic

Why limit your Bitcoind connections?  I have seen mine with 120+ connections, and it seems to work ok.

I get block height change updates 20 to 30 seconds after said block stamp, with around 30 connections, as viewed on blockchain.info.  With around 60 connections it seems to be closer to 10 seconds or less.  I don't have any static connections to "super nodes".

Those block time stamps aren't always correct.  I saw one yesterday that was 8 minutes in the future.

[jurASIC]

Why limit your Bitcoind connections?  I have seen mine with 120+ connections, and it seems to work ok.

Well, sometimes I use Chrome Remote Desktop and when I had a lot of active connections (don't remember how many) Chrome RD would hang up and it was really hard to get back in.  So I am trying not to get locked out...   And less than 20 seconds from the moment I see a new block in the blockchain is not bad at all.

Those block time stamps aren't always correct.  I saw one yesterday that was 8 minutes in the future.

I don't think I've ever noticed that, but I did see some new blocks that had timestamps earlier than the previous block.  For example:

292968   2014-03-28 21:38:43
292967   2014-03-28 21:49:41

[jurASIC]

I do recall a push by the bfgminer author to put longpoll into the bitcoind client to inform mining software of block changes but no such thing exists, so perhaps he does nothing special to make sure you're working on a new block. Cgminer checks every 0.5 second when mining solo to bitcoind.

Can cgminer be set up to mine with a network device?  (I have ASICminer Block Erupter Cubes)  I wanted to try it but I think I would need a miner that I could connect directly by USB to my computer running cgminer.  I don't see an option to specify a network port where it should listen to my miners...  Or is there a way to use it with BE Cubes?

[-ck]

I do recall a push by the bfgminer author to put longpoll into the bitcoind client to inform mining software of block changes but no such thing exists, so perhaps he does nothing special to make sure you're working on a new block. Cgminer checks every 0.5 second when mining solo to bitcoind.

Can cgminer be set up to mine with a network device?  (I have ASICminer Block Erupter Cubes)  I wanted to try it but I think I would need a miner that I could connect directly by USB to my computer running cgminer.  I don't see an option to specify a network port where it should listen to my miners...  Or is there a way to use it with BE Cubes?

Nope, cgminer doesn't act as a proxy to other mining software (and the cubes have their own primitive mining software).

[jurASIC]

Hi Luke,

Which is the latest Stable release of BFGMiner? In your post https://bitcointalk.org/index.php?topic=168174.0 you announce version 3.10.0 but below the latest stable release is listed as 3.5.7 (should I use this one as latest stable release).  Thanks!

[Luke-Jr]

Hi Luke,

Which is the latest Stable release of BFGMiner? In your post https://bitcointalk.org/index.php?topic=168174.0 you announce version 3.10.0 but below the latest stable release is listed as 3.5.7 (should I use this one as latest stable release).  Thanks!

Use the oldest maintained (listed) version that supports the functionality you need.
They both get bug fixes, but the newer version has more features (which are less tested).

[vmfx]

jurASIC >>

I have the same problem with the failover pool in bfgminer...

>>
[2014-03-31 13:01:28] Pool 1 is hiding block contents from us
>>

After which the current block in BFGMiner never gets updated... it just sits there.

Not sure why this is ? Luke ?

here are my settings...

./bfgminer -o http://127.0.0.1:8332 -u user -p pass  -o  http://stratum.btcguild.com:3333 -u user_worker -p pass -S Antminer:all --set-device antminer:clock=x0881  --scan-time 2 --queue 2 --expiry 2 --no-submit-stale --coinbase-addr mycoinbaseaddr --coinbase-sig "mysig"

then i updated to the latest source code and now i get this

 [2014-05-19 23:29:32] Started bfgminer 3.99.0
 [2014-05-19 23:29:32] Probing for an alive pool
 [2014-05-19 23:29:32] Network difficulty changed to 8.85G (63.38P)
 [2014-05-19 23:29:32] Pool 0 http://127.0.0.1:8332 alive
 [2014-05-19 23:29:32] No suitable long-poll found for http://127.0.0.1:8332
 [2014-05-19 23:30:32] Block change for http://127.0.0.1:8332 detection via http://stratum.btcguild.com:3333 stratum
 [2014-05-19 23:30:33] Pool 1 http://stratum.btcguild.com:3333 alive


and then it just hangs there again...

[telshena]

I can confirm the same results experienced by jurASIC when modifying -scantime and -expiry in BFGminer 4.10.0.

I too experienced a significant delay in the new block detection time when solo mining using BFGminer. Using a stopwatch, I timed the following:

-Start: Exact moment (minus reaction time) that the Bitcoin Core debug window showed a new block number in the Block Chain section on the Information tab. Note, this was based on the actual time Bitcoin Core (v 0.9.3.0-g40d2041-beta) received information of a new block with 39 connections (30 in/8 out), not the timestamp of said block.

-Stop: Exact moment (again, minus reaction time) that BFGminer showed 'New block detected on network'.

On two independent, subsequent tests (Block# 327012 and 327013), this resulted in an average new block detection latency of 1 minute 14 seconds. Keep in mind, BFGminer and Bitcoin Core are running on the same computer, thus WAN and/or LAN latency is not an issue.

As one would assume, the target command to alter would be -scantime, given the fact this dictates the upper bound on time spent scanning current work. I played with the argument here, changing it to 0, then 1 (would not accept -1), to no avail.

However, when changing the -expiry argument from the default 120 to 1, I experienced almost simultaneous synchronization between Bitcoin Core and BFGminer regarding the detection of a new block. Since -expiry only pertains to the upper bound on how many seconds after getting work we consider a share from it stale, I am not sure how this pertains to situations when solo mining. From what I've gathered, Bitcoin Core running in server mode does not have long poll capability. Given there are two different commands regarding expiry (e.g. -expiry and -expiry-lp), I wouldn't even know where to start on rationalizing how the expiry command impacts new block detection when solo mining.

Can any resident experts weigh in on this? As jurASIC noted, latency regarding new block detection does have a substantial impact on solo miners, as time wasted working on a block that has already been solved is an even further handicap to the solo miner. However, if altering this argument is somehow harming the solo miner's ability to submit block solutions, it would be good to reopen discussion on the specifics of what the -expiry impact is when not submitting shares to a pool.

Not sure this warrants a new thread, as the issue is quite specific; however, I do hope to revive discussion on this issue in hopes to clarify it for future solo miners. Perhaps it's something worth noting in the README...