Mining protocol bandwidth comparison: GBT, Stratum, and getwork

[kano]

.... you are increasing transaction confirm times by 4 times with GBT versus Stratum.....

... blah blah blah ...

-wk

P.S. - The OP was actually about bandwidth usage, not confirmation times... FWIW, I'm at 0.49 shares per 2KB with stock-settings bfgminer on Eligius using GBT with my four x6500s (~1.5GH share acceptance rate), after about 10 hours.... which is actually better than Luke-Jr's result, for whatever reason.  Seems stock setting is a scantime of 60 seconds and expiry of 120 seconds... which IIRC is the same as cgminer.

No it's not the same - YRI

Again read what you copied from me: "increasing" - yeah tricky word that.

As ckolivas pointed out above, yes I'm referring to how long you think work should be worked on before getting new work with possibly new transactions.

How long you have work determines exactly that, your effect on transaction confirm times.
Stratum on the original stratum pools and GBT in cgminer use the figure of 30s to decide when new work should be sent/received.
BarbieMiner GBT uses 120s (not 80s) thus his GBT increases it by 4x what cgminer does.

[Luke-Jr]

Note that while kano is correct that updating the template less often means a delay of the first confirmation of transactions sent in that time period, it's nowhere near the relevance he claims. The difference between 30 seconds and 120 seconds* is 90 seconds or a minute and a half. That's the extent of the confirmation delays as well. The fact that blocks can take that much time just to propagate the network makes it even less relevant. If "oh no, I sent my transaction 90 seconds too late!" is more of a concern to you than "oh no, someone double-spent to me and I'm scammed!" then I don't know what to say.

* No, he isn't right about stratum v GBT update times being 30/120 - that's entirely pool dependent really (and none I know of are as high as 120 in practice). I'm just using it as a worst-case example.

[kano]

The "Bytes Received" counter is cgminer is simply what CURL tells us it has (sent and) received.

If you think there is some problem with that - then feel free to point out what it is - or go bug the CURL developers and tell them their code is wrong.

No, you failed to read cURL's documentation, which describes what it actually does. cURL does not have a counter for bytes sent/received on the network.
...

No, you failed (as usual you said nothing yet again) to state that your issue is that you may be compressing the data thus the network bytes may (or may not) be compressed.

However, if you bothered to read the curl documentation yourself, it says:

CURLINFO_SIZE_UPLOAD
Pass a pointer to a double to receive the total amount of bytes that were uploaded.

CURLINFO_SIZE_DOWNLOAD
Pass a pointer to a double to receive the total amount of bytes that were downloaded. The amount is only for the latest transfer and will be reset again for each new transfer.

So ... yes I can add another counter for network bytes which MAY be different depending upon the pool and the compression options available.
I'll add it soon.

The current counter is still correct.

You're still hiding things.
Typical and common.
You know this depends on a number of settings, but don't want people to know that they may well still be sending and receiving 50x the data with GBT vs Stratum depending on those settings.

[crazyates]

Lots of good discussion, and I love the flip-flop point-counterpoint going on here.

Third, there has still not been any valid explanation for how sending the miner all the transactions in their entirety actually improves the security of the network. Unless someone is running mining software and a local bitcoin node and is comparing the values from each, and then decides what transactions overlap, and what are valid transactions the pool has chosen to filter out, and then determined that the data is enough of the transaction list to be a true set of transactions, having the transactions does not serve any purpose. Pool security validity software should be developed that does this, and people should use said approach if they care to confirm the pool is behaving. luke's software does not remotely do this to verify the transactions. It simply grabs them and then if it doesn't get them claims the pool is doing something untoward if it doesn't match the template. The transactions could be anything. It also completely ignores the fact that the vast majority of miners run their mining software on machines that aren't actually running bitcoin nodes with which to check the transactions. While the main bitcoin devs might wish this to be the case, it's not remotely the reality and is not going to become it.

Anyone care to comment on this? We can argue about transaction times and network usage, but those are all trivial compared to this. The above issue has to be addressed first, and then the little stuff can be worked out (in whichever protocol and proper implementation of said protocol).

[kano]

Lots of good discussion, and I love the flip-flop point-counterpoint going on here.

Third, there has still not been any valid explanation for how sending the miner all the transactions in their entirety actually improves the security of the network. Unless someone is running mining software and a local bitcoin node and is comparing the values from each, and then decides what transactions overlap, and what are valid transactions the pool has chosen to filter out, and then determined that the data is enough of the transaction list to be a true set of transactions, having the transactions does not serve any purpose. Pool security validity software should be developed that does this, and people should use said approach if they care to confirm the pool is behaving. luke's software does not remotely do this to verify the transactions. It simply grabs them and then if it doesn't get them claims the pool is doing something untoward if it doesn't match the template. The transactions could be anything. It also completely ignores the fact that the vast majority of miners run their mining software on machines that aren't actually running bitcoin nodes with which to check the transactions. While the main bitcoin devs might wish this to be the case, it's not remotely the reality and is not going to become it.

Anyone care to comment on this? We can argue about transaction times and network usage, but those are all trivial compared to this. The above issue has to be addressed first, and then the little stuff can be worked out (in whichever protocol and proper implementation of said protocol).

Or it could be admitted by the GBT zealots that anyone who can program can write a program to do exactly this without even needing the miner to do it.
It's just a few simple lines of code to hook a whole module to do whatever you like to hook into bitcoind to see what transactions are available and what transactions were put into each block.

Not only that, but doing this will easily point out all these imaginary evil pools that gmaxwell and his boss Luke-Jr say exist and make it blatantly simple to say that a pool is doing these nefarious things and thus anyone would have the simple proof to bring out the light of truth and salvation about these evil pools ... and then everyone mining on them would shout praise be to Luke for being the BTC saviour - everyone go crucify the saviour ...

However, there is also a complete and major fallacy with the idea that either of these methods will completely resolve any such security problem.
They both have the same problem of not knowing the list of transactions that any particular bitcoind contains ... ... ...

[2112]

This is why I, and most other Bitcoin developers, work to improve standardization and make Bitcoin more friendly to multiple competing implementations maintained by different people. Remember that GBT is the standard developed by free cooperation of many different people with different projects, whereas stratum was the protocol developed behind closed doors by a single individual.

Stratum family of protocols was not developed behind closed doors. The development started right here with a lively discussion:

https://bitcointalk.org/index.php?topic=55842.0

From the very beginning it was ignored by the core development team because of its key features:

1) allows a clean break from the polling only (RPC) behaviour deeply embedded into the archotecture of the Satoshi's client.

2) shows a way forward that lies after abandoning the legacy of long-poll (one of the most horrible hacks in the Bitcoin millieu) and correctly using two-way transport ability of a single TCP/IP socket.

Those things must have been a complete cultural shock to the core development team: a protocol that not only acknowledges the essential asynchronicity of Bitcoin network, but exploits it to reduce the network resource usage.

I'm going to assume that developers still living in the world where everything has to be polled for will spare no means to try to disparage anyone from outside of their group.

Bitcoin is essentially asynchronous and anyone who tries to hide that fact is going to be working on a very bad software project. Completely asynchronous implementations like 0MQ or FIX are currently too advanced for an average Bitcoin implementer. Stratum is somewhere inbetween and is a way for the average Bitcoin implementer to learn modern software architecture.

[makomk]

No disrespect intended for your thoughtful response, but this made me chuckle.  Deepbit was (is?) frequently attempting to mine forks of its own blocks. It's only Luke's paranoia code that caused anyone to actually notice it.

Actually, I and a number of other people noticed that Deepbit was broken entirely by accident because it broke MPBM's code for detecting stale work every time it started mining two different forks.

[2112]

We have a very suspicious community that will continually check the pools' actions.

No disrespect intended for your thoughtful response, but this made me chuckle.  Deepbit was (is?) frequently attempting to mine forks of its own blocks. It's only Luke's paranoia code that caused anyone to actually notice it.  Look at what happened with with the BIP16 enforcement— discussed and announced months in advance there were popular pools that didn't update for it— for some (e.g. 50BTC) for a long as a month, and they kept hundreds of GH/s of mining during that time. A lot of malicious activity is indistinguishable from the normal sloppiness of Bitcoin services, and what does get caught takes hours (or weeks, months...) for people to widely notice much less respond to... an attack event can be over and done with far faster than that.

The only way to have any confidence that the centralized pools aren't single points of failure is making them so they aren't— so that most malicious activity cause miners to fall over to other pools that aren't misbehaving... but most of those checks can't be implemented without actually telling miners what work they're contributing to.   I'll leave it to Luke to point to the checks he currently implements and sees being implemented in the future.

With your code commits you reinforce centralization by marginalizing anyone who can't or doesn't want to use GCC and has implementation ideas differing from the ones you espouse.

So many possible responses… (1) You must have me confused with someone else— I write portable software. (2) show me this person who "can't" use GCC, (3) but not officially supporting VC is more about being conservative and making the software testable, especially since we have a significant shortage of testing resources, especially on windows (and Microsoft's standards conformance is lackluster to say the least)— that a compiler bug caused miscompilation that ate someone coin but it was fine on the compiler the tester used is little consolation. (4) Although you do seem to have me confused for your employee, as while I'm happy for you to use things I write I'm certainly not writing it to your specifications without a sizable paycheck. (5) But if you'd like to submit some clean portability fixes to the reference client I'll gladly review them (6) though having heterogeneous software is the direct opposite of people blindly copying code with so little attention that they can't bother to do a little work in their environment, so changes that have risk or complexity may not be worth it if they're only for the sake of helping forks and blind copying. (7) Finally, why have you made this entirely offtopic reply here? The thread is about protocols between pools and miners, and has basically nothing to do with the reference client not providing you with MSVC project files.

I'm quoting in full just in case. In my experience the various attempts at disparaging Microsoft software stem from lack of understanding of its capabilities and the requirements it fulfills in many markets. Close to a half of bitcoind code is just a re-implementation of database (but not-really-a-database), of which multitute is available on Windows, together with the plentitude of available APIs. Probably close to a half of bitcoin-qt code is just a re-implementation of a data-bound control that are taught in the first semester of Visual Basic or C# or C++ programming.

I actually admire Gavin Andresen for willingness to simply admit that he's not familiar with various available database/financial APIs and learning them would detract him too much from the short term goals that he sees for Bitcoin.

Anyone can also note that various moral and ideaological opponents of Microsoft suddenly change their stance when talking about Apple. And Apple OSX/iOS is even worse prison-like-system than Microsoft Windows are.

But lets get back to the supposed superior security of the GBT and value that such security provides for Bitcoin. Lets look a the recent successfull double-spends agains SatoshiDICE losing bets. If the GBT was really security-oriented then we would've heard about this from many simultaneous GBT users. But we've heard about this voluntarily from the cheater himself. The list of would-be double-spenders would be freely circulated here on this forum.

I'm having trouble believing anyone who simultaneously claims that he has long-term financial security outlook but is oblivious to the attempts to abuse the protocol happening right now.

Combined with the above is the fact that you seem to pop up in every thread discussing alternate Bitcoin implementations and protocols and start disparaging their authors.

I don't know what are yours and Luke's goals, but I'm getting more and more convinced that they aren't the ones that you are openly defending here. There is a possibility that you are completely incorruptible like Felix Dzerzhinsky and you will show up in your armoured train anywhere you observe some counter-revolutionary activity that is trying to corrupt the ideals of the Bitcoin revolution.

Edit: I may be suffering from a case of mixed revolutions. Substitite Maximilien de Robespierre and guillotine in the above paragraph.

[kuzetsa]

((...snip...))
With your code commits you reinforce centralization by marginalizing anyone who can't or doesn't want to use GCC and has implementation ideas differing from the ones you espouse.

The marginalization of Windows users and developers (like Casascius) is very glaring example: lots more people would contribute to the project if the core developers would actually download Microsoft Visual Studio Express, purge the code base from GNU-isms and made sure that all future changes developed on Linux don't break Visual Studio builds.

+1

... and that would make signed binaries much easier too:

Bitcoin Signed Binaries (posted May 29, 2011, and STILL never been fixed because I can't be arsed to hack such things against a flaky mingw gcc build)

Edited to add:

((...snip...)) it's not like MSVS is free software - so you're asking free individuals to become less free (by agreeing to MSVS's terms) so that you can avoid using free software? I don't get it.

Visual Studio Express 2012 ---> With Visual Studio Express tools, you can build the next great app for Windows 8, Windows Phone, and the web. Best thing about them? They’re entirely free.

^I'm pretty sure you don't agree with their definition of "free" though, so whatever LOL

[Luke-Jr]

url=http://www.microsoft.com/visualstudio/eng/products/visual-studio-express-products]Visual Studio Express 2012 [/url] ---> With Visual Studio Express tools, you can build the next great app for Windows 8, Windows Phone, and the web. Best thing about them? They’re entirely free.

^I'm pretty sure you don't agree with their definition of "free" though, so whatever LOL

Because they aren't free. If you want to sell your soul to Microsoft for it, go ahead and use their malware to sign your binaries yourself.

[kuzetsa]

url=http://www.microsoft.com/visualstudio/eng/products/visual-studio-express-products]Visual Studio Express 2012 [/url] ---> With Visual Studio Express tools, you can build the next great app for Windows 8, Windows Phone, and the web. Best thing about them? They’re entirely free.

^I'm pretty sure you don't agree with their definition of "free" though, so whatever LOL

Because they aren't free. If you want to sell your soul to Microsoft for it, go ahead and use their malware to sign your binaries yourself.

You seem to have messed up with the URL tag in that quote... I think it's cute so I'm quoting your error as-is.


Also, What?

I don't think microsoft or their compiler suite is malware (which somehow magically steals souls)... As a fact-respecting scientist, my usual baseline state don't involve a soul or other philosophical / religious / ethical / GNU baggage, so I'm not worried about this... uh... "philosophically / religiously non-free" distorted price which you speak of, and therefore, somehow different than "free of charge" (zero monetary cost) as most people understand the word...


As far as bandwidth goes, I'll stick with stratum, thanks though for helping prove that even your own implementation of stratum makes better sense than GBT...

Most miners / bitcoin users are probably:

1) windows users without "free license" GNU-ism baggage
2) not watching their mining software unless it stops working
3) not watching the contents of the transactions they're (blindly) verifying 24/7
4) not worried about your "philosophically / religiously non-free" dilemma, since most of their software is compiled using commercial compiler toolchains.

The logic would follow that most miners / bitcoin users don't care if your use of MSVC-compatible code will allegedly hurt their soul, or some sort of philosophical / religious / ethical / GNU-ism or other baggage with regard to how they mine or spend bitcoin... Stratum protocol is also acceptable for above-listed reasons 2 and 3...

OK thanks, carry on.

[MPOE-PR]

I understand your arguments, but I'm unable to reconcile them with your actions.

With your writing in English you are advocating distributed systems and heterogenous implementations.

With your code commits you reinforce centralization by marginalizing anyone who can't or doesn't want to use GCC and has implementation ideas differing from the ones you espouse.

Quoth the wot,

10063   mircea_popescu   106   gmaxwell   2012-04-08 16:54:19   -10   hypocritical idiot.

Not an accident.

[kano]

...
3) not watching the contents of the transactions they're (blindly) verifying 24/7
4) not worried about your "philosophically / religiously non-free" dilemma, since most of their software is compiled using commercial compiler toolchains.

The logic would follow that most miners / bitcoin users don't care if your use of MSVC-compatible code will allegedly hurt their soul, or some sort of philosophical / religious / ethical / GNU-ism or other baggage with regard to how they mine or spend bitcoin... Stratum protocol is also acceptable for above-listed reasons 2 and 3...

OK thanks, carry on.

You've missed the point that a few of us have asked and been given no answer.

What does verifying the transactions do?

It verifies that the list of transactions provided by the pool ... matches the list of transactions in the block.
Well ... good to know ... but what does that do?
Nothing. There are no 'this is a bad transaction' checks anywhere.

Anyway, once a block is actually found - the list of transaction MUST exist for everyone on the bitcoin network to see or the block will be rejected.

So if people are concerned about mining transactions that some psychotic, religious nutcase thinks are damaging to bitcoin, then oddly enough those dangerous, damaging transactions will be known by everyone as soon as they land in a block.
... and then they can go rant about the pool that did it and lots of people will leave the pool.
If they never get in a block ... well aint that good

Though I will agree that this doesn't always work, there was this pool called Eligius that attacked an alt-coin with the pools hash power ... and also drastically restricted Bitcoin transactions for about 5 months ... and clearly stated that he was doing it in his pool thread ... that no one seemed to care about.

So ... basically ... even adding some 'this is an evil transaction' code into the miner, most likely it will have no effect anyway.
... and we'd also have some centralised 'authority' saying what is an 'evil' transaction ... ... ... ... ... but fortunately only centralised for those who used his holiness' cheap clone miner.

[wizkid057]

Though I will agree that this doesn't always work, there was this pool called Eligius that attacked an alt-coin with the pools hash power ... and also drastically restricted Bitcoin transactions for about 5 months ... and clearly stated that he was doing it in his pool thread ... that no one seemed to care about.

While this is semi-off-topic, I feel obligated to point out/clarify that Eligius is under new management. 
There are not arbitrary limits on transaction count of block size in place.  The only thing of note in place regarding transactions, that is not part of the reference client, is the filtering of Satoshidice transactions, which means we simply do not include bets directed at Satoshidice in our blocks.  (For the record, this is publicly noted in the original post from when this was implemented here)  The decision to leave that particular piece of code in place was a personal one, as I do believe that SatoshiDice could use a much less wasteful method of doing business.  Passive aggressive vs SatoshiDice.

Anyways, I think that having the transaction data available for miners to verify is worthwhile.  While the software to actually allow any real verification of this data, aside from validating it against what the pool is asking be hashed, is in it's infancy, I think it has potential to help secure the network even more by allowing miners to choose to allow their software to check their transaction list against their own bitcoind's transaction list, note any major issues, transactions that would not be accepted by a standard client, etc.

Obviously not every single miner is interested in doing this, and I'm reasonably certain no one has ever said otherwise.  But, it would not be a bad thing IF every miner were.

----

In any case, this thread is about the bandwidth use of the various mining protocols.

So far, through my own testing, my number's concur with the OP's to within an acceptable margin for error. (No discrepancy sufficient enough to change the order of most to least usage).

I however, do not see any real performance differences between the protocols as far as pool-side accepted shares over time, so, as far as I'm concerned, the bandwidth usage is a moot issue since I would venture to guess that majority of miners are using residential internet connections which are generally unmetered, and even the worst protocol bandwidth-wise (secure stratum) does not use a substantial amount of bandwidth, averaging about 36kbit/sec... and the cheapest broadband connection available in my area is 1500 kbit/sec. So, I think bandwidth usage is a moot issue for mining, especially considering these are non-vardiff tests too.

Have fun, try to stay on topic here...

-wk

[kano]

...
Anyways, I think that having the transaction data available for miners to verify is worthwhile.  While the software to actually allow any real verification of this data, aside from validating it against what the pool is asking be hashed, is in it's infancy, I think it has potential to help secure the network even more by allowing miners to choose to allow their software to check their transaction list against their own bitcoind's transaction list, note any major issues, transactions that would not be accepted by a standard client, etc.

Obviously not every single miner is interested in doing this, and I'm reasonably certain no one has ever said otherwise.  But, it would not be a bad thing IF every miner were.
...

As I stated in here before, any test that expects 2 bitcoind's to have the same transaction list is naive and will fail.
If a pool restarts it's bitcoind, the memory pool is empty ...
If you restart your bitcoind, the memory pool is empty ...

In any case, this thread is about the bandwidth use of the various mining protocols.

So far, through my own testing, my number's concur with the OP's to within an acceptable margin for error. (No discrepancy sufficient enough to change the order of most to least usage).

I however, do not see any real performance differences between the protocols as far as pool-side accepted shares over time, so, as far as I'm concerned, the bandwidth usage is a moot issue since I would venture to guess that majority of miners are using residential internet connections which are generally unmetered, and even the worst protocol bandwidth-wise (secure stratum) does not use a substantial amount of bandwidth, averaging about 36kbit/sec... and the cheapest broadband connection available in my area is 1500 kbit/sec. So, I think bandwidth usage is a moot issue for mining, especially considering these are non-vardiff tests too.

Have fun, try to stay on topic here...

-wk

I'll reply to your other comment later ... since it's better to supply numbers rather than make vague sweeping statements like you have.
The cgminer API reports those numbers so it's easy to see what they are ... rather than just say something without proof.

Edit: though I will point out that after I added the extra 'network' byte counter to cgminer, I went and looked at the clone code ... and noticed he was ignoring some data ....................
Here's a clue (that he obviously didn't have one of )
https://github.com/bagder/curl/blob/master/src/tool_cb_dbg.c

[Luke-Jr]

The cgminer API reports those numbers so it's easy to see what they are ... rather than just say something without proof.

Edit: though I will point out that after I added the extra 'network' byte counter to cgminer, I went and looked at the clone code ... and noticed he was ignoring some data ....................
Here's a clue (that he obviously didn't have one of )
https://github.com/bagder/curl/blob/master/src/tool_cb_dbg.c

Of course, as you already know (since I reported it to you), cgminer's counters lie about bandwidth usage. And since you haven't reported any bug to me, you have nobody to blame for that but yourself - IF it's true (which I doubt, since you're not competent and I did in fact read the code you referenced).

[kano]

The cgminer API reports those numbers so it's easy to see what they are ... rather than just say something without proof.

Edit: though I will point out that after I added the extra 'network' byte counter to cgminer, I went and looked at the clone code ... and noticed he was ignoring some data ....................
Here's a clue (that he obviously didn't have one of )
https://github.com/bagder/curl/blob/master/src/tool_cb_dbg.c

Of course, as you already know (since I reported it to you), cgminer's counters lie about bandwidth usage. And since you haven't reported any bug to me, you have nobody to blame for that but yourself - IF it's true (which I doubt, since you're not competent and I did in fact read the code you referenced).

Well I've pointed out the problem and you've ignored it.
To be expected. Nothing new. Source code in front of your face is too difficult for you to understand ...

cgminer doesn't lie about the byte count, it is the correct byte count transferred.
The old version just didn't report the network byte count which can, of course, be different to the data content byte count transferred, if the data is being compressed.
What you didn't bother to point out to the fools who believe you, is that will depend on the pool ...

[wizkid057]

The cgminer API reports those numbers so it's easy to see what they are ... rather than just say something without proof.

Edit: though I will point out that after I added the extra 'network' byte counter to cgminer, I went and looked at the clone code ... and noticed he was ignoring some data ....................
Here's a clue (that he obviously didn't have one of )
https://github.com/bagder/curl/blob/master/src/tool_cb_dbg.c

Of course, as you already know (since I reported it to you), cgminer's counters lie about bandwidth usage. And since you haven't reported any bug to me, you have nobody to blame for that but yourself - IF it's true (which I doubt, since you're not competent and I did in fact read the code you referenced).

I've also concluded that, for whatever reason, cgminer does not correctly report bandwidth usage consistent with usage reported by sniffing data directly on the box in question between the pool and box, which is the method I personally used for my tests.  I have several GB of PCAP to back up my "vague sweeping statements".  However, I find it wasteful to post all of that raw data when a summary of the findings is all that is needed.

Also, the new "E" that calculates bandwidth usage in bfgminer as a measure of number of shares per 2KB was in fact accurate to within 1% of my PCAP data. My assumption is that I'm including some packet header data that "E" is not, which would account for the error mostly.

cgminer's counters however almost never matched my PCAP data, being off by huge margins that I could not logically account for, thus I deemed that particular output from the miner useless.

After disabling gzip in bfgminer, both cgminer and bfgminer yield statistically identical results for bandwidth usage for all protocols supported by both softwares.  With gzip compression enabled in bfgminer code, bfgminer, obviously, uses less bandwidth.

All long tests were done against eloipool, and some shorter comparison tests done against various other pools with expected results.

[kano]

The cgminer API reports those numbers so it's easy to see what they are ... rather than just say something without proof.

Edit: though I will point out that after I added the extra 'network' byte counter to cgminer, I went and looked at the clone code ... and noticed he was ignoring some data ....................
Here's a clue (that he obviously didn't have one of )
https://github.com/bagder/curl/blob/master/src/tool_cb_dbg.c

Of course, as you already know (since I reported it to you), cgminer's counters lie about bandwidth usage. And since you haven't reported any bug to me, you have nobody to blame for that but yourself - IF it's true (which I doubt, since you're not competent and I did in fact read the code you referenced).

I've also concluded that, for whatever reason, cgminer does not correctly report bandwidth usage consistent with usage reported by sniffing data directly on the box in question between the pool and box, which is the method I personally used for my tests.  I have several GB of PCAP to back up my "vague sweeping statements".  However, I find it wasteful to post all of that raw data when a summary of the findings is all that is needed.

Also, the new "E" that calculates bandwidth usage in bfgminer as a measure of number of shares per 2KB was in fact accurate to within 1% of my PCAP data. My assumption is that I'm including some packet header data that "E" is not, which would account for the error mostly.

cgminer's counters however almost never matched my PCAP data, being off by huge margins that I could not logically account for, thus I deemed that particular output from the miner useless.

After disabling gzip in bfgminer, both cgminer and bfgminer yield statistically identical results for bandwidth usage for all protocols supported by both softwares.  With gzip compression enabled in bfgminer code, bfgminer, obviously, uses less bandwidth.

All long tests were done against eloipool, and some shorter comparison tests done against various other pools with expected results.

Loser ... using the version of cgminer that doesn't have the change to include the Network byte count ... D'oh
If you are going to make comments about cgminer - then do that when you know what you are talking about.

Anyway, the clone is ignoring data in certain circumstances ... as per that code from the curl developers shows ... that clearly both of you are too dumb to even notice the problem when I stick the code in front of you - lulz.

Meanwhile ... a run log ...



And anyone wanting to generate that with a customsummarypage in miner.php

Code:

#
$protopage = array(
 'DATE' => null,
 'RIGS' => null,
 'CONFIG' => array('GPU Count', 'PGA Count', 'Pool Count', 'Strategy',
                        'Device Code', 'OS', 'Failover-Only'),
 'SUMMARY' => array('Elapsed', 'MHS av', 'Found Blocks=Blks', 'Difficulty Accepted=Diff Acc',
                        'Difficulty Rejected=Diff Rej', 'Hardware Errors=HW Errs',
                        'Network Blocks=Net Blks', 'Utility', 'Work Utility'),
 'POOL+STATS' => array('STATS.ID=ID', 'POOL.URL=URL', 'POOL.Accepted=Acc',
                        'POOL.Difficulty Accepted=Diff Acc',
                        'POOL.Difficulty Rejected=Diff Rej', 'POOL.Has GBT=GBT',
                        'STATS.Max Diff=Max Work Diff',
                        'STATS.Times Sent=#Sent', 'STATS.Bytes Sent=Byte Sent',
                        'STATS.Net Bytes Sent=Net Sent', 'STATS.Times Recv=#Recv',
                        'STATS.Bytes Recv=Byte Recv', 'STATS.Net Bytes Recv=Net Recv'));
#
$protosum = array(
 'SUMMARY' => array('MHS av', 'Found Blocks', 'Difficulty Accepted', 'Difficulty Rejected',
                        'Hardware Errors', 'Utility', 'Work Utility'),
 'POOL+STATS' => array('POOL.Accepted', 'POOL.Difficulty Accepted', 'POOL.Difficulty Rejected',
                        'STATS.Times Sent', 'STATS.Bytes Sent', 'STATS.Net Bytes Sent',
                        'STATS.Times Recv', 'STATS.Bytes Recv', 'STATS.Net Bytes Recv'));
#
$protoext = array(
 'POOL+STATS' => array(
        'where' => null,
        'group' => array('POOL.URL', 'POOL.Has GBT'),
        'calc' => array('POOL.Accepted' => 'sum', 'POOL.Difficulty Accepted' => 'sum',
                        'POOL.Difficulty Rejected' => 'sum',
                        'STATS.Max Diff' => 'max',
                        'STATS.Times Sent' => 'sum', 'STATS.Bytes Sent' => 'sum',
                        'STATS.Net Bytes Sent' => 'sum', 'STATS.Times Recv' => 'sum',
                        'STATS.Bytes Recv' => 'sum', 'STATS.Net Bytes Recv' => 'sum'),
        'having' => array(array('STATS.Bytes Recv', '>', 0)))
);
#

... and of course you add this to your list of customsummarypages:
'Proto' => array($protopage, $protosum, $protoext)

[wizkid057]

The cgminer API reports those numbers so it's easy to see what they are ... rather than just say something without proof.

Edit: though I will point out that after I added the extra 'network' byte counter to cgminer, I went and looked at the clone code ... and noticed he was ignoring some data ....................
Here's a clue (that he obviously didn't have one of )
https://github.com/bagder/curl/blob/master/src/tool_cb_dbg.c

Of course, as you already know (since I reported it to you), cgminer's counters lie about bandwidth usage. And since you haven't reported any bug to me, you have nobody to blame for that but yourself - IF it's true (which I doubt, since you're not competent and I did in fact read the code you referenced).

I've also concluded that, for whatever reason, cgminer does not correctly report bandwidth usage consistent with usage reported by sniffing data directly on the box in question between the pool and box, which is the method I personally used for my tests.  I have several GB of PCAP to back up my "vague sweeping statements".  However, I find it wasteful to post all of that raw data when a summary of the findings is all that is needed.

Also, the new "E" that calculates bandwidth usage in bfgminer as a measure of number of shares per 2KB was in fact accurate to within 1% of my PCAP data. My assumption is that I'm including some packet header data that "E" is not, which would account for the error mostly.

cgminer's counters however almost never matched my PCAP data, being off by huge margins that I could not logically account for, thus I deemed that particular output from the miner useless.

After disabling gzip in bfgminer, both cgminer and bfgminer yield statistically identical results for bandwidth usage for all protocols supported by both softwares.  With gzip compression enabled in bfgminer code, bfgminer, obviously, uses less bandwidth.

All long tests were done against eloipool, and some shorter comparison tests done against various other pools with expected results.

Loser ... using the version of cgminer that doesn't have the change to include the Network byte count ... D'oh
If you are going to make comments about cgminer - then do that when you know what you are talking about.

Anyway, the clone is ignoring data in certain circumstances ... as per that code from the curl developers shows ... that clearly both of you are too dumb to even notice the problem when I stick the code in front of you - lulz.

Meanwhile ... a run log ...



And anyone wanting to generate that with a customsummarypage in miner.php

Code:

#
$protopage = array(
 'DATE' => null,
 'RIGS' => null,
 'CONFIG' => array('GPU Count', 'PGA Count', 'Pool Count', 'Strategy',
                        'Device Code', 'OS', 'Failover-Only'),
 'SUMMARY' => array('Elapsed', 'MHS av', 'Found Blocks=Blks', 'Difficulty Accepted=Diff Acc',
                        'Difficulty Rejected=Diff Rej', 'Hardware Errors=HW Errs',
                        'Network Blocks=Net Blks', 'Utility', 'Work Utility'),
 'POOL+STATS' => array('STATS.ID=ID', 'POOL.URL=URL', 'POOL.Accepted=Acc',
                        'POOL.Difficulty Accepted=Diff Acc',
                        'POOL.Difficulty Rejected=Diff Rej', 'POOL.Has GBT=GBT',
                        'STATS.Max Diff=Max Work Diff',
                        'STATS.Times Sent=#Sent', 'STATS.Bytes Sent=Byte Sent',
                        'STATS.Net Bytes Sent=Net Sent', 'STATS.Times Recv=#Recv',
                        'STATS.Bytes Recv=Byte Recv', 'STATS.Net Bytes Recv=Net Recv'));
#
$protosum = array(
 'SUMMARY' => array('MHS av', 'Found Blocks', 'Difficulty Accepted', 'Difficulty Rejected',
                        'Hardware Errors', 'Utility', 'Work Utility'),
 'POOL+STATS' => array('POOL.Accepted', 'POOL.Difficulty Accepted', 'POOL.Difficulty Rejected',
                        'STATS.Times Sent', 'STATS.Bytes Sent', 'STATS.Net Bytes Sent',
                        'STATS.Times Recv', 'STATS.Bytes Recv', 'STATS.Net Bytes Recv'));
#
$protoext = array(
 'POOL+STATS' => array(
        'where' => null,
        'group' => array('POOL.URL', 'POOL.Has GBT'),
        'calc' => array('POOL.Accepted' => 'sum', 'POOL.Difficulty Accepted' => 'sum',
                        'POOL.Difficulty Rejected' => 'sum',
                        'STATS.Max Diff' => 'max',
                        'STATS.Times Sent' => 'sum', 'STATS.Bytes Sent' => 'sum',
                        'STATS.Net Bytes Sent' => 'sum', 'STATS.Times Recv' => 'sum',
                        'STATS.Bytes Recv' => 'sum', 'STATS.Net Bytes Recv' => 'sum'),
        'having' => array(array('STATS.Bytes Recv', '>', 0)))
);
#

... and of course you add this to your list of customsummarypages:
'Proto' => array($protopage, $protosum, $protoext)

I'm pretty sure looking any code is not even necessary considering I'm using PCAP data for my tallies. Learn to read.

Second, it's good to know that you're psychic now and know what version of the software I'm using without even Asking! I'll keep that in mind on my next trip to Vegas.

Edit to clarify: PCAP data matches bfgminers bandwidth efficiency number, while no cgminer api output from any version to date matches. Feel free to run a capture and run the numbers on your own if you don't believe me.