1GH/s, 20w, $700 (was $500) — ButterflyLabs 3rd party testing for Dummies

[freequant]

I noticed that many people here are totally lost about what has be told and done, told but not done, done but not told and neither told nor done regarding the third party testing of ButterflyLab BitForce Single product.
With all the stuff scattered over two lengthy threads and weeks of endless debate, it is geting so confusing that even Inaba seems to be lost about what he has said and done.

This is why I have written for you the little pocket guide of "ButterflyLabs 3rd party testing for Dummies".
All the quotations are taken from the "is it a scam" and "is it for real" threads (courtesy of BFL and Inaba).
The emphasis in bold and comments are from me.

I have tried to remain objective, but I am of course totally biaised  , so do not base your opinion solely on my comments.
Please base your opinion on the facts, quotations, and linked messages.

This is a recommanded reading for :
- People waiting for their pre-order to be delivered
- Inaba, to help him sort out what he has said already
- Everyone else who wonders what the heck is going on with that ButterflyLabs testing nonsense.

----------------------

Yes, that's all I've seen and yes I've reviewed it.  I've not had a lot of time to go into minute detail and step through the code as it would potentially operate, but I see nothing nefarious in it. I informed them that whatever they sent me is what I would be compiling and using on my own hardware that they have not had access to prior to the test... so what I've reviewed is what I'll be using to conduct the test(s).

Friday 25 November, no test have been done yet.
Inaba starts defining the experimental protocol and commits to use his own hardware and compile himself the binaries.

Commitments about the experimentation protocol:
- miner run on own computer
- self built software

I just got home after meeting with Sonny in person from BFL and their engineer via Skype.  We had a fairly long conversation and I live over 30 miles away, so that's why it's taken me this long.
We did some simulated testing, but no live testing tonight.  We elected to put that off until tomorrow or possibly Sunday to allow for proper testing

Inaba announces the tentative date for a first test : Saturday 26 Nov or Sunday 27 Nov.
The basic demo is done, but was more like simulation testing. Let's call that preliminary test "the basic demo".
The "proper testing" will be on Saturday 26 Nov or Sunday 27 Nov.

Status of the experiment:
- "basic demo" test done
- "proper testing" test pending => planned for 26 or 27 Nov

As I said yesterday, we'd either be doing the demo Saturday or Sunday.  Right now, we have it planned for Sunday evening here in CST, probably around 18:00.  I've been pretty swamped today, but tomorrow is mostly open.

The test (as called earlier "proper testing") will be on Sunday 27 Nov.

Status of the experiment:
- "basic demo" test done
- "proper testing" test pending => planned for 27 Nov

The demo plan is that I will:
A) connect it to a non-routable development side of the pool, so that the box is unable to communicate with the internet.  I will then let it submit shares to the pool and I will have one of the getwork servers in debug mode and I will see what is sent out and what's sent back.  As I found no evidence of any wireless communications on the board, and since the computer it's connected to will not be on the internet, it won't have any way of falsifying the shares submitted.

Commitment to do the necessary to segregate the computer used for the test from the Internet.
Full awareness that any possibility of external communication would make the test invalid.

Commitments about the experimentation protocol:
- miner run on own computer
- self built software
- pool run on own server
- segregated network environment

There are some technical issues with the timing between the board and the software that they are working out, apparently.  I understand this will probably bring all the conspiracy theorists to the fore, so have at it.  For everyone else, I'm told that they will be ready in a couple days or so for a live demo.

Technical problem, actual (proper testing) test delayed.
Test rescheduled for in a couple days.
This (proper testing) test is called "live demo" here. This is an important term. Let's remember it.
So the "live demo" will take place in a couple days.

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing") test pending => posponed to "in a couple days or so"

they agreed to do the demo tomorrow afternoon.

The "live demo" will take place on 30 Nov afternoon.

As Inaba mentioned, we'll be bringing a live running unit to his data center tomorrow for a quick look and performance demonstration.  Since we're not at our final figures, he's agreed to simply confirm or deny if the unit is generally within the performance class expected.  I think rather than fuzzy numbers, everyone just wants to know if this unicorn is real.  Fair enough...  we expect to clarify that once and for all tomorrow afternoon.. We'll follow up with a more formal demo once we're finished.

This is a BFL anouncement.
The test in question (so the "live demo") will take place 30 Nov afternoon CST.
This test involved running a "live unit" (in case it wasn't clear enough this is a live demo) in Inaba's data center.
According to BFL's own words, the test aims to no less than proving that the product is real.

We plan on meeting at 16:00

The test in question (the "live demo") will take place 30 Nov afternoon at 16:00 CST.
No correction to what BFL said, so it is implicitely assumed that we are talking about the "live demo" that will prove that the product is real.

At any rate, we did a small demo of the hardware last night
[...]
In any case, the first test ran at a hashrate quite a bit lower than projected, but returned the expected nonce
[...]
But in either case, I feel that the test we conducted showed a POC that adequately demonstrates that at least the hardware does what it's designed to do

The test in question (the "live demo") took place on 30 Nov afternoon CST.
Positive feed back : the hardware does what it was designed to do.
At this point, no mention if the experimental protocol that was agreed on as been followed.

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing") test done

The purpose of the pre-release demo was simply to answer the question of 'real' vs 'fake'.I think most reasonable people would agree that has been resolved.

Inaba confirms after the fact that the purpose of the earlier "live demo", also called "pre-release demo", was to confirm if the product was real of fake.
Strong suggestion that the test was successfull in showing that the product is in fact real.
At this point, no mention if the experimental protocol that was agreed on as been followed, and therefore whether the test was reliable.

Commitments about the experimentation protocol:
- miner run on own computerassumed
- self built software assumed
- pool run on own server assumed
- segregated network environment assumed

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

I have reported exactly what I've seen, without bias and within the ground rules that were set forth PRIOR to the test.

Inaba confirms after the fact that the test (the "live demo") followed the experimental protocol that was agreed.
He also mentions that he reported exactly, and without bias.
This effectively clears any doubt about the reliability of the test, the applicablity of the agreed commitment, and therefore the correctness of the conclusions given earlier.

Commitments about the experimentation protocol:
- miner run on own computerapplicable
- self built software applicable
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

Strange as it may look it does appear like the screenshot BFL Labs posted was running on a Lenovo W520 laptop ( which Inaba also seems to own in the other pics earlier on ). Maybe they ran the demo on Inaba's laptop and BFL posted the screenshot here or they oddly have the exact same laptop model or they are the same person after all ?

One week later, the original "scam thread" has been closed and a new "is it for real" thread has been created.
On 9 Dec, Bulanula uncovers that the laptop used for the "basic demo" (which photo have been published) is the same as the one used by BFL for their screen shots.

That is BFL's laptop, not mine.  The one I plan on using for the test is an HP that BFL has not seen/had access to yet.

Inaba confirms that the laptop that was used for the "basic demo" and the "live demo" is BFL's own laptop.
His own laptop was actually never seen or accessed to by BFL.

Earlier commitments about the experimentation protocol:
- miner run on own computerapplicable not applicable
- self built software applicable
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

[NR]
It is doubtful that the drivers / miner built by Inaba for his own computer / distribution would work straight away on BFL's own laptop under Ubuntu.
It is also rather unlikely that Inaba thought of transferring the drivers he built to some portable storage, but didn't think of bringing his own laptop as agreed.
Even assuming that Inaba really gave the binaries he built himself to BFL, the laptop being BFL's, the path could have been set so as to run their own version of the binary.
Under these conditions, the condition "self built software" cannot be ascertained.


Earlier commitments about the experimentation protocol:
- miner run on own computerapplicable not applicable
- self built softwareapplicable doubtful
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done

That is BFL's laptop, not mine.  The one I plan on using for the test is an HP that BFL has not seen/had access to yet.

So let me get this straight... you wrote all that stuff about compiling the code from source and about how the board and test machine didn't have any kind of internet connection, and then you neglected to mention that you'd used a laptop set up and provided by BFL which could've had a hidden wireless connection to anywhere.  Did you even make sure that the binaries you were running actually corresponded to the source code you received, or did you use binaries that BFL provided too?

Neglected to mention it for what?  We haven't done the test yet...  How can I not mention something that hasn't even happened?

Makomk question regarding the objectivity and reliability of the live test.
Inaba clarifies the situation as he understands / remember it : the live test did not take place.
[NR] Note for people who are not familiar with block chain reorganizations : the live test that took place on 30 Nov at 16:00 CST, did in fact not take place (i.e. the live test has been orphaned).

Earlier commitments about the experimentation protocol:
- miner run on own computerapplicable not applicable
- self built softwareapplicable doubtful
- pool run on own serverapplicable
- segregated network environmentapplicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done not done

So my plan was published after the first meeting and we haven't done a second, live test yet.  So I'm a little confused as to what you are taking issue with?

Later in the same post, after referring to quotations of the "basic demo" test and highlighting that this was not a live test, Inaba states that there hasn't been a second live test (which also means that there was a first live test).
[NR] Note for people who, like Inaba, would be confused at this point : the live test that took place on 30 Nov at 16:00 CST, and that subsequently did not take place, actually took place but it was the first (this is important).

Earlier commitments about the experimentation protocol:
- miner run on own computerapplicable not applicable
- self built softwareapplicable doubtful
- pool run on own serverapplicable
- segregated network environment applicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done not done kind of done
- "second live demo" test pending

I know what wireless is available in the building we were in and I know the response times on those routers - they are way too congested to be of any value for that sort of thing.  For 3G, I agree, a few hundred ms, so you're looking at .5 seconds on a good day, probably longer on a bad one, so lets go with .5 round trip.  That leaves 3.5 seconds to hash ~4.2 billion nonces, which is (off the top of my head) about 1.3 (?) GH/s?  That means either a custom build FPGA rig hidden somewhere (expensive) or a multi GPU rig hidden somewhere with custom mining software able to split the data into partitioned nonce ranges (less expensive, more complex) - and all this would have to be done in LESS than 3.5 seconds, meaning they'd need I'd say at least double the hashrate to overcome the latency issues, so 2.6 GH/s.

Inaba recognizes that after all, it is possible that BFL's laptop was able to communicate outside (albeit with a high latency) using 3G or WiFi routers that he knows of in the data center.
Under these conditions, the "segregated network environment" doesn't apply.
The fact he knows of the WiFi router in the building suggests that he could scan the WiFi, which suggests in turn that the data center building isn't isolated from WiFi signal.

Earlier commitments about the experimentation protocol:
- miner run on own computerapplicable not applicable
- self built softwareapplicable doubtful
- pool run on own serverapplicable
- segregated network environmentapplicable not applicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done not done kind of done
- "second live demo" test pending

Additionally they were ready and willing to go to the datacenter which is essentially one giant faraday cage and would have prevented any sort of reliable (if any) data connection over wireless (either Wifi or 3G).

Later in the same post, in fact the datacenter is a giant faraday cage that would have prevented WiFi and 3G to get through.
[NR] Note : the faraday cage there prevented BFL to connect from inside the building to the routers that Inaba detected in spite of the faraday cage by scanning from inside the building.

Earlier commitments about the experimentation protocol:
- miner run on own computerapplicable not applicable
- self built softwareapplicable doubtful
- pool run on own serverapplicable
- segregated network environmentapplicable not applicable kind of applicable

Status of the experiment:
- "basic demo" test done
- "live demo" (aka "proper testing" aka "pre-release demo") test done not done kind of done
- "second live demo" test pending

----------------------

Conclusion :

Meeeehh 

[1713491666]

1713491666

[1713491666]

1713491666

[1713491666]

1713491666

[1713491666]

1713491666

[1713491666]

1713491666

[Inaba]

There's a few things that are inaccurate here.  First is the "Ground rules" statement referred to my inability to give hard numbers, not to test protocol.  This is an important point and at the time I was aware of no one who was confused by that statement or I would have clarified it.

Secondly, the building we have been in is my office building, not the data center, thus why there is wifi available.  Also why I know what the wifi available in that building is and what it congestion rate is.  At no time have we (BFL and I) been to the data center and I don't believe I have stated we have done any testing inside the data center at this point.

The "small demo" was NOT a live demo, and I stated as much.  It was a demo of me taking (by hand) a dataset from my pool and pasting it into the mining testbed to return the nonces I reported and offered to independent verification.

I thought I had mentioned it, but perhaps not, I have yet to receive software source that will compile from BFL.  The one set of software I received and reviewed would not compile and I am suppose to receive a new codeset this weekend.  However, given the nature of the codeset and it's author, I'm inclined to believe that is more likely legit than not, but that does not mean I will not be reviewing it regardless.

[freequant]

I will answer to your points, one per post, to keep things clearly separated.

There's a few things that are inaccurate here.  First is the "Ground rules" statement referred to my inability to give hard numbers, not to test protocol.  This is an important point and at the time I was aware of no one who was confused by that statement or I would have clarified it.

It was indeed extremely ambiguous because, we (as the community) could not care less if you respected your agreement with BFL, and could not care more if you did not respect your agreement with us.
What the "the ground rules that were set forth PRIOR to the test" refer to is a question of context.
If you were answering to BFL, it would make sense that the above would refer to your agreement with them.
And if you were answering to the community (which was the case here), it was only logical that the ground rules would refer to our agreement on the experimental conditions.

[freequant]

Secondly, the building we have been in is my office building, not the data center, thus why there is wifi available.  Also why I know what the wifi available in that building is and what it congestion rate is.  At no time have we (BFL and I) been to the data center and I don't believe I have stated we have done any testing inside the data center at this point.

Both the test of 25th Nov and 30th Nov were meant to take place in your datacenter according to yourself or BFL.
Let me put things back in context.

It's about noon on the 25th here in KC.
I plan on meeting them in about 6 hours from now (or midnight UTC).  After that, we'll decide if we have enough time to do the live test or if we'll just do the hardware portion.

Without question, the context here is the first meeting with BFL planned on 25th at 18:00 CST.
You are planning to do the live test if enough time.

I have no outside individuals coming, but if anyone is local that wants to come along, I can probably accommodate 1 or two additional people.  BFL said they might have one or two additional people... I need to keep it small as my DC probably doesn't want a ton of people marching into one of the server rooms.

In the next post, you mention that the test will take place in your data center.
There is no ambiguity of context : this is the next post, right after previous one where you were speeking about the meeting with BFL.
You even invited people to join, so assuming some people had joined, they would have been waiting in front of your datacenter.

At this point :
- you mentionned you would try to do the live test
- you mentionned it will take place in the data center.

So everyone who followed so far would naturally expect :
- that you will bring your laptop
- that the test will indeed take place in the data center

Hard to take pictures in a server room? Let me guess, poor pictures that barely show the heat sink that we all know well? Nothing of the chips? I'm just being irritating.

Luckily, the next day someon mentions about the data center.

No, we were not in the DC tonight.  And no, the pictures are fairly comprehensive as far as that goes.

And you clarify : the test did not take place in the data center after all (based on the photo, it looks like a restaurant, or someone's home).
So on 25 Nov, you wanted to do the live test in the data center with your laptop, but none of that happened.
You mentionned that the live test was postponed, but you didn't mention about the computer and the location until people ask more explicitely.
What happened? Who changed the plan and why? Why not be more clear about the change of place and the circumstances?

So anyway, that was the first test, and the first confusion.

Now let's look the announcement of the second "pre-release" test.

As Inaba mentioned, we'll be bringing a live running unit to his data center tomorrow for a quick look and performance demonstration.  Since we're not at our final figures, he's agreed to simply confirm or deny if the unit is generally within the performance class expected.  I think rather than fuzzy numbers, everyone just wants to know if this unicorn is real.  Fair enough...  we expect to clarify that once and for all tomorrow afternoon.. We'll follow up with a more formal demo once we're finished.

Here, it is the BFL official annoucement for the test of 30th Nov.
By BFL's own words, it is unambiguously stated that the test will take place in the data center.
By BFL's own words, it is a pre-release (so almost final) test with a live unit, so that should clearly qualify as the previously mentionned "live test".
By BLF's own words, it is going to establish the proof that the their product is real, so we are talking about a test with a formal experimental protocol here.

Following BFL's announcement, you did not make any comment to retarget before or after the test.

So everyone who followed so far would naturally expect :
- that you will bring your laptop
- that you will apply the experimental protocol we agreed one.
- that this was the so-called "live test" you were talking about all along.
- that the test will indeed take place in the data center

Now, you just clarified : this was not in the data center (in spice of BFL's announcement)
You clarified a few days ago that you in fact did not use your laptop, that the network was in fact not really segregated, and that this was in fact not the live test after all.

So on 30 Nov, as on 25 Nov before that, you wanted to do the live test in the data center with your laptop but, again, none of that happened.
Same pattern as last time, you mentionned vaguely that you would do more testing (but not that this wasn't the live test), and omitted to mention about the computer and the location until people ask more explicitely.
What happened again? Who changed again the plan and why? Again why not be more clear about the change of place and the circumstances?

[sadpandatech]

First I need to criticize you a bit, Freequant. Assuming you are being sincere in your questioning and not just lacking the balls to say "I think you are full of shit". Then is the title including, "testing for Dummies" really condusive to getting ones questions answered? And why the need for so much text? It just seems way overdone for just being thorough.

Second though, I am also curious about why the changes in 'plans'. But not from the perspective of any kind of assumption that it implies anything.

Was it BFL or Inaba that changed the venue for meeting one?
Meeting two?
Why?
I too naturally expected that it was Inaba's laptop that I was looking at in the pictures of test 1. And from some of the verbage that mentioned the unit being plugged into the laptop. I am not certain, but don't recall the indentifiers of 'mine' or 'theirs' being present in the text.

  There were more questions in my mind but those and the ones I pressent here seemed meaningless to me at the time. And, I would have assumed anyone else following along would not be so concerned since the final result was that there was still not a unit that was in a ready state for a truely proper test. I do agree testing and reporting should be as complete as is possible. But I also appreciate that as far as we know Inaba is doing this on his own time. For no agreed compensation. And has said nothing to make any intelligent person want to rush out and pay for something they would not do otherwise. I mean, if the intention of being so critical is to help 'protect' the community that's great. But at the same time it also basicly is saying to everyone that they are stupid and not capable of making an informed decision on their own.

  Well, cheers to you both for taking action on behalf of the community whatever the outcome may be.

 
P.S. Fix the freaking "1GH/s, 20w, $700 (was $500)". It comes off as being very purposely biased to everyone who knows you are smarter than that...
The unit was never priced at $500. It was $599. There were pre-orders with a set date at $499. Anyone who ordered by that date has not been asked to pay more money.

[plastic.elastic]

The unit was never priced at $500. It was $599. There were pre-orders with a set date at $499. Anyone who ordered by that date has not been asked to pay more money.

Tell the other OP to fix his title too if you really care. You're being a nitpicking?

As far as the laptop, i always assume it was BFL's, as its obvious to me from the pics (i doubt Inaba would have left the username and computer name as default. BFL, on the other hand, would do a fresh install of ubuntu to avoid exposing any personal identity)

I'm surprised someone thought it was Inaba's laptop from the pics.

Now, isnt the test scheduled for this weekend?

[sadpandatech]

The unit was never priced at $500. It was $599. There were pre-orders with a set date at $499. Anyone who ordered by that date has not been asked to pay more money.

Tell the other OP to fix his title too if you really care. You're being a nitpicking?

  I did.......

https://bitcointalk.org/index.php?topic=53530.msg639056#msg639056

[freequant]

The "small demo" was NOT a live demo, and I stated as much.

There are countless occurences of you and BFL referring to the test of 25 and then 30 Nov as "proper testing", "live test" "prerelease demo" and whatnot.
You clearly mentionned after the fact that 25 Nov wasn't a live test.
I have found no evidence of you telling anywhere that 30 Nov wasn't a live demo (appart from your quotation of yesteday, of course).
Anyway, whether the test of 30 Nov was to be called a "live test", a "prerelease demo" or "proper testing" or a "small demo" isn't the real problem.

The problem is that the test of 30 Nov was arguably meant to prove that the product of BFL was legit, and that no countrary statement has been made to moderate the original announcement.

As Inaba mentioned, we'll be bringing a live running unit to his data center tomorrow for a quick look and performance demonstration.  Since we're not at our final figures, he's agreed to simply confirm or deny if the unit is generally within the performance class expected.  I think rather than fuzzy numbers, everyone just wants to know if this unicorn is real.  Fair enough...  we expect to clarify that once and for all tomorrow afternoon.. We'll follow up with a more formal demo once we're finished.

BFL calls the test of 30 Nov the "pre-release demo" so let's keep this term to stop playing with the words.
The "pre-release demo" had for ambition to establish once and for all the their product is legit.
Following the announcement, you didn't comment to moderate the announcement of BFL or clarify the location or circumstances.

Hi Guys,

I'd like to remind everyone that the performance figures Inaba witnessed were on a tuned down debug unit and are not the final performance figures expected for delivery.  Nonetheless, Inaba has confirmed performance is in the class expected.  The reasons for demoing a tuned down unit were clearly stated to Inaba beforehand and to the forum in general.  That is that we are still in development.

The purpose of the pre-release demo was simply to answer the question of 'real' vs 'fake'.   I think most reasonable people would agree that has been resolved.  The next step is final performance and when we complete our dev tuning, you'll have those figures to comment on.  Til then, troll fest!  Smiley

Regards,
BFL

After the test, BFL confirm with triumph that the test was a success in establishing that their product is legit.
Again, you didn't comment to moderate the announcement of BFL.
Later posts show that you actually vouch for their conclusion.

But in either case, I feel that the test we conducted showed a POC that adequately demonstrates that at least the hardware does what it's designed to do

Here you are pretty affirmative about the fact the test demonstrates that their product is legit.

Well stable, the BFL unit is > 4x the hashrate than ztek and more than double ngzhang. Power draw is also more than double ngzhang (Dunno what ztek power draw is).  Even allowing for a 10% efficiency decrease for "real world" scenario vs the test data, the numbers still hold.

Again, this is pretty affirmative. You even go to the length of doing comparative marketing on behalf of BFL.

To summarize :

You knew that our requirement to establish that BFL's product is legit was to follow a certain number of constraints, including limitations on the location, hardware, network access and software build.
You acknowledged, and made a rigourous protocol that was covering all these requirements and commited to apply it for all the tests.
So you knew without any ambiguity that there was no way we could ever reach the conclusion "BFL's product is legit" without abiding to the rules that we had agreed on.

On 30th Nov, you and BFL announced a test that had all the appearance of a formal test, and aimed at no less than prove that BFL's product is legit.
But the test that you performed was not valid as it did not follow in any way the protocol that was agreed on.
At that point you knew that there was a flaw in the way the test had been executed, and that no evidence had been made.
Nonetheless on 1st Dec, you and BFL announced that the test was successful at establishing that BFL product is legit.

The least that can be said while remaining objective is that the way you reported about the testing so far has been extremely misleading.
I won't make any further assumptions here simply because I think there is no ground for making additional assumptions.

[freequant]

First I need to criticize you a bit, Freequant. Assuming you are being sincere in your questioning and not just lacking the balls to say "I think you are full of shit".

So you think that people under a pseudonym calling each other names without any actual evidence are being courageous?
We must have a very different notion of courage.

To answer what I think is your question, I am not at all sure what to think about Inaba.
As I said earlier, in one hand, if BFL turns out to be scam, Inaba would be the ideal candidate for the role of the con artist, but he could also be a naive victim however unlikely that would be.
In the other hand, it is also very possible that BFL is legit in spite of their questionnable marketing policy, in which case Inaba would really be the helpful guy he appears to be.
Short of evidence, I am not going to jump to any hastly conclusion, nor to call Inaba and BFL names, neither am I going to believe BFL and Inaba without questionning.

That being said, there is definitely something fishy with this 30 Nov testing so I'd like to understand what happened.
In any case, until evidence is clearly made in favor or agains BFL's legitimacy, I'd like to advise people to apply healthy skepticism in everything they read on this forum, including testimony of seemingly trustable but isolated people.

Then is the title including, "testing for Dummies" really condusive to getting ones questions answered? And why the need for so much text? It just seems way overdone for just being thorough.

You have to expect that tracking what has been said over 87+ pages and two threads is going to take some space.
This is the price to pay to reproduce the entire context, and have a discussion backed with facts and not blurry gut feelings.
Now, if you do not have any doubt about the way the testing was done and reported, no one forces you to read it all..

I mean, if the intention of being so critical is to help 'protect' the community that's great. But at the same time it also basicly is saying to everyone that they are stupid and not capable of making an informed decision on their own.

I am not being critical. I am being inquisitive.
I decided to lay down all the evidence and start from there after Makomk asked exactly what I had on my mind on 9 Dec, and Inaba somehow dodged the question that IMO was legitimate (cf quotation below)

That is BFL's laptop, not mine.  The one I plan on using for the test is an HP that BFL has not seen/had access to yet.

So let me get this straight... you wrote all that stuff about compiling the code from source and about how the board and test machine didn't have any kind of internet connection, and then you neglected to mention that you'd used a laptop set up and provided by BFL which could've had a hidden wireless connection to anywhere.  Did you even make sure that the binaries you were running actually corresponded to the source code you received, or did you use binaries that BFL provided too?

Neglected to mention it for what?  We haven't done the test yet...  How can I not mention something that hasn't even happened?

Well, cheers to you both for taking action on behalf of the community whatever the outcome may be.

If only BFL could stop taking orders for products that they are not ready to ship yet, and resume ordering once they have enough stock to ship within the charge back delay of paypal.
That would eliminate once and for all the growing doubt in the community that they are just trying to buy some time.

P.S. Fix the freaking "1GH/s, 20w, $700 (was $500)". It comes off as being very purposely biased to everyone who knows you are smarter than that...
The unit was never priced at $500. It was $599. There were pre-orders with a set date at $499. Anyone who ordered by that date has not been asked to pay more money.

The title is a copy / paste of previous thread title.
The question of the original price being totally irrelevant to the topic, that shouldn't be a big issue.

[plastic.elastic]

If only BFL could stop taking orders for products that they are not ready to ship yet, and resume ordering once they have enough stock to ship within the charge back delay of paypal.
That would eliminate once and for all the growing doubt in the community that they are just trying to buy some time.

This is the key here for all the sheeps out there.

BFL even has a nerve to tell us "mind your own business"

[plastic.elastic]

If only BFL could stop taking orders for products that they are not ready to ship yet, and resume ordering once they have enough stock to ship within the charge back delay of paypal.
That would eliminate once and for all the growing doubt in the community that they are just trying to buy some time.

This is the key here for all the sheeps out there.

BFL even has a nerve to tell us "mind your own business"

They wont even adjust their power usage on their website. I'm shocked honestly people have not contacted paypal and had them shut down.

It would be nice to have a few Single box working and orders coming in. The "Rig Box" then would be "ready to ship in 6 weeks" and they earn/scam a nice trip to Thailand

[rjk]

<troll>If you guys frequented IRC like normal people instead of skulking on a forum, you might learn some interesting tidbits.</troll>

[makomk]

I thought I had mentioned it, but perhaps not, I have yet to receive software source that will compile from BFL.  The one set of software I received and reviewed would not compile and I am suppose to receive a new codeset this weekend.  However, given the nature of the codeset and it's author, I'm inclined to believe that is more likely legit than not, but that does not mean I will not be reviewing it regardless.

Ah yes, I remember Luke-Jr mentioned he'd been writing mining client code for it that he thought you might be using for your test. It's unfortunate that it seems to be taking so long to get working code. (If it helps, I released some poclbm-based code a while ago that could probably be adapted to the BitForce boards with a little work, though it's hard to be sure given that they haven't released any interface specs or code. It's not a general purpose solution but...)

[Inaba]

So I'm not sure how to reply to this without a "Wall of Text" which I would like to avoid.  It got out of hand in the last thread and I would like to avoid that same mistake here.

With regards to the "ground rules," I am sorry but I fail to see how one could take the "ground rules" to be test protocol both contextually (important) and also literally (less important, but I can see where one could conflate test protocols and ground rules).  Clearly it has happened, at least in one persons case, so I apparently underestimated peoples understanding of the situation.  That said, the ground rules refer to the rules set forth by BFL for what I am/was able to disclose and what I was not able to disclose.

In the next post, you mention that the test will take place in your data center.
There is no ambiguity of context : this is the next post, right after previous one where you were speeking about the meeting with BFL.
You even invited people to join, so assuming some people had joined, they would have been waiting in front of your datacenter.

At this point :
- you mentionned you would try to do the live test
- you mentionned it will take place in the data center.

So everyone who followed so far would naturally expect :
- that you will bring your laptop
- that the test will indeed take place in the data center

I did bring my laptop, but we were unable to get the software operating on it.  The compile issue which I had mentioned earlier.  I find this part to be immaterial though, but I thought I would mention it.  You are also strategically leaving out parts of my posts that are important, and you are altering times that I said things to fit your apparent theory that I have something to hide.  I find this both insulting and offensive, especially since ANYONE can verify for themselves that what you've quoted here is inaccurate on numerous levels.  For example:

I said on post 891 that we did some "simulated testing, but no live testing tonight."  This would indicate that at the very least, part of the "protocol" was not followed.  I then went on in subsequent posts, within 15 minutes (actually the next post I made) clarifying that we were not in the DC - I did not wait an entire to day make mention of this fact... then 40 minutes later, I said that since it wasn't a live test I wasn't concerned with why it said "0" processors as it wasn't a live test... I stated that I was more concerned with the power consumption at that point.  So clearly, within 40 minutes of my first post, this was not any sort of rigorous test.  Then, the following day is when I laid out the protocol.

And you clarify : the test did not take place in the data center after all (based on the photo, it looks like a restaurant, or someone's home).
So on 25 Nov, you wanted to do the live test in the data center with your laptop, but none of that happened.
You mentionned that the live test was postponed, but you didn't mention about the computer and the location until people ask more explicitely.
What happened? Who changed the plan and why? Why not be more clear about the change of place and the circumstances?

I did not mention it because it was not and is not relevant.  Until we (I) am able to test a unit fully on my own terms, it's completely immaterial whether or not we use their hardware or my hardware - in either case, it's a) development hardware, not shipping hardware, b) incomplete hardware, c) incomplete software.  With all of that stacked against the unit, it's really irrelevant what hardware the testing is running on.  But for the record, the "plan" was changed when it was clear that the hardware would not operate in a fully functional manner - at that point I decided it was completely superfluous to waste time going to the datacenter to conduct any "testing" that we might be able to conduct, which we could conduct in my office building instead.

Here, it is the BFL official annoucement for the test of 30th Nov.
By BFL's own words, it is unambiguously stated that the test will take place in the data center.
By BFL's own words, it is a pre-release (so almost final) test with a live unit, so that should clearly qualify as the previously mentionned "live test".
By BLF's own words, it is going to establish the proof that the their product is real, so we are talking about a test with a formal experimental protocol here.

Following BFL's announcement, you did not make any comment to retarget before or after the test.

So everyone who followed so far would naturally expect :
- that you will bring your laptop
- that you will apply the experimental protocol we agreed one.
- that this was the so-called "live test" you were talking about all along.
- that the test will indeed take place in the data center

Now, you just clarified : this was not in the data center (in spice of BFL's announcement)
You clarified a few days ago that you in fact did not use your laptop, that the network was in fact not really segregated, and that this was in fact not the live test after all.

So on 30 Nov, as on 25 Nov before that, you wanted to do the live test in the data center with your laptop but, again, none of that happened.
Same pattern as last time, you mentionned vaguely that you would do more testing (but not that this wasn't the live test), and omitted to mention about the computer and the location until people ask more explicitely.
What happened again? Who changed again the plan and why? Again why not be more clear about the change of place and the circumstances?

You are, again, being inaccurate or taking things out of context.  Regardless of what BFL said the test was going to be, my next post on the subject should have dispelled any of those previous statements as being accurate:

Post 1291:

Sorry it took so long to get this post up, but I have a few other things that had to be taken care last evening and I didn't get home till late and pretty much went straight to bed.  At any rate, we did a small demo of the hardware last night, here is the test data I used, which I pulled from one of the getwork servers of my pool:

Data: fd90c721557226679bfc01bc971be894ec08137d0f36fd923f822e4743f954da
Merkle: 9d0e5b394ed6ae311a0f61b1

Data: e4f4a3eb23855f185379d5833f0eabb9daee8483e43d39a6a9b3888882bfc0fa
Merkle: 29a9690f4ed6aecc1a0f61b1

Both of these were fresh out of my pool

We did a small demo, not a live demo, or an official demo, or any of the other terms that you've been throwing around.  I also stated that I pulled test data from my server - why would I need to do that if the unit was operating properly?  WHy just two midstates?  Why  report the nonces?  Why not just say "Hey yeah, it did what it was suppose to do at a XX hashrate?"  I mean, the fact that I gave details that would seem to indicate something other than a "normal" operating mode was in process didn't raise any red flags?  Perhaps I just give people too much credit for critical thinking.

But to help clarify that, I also helpfully included in the same post:

Another demo once they get some of the technical issues worked out is planned for the near future (no definite date at the moment, but within a week or so I would think - this is just my speculation) with a fully automated mining client running and submitting work to my live development pool. 

If there were technical issues, how could this be a live demo?

My conclusion is that even if the units were to ship with the lower hashrate I tested and the power consumption I tested, they would still be extremely viable pieces of hardware and are also superior to the currently available public offering(s).  Would they be worth $700 as witnessed?  That would be up to the individual to make that decision, but I personally feel that they are at least within the ballpark of most peoples definition of reasonable.  Any improvements on what I actually saw will increase the value, and from the explanations and technical details I received, I do not see any reason that the final product won't be substantially improved from what I actually tested tonight.

Now the above quote (also from the same post) is about the only place I can see where there would be legitimate confusion and I agree, I did not explicitly say that I tested this on their laptop (well, actually it was a combination of my own and theirs, as I was pulling test data from my laptop and emailing it to them so that we could cut and paste the data into the test bench program.)... but as it was not really a valid test beyond showing that the hardware functions, I did not feel that test conditions needed to be spelled out.  But I apologize for any confusion that may have caused.

There are countless occurences of you and BFL referring to the test of 25 and then 30 Nov as "proper testing", "live test" "prerelease demo" and whatnot.
You clearly mentionned after the fact that 25 Nov wasn't a live test.
I have found no evidence of you telling anywhere that 30 Nov wasn't a live demo (appart from your quotation of yesteday, of course).
Anyway, whether the test of 30 Nov was to be called a "live test", a "prerelease demo" or "proper testing" or a "small demo" isn't the real problem.

The problem is that the test of 30 Nov was arguably meant to prove that the product of BFL was legit, and that no countrary statement has been made to moderate the original announcement.

Yes, there are countless occurances of that, however after every "test," it was pretty clearly indicated that things did not go as planned and I constantly updated with a future test is in the works.  If the test was conducted, why have a future test?  That's like looking for your car keys after you've already found them.  As for not finding any evidence, go back and read my previous posts, there's plenty of evidence and I've already gone over some of it above.

The 30 Nov test was meant to prove that, but it was indicated that it did not due to technical difficulties and it was rescheduled.  That was not ambiguous.

After the test, BFL confirm with triumph that the test was a success in establishing that their product is legit.
Again, you didn't comment to moderate the announcement of BFL.
Later posts show that you actually vouch for their conclusion.

So you've said this several times, I just quoted this particular one to address it.  You say that I did not comment to moderate the announcement... but you (and others) also critize me when I do comment on their announcements as being a shill or supporting them, or being blinded by them, etc...  So which is it?  Do you want me to comment/moderate BFL or do you want me to be impartial and let BFL speak for themselves while I report what I see and my own feelings on the subject?  It's impossible to do both.

Here you are pretty affirmative about the fact the test demonstrates that their product is legit.

I am making a statement of fact.  It does what it's designed to do - hash SHA256 blocks.  I did not say it was legitimate, I said "adequately demonstrates that it at least the hardware does what it's designed to do" - I added the bold text.  I still stand by this statement.

Again, this is pretty affirmative. You even go to the length of doing comparative marketing on behalf of BFL.

I do not.  I compare what I saw with what is publicly available tech spec wise for other products in the same category.  This was not on behalf of BFL, this was in response to a query made by another member who was not BFL.

On 30th Nov, you and BFL announced a test that had all the appearance of a formal test, and aimed at no less than prove that BFL's product is legit.
But the test that you performed was not valid as it did not follow in any way the protocol that was agreed on.
At that point you knew that there was a flaw in the way the test had been executed, and that no evidence had been made.
Nonetheless on 1st Dec, you and BFL announced that the test was successful at establishing that BFL product is legit.

There was no test executed as previously described, as much was said, so how or why would I indicate that test protocol was not followed on a test that didn't happen?  I would think that would be self evident.  Perhaps not.

That being said, there is definitely something fishy with this 30 Nov testing so I'd like to understand what happened.
In any case, until evidence is clearly made in favor or agains BFL's legitimacy, I'd like to advise people to apply healthy skepticism in everything they read on this forum, including testimony of seemingly trustable but isolated people.

Why would you not be doing that anyway?  You should always treat these things with healthy skepticism. 

My credibility comes from my involvement with the community for months (Since March or April, I forget) and the fact that I run a reputable, long standing (comparatively) pool.  I volunteered a) because I was curious myself and b) I was literally right across the street, purely by coincidence.  I have not, nor do I care if anyone trusts me as being a legitimate member of the community beyond people trusting in my pool and patronizing it.  As I have said in a previous post, I am doing this as much for my own edification and to base my own decisions on purchase as I am for any other reason - if I can provide a valuable service to the community in the process then I am happy to do.

I also did not dodge any questions; the questions were immaterial.  No official live test has been conducted, the software they sent me at the time did not compile so there was no verification to be done to begin with.  We did not use the software they sent me, but used a test program that required us to cut and paste the data by hand, so comparing binaries would be futile.

AAaaaand another wall 'o text. My apologies. 

[bombo999]

+1

[btc_artist]

Watching

[Electricbees]

Also watching. 

[freequant]

you are altering times that I said things to fit your apparent theory that I have something to hide.  I find this both insulting and offensive, especially since ANYONE can verify for themselves that what you've quoted here is inaccurate on numerous levels.

I had the courtesy not to attribute you negative intentions, and not to charge you of any accusations beside of that of being misleading in your way of reporting things (which is true and backed by facts).
It seems that you do not have the same principles...
So, you charge me, off the top of your head, of "altering times to fit my theory"?

First which theory?
Now that you said that I have a "theory that you have something to hide", please quote any of my answers that proves your claim.

Second which timestamps?
I am not claiming that the timestamps are perfectly accurate because, the first thread being locked, I had to do manual quotations which is error prone (regeneration of epoch timestamps from the dates).
But at any rate, I strongly doubt that this would change in any way any of the facts reported earlier.
I you disagree, please give an example of inaccurate timestamp, and show how correcting it makes any of what I said invalid.

After the test, BFL confirm with triumph that the test was a success in establishing that their product is legit.
Again, you didn't comment to moderate the announcement of BFL.
Later posts show that you actually vouch for their conclusion.

So you've said this several times, I just quoted this particular one to address it.  You say that I did not comment to moderate the announcement... but you (and others) also critize me when I do comment on their announcements as being a shill or supporting them, or being blinded by them, etc...  So which is it?  Do you want me to comment/moderate BFL or do you want me to be impartial and let BFL speak for themselves while I report what I see and my own feelings on the subject?  It's impossible to do both.

Your answer is so full of hot air that I just don't know what to tell you.
There is only one objective truth, and it is made of facts.
As a trusted third party mandated to verify the facts, when to comment and what to tell should have been obvious to you.
 

[bombo999]

here's a new fact .... "freequant is a troll", please stop feeding him, end of fact

[Inaba]

I had the courtesy not to attribute you negative intentions, and not to charge you of any accusations beside of that of being misleading in your way of reporting things (which is true and backed by facts).
It seems that you do not have the same principles...
So, you charge me, off the top of your head, of "altering times to fit my theory"?

First which theory?
Now that you said that I have a "theory that you have something to hide", please quote any of my answers that proves your claim.

The theory that I have something to hide.

Second which timestamps?
I am not claiming that the timestamps are perfectly accurate because, the first thread being locked, I had to do manual quotations which is error prone (regeneration of epoch timestamps from the dates).
But at any rate, I strongly doubt that this would change in any way any of the facts reported earlier.
I you disagree, please give an example of inaccurate timestamp, and show how correcting it makes any of what I said invalid.

I already gave you an example of inaccurate time stamps.  You claim that I waited an entire day to clarify that the first "test" was not in the DC, when in fact it was less than 15 minutes and the next post that I made that clarified that fact.  Post 891 - 893 I believe, but I don't have the time to check those post numbers at the moment.  I will when I get back.

Your answer is so full of hot air that I just don't know what to tell you.
There is only one objective truth, and it is made of facts.
As a trusted third party mandated to verify the facts, when to comment and what to tell should have been obvious to you.

And how is this full of hot air?  People bitch when I backup BFL's statements with my observations and opinions - somehow this makes me a shill.  People bitch when I don't comment on BFL's statements, somehow this translates as me being misleading or having something to hide.  WTF, seriously?  How am I suppose to cater to that?

Pick on, pick the other.  I don't care. I will do either, but I physically can not do both.