Bitcoin software testing effort

[arklan]

So, as some of you may have seen in the Meta section, I posted there regarding a bitcoin software testing subforum. After several days and zero replies, I think I can safely assume that's an idea that's dead on arrival. however! the bitcoin software, specifically referring to the satoshi client here, needs a QA effort. Having spent the last decade in the video game industry doing QA, i felt this was my place to jump in and help.

Unfortunately what little discussion was occurring via the development mailing list last week has altogether ceased, with no progress made. So, I am posting here hoping to stir some interest from volunteers who would be willing to take on some bitcoin quality assurance testing. Initially, all Gavin wants (yes I've spoken with him about this) is a a few basic acceptance tests along the lines of "does this software build run and connect to the network on all our supported platforms?" and related basics.

There is a small stash of donated BTC from the last time Gavin tried to get some testing going that is meant to pay those doing the testing. How that will be distributed is another topic needing discussion - but first I need people to discuss it with who are involved in the testing.

So, is anyone interested in being a part of this effort to test the software of the reference client? I'm also completely open to discussing the testing of other bitcoin clients or software. I'm sure they're all in need. I personally haven't had any contact with their developers or in fact used an alt client at all, hence my focus on the satoshi client.

any feedback at all is welcome.

[jgarzik]

Any level of testing is useful and appreciated.  Various types of testing that are helpful:

• "it works" testing:  Simply run the latest Release Candidate (or latest version, if released).  Make sure all the basics work (for whatever definition of "basics" you desire).  This is the level most accessible to casual users.
• Major features testing:  Develop a short checklist of must-work features, and organize volunteers to work together and go through that checklist, item by item.  Test each major feature on each major platform.
• Stress and fuzz testing:  Attempt to "stress" the system somehow, or randomly corrupt bits of data.  See what breaks.
• Regression testing:  Record bugs fixed, and develop automated test cases that successfully reproduce the bugs on older versions, and verify newer versions remain fixed.
• Unit function testing:  Rigorously exercise each C++ class to ensure it behaves as expected at a micro level.
• Full peer automated testing:  Automated testing of RPC and P2P functions is non-existent, because of the difficulty in doing so.  Find a solution to this problem.
• Data-driven tests: If possible, write software-neutral, data-driven tests.  This enables clients other than the reference one (Satoshi client) to be tested. Embed tests in testnet3 chain, if possible.

The community at large can be a big help simply by doing the first item:  download and run the Release Candidates and the latest version, and report any problems.  Even reporting success is fine by me, for example: "Version 0.7.1 works for me on Windows 7/32-bit" posted on a forum thread.

It is always very difficult to organize any sort of testing regime with open source volunteers that come and go.  Each volunteer chooses their level of involvement.  Any amount of testing and test-case writing, large or small, is helpful to bitcoin.

[jgarzik]

As an aside, typically this is the point where hardware engineers suck in a shocked breath, then gasp "what??? you do not already test every single function?" 

That is the an ideal goal, but it is incredibly difficult given (a) all the different configuration permutations and (b) the need to rely on a mostly volunteer Q/A army.

[arklan]

Thanks for chiming in Jeff. Now I look like slightly less of a loony acting alone.

My understanding of some aspects here is limited, so I want to make sure I understand what I'm talking about before I try and act.

To that end: The reason the full node automated testing of RPC and P2P functionality is difficult is because it would require (at a minimum) two pc's running full bitcoin clients, correct?

If I'm right, I'm going to let that stew in my brain and see what bubbles up. Short of physically using multiple computers, the only immediate thought is renting an VPS or such to do it. Doable, but still a headache. I wonder if there's a way to run multiple virtual machines on the same system at once, and if that would be useful.

Yay! Research time!

[jgarzik]

To that end: The reason the full node automated testing of RPC and P2P functionality is difficult is because it would require (at a minimum) two pc's running full bitcoin clients, correct?

Yes.  More generally, many RPC and P2P requests require a functioning (or simulated) network with remote nodes to query.

A lot of coverage may be gained with simply two nodes, a "local" and "remote" node talking to each other.

However, two nodes will not exercise code important to peer-to-peer software, like good peer selection.

[arklan]

gotcha.  i'll see what i can think up.

[finway]

I'll do "it works" testing.

With HOWTO, i think i can do some other level testing.

[jgarzik]

...and here it a call for "it works" testing, for the current release candidate of 0.7.1:
https://bitcointalk.org/index.php?topic=117874.0

Any testing beyond "it works" is appreciated as well of course.  

Volunteers are encouraged to self-organize in this thread to come up (and execute) more formal test plans.

[arklan]

Hurray! Stuff to do. Will run this later tonight. Need to finish work and set up a VM environment on my home system so I can screw around in safety.