Okay I've thought about it some more and I think the best way forwards is to do away with the --lookup-gap and --thread-concurrency options and just get the user to input the shader count of their device and let cgminer choose. This will always set lookup gap to 2 and set the thread concurrency to the highest it can be that is a multiple of the shader count which would be much easier for users to configure. This then leaves you with the choice of gpu threads and intensity still. Otherwise there is far too much to configure, and most people won't be able to figure out how to do it. I plan to include a readme with the shader count of the various hardware like in that earlier post. What do people think of this?

Eg for a 6850 and 5770:
cgminer --shaders 960,800

I mean so that cgminer could detect the best settings automatically.

See it says 6800 not what 68x0 ? Same with 5700, not what 57x0. It mattters.

There might not be enough info in opencl to do this from cgminer. What does
cgminer.exe -n
report on these cards?

Darn, but yes the displaying library pdcurses is the issue. You can try the old fashioned -T mode which is just plain text. At least you'll have some output.

Great. I guess I can put in a database into the software to lookup the best values after querying the device... maybe soon.

6850 is 960
5770 is 800

Therefore you should try tc of
6850: 7680
5770: 8000

No it wont pick the best. Read again. It will pick the largest. Then find the multiple of your shaders less than or equal to that and YOU set that manually. That's why I put that list of shader counts below...

Tried increasing the size of your dos prompt window before starting it?

Note that you WILL get better hashrates from using cgminer because of massively improved behaviour with respect to getting work, submitting shares and managing longpolls. So no, I don't believe the results are equivalent overall.

Check the help...
--lookup-gap <arg>  Set GPU lookup gap for scrypt mining, comma separated
--thread-concurrency <arg> Set GPU thread concurrency for scrypt mining, comma separated                                                               

Okay so I've had an extensive discussion with mtrlt about the code and I've done a lot of debugging and I've learnt a lot.

First of all, the values you can safely plug into linux are NOT compatible with what you can plug into windows. There are different restrictions on the allocatable memory dependent on driver/OS combination. Therefore you cannot compare results from the two.

Second, there IS a MEANINGFUL upper limit to aggression or in this case, intensity. It is where the power of 2 is greater than the concurrent threads.

Eg concurrent threads of 8192 has an upper limit of 13 intensity because 2^13 is 8192. You CAN go over this value, but you are absolutely guaranteed to start producing invalid results. How many invalid results you get for the potential rise in hashrate is highly hardware dependent.

The previous release code did no boundary checking or any testing of the device. I have now updated the git tree to test just how much memory it can allocate and it will now AUTOMATICALLY TUNE to the maximum values that are likely to work. I suggest you start it in debug mode with -D to see what it reports as the concurrent threads, and then find the value that is the largest multiple of number of shaders in the device. Eg a 6950 has 134217728 max memory, this works out to concurrency 2048 but it only has 1408 shaders so setting concurrent_threads to 1408 will likely make it faster.

Changing lookup_gap has 2 effects. The larger it is, the higher you can go with thread_concurrency. However, speed also is dependent on architecture design, and virtually all GPUs are fastest at a gap of 2. If you choose a custom gap without choosing a thread concurrency, cgminer will choose the concurrency for you. If you don't choose a gap, it will select 2 for you.

About GPU threads: You should run as many as you can start without cgminer crashing or failing. They do NOT correlate with shaders, compute units, ram or anything else as any meaningful multiple or anything like that.

Now finally, and you can believe me or not on this, but raper sends work to the GPU WITHOUT CHECKING if it was accepted, and gets the return buffer WITHOUT CHECKING if it actually did any work, and then adds the number of hashes it would have expected the GPU to do with that work sent to it. This means that when you start with lots of threads, some of them may not even be doing anything. Or if you've set some borderline invalid values, it will appear to be working fine, report back a big hashrate, but generate less valid shares. So I implore you to check the share rate generation and pretty much ignore the reported hashrate when comparing notes. Remember that cgminer AND raper use virtually identical kernels so should hash at virtually identical rates.

Summary: Start cgminer without setting worksize, vectors, lookup gap or thread concurrency, but in debug mode with -D -T (I made this example up, not sure what it really is)

 [2012-07-23 21:07:18] GPU 0: selecting lookup gap of 2                    
 [2012-07-23 21:07:18] GPU 0: selecting thread concurrency of 2048    

Then if you're on a 5770, you can google it has 800 processing elements, so pick the highest multiple of that while staying under the thread concurrency, so 1600. The nearest power of 2 is 2048 so an intensity of 12.

Give that a go and let's see what happens. I expect different results on windows and linux. Use this table as a guide for what multiples to make concurrent threads.

...and the signal to noise ratio drops to the usual levels 

As per readme, tried?:
sudo modprobe ftdi_sio vendor=0x0403 product=0x6014

There's definitely not enough resources at 10240 gap 2 to run 5 actual threads on a 7970, and since the values don't line up with raper, who knows what it's doing, so long as we get good performance... I've lost interest at looking at the raper code itself since I got the kernel working well.

Yeah I'm not really sure why I can use 10240 on raper and not on this since they're structurally virtually the same kernels.

I don't get an increase in rejects with 4 threads, and I let it choose the default worksize for 7970 which is 64, not 256.

Yes this definitely needs sdk2.6+

EDIT: and make sure you delete any .bins already generated with older SDKs if you upgrade.

Found a sweet spot with my 7970s. Memory 1375 Engine 1135. Increasing engine slows it down beyond that. There is definitely a relationship between engine and memory clock, scrypt settings and even motherboard speed. Assuming higher is better is not going to necessarily be true.

Thanks

I'm pretty sure we'll need to create a database with suitable values. Every card seems to want something completely different. My 7970s really need all 1375MHz of memory or hashrate drops off. Ironically they seem to do the opposite with engine clock - there's a ceiling to how high the hashrate gets and then turning the engine up further doesn't speed things up any more.

Yes cgminer supports higher difficulty shares, which is why I've been trying to get BTC pools to start supporting it with all this faster hardware coming around. Ironically with LTC being much easier to mine difficulty 1 shares, the LTC pools needed to support higher difficulty shares first. P2pool however doesn't actually ask cgminer for higher difficulty shares, it asks for difficulty 1 shares and then internally decides if it's a "true share" based on the target difficulty it meets. It works either way, but cgminer uses less CPU so it makes much more sense to allow the mining software to do the testing. In the scrypt version of cgminer, it actually does the target difficulty even on the GPU for anything less than 4,294,967,295 where difficulty 1 shares on litecoin are only 65,535.

Time is averaged over 5 seconds, and at those high intensities the threads wont report in for over 5 seconds potentially so you'll get serious aliasing artefacts.