OFFICIAL CGMINER mining software thread for linux/win/osx/mips/arm/r-pi 4.11.1

[Fiyasko]

So I got some debugging on a 6990 and found that, of course, it does things differently to the 5970.

I've committed some changes to the git tree which should detect 5970s and 6990s reliably on linux, and mumble mumble something maybe on windows. I have at least one report of success on windows with 5970 already.

Give it a go and report back!

I gave that windows build a spin and unfortunatly it doesn't work like it should.

I have 2x6990, so 4 GPU's visible for cgminer.  I've ran it with 3 gpu's enabled, one disabled.  By looking at the temperature, it is clear that cgminer is not showing the correct temperature for the correct core.  Also, this version does not show temperatures for cores that are disabled, I think it would be useful if we still can see that.
Next remark: I see (always have, unchanged in this version) for my 2 first cores the RPM's for the fans, but the other 2 just show a percentage...

Im going to assume that that's Because you 6990's only have 1fan Per GPU? Durr? AKA 2 fans, AKA 2 readouts

[-ck]

So I got some debugging on a 6990 and found that, of course, it does things differently to the 5970.

I've committed some changes to the git tree which should detect 5970s and 6990s reliably on linux, and mumble mumble something maybe on windows. I have at least one report of success on windows with 5970 already.

Give it a go and report back!

I gave that windows build a spin and unfortunatly it doesn't work like it should.

I have 2x6990, so 4 GPU's visible for cgminer.  I've ran it with 3 gpu's enabled, one disabled.  By looking at the temperature, it is clear that cgminer is not showing the correct temperature for the correct core.  Also, this version does not show temperatures for cores that are disabled, I think it would be useful if we still can see that.
Next remark: I see (always have, unchanged in this version) for my 2 first cores the RPM's for the fans, but the other 2 just show a percentage...

This is a windows machine, yes? Was this one where the two GPUs from the same card would be not one after the other? I don't know if I can pick those up at all. As I've said before, false positives for linked GPUs would be a big problem whereas false negatives would be no worse than current releases of cgminer. I'll try and make a debugging version for windows which might give me more information.

[jjiimm_64]

So I got some debugging on a 6990 and found that, of course, it does things differently to the 5970.

I've committed some changes to the git tree which should detect 5970s and 6990s reliably on linux, and mumble mumble something maybe on windows. I have at least one report of success on windows with 5970 already.

Give it a go and report back!

I gave that windows build a spin and unfortunatly it doesn't work like it should.

I have 2x6990, so 4 GPU's visible for cgminer.  I've ran it with 3 gpu's enabled, one disabled.  By looking at the temperature, it is clear that cgminer is not showing the correct temperature for the correct core.  Also, this version does not show temperatures for cores that are disabled, I think it would be useful if we still can see that.
Next remark: I see (always have, unchanged in this version) for my 2 first cores the RPM's for the fans, but the other 2 just show a percentage...

This is a windows machine, yes? Was this one where the two GPUs from the same card would be not one after the other? I don't know if I can pick those up at all. As I've said before, false positives for linked GPUs would be a big problem whereas false negatives would be no worse than current releases of cgminer. I'll try and make a debugging version for windows which might give me more information.

I was the one with the out of order 5970's on win7

miner14 is a win7 box
http://cgminerweb.com/example.miner.php.html#miner14

miner16 would be a more normal view for 5970s

[-ck]

Righto, well still some work to do then 

[DeathAndTaxes]

Righto, well still some work to do then  

Want me to really ruin your day?

For me the order of GPU (for 3x5970 rig running windows 7) in the config file is different than the order of the GPU as reported AND the pairs of cards are in differing order too.  I never brought it up because I (via trial & error) got it working and nobody else has ever reported it BUT

I have a 3x5970 watercooled rig.  The only thing I can think that might contribute is that 2 of the cards are ATI reference and one is an XFX black edition.

Code:

Config File -> cgminer display
1st -> GPU #0
2nd -> GPU #2
3rd -> GPU #1
4th -> GPU #4
5th -> GPU #5   <- BTW this is physically the first card, and the one monitors are connected to
6th -> GPU $6

It took forever for me to figure this out.  I got suspicious because when I raised the overclocked on the fastest GPU it was always the slowest GPU that crashed and lower its clock even down to stock didn't help.  So to test my theory I intentionally set the clocks to something like:

Code:

"gpu-engine" : "100,200,300,400,500,600",

   

and then checked the shares submitted after an hour.


Does AMD API exposed a card serial # or any other unique identifying information?

BTW I don't have this problem in Linux.

[-ck]

Does AMD API exposed a card serial # or any other unique identifying information?

Nope nope, as I've said numerous times before this is the most unsatisfying part of it.
The opencl information is useless for determining which card is which. Furthermore if you have 2 monitors connected to a single GPU, it will come up as 2 unique opencl devices.
Then motherboards decide to order cards backwards sometimes with the pci bus id order being the opposite of the device order we end up getting.
Some motherboards do not order their PCIE lanes in numerical order as well, with lanes going 1,3,2 or other randomness.
Then the ATI Display Library gives unique adapter ID information that has absolutely nothing in common with the opencl devices.
The ADL will also give you one for each device, including the ones that don't have OpenCL support so if you have a card that can't mine in your setup, the number of devices cannot match.
There are also unique "thermal devices" in the ADL which had the opportunity to designate GPU 0 and GPU 1 in shared devices (like 6990) but instead they simply come up as unique devices.
Shared GPUs in single cards also do not have a single identifying feature whatsoever to say they're on the same card.
Then of course, windows does something different to linux, and osx doesnt even have an ADL.
All I can do is enumerate them in the order they appear and *hope* they align.
Then I use surrogate markers with circumstantial evidence that the GPUs are on the same card.
As far as I'm concerned, getting any of these bastards behaving in concert is somewhat of a miracle.
All in all it's a mindfuck of epic proportions.

[DeathAndTaxes]

Does AMD API exposed a card serial # or any other unique identifying information?

Nope nope, as I've said numerous times before this is the most unsatisfying part of it.
The opencl information is useless for determining which card is which. Furthermore if you have 2 monitors connected to a single GPU, it will come up as 2 unique opencl devices.
Then motherboards decide to order cards backwards sometimes with the pci bus id order being the opposite of the device order we end up getting.
Some motherboards do not order their PCIE lanes in numerical order as well, with lanes going 1,3,2 or other randomness.
Then the ATI Display Library gives unique adapter ID information that has absolutely nothing in common with the opencl devices.
There are also unique "thermal devices" in the ADL which had the opportunity to designate GPU 0 and GPU 1 in shared devices (like 6990) but instead they simply come up as unique devices.
Shared GPUs in single cards also do not have a single identifying feature whatsoever to say they're on the same card.
Then of course, windows does something different to linux, and osx doesnt even have an ADL.
All I can do is enumerate them in the order they appear and *hope* they align.
Then I use surrogate markers with circumstantial evidence that the GPUs are on the same card.
As far as I'm concerned, getting any of these bastards behaving in concert is somewhat of a miracle.
All in all it's a mindfuck of epic proportions.

I agree.  Sad AMD doesn't provide a serial # via the OpenCL driver.  Then at least one could simply explicitly declare everything via config file

Like I said via trial and error I got it to work and it is only on my Windows workstation so I am not that weird.  It just seems "incomplete" to me that AMD doesn't provide better enumeration tools especially via the heterogeneous nature of OpenCL devices.

[-ck]

Does his mean you don't need another Linux/6990 box?

Correct, thanks for the offer though. I just happened to bug someone online and get the access I needed. Linux is working mostly ok now with these dual card GPUs. Windows on the other hand... as you can see.

[jake262144]

...All in all it's a mindfuck of epic proportions.

Never did any GPU programming and now I hope I won't ever have to.
If the lane IDs are getting screwed up that's likely due to some insane PCIE line multiplexing and redriving being done by some board designs...

There, Conman, a little something has just been sent your way.

[jjiimm_64]

There, Conman, a little something has just been sent your way.

https://bitcointalk.org/index.php?topic=61027.0

[jake262144]

https://bitcointalk.org/index.php?topic=61027.0

Nah, I don't need that. Always been donating straight to Con's address.
Still, that's a commendable enterprise.

[-ck]

Thanks.

Here is a cgdebug.exe special build that outputs extra information about each device.

http://ck.kolivas.org/apps/cgminer/temp/cgdebug.exe

For those on windows with 5970 or 6990 that are out of order, please run this like you would run cgminer, but with -T and stop it once it starts mining. Then please post the lines that look like this:

Code:

[2012-01-22 17:31:58] lpAdapterID 20016752 iBusNumber 1 iDeviceNumber 0 iFunctionNumber 0 iVendorID 4098 strAdapterName AMD Radeon HD 6900 Series  strDisplayName :0.0 lpInfo.strUDID  256:26393:4098:12583:5762 

And then please describe the layout of where the GPUs of each gpu "twin" appears.

Thanks!

[kano]

one tiny suggestion, you could make the api socket work a lot faster after a restart by setting SO_REUSEADDR with setsockopt.  I do not know if this might have negative effects on systems other linux, but it works fine here and makes socket always open on first try (no more "API bind to port %d failed - trying again in 15sec", which I was seeing quite a lot).

doubt you actually need the code from me heh.  but in case, at line 889 of api.c, after sock is returned from socket()..

int optval_reuseaddr = 1;
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &optval_reuseaddr, sizeof(optval_reuseaddr));

or something like that..  makes life nicer.

Thanks I'll probably put SO_REUSEADDR in next time I'm making changes.
(but I'll try test a few possible issues with it first in case it causes something else)

The current code doesn't need it if you quit from the API, but if you quit directly from cgminer (pressing 'q') the socket is killed and thus doesn't clean up (i.e. doesn't get to call shutdown() )
Though really I should make the socket thread shutdown tidily by calling shutdown() also when you 'q' cgminer, except none of the other threads do, so I thought it sorta pointless to add the extra interrupt code.

[DeathAndTaxes]

For those on windows with 5970 or 6990 that are out of order, please run this like you would run cgminer, but with -T and stop it once it starts mining. Then please post the lines that look like this:

...

And then please describe the layout of where the GPUs of each gpu "twin" appears.

Code:

C:\bitcoin\cgminer-2.1.2>cgdebug -T
[2012-01-27 05:51:04] iAdapterIndex  0 strUDID PCI_VEN_1002&DEV_689C&SUBSYS_20421002&REV_00_8&36D7B016&0&004000100008A iBusNumber 10 iDeviceNumber 0 FunctionNumber 0 iVendorID 1002 strAdapterName  ATI Radeon HD 5900 Series
[2012-01-27 05:51:04] iAdapterIndex  2 strUDID PCI_VEN_1002&DEV_689C&SUBSYS_20421002&REV_00_8&D94A8A1&0&004000180008A iBusNumber 14 iDeviceNumber 0 iFunctionNumber 0 iVendorID 1002 strAdapterName  ATI Radeon HD 5900 Series
[2012-01-27 05:51:05] iAdapterIndex  4 strUDID PCI_VEN_1002&DEV_689C&SUBSYS_25421002&REV_00_8&21A460A4&0&002000100008A iBusNumber 9 iDeviceNumber 0 iFunctionNumber 0 iVendorID 1002 strAdapterName  ATI Radeon HD 5900 Series
[2012-01-27 05:51:05] iAdapterIndex  7 strUDID PCI_VEN_1002&DEV_689C&SUBSYS_25421002&REV_00_8&79EA6D1&0&002000180008A iBusNumber 13 iDeviceNumber 0 iFunctionNumber 0 iVendorID 1002 strAdapterName  ATI Radeon HD 5900 Series
[2012-01-27 05:51:05] iAdapterIndex 10 strUDID PCI_VEN_1002&DEV_689C&SUBSYS_30201682&REV_00_8&2239DA41&0&002000000008A iBusNumber 5 iDeviceNumber 0 iFunctionNumber 0 iVendorID 1002 strAdapterName  ATI Radeon HD 5900 Series
[2012-01-27 05:51:05] iAdapterIndex 13 strUDID PCI_VEN_1002&DEV_689C&SUBSYS_30201682&REV_00_8&376D29B3&0&004000000008A iBusNumber 6 iDeviceNumber 0 iFunctionNumber 0 iVendorID 1002 strAdapterName  ATI Radeon HD 5900 Series

One thing I can help decode is:
SUBSYS_20421002
SUBSYS_25421002
SUBSYS_30201682

The GPU-Z utility indicates the suffix 1002 is ATI and 1682 is XFX

What is interesting is I have 3 5970 and 3 different prefixes (2042, 2542, 3020).  Probably coincidence but one can hope.

Is it possible this is a GPU level serial #?
[2012-01-27 05:51:04] iAdapterIndex  0 strUDID PCI_VEN_1002&DEV_689C&SUBSYS_20421002&REV_00_8&36D7B016&0&004000100008A iBusNumber 10 iDeviceNumber 0 FunctionNumber 0 iVendorID 1002 strAdapterName  ATI Radeon HD 5900 Series

Remember on my Win7 workstation the card order isn't consistent.

In cgminer display:

Code:

GPU #0 & GPU #2 = 2nd physical card (2nd expansion slot from CPU)
GPU #1 & GPU #3 = 3rd physical card (3rd expansion slot from CPU)
GPU #5 & GPU #6 = 1st physical card (1st expansion slot from CPU)

In cgminer.conf:

Code:

"gpu-engine" : "Card2-GPU1,Card2-GPU2,Card3-GPU1,Card3-GPU2,Card1-GPU1,Card1-GPU2",

Hey ckolivas I just thought of something.  On two of my cards the system enumerates the first GPU of each card first and the the second GPU of each card but the third GPU is both GPU in a row.   So C1, C2, C1, C2, C3, C3 right?  Now other people have describe both methods (C1, C1, C2, C2, C3, C3  or C1, C2, C3, C1, C2, C3).

I am wondering if crossfire has something to do with the way Windows orders the card. The 5970 and 6990 have an internal crossfire bridge.  My understanding is the card is simply a PCIe switch which converts one physical PCIe slot into two seperate "virtual" PCIe slots.  The two GPU are then wired to that and have a cross fire bridge hardwired between them.  Logically it is no different than using a 2:1 PCIe expansion adapter and plugging into 2 5870s.   I wonder if I have crossfire accidentally turned on for just one of my cards thus 2 cards are ordered one way and the last card is ordered differently.  I can't check remotely but it the only thing I can think of to explain the strange combination of both GPU ordering patterns.

That might explain how/why people have one of two different ordering patterns (A, B, C, A, B, C) or (A, A, B, B, C, C).  One pattern is used when crossfired, one when not?  Maybe it has to do w/ what the cross fire status was when windows was first installed?  I am rambling now ...

TL/DR maybe when people post their ordering data they should also check crossfire status in either GPU-Z or CCC.

[-ck]

Thanks very much for that D&T

You said:

A iBusNumber 10
B iBusNumber 14
C iBusNumber 9
D iBusNumber 13
E iBusNumber 5
F iBusNumber 6

and:
GPU #0 & GPU #2 = 2nd physical card (2nd expansion slot from CPU)
GPU #1 & GPU #3 = 3rd physical card (3rd expansion slot from CPU)
GPU #5 & GPU #6 = 1st physical card (1st expansion slot from CPU)

This is reassuring. Using my existing logic, cgminer should already be grouping A+C, B+D and E+F which corresponds with your layout. This should actually already be working. GPU 0 and GPU 2 should be showing the same fanspeed and so on. Is it not working?

I'm trying to recall if you watercooled or something, because cgminer also relies on one card having a fancontrol while the other does not. Is there any way you have removed the fancontrol? Ultimately all I'm trying to do with this code is make the fan speed control work from the temps from both GPUs.

If not, I could always remove the test for one having a fancontrol and the other not having and relying entirely on the bus numbers.

[DeathAndTaxes]

This is reassuring. Using my existing logic, cgminer should already be grouping A+C, B+D and E+F which corresponds with your layout. This should actually already be working. GPU 0 and GPU 2 should be showing the same fanspeed and so on. Is it not working?

It "does".  Just (at least in my instance) it seems weird it is
A+C
B+D
E+F

instead of
A+D
B+E
C+F

or
A+B
C+D
E+F

but this is the only rig that has a single different card "XFX Black Edition" so I wonder if that is making it "weird".  My other 14 5970s are all ATI branded reference. 

I'm trying to recall if you watercooled or something, because cgminer also relies on one card having a fancontrol while the other does not. Is there any way you have removed the fancontrol? Ultimately all I'm trying to do with this code is make the fan speed control work from the temps from both GPUs.

It is watercooled so there are no fans connected.  I didn't do anything to the card other than unplug the fan header. It does report fan speed (one per pair of GPUs).  I have GPU-fan set to 0% for all 6 cards in config file.  It reports some low RPM like 400 or something.  Weirdly that XFX bastard reports no fan speed (or temp) for either GPU.

Honestly I wish I had never got this XFX black edition.  The more I think about it the more likely I am to ebay it and grab another ATI branded reference card.

When I use windows remote desktop I lose all temp/fan/clock data so when I get home I will grab a screenshot of what cgminer shows when mining.

I will report w/ screenshot of cgminer mining, the data debug dump and the config file all in one post.  Hopefully if others do the same you can get enough data points.

[-ck]

I was trying to identify linked GPUs with this latest code to prevent frying a GPU by disabling the one that had the fan control while the one without the fan control was still burning.

Hmmmm D&T... I was thinking that I can't fix the ordering, as that happens on a bios+mobo+os+driver+/- crossfire basis 

There is a way though... hmm but it would require running through all the ADL code first and then renumbering everything... hmmmmmm

* ckolivas hmms some more...

I think I have a way.

* ckolivas ponders.

BBIAW...

[The00Dustin]

Hmmmm D&T... I was thinking that I can't fix the ordering, as that happens on a bios+mobo+os+driver+/- crossfire basis 

There is a way though... hmm but it would require running through all the ADL code first and then renumbering everything... hmmmmmm

* ckolivas hmms some more...

I think I have a way.

* ckolivas ponders.

BBIAW...

I think D&T's concern was that you assumed it was always A+C,B+D,E+F more than what order they displayed in.  Based on your acknowledgements RE: bios+mobo+os+driver+/-crossfire that isn't the case and you may be able to identify them appropriately regardless.  However, if you can and want to make them display in a logical order, more power to you. 

[-ck]

Yes I understood that. I just didn't expect it was possible, but this extra debugging has shown me it is. I just need to actually code it up now.

[DeathAndTaxes]

Yes I understood that. I just didn't expect it was possible, but this extra debugging has shown me it is. I just need to actually code it up now.

That would be awesome and would certainly warrant a donation from any miner who uses multiple dual GPU cards.  Well certainly from the non-cheapo ones.