Need Help getting DUAL-GPU radeon's to mine in linux (resolved in centos)

[Viceroy]

The following issue resolved by reinstalling OS here.

I had cgminer working beautifully and then one day I went did something stupid... I "upgraded" my software.  Now typically you want to patch your computer to make sure there are no known exploits available to hackers who might want to hijack your machine.  That is NOT the case with fragile systems like these bitcoin mining 'rigs'.  If you have a working system, do yourself a favor and leave it the F alone.

Now, onto the problem.  I cannot get OpenGL to work.  I've tried several versions of Catalyst and they all seem to work fine but the OpenGL calls just seem to fail.  So I started reading about Catalyst and found something very weird... looks like they took GPU support OUT of the SDK.  Am I reading this correctly?  

If they took out the GPU support how do I now compile the miner software?

Here's the beginning of my search for answers:

https://bitcointalk.org/index.php?topic=28402.msg1711097#msg1711097

My current status remains the same...

Code:

$ ./cgminer -n
 [2013-03-31 14:05:49] CL Platform 0 vendor: Advanced Micro Devices, Inc.
 [2013-03-31 14:05:49] CL Platform 0 name: AMD Accelerated Parallel Processing
 [2013-03-31 14:05:49] CL Platform 0 version: OpenCL 1.1 AMD-APP-SDK-v2.4 (595.10)
 [2013-03-31 14:05:49] Error -1: Getting Device IDs (num)
 [2013-03-31 14:05:49] clDevicesNum returned error, no GPUs usable
 [2013-03-31 14:05:49] 0 GPU devices max detected

Though under my current OS I can actually see multiple cards:

Code:

$ lspci -nn | grep VGA
06:00.0 VGA compatible controller [0300]: ATI Technologies Inc Barts XT [ATI Radeon HD 6800 Series] [1002:6738]
0e:00.0 VGA compatible controller [0300]: ATI Technologies Inc Barts XT [ATI Radeon HD 6800 Series] [1002:6738]

ATI *seems* fine:

Code:

$ aticonfig --lsa
* 0. 06:00.0 AMD Radeon HD 6800 Series
  1. 0e:00.0 AMD Radeon HD 6800 Series
  2. 07:00.0 AMD Radeon HD 6800 Series
  3. 0f:00.0 AMD Radeon HD 6800 Series

* - Default adapter

But fglrx just will not work:

Code:

$ fglrxinfo
Error: unable to open display (null)

Oh yea, you think you figured it out?  I wasn't setting my display?

Code:

$ fglrxinfo
Error: unable to open display (null)

$ export DISPLAY=:0

$ fglrxinfo
X Error of failed request:  BadRequest (invalid request code or no such operation)
  Major opcode of failed request:  139 (ATIFGLEXTENSION)
  Minor opcode of failed request:  66 ()
  Serial number of failed request:  13
  Current serial number in output stream:  13

Here's one I haven't seen before:

Code:

./cgminer -n
Setting of real/effective user Id to 0/0 failed
FATAL: Module fglrx not found.
Error! Fail to load fglrx kernel module! Maybe you can switch to root user to load kernel module directly
X Error of failed request:  BadRequest (invalid request code or no such operation)
  Major opcode of failed request:  136 ()
  Minor opcode of failed request:  19
  Serial number of failed request:  12
  Current serial number in output stream:  12

After reading about SDK's after 12.8 not offering GPU support I checked and found indeed they are not there!:

Code:

$ ldconfig -p | grep AMD
        libglut.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libglut.so
        libglut.so (libc6) => /opt/AMDAPP/lib/x86/libglut.so
        libGLEW.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libGLEW.so
        libGLEW.so (libc6) => /opt/AMDAPP/lib/x86/libGLEW.so
        libAMDXvBA.so.1 (libc6,x86-64) => /usr/lib64/libAMDXvBA.so.1
        libAMDXvBA.so.1 (libc6) => /usr/lib/libAMDXvBA.so.1

Compared to a known working system:

Code:

$ ldconfig -p | grep AMD
 libglut.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libglut.so
 libglut.so (libc6) => /opt/AMDAPP/lib/x86/libglut.so
 libamdocl64.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libamdocl64.so
 libamdocl32.so (libc6) => /opt/AMDAPP/lib/x86/libamdocl32.so
 libOpenCL.so.1 (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libOpenCL.so.1
 libOpenCL.so.1 (libc6) => /opt/AMDAPP/lib/x86/libOpenCL.so.1
 libOpenCL.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libOpenCL.so
 libOpenCL.so (libc6) => /opt/AMDAPP/lib/x86/libOpenCL.so
 libGLEW.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libGLEW.so
 libGLEW.so (libc6) => /opt/AMDAPP/lib/x86/libGLEW.so
 libAMDXvBA.so.1 (libc6,x86-64) => /usr/lib64/libAMDXvBA.so.1
 libAMDXvBA.so.1 (libc6) => /usr/lib/libAMDXvBA.so.1

Unfortunately getting those to match did not fix my problem:

Code:

Working:
 ldconfig -p | grep AMD
 libglut.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libglut.so
 libglut.so (libc6) => /opt/AMDAPP/lib/x86/libglut.so
 libamdocl64.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libamdocl64.so
 libamdocl32.so (libc6) => /opt/AMDAPP/lib/x86/libamdocl32.so
 libOpenCL.so.1 (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libOpenCL.so.1
 libOpenCL.so.1 (libc6) => /opt/AMDAPP/lib/x86/libOpenCL.so.1
 libOpenCL.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libOpenCL.so
 libOpenCL.so (libc6) => /opt/AMDAPP/lib/x86/libOpenCL.so
 libGLEW.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libGLEW.so
 libGLEW.so (libc6) => /opt/AMDAPP/lib/x86/libGLEW.so
 libAMDXvBA.so.1 (libc6,x86-64) => /usr/lib64/libAMDXvBA.so.1
 libAMDXvBA.so.1 (libc6) => /usr/lib/libAMDXvBA.so.1

Not Working:
 ldconfig -p | grep AMD
 libglut.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libglut.so
 libglut.so (libc6) => /opt/AMDAPP/lib/x86/libglut.so
 libamdocl64.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libamdocl64.so
 libamdocl32.so (libc6) => /opt/AMDAPP/lib/x86/libamdocl32.so
 libOpenCL.so.1 (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libOpenCL.so.1
 libOpenCL.so.1 (libc6) => /opt/AMDAPP/lib/x86/libOpenCL.so.1
 libOpenCL.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libOpenCL.so
 libOpenCL.so (libc6) => /opt/AMDAPP/lib/x86/libOpenCL.so
 libGLEW.so (libc6,x86-64) => /opt/AMDAPP/lib/x86_64/libGLEW.so
 libGLEW.so (libc6) => /opt/AMDAPP/lib/x86/libGLEW.so
 libAMDXvBA.so.1 (libc6,x86-64) => /usr/lib64/libAMDXvBA.so.1
 libAMDXvBA.so.1 (libc6) => /usr/lib/libAMDXvBA.so.1

Now I've identified another mismatch between the working and non-working systems...

Working:

Code:

$  glxinfo| grep -i version
server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4
OpenGL version string: 4.2.11566 Compatibility Profile Context
OpenGL shading language version string: 4.20
server glx version string: 1.4
client glx version string: 1.4
GLX version: 1.4
OpenGL version string: 4.2.11566 Compatibility Profile Context
OpenGL shading language version string: 4.20

Not Working:

Code:

$  glxinfo| grep -i version
server glx version string: 1.2
client glx version string: 1.4
GLX version: 1.2
OpenGL version string: 1.2 (4.2.11631 Compatibility Profile Context)

Anyone have a good handle on What GLX version is tied to What Catalyst package?


Happy to reformat and start again with any known working configuration of multiple GPU's operating cleanly under Linux.


Working to set up some documentation and test ideas here.

Any and all help appreciated.

The previous issue was resolved by reinstalling OS here.

[Gator-hex]

I cannot get OpenGL to work.

I think you mean OpenCL?

Some OpenCL versions are specific to the Catalyst versions.

Driver Conformance
AMD APP SDK v2.8    AMD Catalyst™ 12.10 (8.961) onwards
AMD APP SDK v2.7    AMD Catalyst™ 12.4 (8.961)
AMD APP SDK v2.6    AMD Catalyst™ 11.12 (8.92)
AMD APP SDK v2.5    AMD Catalyst™ 11.7 (8.872)

http://developer.amd.com/tools/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/
 
Some people claim the oldest one is actually the fastest.

[Viceroy]

I cannot get OpenGL to work.

I think you mean OpenCL

Some OpenCL versions are specific to the Catalyst versions.

Driver Conformance
AMD APP SDK v2.8    AMD Catalyst™ 12.10 (8.961) onwards
AMD APP SDK v2.7    AMD Catalyst™ 12.4 (8.961)
AMD APP SDK v2.6    AMD Catalyst™ 11.12 (8.92)
AMD APP SDK v2.5    AMD Catalyst™ 11.7 (8.872)

http://developer.amd.com/tools/heterogeneous-computing/amd-accelerated-parallel-processing-app-sdk/downloads/
 

Excellent observation, thank you.  Suggested packages?

README in cgminer states: 
The best version for Radeon 5xxx and 6xxx is v2.5, while 7xxx cards need v2.6 or later, 2.7 seems the best.

So I guess I need 2.5.  But since I have 2.4 working on my Centos 6, I'm gonna re-install that.

[Gator-hex]

I'm still using Ye-olde LinuxCoin on a USB stick.

https://bitcointalk.org/index.php?topic=7374.0;all

I haven't messed with it. We have a saying, "if it ain't broke, don't fix it!"

There is a more up to date Bitcoin mining distro with a different name somewhere on BitcoinTalk.

[Viceroy]

I'm still using LinuxCoin on a USB stick.

https://bitcointalk.org/index.php?topic=7374.0;all

We have a saying, "if it ain't broke, don't fix it!"  

Yea, I'm cautious about moving off Centos as I haven't found ANY other linux that lets me see multiple cards with lspci.  Ubuntu based distros won't boot on my machine with two cards installed... screen goes black during install.  Are you running dual gpu cards in your rigs?  (That said, I do have a usb flash drive laying about if anyone has a link to the iso)

Every time I try to install ubuntu/mint or debian I'm cool until the install begins.  As soon as the system tries to 'turn on' my video cards I can't see a thing.  Even when I was successful (remove second card... everything works fine) I put the second card in and the system locks up or refuses to boot or refuses to light up my screens.  Centos/Fedora have been the only systems that let me see multiple 6870x2's with lspci and subsequently aticonfig.

[Viceroy]

Can anyone help me understand this behavior?

I am connecting remotely through a putty ssh session.
I have xming set up on my windows box so I can run x sessions.

Before I set the display variable:

Code:

$ lspci | grep VGA
06:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Barts XT [Radeon HD 6800 Series]

$ fglrxinfo
display: localhost:10.0  screen: 0
OpenGL vendor string: ATI Technologies Inc.
OpenGL renderer string: AMD Radeon HD 6800 Series
OpenGL version string: 1.2 (4.2.11631 Compatibility Profile Context)

$ glxgears
10169 frames in 5.0 seconds = 2019.947 FPS
9102 frames in 5.0 seconds = 1808.066 FPS
7802 frames in 5.0 seconds = 1556.768 FPS
10645 frames in 5.0 seconds = 2121.250 FPS
8972 frames in 5.0 seconds = 1793.257 FPS
8842 frames in 5.0 seconds = 1758.980 FPS
^C

$ aticonfig --lsa
* 0. 06:00.0 AMD Radeon HD 6800 Series
  1. 07:00.0 AMD Radeon HD 6800 Series

* - Default adapter

Then I set the display variable:

Code:

$ export DISPLAY=:0

$ fglrxinfo
X Error of failed request:  BadRequest (invalid request code or no such operation)
  Major opcode of failed request:  152 (GLX)
  Minor opcode of failed request:  19 (X_GLXQueryServerString)
  Serial number of failed request:  12
  Current serial number in output stream:  12

$ glxgears
X Error of failed request:  BadRequest (invalid request code or no such operation)
  Major opcode of failed request:  152 (GLX)
  Minor opcode of failed request:  19 (X_GLXQueryServerString)
  Serial number of failed request:  12
  Current serial number in output stream:  12

$ aticonfig --lsa
* 0. 06:00.0 AMD Radeon HD 6800 Series
  1. 07:00.0 AMD Radeon HD 6800 Series

* - Default adapter

(system specifications can be found at the top of this post in the first link).

[jspielberg]

Not sure it will help... but this is the thread I used way back to get my original 5780 and 5780Ares miner going.

https://bitcointalk.org/index.php?topic=3359.0

Warning though it is ubuntu based and not Redhat/centos/fedora... but maybe some of the ideas are independent of the packager.

Best of luck -
- jeff

[Viceroy]

I just built a new machine with my step by step instruction guide to building a bitcoin miner under centos

[a1phanumrc]

I just built a new machine with my step by step instruction guide to building a bitcoin miner under centos

Excellent guide. Does this mean you resolved the issue above?

I've used CentOS for many years in production environments -- Esp. on VoIP boxes since any downtown will be immediately noticed by clients. Just curious, have you looked into the following project? A Raspberry Pi (or two for redundancy) on each Avalon seems an elegant solution. Could be ported to CentOS easily.

https://bitcointalk.org/index.php?topic=137934.0
http://mineforeman.com/minepeon/

[Viceroy]

I just built a new machine with my step by step instruction guide to building a bitcoin miner under centos

Excellent guide. Does this mean you resolved the issue above?

I've used CentOS for many years in production environments -- Esp. on VoIP boxes since any downtown will be immediately noticed by clients. Just curious, have you looked into the following project? A Raspberry Pi (or two for redundancy) on each Avalon seems an elegant solution. Could be ported to CentOS easily.

https://bitcointalk.org/index.php?topic=137934.0
http://mineforeman.com/minepeon/

Thank you for taking a look at the guide and thank you for that lead, looks like an interesting project... I'll definitely tune in.


The previous issue was resolved by reinstalling OS here.

Thread locked.