Bitcoin Forum
November 21, 2017, 08:11:55 PM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 ... 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 [54] 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 ... 287 »
  Print  
Author Topic: [OS] nvOC easy-to-use Linux Nvidia Mining v0019-1.4  (Read 327100 times)
cryptosharks
Newbie
*
Offline Offline

Activity: 8


View Profile
June 22, 2017, 10:42:02 AM
 #1061

Hey @fullzero,

I have encountered an issue with ccminer after a reboot:

error while loading shared libraries: libcudart.so.x: cannot open shared object file: No such file or directory

Also, I have found a solution:

http://www.cs.virginia.edu/~mwb7w/cuda_support/libcudart.html

The above has been proven to get ccminer to start without any issues.

Also, I've been testing the latest nvOC version on 5 mining rigs - so far all looking good.

Cheers,
Join ICO Now A blockchain platform for effective freelancing
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1511295115
Hero Member
*
Offline Offline

Posts: 1511295115

View Profile Personal Message (Offline)

Ignore
1511295115
Reply with quote  #2

1511295115
Report to moderator
1511295115
Hero Member
*
Offline Offline

Posts: 1511295115

View Profile Personal Message (Offline)

Ignore
1511295115
Reply with quote  #2

1511295115
Report to moderator
1511295115
Hero Member
*
Offline Offline

Posts: 1511295115

View Profile Personal Message (Offline)

Ignore
1511295115
Reply with quote  #2

1511295115
Report to moderator
Nexillus
Full Member
***
Offline Offline

Activity: 132


View Profile
June 22, 2017, 11:15:03 AM
 #1062

Fullzero,

I want to say that ver16 is much more stable then ver15 and great work on it overall!

Thank you so much for putting this together, disseminating it it and all the hard work you do!

Bitrated user: nexillus.
car1999
Full Member
***
Offline Offline

Activity: 168


View Profile
June 22, 2017, 11:33:48 PM
 #1063

I've run nvOC 7 hours on a gtx1050ti x 7 rig, very stable, very easy, awesome project, thanks!
pixelizedchaos
Newbie
*
Offline Offline

Activity: 18

You only die when you are forgotten.


View Profile
June 22, 2017, 11:47:46 PM
 #1064

Hello! sorry but somewhat of a noob. I am really interested in SSH'ing into my rig from external networks, I have used Team viewer, but with local SSH it seems a lot more reliable.

So essentially I have two questions:
1) When I am connected through SSH and let's say I want to restart the gnome terminal back on the miner instead of my computer what command do I use to restart the PID that I initially closed.
2) How would I go about setting up a remote server which I can ssh into remotely, without being in the local area network?

Any help is appreciated! Also absolutely love your work, plan on showcasing it in videos once I learn a bit more about it! Thanks so much!
(Ps. if anyone has used ASUS Dual Series 1070's and found the optimal OC for them please let me know!)

I use [OS] nvOC easy-to-use Linux Nvidia Mining <3
fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:24:47 AM
 #1065

When I try to run lbry I get an error. Something to the effect of "980ti: invalid device function"

Works fine with ZEC and ZEN

I removed a timeout before the cmd that adds the link at each boot to solve this problem.  i thought I would be ok with all USBs this way but it apparently still needs the timeout.

You can fix this permentely by opening the home directory and opening the 2unix file.  on the line above the line:

Code:
sudo ldconfig /usr/local/cuda/lib64

add:

Code:
sleep 10

You can also open the guake terminal ( f12 )

and manually enter the cmd at any time (but this will only solve for the current boot):
Code:
sudo ldconfig /usr/local/cuda/lib64


If you download and replace your existing oneBash with the newest one From the link on the OP it should solve this problem as well.


fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:37:24 AM
 #1066

Hi All,

I have decided to give nvOC a go, I am using an older Gigabyte GA-EX58-UD4P (Socket LGA1366) board and an i7 920 CPU.  I have two EVGA GTX 1080ti on risers in both PCI-E x16 slots.
I am able to boot fine but things just are not stable, I can run for about 15 hours but later GPU0 drops to about 150-170w and SOLs are suffering, I can reboot and get it working but it seems to happen again.

I have no idea if any BIOS changes need to be made to get this running stable or not, does anyone have any idea what I can try?

Also if I kill the mining process and make changes to the onebash config it doesn't seem to accept the changes, more specially the fan speed.  I attempt to launch the miner again and get the below error but the process does start;


Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.


When the mining process starts do you see OC messages like this?

https://ip.bitcointalk.org/?u=https%3A%2F%2Fs13.postimg.org%2Fq08huqnyv%2FIMG_0270.jpg%26t%3D577%26c%3Dyq0szP4ICxv47w&t=577&c=q_tbQ_FaeljjYQ

If you don't:

At any point did you boot with the monitor connected to the motherboard?

Did you at anytime boot with only one GPU attached?

If either of these is the case: ensure the monitor is attached to the primary GPU ( the one connected to the 16x slot closest to the CPU )

then follow this process:

https://bitcointalk.org/index.php?topic=1854250.msg19449945#msg19449945

I only booted with a monitor connected the first time, now it is completely headless.
Yes I might have booted with only one GPU attached at one point.
Thank you I will follow that process.  If I re-image the USB key and boot completely headless should I be seeing the OC messages via SSH as well?

With v0015 you would have to enable openssh server before you could SSH in.  You would also have to be able to know the rigs IP; which can be done in several ways.

With v0016 you can enable openssh server in oneBash, I would recommending trying it and seeing if it solves the problem.

I have now also tested with v0016, the mining process starts fine with all OC settings when viewing directly on the rig.  However I am still getting the error via SSH when trying to start.

Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.

What type of OS is the client computer; and what are the IPs of the client and rig?

Windows 10 running putty, Rig: 192.168.1.19 Client: 192.168.1.6

So when you enter

m1@192.168.1.19   

using port 22

with SSH selected

and click open it gives you the error above?


Sorry let me try and clear some things up.  I am able to SSH into the rig just fine but when executing the miner via ssh, this is when i am getting the error, so clock settings, fan speed and power limits are not being set but the mining process still starts.

I also really appreciate your support and rapid response.

Are you killing the existing mining process before launching another via SSH?

Also; when you are launching the mining process via SSH are you using the cmd:

Code:
bash '/media/m1/1263-A96E/oneBash'

I am SSHing and executing the following commands.

ps aux | grep gnome-terminal - finding the gnome-server PID
kill PID
screen -S rig1
bash '/media/m1/1263-A96E/oneBash'

Just before the miner starts I see the above error in place of the "attribute" commands but the mining process still starts.


after you have SSHed in enter the cmd:

Code:
echo $DISPLAY

and tell me what it outputs

That does not output anything at all.

Ok, tomorrow I will try to replicate this error; and see if I can figure out what is happening.  My guess is X11 is having a problem trying to output graphically.

If you have a linux computer other than your rig and you SSH into the rig with it; does it have the same error?

I tested from a linux VM I have and the same result Sad


I tested this today and found there is a new problem resulting (most likely) from my adding support for up to 14 GPUs. 

I found that if I waited the screen would still connect to the mining process after failing to connect 3 or 4 times.

If you are using linux
you can resolve this error by adding the following argument when SSHing into the rig:

Code:
-X

so that from a terminal I would enter:

Code:
ssh m1@rigipaddress -X

so for a rig with an ip of 192.168.1.22. I would use:

Code:
ssh m1@192.168.1.22 -X

If you are using putty with windows the -X will not work; but after showing the error 3 or 4  times the screen should still connect to the mining process


I also tested using the -dmS argument when calling screen.

if you call screen with:
Code:
screen -dmS rig1

screen with start as a background process (so you can disconnect your ssh session and the miner will continue to mine on its own)

after starting screen you will need to connect to the screen (as it is running in the background)

you do this by entering:

Code:
screen -r

you can close the ssh session whenever and then reSSH in and enter:

Code:
screen -r

to return to the mining process whenever desired.


fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:39:43 AM
 #1067

Hi,

Good work.

However it takes several minutes to boot from my USB 3.0 stick?
The booting seems have some problem.

V16 changed name to amdOC?

I think you downloaded the amd Beta; it is only for amd GPUs:

scroll further down on the OP and you should find the v0016 download link.

bdiddle
Member
**
Online Online

Activity: 62


View Profile
June 23, 2017, 12:42:38 AM
 #1068

Is there an easy way to switch mining pool/coin without reloading one bash? Like, if I just want it to switch to ZCL rather than ZEC.

I looked on the first page and didn't see an easy way to do that.
fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:45:23 AM
 #1069

Hey @fullzero,

I have encountered an issue with ccminer after a reboot:

error while loading shared libraries: libcudart.so.x: cannot open shared object file: No such file or directory

Also, I have found a solution:

http://www.cs.virginia.edu/~mwb7w/cuda_support/libcudart.html

The above has been proven to get ccminer to start without any issues.

Also, I've been testing the latest nvOC version on 5 mining rigs - so far all looking good.

Cheers,

Thanks for effort and bringing this to my attention:  see these posts:

https://bitcointalk.org/index.php?topic=1854250.msg19720949#msg19720949


https://bitcointalk.org/index.php?topic=1854250.msg18578060#msg18578060


fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:47:44 AM
 #1070

Is there an easy way to switch mining pool/coin without reloading one bash? Like, if I just want it to switch to ZCL rather than ZEC.

I looked on the first page and didn't see an easy way to do that.

No there is currently no way to do that other than to modify oneBash.

You can modify oneBash remotely with nano or on rig with gedit.

fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:53:27 AM
 #1071

I will respond to all posts later today: sorry if I have taken a longtime to respond.  I will make sure I respond to everyone at some point tonight.

v0016 is out see the OP for a link.

I didn't implement all requested changes, but I think I got most of them.  If I missed yours with v0016 I will get to it soon.

I have only tested this build on 8 rigs; so let me know if you find any bugs.  Grin

Change Log:
v0016
  auto repair xorg: fixes single GPU not OCing bug (will auto detect / repair / reboot if using a single GPU or the xorg file has otherwise been damaged)
  implemented Maxximus007 suggestion to enable expect-less oneBash
  Thanks Maxximus007  Smiley
  powerlimit not setting bug is fixed (even for very slow USB keys)
  added Teamviewer with oneBash YES / NO switch to enable / disable it
  added openssh-server switch in oneBash YES / NO switch to enable / disable it
  added Desktop Sharing RDP program (enable / setup with GUI on the taskbar)
  added ifconfig output to the top of mining process (easy to get rig IP)
  added EWBF v3.4
  added Claymore v9.5
  added EWBF and Claymore version selectors to oneBash
  added EXTENTION_ARGUMENTS for each Ethash COIN (to add additional Claymore parameters)
  increased Claymore stability
  reduced boot time
  will now work with slow USB keys
    VTC COIN selection with VTC or BTC payout
    ZEN COIN selection
    NICE COIN selection




keep getting low disk space
is something logging now?

There should be no logging other than the standard Ubuntu syslog; which should take up almost no space.

In v0016; to make the image smaller: I reduced the size of the primary partition by almost 1gb, which also reduced the free space available to the system. 

It is possible I removed too much, and should enlarge the partition for the next version.  Let me know if anything other than low disk messages happen; or if it appears performance is degraded in anyway as a result.



You might want to run sudo apt update, sudo apt upgrade, sudo apt autoremove. This will remove around 1.5 Gb obsolete files. Apt was complaining about a file in /lib/modules/4.4.0-65.generic/kernel/drivers/media/usb/hdpvr/hdpvr.ko. I removed that PID file by hand, and ran sudo apt autoremove again. Currently 3.3Gb free space, that should be plenty.
** disclaimer ** it worked for me, YMMV

Most of the larger 'obsolete files' are system files for chipsets and cpus other than the ones your rig is currently using.  If you don't plan on using that USB key with a different chipset / CPU; there shouldn't be any problems removing them and getting more space.

fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:55:01 AM
 #1072

Version 0016 installed, I just tried teamviewer, PL is working good.
I tried with 9x gtx 1060 3gb mining ETHEREUM at 22.5 each with 880 watts on the wall. TOTAL 202 Mh/s

POWERLIMIT="YES"
POWERLIMIT_WATTS=84
__CORE_OVERCLOCK=100
MEMORY_OVERCLOCK=1400

Working temp 58C - 64C / fan at 75%

I had the issue of SSHing and executing bash '/media/m1/1263-A96E/oneBash',
it gave me this error,:

Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help` for usage information.

but now that I have teamviewer then I don´t need that anymore...

thanks fullzero

If you ever decide to SSH again for whatever reason look at this post:

https://bitcointalk.org/index.php?topic=1854250.msg19721060#msg19721060

fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 12:57:30 AM
 #1073

Amazing work ont eh update.

Got it working with no issues but then received the following error.  Dont know what to make of it.


 DCR: Stratum - connecting to 'dcr.suprnova.cc' <217.182.138.167> port 3252
ETH: Stratum - Connected (eth.suprnova.cc:5000)
 DCR: Stratum - Connected (dcr.suprnova.cc:3252)
 DCR: Authorized
 DCR: 06/22/17-00:04:37 - New job from dcr.suprnova.cc:3252

Received unknown response: {"id":null,"method":"mining.notify","params":["3247","129656acf5804305816f3dc0b32b47077c8cf4b8724f181efc529a93e4c1385b","999803e4fef6e5c5edd9968b3e00398713ca6f8d7f24812ea49ebf8373f11e8d",true]}
GPU0 t=54C fan=0%, GPU1 t=52C fan=0%, GPU2 t=49C fan=0%
ETH: Job timeout, disconnect, retry in 20 sec...


anybody have any thoughts?

not sure

show me what these sections in your oneBash look like:

Code:
ETHERMINEdotORG="NO"
CLAYMORE_VERSION="9_5"    # choose 9_5  or  9_4  or  8_0

ETH_WORKER="nvOC"
ETH_ADDRESS="0xe12bdd454997e443ec0cae6bebb6bb3c74242aae"
ETH_POOL="eth-us-east1.nanopool.org:9999"
ETH_EXTENTION_ARGUMENTS=""    # add any additional claymore arguments desired here

DCR_WORKER="nvOC"
DCR_ADDRESS="fullzero22"
DCR_POOL="stratum+tcp://dcr.suprnova.cc:3252"

and I might be able to figure it out.

lbrasi
Newbie
*
Offline Offline

Activity: 25


View Profile
June 23, 2017, 01:03:31 AM
 #1074

Hi All,

I have decided to give nvOC a go, I am using an older Gigabyte GA-EX58-UD4P (Socket LGA1366) board and an i7 920 CPU.  I have two EVGA GTX 1080ti on risers in both PCI-E x16 slots.
I am able to boot fine but things just are not stable, I can run for about 15 hours but later GPU0 drops to about 150-170w and SOLs are suffering, I can reboot and get it working but it seems to happen again.

I have no idea if any BIOS changes need to be made to get this running stable or not, does anyone have any idea what I can try?

Also if I kill the mining process and make changes to the onebash config it doesn't seem to accept the changes, more specially the fan speed.  I attempt to launch the miner again and get the below error but the process does start;


Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.


When the mining process starts do you see OC messages like this?

https://ip.bitcointalk.org/?u=https%3A%2F%2Fs13.postimg.org%2Fq08huqnyv%2FIMG_0270.jpg%26t%3D577%26c%3Dyq0szP4ICxv47w&t=577&c=q_tbQ_FaeljjYQ

If you don't:

At any point did you boot with the monitor connected to the motherboard?

Did you at anytime boot with only one GPU attached?

If either of these is the case: ensure the monitor is attached to the primary GPU ( the one connected to the 16x slot closest to the CPU )

then follow this process:

https://bitcointalk.org/index.php?topic=1854250.msg19449945#msg19449945

I only booted with a monitor connected the first time, now it is completely headless.
Yes I might have booted with only one GPU attached at one point.
Thank you I will follow that process.  If I re-image the USB key and boot completely headless should I be seeing the OC messages via SSH as well?

With v0015 you would have to enable openssh server before you could SSH in.  You would also have to be able to know the rigs IP; which can be done in several ways.

With v0016 you can enable openssh server in oneBash, I would recommending trying it and seeing if it solves the problem.

I have now also tested with v0016, the mining process starts fine with all OC settings when viewing directly on the rig.  However I am still getting the error via SSH when trying to start.

Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.

What type of OS is the client computer; and what are the IPs of the client and rig?

Windows 10 running putty, Rig: 192.168.1.19 Client: 192.168.1.6

So when you enter

m1@192.168.1.19   

using port 22

with SSH selected

and click open it gives you the error above?


Sorry let me try and clear some things up.  I am able to SSH into the rig just fine but when executing the miner via ssh, this is when i am getting the error, so clock settings, fan speed and power limits are not being set but the mining process still starts.

I also really appreciate your support and rapid response.

Are you killing the existing mining process before launching another via SSH?

Also; when you are launching the mining process via SSH are you using the cmd:

Code:
bash '/media/m1/1263-A96E/oneBash'

I am SSHing and executing the following commands.

ps aux | grep gnome-terminal - finding the gnome-server PID
kill PID
screen -S rig1
bash '/media/m1/1263-A96E/oneBash'

Just before the miner starts I see the above error in place of the "attribute" commands but the mining process still starts.


after you have SSHed in enter the cmd:

Code:
echo $DISPLAY

and tell me what it outputs

That does not output anything at all.

Ok, tomorrow I will try to replicate this error; and see if I can figure out what is happening.  My guess is X11 is having a problem trying to output graphically.

If you have a linux computer other than your rig and you SSH into the rig with it; does it have the same error?

I tested from a linux VM I have and the same result Sad


I tested this today and found there is a new problem resulting (most likely) from my adding support for up to 14 GPUs. 

I found that if I waited the screen would still connect to the mining process after failing to connect 3 or 4 times.

If you are using linux
you can resolve this error by adding the following argument when SSHing into the rig:

Code:
-X

so that from a terminal I would enter:

Code:
ssh m1@rigipaddress -X

so for a rig with an ip of 192.168.1.22. I would use:

Code:
ssh m1@192.168.1.22 -X

If you are using putty with windows the -X will not work; but after showing the error 3 or 4  times the screen should still connect to the mining process


I also tested using the -dmS argument when calling screen.

if you call screen with:
Code:
screen -dmS rig1

screen with start as a background process (so you can disconnect your ssh session and the miner will continue to mine on its own)

after starting screen you will need to connect to the screen (as it is running in the background)

you do this by entering:

Code:
screen -r

you can close the ssh session whenever and then reSSH in and enter:

Code:
screen -r

to return to the mining process whenever desired.



Tested with -X on the SSH command from my linux VM and now I am getting the below;

ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUGraphicsClockOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUMemoryTransferRateOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUGraphicsClockOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUMemoryTransferRateOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment '[gpu:0]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:0' (No targets match target
       specification), specified in assignment '[fan:0]/GPUTargetFanSpeed=65'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment '[gpu:1]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:1' (No targets match target
       specification), specified in assignment '[fan:1]/GPUTargetFanSpeed=65'.
fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 01:08:33 AM
 #1075

Hi All,

I have decided to give nvOC a go, I am using an older Gigabyte GA-EX58-UD4P (Socket LGA1366) board and an i7 920 CPU.  I have two EVGA GTX 1080ti on risers in both PCI-E x16 slots.
I am able to boot fine but things just are not stable, I can run for about 15 hours but later GPU0 drops to about 150-170w and SOLs are suffering, I can reboot and get it working but it seems to happen again.

I have no idea if any BIOS changes need to be made to get this running stable or not, does anyone have any idea what I can try?

Also if I kill the mining process and make changes to the onebash config it doesn't seem to accept the changes, more specially the fan speed.  I attempt to launch the miner again and get the below error but the process does start;


Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.


When the mining process starts do you see OC messages like this?

https://ip.bitcointalk.org/?u=https%3A%2F%2Fs13.postimg.org%2Fq08huqnyv%2FIMG_0270.jpg%26t%3D577%26c%3Dyq0szP4ICxv47w&t=577&c=q_tbQ_FaeljjYQ

If you don't:

At any point did you boot with the monitor connected to the motherboard?

Did you at anytime boot with only one GPU attached?

If either of these is the case: ensure the monitor is attached to the primary GPU ( the one connected to the 16x slot closest to the CPU )

then follow this process:

https://bitcointalk.org/index.php?topic=1854250.msg19449945#msg19449945

I only booted with a monitor connected the first time, now it is completely headless.
Yes I might have booted with only one GPU attached at one point.
Thank you I will follow that process.  If I re-image the USB key and boot completely headless should I be seeing the OC messages via SSH as well?

With v0015 you would have to enable openssh server before you could SSH in.  You would also have to be able to know the rigs IP; which can be done in several ways.

With v0016 you can enable openssh server in oneBash, I would recommending trying it and seeing if it solves the problem.

I have now also tested with v0016, the mining process starts fine with all OC settings when viewing directly on the rig.  However I am still getting the error via SSH when trying to start.

Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.

What type of OS is the client computer; and what are the IPs of the client and rig?

Windows 10 running putty, Rig: 192.168.1.19 Client: 192.168.1.6

So when you enter

m1@192.168.1.19   

using port 22

with SSH selected

and click open it gives you the error above?


Sorry let me try and clear some things up.  I am able to SSH into the rig just fine but when executing the miner via ssh, this is when i am getting the error, so clock settings, fan speed and power limits are not being set but the mining process still starts.

I also really appreciate your support and rapid response.

Are you killing the existing mining process before launching another via SSH?

Also; when you are launching the mining process via SSH are you using the cmd:

Code:
bash '/media/m1/1263-A96E/oneBash'

I am SSHing and executing the following commands.

ps aux | grep gnome-terminal - finding the gnome-server PID
kill PID
screen -S rig1
bash '/media/m1/1263-A96E/oneBash'

Just before the miner starts I see the above error in place of the "attribute" commands but the mining process still starts.


after you have SSHed in enter the cmd:

Code:
echo $DISPLAY

and tell me what it outputs

That does not output anything at all.

Ok, tomorrow I will try to replicate this error; and see if I can figure out what is happening.  My guess is X11 is having a problem trying to output graphically.

If you have a linux computer other than your rig and you SSH into the rig with it; does it have the same error?

I tested from a linux VM I have and the same result Sad


I tested this today and found there is a new problem resulting (most likely) from my adding support for up to 14 GPUs. 

I found that if I waited the screen would still connect to the mining process after failing to connect 3 or 4 times.

If you are using linux
you can resolve this error by adding the following argument when SSHing into the rig:

Code:
-X

so that from a terminal I would enter:

Code:
ssh m1@rigipaddress -X

so for a rig with an ip of 192.168.1.22. I would use:

Code:
ssh m1@192.168.1.22 -X

If you are using putty with windows the -X will not work; but after showing the error 3 or 4  times the screen should still connect to the mining process


I also tested using the -dmS argument when calling screen.

if you call screen with:
Code:
screen -dmS rig1

screen with start as a background process (so you can disconnect your ssh session and the miner will continue to mine on its own)

after starting screen you will need to connect to the screen (as it is running in the background)

you do this by entering:

Code:
screen -r

you can close the ssh session whenever and then reSSH in and enter:

Code:
screen -r

to return to the mining process whenever desired.



Tested with -X on the SSH command from my linux VM and now I am getting the below;

ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUGraphicsClockOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUMemoryTransferRateOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUGraphicsClockOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUMemoryTransferRateOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment '[gpu:0]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:0' (No targets match target
       specification), specified in assignment '[fan:0]/GPUTargetFanSpeed=65'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment '[gpu:1]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:1' (No targets match target
       specification), specified in assignment '[fan:1]/GPUTargetFanSpeed=65'.


When you enter:

Code:
lspci | grep VGA

what is the output?


lbrasi
Newbie
*
Offline Offline

Activity: 25


View Profile
June 23, 2017, 01:16:17 AM
 #1076

Hi All,

I have decided to give nvOC a go, I am using an older Gigabyte GA-EX58-UD4P (Socket LGA1366) board and an i7 920 CPU.  I have two EVGA GTX 1080ti on risers in both PCI-E x16 slots.
I am able to boot fine but things just are not stable, I can run for about 15 hours but later GPU0 drops to about 150-170w and SOLs are suffering, I can reboot and get it working but it seems to happen again.

I have no idea if any BIOS changes need to be made to get this running stable or not, does anyone have any idea what I can try?

Also if I kill the mining process and make changes to the onebash config it doesn't seem to accept the changes, more specially the fan speed.  I attempt to launch the miner again and get the below error but the process does start;


Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.


When the mining process starts do you see OC messages like this?

https://ip.bitcointalk.org/?u=https%3A%2F%2Fs13.postimg.org%2Fq08huqnyv%2FIMG_0270.jpg%26t%3D577%26c%3Dyq0szP4ICxv47w&t=577&c=q_tbQ_FaeljjYQ

If you don't:

At any point did you boot with the monitor connected to the motherboard?

Did you at anytime boot with only one GPU attached?

If either of these is the case: ensure the monitor is attached to the primary GPU ( the one connected to the 16x slot closest to the CPU )

then follow this process:

https://bitcointalk.org/index.php?topic=1854250.msg19449945#msg19449945

I only booted with a monitor connected the first time, now it is completely headless.
Yes I might have booted with only one GPU attached at one point.
Thank you I will follow that process.  If I re-image the USB key and boot completely headless should I be seeing the OC messages via SSH as well?

With v0015 you would have to enable openssh server before you could SSH in.  You would also have to be able to know the rigs IP; which can be done in several ways.

With v0016 you can enable openssh server in oneBash, I would recommending trying it and seeing if it solves the problem.

I have now also tested with v0016, the mining process starts fine with all OC settings when viewing directly on the rig.  However I am still getting the error via SSH when trying to start.

Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.

What type of OS is the client computer; and what are the IPs of the client and rig?

Windows 10 running putty, Rig: 192.168.1.19 Client: 192.168.1.6

So when you enter

m1@192.168.1.19   

using port 22

with SSH selected

and click open it gives you the error above?


Sorry let me try and clear some things up.  I am able to SSH into the rig just fine but when executing the miner via ssh, this is when i am getting the error, so clock settings, fan speed and power limits are not being set but the mining process still starts.

I also really appreciate your support and rapid response.

Are you killing the existing mining process before launching another via SSH?

Also; when you are launching the mining process via SSH are you using the cmd:

Code:
bash '/media/m1/1263-A96E/oneBash'

I am SSHing and executing the following commands.

ps aux | grep gnome-terminal - finding the gnome-server PID
kill PID
screen -S rig1
bash '/media/m1/1263-A96E/oneBash'

Just before the miner starts I see the above error in place of the "attribute" commands but the mining process still starts.


after you have SSHed in enter the cmd:

Code:
echo $DISPLAY

and tell me what it outputs

That does not output anything at all.

Ok, tomorrow I will try to replicate this error; and see if I can figure out what is happening.  My guess is X11 is having a problem trying to output graphically.

If you have a linux computer other than your rig and you SSH into the rig with it; does it have the same error?

I tested from a linux VM I have and the same result Sad


I tested this today and found there is a new problem resulting (most likely) from my adding support for up to 14 GPUs. 

I found that if I waited the screen would still connect to the mining process after failing to connect 3 or 4 times.

If you are using linux
you can resolve this error by adding the following argument when SSHing into the rig:

Code:
-X

so that from a terminal I would enter:

Code:
ssh m1@rigipaddress -X

so for a rig with an ip of 192.168.1.22. I would use:

Code:
ssh m1@192.168.1.22 -X

If you are using putty with windows the -X will not work; but after showing the error 3 or 4  times the screen should still connect to the mining process


I also tested using the -dmS argument when calling screen.

if you call screen with:
Code:
screen -dmS rig1

screen with start as a background process (so you can disconnect your ssh session and the miner will continue to mine on its own)

after starting screen you will need to connect to the screen (as it is running in the background)

you do this by entering:

Code:
screen -r

you can close the ssh session whenever and then reSSH in and enter:

Code:
screen -r

to return to the mining process whenever desired.



Tested with -X on the SSH command from my linux VM and now I am getting the below;

ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUGraphicsClockOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUMemoryTransferRateOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUGraphicsClockOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUMemoryTransferRateOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment '[gpu:0]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:0' (No targets match target
       specification), specified in assignment '[fan:0]/GPUTargetFanSpeed=65'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment '[gpu:1]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:1' (No targets match target
       specification), specified in assignment '[fan:1]/GPUTargetFanSpeed=65'.


When you enter:

Code:
lspci | grep VGA

what is the output?


01:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1)
jamesb16
Newbie
*
Offline Offline

Activity: 1


View Profile
June 23, 2017, 01:16:50 AM
 #1077

im trying to get this to work but i am struggling big time


im using asus prime z270 - a

i have bios set as instructed

i boot up using the img as instructed then nothing happens


cant get any image on the monitor tried hdmi & dvi


fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 01:24:15 AM
 #1078

im trying to get this to work but i am struggling big time


im using asus prime z270 - a

i have bios set as instructed

i boot up using the img as instructed then nothing happens


cant get any image on the monitor tried hdmi & dvi

just responded to your pm

Also have you been connecting the monitor directly to the motherboard or to the primary GPU?

fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 01:33:47 AM
 #1079

Hi All,

I have decided to give nvOC a go, I am using an older Gigabyte GA-EX58-UD4P (Socket LGA1366) board and an i7 920 CPU.  I have two EVGA GTX 1080ti on risers in both PCI-E x16 slots.
I am able to boot fine but things just are not stable, I can run for about 15 hours but later GPU0 drops to about 150-170w and SOLs are suffering, I can reboot and get it working but it seems to happen again.

I have no idea if any BIOS changes need to be made to get this running stable or not, does anyone have any idea what I can try?

Also if I kill the mining process and make changes to the onebash config it doesn't seem to accept the changes, more specially the fan speed.  I attempt to launch the miner again and get the below error but the process does start;


Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.


When the mining process starts do you see OC messages like this?

https://ip.bitcointalk.org/?u=https%3A%2F%2Fs13.postimg.org%2Fq08huqnyv%2FIMG_0270.jpg%26t%3D577%26c%3Dyq0szP4ICxv47w&t=577&c=q_tbQ_FaeljjYQ

If you don't:

At any point did you boot with the monitor connected to the motherboard?

Did you at anytime boot with only one GPU attached?

If either of these is the case: ensure the monitor is attached to the primary GPU ( the one connected to the 16x slot closest to the CPU )

then follow this process:

https://bitcointalk.org/index.php?topic=1854250.msg19449945#msg19449945

I only booted with a monitor connected the first time, now it is completely headless.
Yes I might have booted with only one GPU attached at one point.
Thank you I will follow that process.  If I re-image the USB key and boot completely headless should I be seeing the OC messages via SSH as well?

With v0015 you would have to enable openssh server before you could SSH in.  You would also have to be able to know the rigs IP; which can be done in several ways.

With v0016 you can enable openssh server in oneBash, I would recommending trying it and seeing if it solves the problem.

I have now also tested with v0016, the mining process starts fine with all OC settings when viewing directly on the rig.  However I am still getting the error via SSH when trying to start.

Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.

What type of OS is the client computer; and what are the IPs of the client and rig?

Windows 10 running putty, Rig: 192.168.1.19 Client: 192.168.1.6

So when you enter

m1@192.168.1.19   

using port 22

with SSH selected

and click open it gives you the error above?


Sorry let me try and clear some things up.  I am able to SSH into the rig just fine but when executing the miner via ssh, this is when i am getting the error, so clock settings, fan speed and power limits are not being set but the mining process still starts.

I also really appreciate your support and rapid response.

Are you killing the existing mining process before launching another via SSH?

Also; when you are launching the mining process via SSH are you using the cmd:

Code:
bash '/media/m1/1263-A96E/oneBash'

I am SSHing and executing the following commands.

ps aux | grep gnome-terminal - finding the gnome-server PID
kill PID
screen -S rig1
bash '/media/m1/1263-A96E/oneBash'

Just before the miner starts I see the above error in place of the "attribute" commands but the mining process still starts.


after you have SSHed in enter the cmd:

Code:
echo $DISPLAY

and tell me what it outputs

That does not output anything at all.

Ok, tomorrow I will try to replicate this error; and see if I can figure out what is happening.  My guess is X11 is having a problem trying to output graphically.

If you have a linux computer other than your rig and you SSH into the rig with it; does it have the same error?

I tested from a linux VM I have and the same result Sad


I tested this today and found there is a new problem resulting (most likely) from my adding support for up to 14 GPUs. 

I found that if I waited the screen would still connect to the mining process after failing to connect 3 or 4 times.

If you are using linux
you can resolve this error by adding the following argument when SSHing into the rig:

Code:
-X

so that from a terminal I would enter:

Code:
ssh m1@rigipaddress -X

so for a rig with an ip of 192.168.1.22. I would use:

Code:
ssh m1@192.168.1.22 -X

If you are using putty with windows the -X will not work; but after showing the error 3 or 4  times the screen should still connect to the mining process


I also tested using the -dmS argument when calling screen.

if you call screen with:
Code:
screen -dmS rig1

screen with start as a background process (so you can disconnect your ssh session and the miner will continue to mine on its own)

after starting screen you will need to connect to the screen (as it is running in the background)

you do this by entering:

Code:
screen -r

you can close the ssh session whenever and then reSSH in and enter:

Code:
screen -r

to return to the mining process whenever desired.



Tested with -X on the SSH command from my linux VM and now I am getting the below;

ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUGraphicsClockOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment
       '[gpu:0]/GPUMemoryTransferRateOffset[3]=0'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUGraphicsClockOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment
       '[gpu:1]/GPUMemoryTransferRateOffset[3]=50'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:0' (No targets match target
       specification), specified in assignment '[gpu:0]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:0' (No targets match target
       specification), specified in assignment '[fan:0]/GPUTargetFanSpeed=65'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'gpu:1' (No targets match target
       specification), specified in assignment '[gpu:1]/GPUFanControlState=1'.


ERROR: Error querying enabled displays on GPU 0 (Missing Extension).


ERROR: Error querying connected displays on GPU 0 (Missing Extension).



ERROR: Error resolving target specification 'fan:1' (No targets match target
       specification), specified in assignment '[fan:1]/GPUTargetFanSpeed=65'.


When you enter:

Code:
lspci | grep VGA

what is the output?


01:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1)


try this

on the rig open a guake terminal and enter:

gksu gedit '/etc/X11/xorg.conf'

then select all and delete:

replace with this:

Code:
Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    Screen      1  "Screen1" 1920 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    Option         "Xinerama" "0"
EndSection

Section "Files"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "CLB  fit Headless"
    HorizSync       30.0 - 83.0
    VertRefresh     56.0 - 76.0
EndSection

Section "Monitor"
    Identifier     "Monitor1"
    VendorName     "Unknown"
    ModelName      "CLB  fit Headless"
    HorizSync       30.0 - 83.0
    VertRefresh     56.0 - 76.0
EndSection


Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 1080 Ti"
    BusID          "PCI:01:00:0"
EndSection

Section "Device"
    Identifier     "Device1"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 1080 Ti"
    BusID          "PCI:02:00:0"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "AllowEmptyInitialConfiguration" "True"
    Option         "Coolbits" "28"
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DFP-1"
    Option         "metamodes" "nvidia-auto-select +0+0"
    Option         "SLI" "Off"
    Option         "MultiGPU" "Off"
    Option         "BaseMosaic" "off"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Screen1"
    Device         "Device1"
    Monitor        "Monitor1"
    DefaultDepth    24
    Option         "AllowEmptyInitialConfiguration" "True"
    Option         "Coolbits" "28"
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DFP-1"
    Option         "metamodes" "nvidia-auto-select +0+0"
    Option         "SLI" "Off"
    Option         "MultiGPU" "Off"
    Option         "BaseMosaic" "off"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

save

then logout

login

see if this solved the problem.

Not sure if this will work; but its worth a try

fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
June 23, 2017, 01:38:04 AM
 #1080

Fullzero,

I want to say that ver16 is much more stable then ver15 and great work on it overall!

Thank you so much for putting this together, disseminating it it and all the hard work you do!

I've run nvOC 7 hours on a gtx1050ti x 7 rig, very stable, very easy, awesome project, thanks!

Glad its working as intended.   Smiley

Pages: « 1 ... 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 [54] 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 ... 287 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!