Bitcoin Forum
November 21, 2017, 08:12:09 PM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 ... 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 [82] 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 ... 287 »
  Print  
Author Topic: [OS] nvOC easy-to-use Linux Nvidia Mining v0019-1.4  (Read 327109 times)
fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
July 07, 2017, 07:01:54 PM
 #1621


I can't even get the bios to post. I swapped all the parts minus the mobo from bad rig to good rig, and they worked fine. so i'm guessing I have my answer

This may have been suggested before, but I'm lazy and don't like to read so I'll just toss this out there and take the heat later.    Cool

Have you tried adding power to an ATX4P connection on the motherboard (or molex 4 pin connector if it has one) to supply additional power to the PCIe connections?  Even if the risers are separately powered, sometimes the board can't produce enough juice to the PCIe slots to support GPUs in all 6 positions. 

I have a Gigabyte Z170x-UD5 TH mobo that had that bootloop issue that drove me nuts for a few days straight. It wouldn't run more than 5 cards when there are 6 slots (that was with all gtx 1070s) so I finally discovered the purpose of the ATX4P connector and added a SATA power cable to the board and now it boots fine with 5 1070s on risers and a 1080ti in the last onboard slot.

Sorry if this is redundant info.

Sometimes hardware is bad.  Having a redundant setup is very helpful in identifying trouble components as you have.

1511295129
Hero Member
*
Offline Offline

Posts: 1511295129

View Profile Personal Message (Offline)

Ignore
1511295129
Reply with quote  #2

1511295129
Report to moderator
1511295129
Hero Member
*
Offline Offline

Posts: 1511295129

View Profile Personal Message (Offline)

Ignore
1511295129
Reply with quote  #2

1511295129
Report to moderator
Join ICO Now Coinlancer is Disrupting the Freelance marketplace!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1511295129
Hero Member
*
Offline Offline

Posts: 1511295129

View Profile Personal Message (Offline)

Ignore
1511295129
Reply with quote  #2

1511295129
Report to moderator
1511295129
Hero Member
*
Offline Offline

Posts: 1511295129

View Profile Personal Message (Offline)

Ignore
1511295129
Reply with quote  #2

1511295129
Report to moderator
TheCoinMine
Jr. Member
*
Offline Offline

Activity: 40


View Profile
July 07, 2017, 07:15:48 PM
 #1622


I can't even get the bios to post. I swapped all the parts minus the mobo from bad rig to good rig, and they worked fine. so i'm guessing I have my answer

This may have been suggested before, but I'm lazy and don't like to read so I'll just toss this out there and take the heat later.    Cool

Have you tried adding power to an ATX4P connection on the motherboard (or molex 4 pin connector if it has one) to supply additional power to the PCIe connections?  Even if the risers are separately powered, sometimes the board can't produce enough juice to the PCIe slots to support GPUs in all 6 positions. 

I have a Gigabyte Z170x-UD5 TH mobo that had that bootloop issue that drove me nuts for a few days straight. It wouldn't run more than 5 cards when there are 6 slots (that was with all gtx 1070s) so I finally discovered the purpose of the ATX4P connector and added a SATA power cable to the board and now it boots fine with 5 1070s on risers and a 1080ti in the last onboard slot.

Sorry if this is redundant info.



I'm only running 3 cards so idk if that is super applicable but if nothing else works I'll see if that helps. Thanks for the suggestion
TenaciousJ
Full Member
***
Offline Offline

Activity: 123


View Profile
July 07, 2017, 07:30:33 PM
 #1623



Powerstates are weird in Linux; usually don't change if you issue the command to change them.  I suspect this particular driver disallows nvidia-settings control over them.

Also you will need to use a higher OC offset to match the results from windows; as the OC curve is different in linux.

It seems like the power state will not change if the miner is running since the value is locked while in use by the miner, probably because the xserver p-state setting is initialized when linux loads but the miner changes the p-state settings after that which resets the values overriding the o/s default settings. Once that's done, you have to stop the miner, reset the p-state in nvidia config panel  and then run the miner again... but the problem is that oneBash changes the setting back when it loads.  I've tested setting the cards individually to 'prefer max performance' in nvidia's control panel while miner is running.  P-state doesn't change.  As soon as the miner, is closed the p-state will go up to max settings as you'd expect.

One way I think you could avoid this is by not setting power states in oneBash, but rather set them manually in the nvidia xserver settings so they are in place before oneBash runs.  I'm sure there's a way to automate the P-state through nvidia xserver config file, but I'd have to dig around to find out how precisely to do it.  That also means you'd have to remove the p-state config from the oneBash options to avoid it being changed back by the miner script.

fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
July 07, 2017, 07:39:36 PM
 #1624



Powerstates are weird in Linux; usually don't change if you issue the command to change them.  I suspect this particular driver disallows nvidia-settings control over them.

Also you will need to use a higher OC offset to match the results from windows; as the OC curve is different in linux.

It seems like the power state will not change if the miner is running since the value is locked while in use by the miner, probably because the xserver p-state setting is initialized when linux loads but the miner changes the p-state settings after that which resets the values overriding the o/s default settings. Once that's done, you have to stop the miner, reset the p-state in nvidia config panel  and then run the miner again... but the problem is that oneBash changes the setting back when it loads.  I've tested setting the cards individually to 'prefer max performance' in nvidia's control panel while miner is running.  P-state doesn't change.  As soon as the miner, is closed the p-state will go up to max settings as you'd expect.

One way I think you could avoid this is by not setting power states in oneBash, but rather set them manually in the nvidia xserver settings so they are in place before oneBash runs.  I'm sure there's a way to automate the P-state through nvidia xserver config file, but I'd have to dig around to find out how precisely to do it.  That also means you'd have to remove the p-state config from the oneBash options to avoid it being changed back by the miner script.

In my tests different GPUs had different levels of responsiveness to the power state cmds.  Maybe the new driver will work better overall.

There is no explict power state implementation in oneBash; if it is changed it is due to internal workings of the nvidia api when either OC or powerlimit is applied.  It is possible that adding power state cmds before the launching of the mining client, but after the powerlimit and OC will work.



min3333r
Newbie
*
Offline Offline

Activity: 11


View Profile
July 07, 2017, 07:48:30 PM
 #1625

your xorg.conf is most likely the problem

I would try re imaging a USB; although depending on your version you might not need to: what version are you using?

Also I don't recommend using dd to image USBs or SSDs; use etcher for linux instead.

Tried using hdd raw copy on win machine and etcher - same issue.

Tried burning the image on SSD and USB - same.

Using v0017 and latest oneBash

Could anyone provide me their xorg.conf which works with fan speed changing?

Thanks
TenaciousJ
Full Member
***
Offline Offline

Activity: 123


View Profile
July 07, 2017, 08:46:56 PM
 #1626



Powerstates are weird in Linux; usually don't change if you issue the command to change them.  I suspect this particular driver disallows nvidia-settings control over them.

Also you will need to use a higher OC offset to match the results from windows; as the OC curve is different in linux.

It seems like the power state will not change if the miner is running since the value is locked while in use by the miner, probably because the xserver p-state setting is initialized when linux loads but the miner changes the p-state settings after that which resets the values overriding the o/s default settings. Once that's done, you have to stop the miner, reset the p-state in nvidia config panel  and then run the miner again... but the problem is that oneBash changes the setting back when it loads.  I've tested setting the cards individually to 'prefer max performance' in nvidia's control panel while miner is running.  P-state doesn't change.  As soon as the miner, is closed the p-state will go up to max settings as you'd expect.

One way I think you could avoid this is by not setting power states in oneBash, but rather set them manually in the nvidia xserver settings so they are in place before oneBash runs.  I'm sure there's a way to automate the P-state through nvidia xserver config file, but I'd have to dig around to find out how precisely to do it.  That also means you'd have to remove the p-state config from the oneBash options to avoid it being changed back by the miner script.

In my tests different GPUs had different levels of responsiveness to the power state cmds.  Maybe the new driver will work better overall.

There is no explict power state implementation in oneBash; if it is changed it is due to internal workings of the nvidia api when either OC or powerlimit is applied.  It is possible that adding power state cmds before the launching of the mining client, but after the powerlimit and OC will work.




How tricky would it be to use the nvidia-settings commands from within oneBash to run the following command recursively for each card that's detected when it runs, rather than setting a specific power limit or clock offset?

Enable PowerMizer (Prefer Maximum Performance)

nvidia-settings -a '[gpu:0]/GPUPowerMizerMode=1'


essentially this, but with more efficient code because I never was good at foreach loops and such...

if [ $POWERLIMIT == "NO" ]
then
sudo nvidia-settings -a '[gpu:0]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:1]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:2]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:3]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:4]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:5]/GPUPowerMizerMode=1'
fi


On a side note, I updated to the most recent nvidia open source drivers (v. 381) and now my 1080 ti is recognized, so that's a plus.  Before it just listed as 'graphical device'

lbrasi
Newbie
*
Offline Offline

Activity: 25


View Profile
July 08, 2017, 04:13:34 AM
 #1627

I have had a lot of requests for this; so here is a new oneBash and modded switch file which implement full integration of SALFTER_NICEHASH_PROFIT_SWITCHING

see the OP for links:

Replace your current oneBash with the new one.

extract switch and move it to the:
Code:
/home/m1

directory

(the one which opens when you click the Files icon on the left)

configure the following in oneBash

Code:
SALFTER_NICEHASH_PROFIT_SWITCHING="YES"

# LOCAL will attach the mining process to the guake terminal
# REMOTE will leave it unattached / ready for SSH
LOCALorREMOTE="LOCAL"       # LOCAL  or  REMOTE

CURRENCY=USD
POWER_COST=0.10
MINIMUM_PROFIT=0.0
# this is salfters BTC address:
PAYMENT_ADDRESS=1TipsGocnz2N5qgAm9f7JLrsMqkb3oXe2
WORKER_NAME=nv$IP_AS_WORKER

daggerhashimoto_POWERLIMIT_WATTS=125
__daggerhashimoto_CORE_OVERCLOCK=100
daggerhashimoto_MEMORY_OVERCLOCK=100
_______daggerhashimoto_FAN_SPEED=75

equihash_POWERLIMIT_WATTS=125
__equihash_CORE_OVERCLOCK=100
equihash_MEMORY_OVERCLOCK=100
_______equihash_FAN_SPEED=75

neoscrypt_POWERLIMIT_WATTS=125
__neoscrypt_CORE_OVERCLOCK=100
neoscrypt_MEMORY_OVERCLOCK=100
_______neoscrypt_FAN_SPEED=75

lyra2rev2_POWERLIMIT_WATTS=125
__lyra2rev2_CORE_OVERCLOCK=100
lyra2rev2_MEMORY_OVERCLOCK=100
_______lyra2rev2_FAN_SPEED=75

lbry_POWERLIMIT_WATTS=125
__lbry_CORE_OVERCLOCK=100
lbry_MEMORY_OVERCLOCK=100
_______lbry_FAN_SPEED=75

pascal_POWERLIMIT_WATTS=125
__pascal_CORE_OVERCLOCK=100
pascal_MEMORY_OVERCLOCK=100
_______pascal_FAN_SPEED=75

remember to thank salfter if you use this  Smiley



Thanks for implementing this, but for some odd reason I keep getting two instances of the miner screen running which causes the system to crash, I will do some more testing to try and figure out what is going on.

EDIT: Actually the kill code does not seem to work causing multiple miner screens, this is how the system is crashing.
TenaciousJ
Full Member
***
Offline Offline

Activity: 123


View Profile
July 08, 2017, 06:02:37 AM
 #1628

I finally got my system stable with 6x 1070's after swapping mobos to an asrock z270 (the ga-z170x-ud5 th mobo i had the cards on seemed stable, but froze up and then went back to the old bootloop scenario again) - I just have one glitch that keeps creeping up. 1 gpu is only being utilized at about 66% consistently, where the rest are at 99% - overall it drops my hashrate 100 sol/s.  Any clue what might cause that to happen?   I'm running v. 0017.



tempgoga
Newbie
*
Offline Offline

Activity: 29


View Profile
July 08, 2017, 08:12:14 AM
 #1629


i'm actually dreading the 0018 release... i've made so many changes to the 0017 oneBash and nvOC in general that moving them to the new version is gonna take forever.. currently working on automatic rig reboot in the event of a system freeze from the miner crashing.

@ tempgoga - I'm curious on your approach, this is next on my plate (without a remote power switch) the easiest  - that seems applicable to my setup - is to filter the Genoil watchdog script for a reduced hash rate threshold instead of just 'error' since that script filters stdout from Genoil so technically the info is there to capture already.
I haven't gotten around to this yet but I'm hoping this weekend.

Another idea that I'm thinking of is some sort of port-knocking from a remote machine - it could be enough since the rigs usually are responsive to ssh or local scripts after they "soft crash" with the video cards - but this won't help in case of a complete freeze.
Then you need a remote power cycle ability which is a whole different level of infra.

Cheers!


Sorry for the late response, right now i'm trying to initiate a system reboot in the event that Xorg service takes up 98% or more cpu for for 10 seconds or longer, which happens every time any miner crashes, Xorg always shoots up to 99-100% cpu and stays there, trying to use monit for this, will update if it works.

i like your port-knocking idea.
IAmNotAJeep
Jr. Member
*
Offline Offline

Activity: 44


View Profile
July 08, 2017, 12:34:21 PM
 #1630


i'm actually dreading the 0018 release... i've made so many changes to the 0017 oneBash and nvOC in general that moving them to the new version is gonna take forever.. currently working on automatic rig reboot in the event of a system freeze from the miner crashing.

@ tempgoga - I'm curious on your approach, [... snip]

Sorry for the late response, right now i'm trying to initiate a system reboot in the event that Xorg service takes up 98% or more cpu for for 10 seconds or longer, which happens every time any miner crashes, Xorg always shoots up to 99-100% cpu and stays there, trying to use monit for this, will update if it works.

i like your port-knocking idea.

Hey monit looks nice, I'm about half way there to taking the output of genoil and turing it into a hearbeat, then to remotely cycle the server, the hard part is how to define the conditions of when to trigger that event.
Nexillus
Full Member
***
Offline Offline

Activity: 132


View Profile
July 08, 2017, 01:27:27 PM
 #1631

Got my second right up and running, currently tuning the 1060's with Ethminer, genoil fork on VER0017 nvOC.

So far getting around 149MH/s total on 6 cards with these settings. Short term stability is their, long term us currently unknown.

Getting about 24.83MH/s on each.

5 cards:
PL: 125
Core: -100
Mem: 1700

1 card:
PL: 125
Core: -100
Mem: 1700

Anybody else have settings for their 1060s as I am curious to what others have gotten so far.

Besides stability for the OC, going to be stepping down the power to optimize it.

Bitrated user: nexillus.
IAmNotAJeep
Jr. Member
*
Offline Offline

Activity: 44


View Profile
July 08, 2017, 03:00:55 PM
 #1632

Got my second right up and running, currently tuning the 1060's with Ethminer, genoil fork on VER0017 nvOC.

So far getting around 149MH/s total on 6 cards with these settings. Short term stability is their, long term us currently unknown.

Getting about 24.83MH/s on each.

5 cards:
PL: 125
Core: -100
Mem: 1700

1 card:
PL: 125
Core: -100
Mem: 1700

Anybody else have settings for their 1060s as I am curious to what others have gotten so far.

Besides stability for the OC, going to be stepping down the power to optimize it.
Depending on the mobo I can get 25/26 MHs per 1060 under genoil with the watchdog script with -75 and +1450 with power limit at 82W.
With Claymore I was able to crank it up to 170/1700/82 and get another 1-2 MHs per card and it was relatively stable but when it did crash it did not recover without intervention  so for now I'm going with Genoil and finding different ways to deal with the remaining instability (which to me means a reboot ever 1-2 days at the moment).


Nexillus
Full Member
***
Offline Offline

Activity: 132


View Profile
July 08, 2017, 03:59:20 PM
 #1633

Got my second right up and running, currently tuning the 1060's with Ethminer, genoil fork on VER0017 nvOC.

So far getting around 149MH/s total on 6 cards with these settings. Short term stability is their, long term us currently unknown.

Getting about 24.83MH/s on each.

5 cards:
PL: 125
Core: -100
Mem: 1700

1 card:
PL: 125
Core: -100
Mem: 1700

Anybody else have settings for their 1060s as I am curious to what others have gotten so far.

Besides stability for the OC, going to be stepping down the power to optimize it.
Depending on the mobo I can get 25/26 MHs per 1060 under genoil with the watchdog script with -75 and +1450 with power limit at 82W.
With Claymore I was able to crank it up to 170/1700/82 and get another 1-2 MHs per card and it was relatively stable but when it did crash it did not recover without intervention  so for now I'm going with Genoil and finding different ways to deal with the remaining instability (which to me means a reboot ever 1-2 days at the moment).




That is awesome to hear! I am right below the 25MH/s. With your powerlimit you suggested, I haven't gotten to but it gave me a good baseline. I was not aware the 1060's are that power efficient even under a heavy OC. My 1070s I could barely go below 115 with such heavy OC on it.

Biggest reason I want Genoil is the no fee attached to the miner compared to claymore.

What watchdog script you talking about? Or did you make it?  

Bitrated user: nexillus.
achalmersman
Newbie
*
Offline Offline

Activity: 16


View Profile
July 08, 2017, 06:01:44 PM
 #1634

Claymore 9.7 is out and on my Windows10 machine I went from Dual Mining ETH: 94.5 MH/s and SC: 945 MH/s increased to ETH: 99 MH/s and SC: 990 MH/s.  No other changes and exactly the same OC.  We will see if it stays stable.  Possible addition to v18? 

I am just unable to keep my nvOC stable.  It keeps hanging and locking up the OS.  I am currently loading Windows 10 on my rig just to see if I have a hardware issue that I can't figure out.  If Windows 10 stays stable I don't know what I will do.  I REALLY like nvOC.  If I had another stick of ram laying around I would stick it in.  I will order one if I have to.
andreih
Newbie
*
Offline Offline

Activity: 7


View Profile
July 08, 2017, 07:01:06 PM
 #1635

I just copied the content of the claymore 9.7 into 9.5 folder and run the miner. So far I'm getting 190mh/s mining eth from 6 nvidia gtx 1070 as for the 9.5 - the version on the 0017 onebash I was getting 183mh/s. Thinking this is a must to be implemented as soon as possible. PS- sry for my english Wink)
tempgoga
Newbie
*
Offline Offline

Activity: 29


View Profile
July 08, 2017, 07:13:51 PM
 #1636

For everyone that have stability problems where the OS hangs or slows down to a halt, this is due to Xorg taking 100% of one of the cpu cores, as far as i can tell this usually only begins after some kind of soft crash from overclocking/undervolting, i've googled my ass off and found a few references of people claiming to have solved this problem by configuring Xorg to run from the integrated intel GPU and leaving the nvidia cards to pure computations, my linux know how is Very limited so this is more of a question for fullzero.

what do you think Fullzero?
fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
July 08, 2017, 07:17:54 PM
 #1637

your xorg.conf is most likely the problem

I would try re imaging a USB; although depending on your version you might not need to: what version are you using?

Also I don't recommend using dd to image USBs or SSDs; use etcher for linux instead.

Tried using hdd raw copy on win machine and etcher - same issue.

Tried burning the image on SSD and USB - same.

Using v0017 and latest oneBash

Could anyone provide me their xorg.conf which works with fan speed changing?

Thanks

v0017 should automatically restore your oneBash if it is altered.  There is a backup in the system already.

open the guake terminal and enter:

Code:
sudo cp '/etc/X11/xorg.conf.backup' '/etc/X11/xorg.conf'

then logout and login

andreih
Newbie
*
Offline Offline

Activity: 7


View Profile
July 08, 2017, 07:18:51 PM
 #1638

I put it to a ssd and runs very smooth
fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
July 08, 2017, 07:20:35 PM
 #1639



Powerstates are weird in Linux; usually don't change if you issue the command to change them.  I suspect this particular driver disallows nvidia-settings control over them.

Also you will need to use a higher OC offset to match the results from windows; as the OC curve is different in linux.

It seems like the power state will not change if the miner is running since the value is locked while in use by the miner, probably because the xserver p-state setting is initialized when linux loads but the miner changes the p-state settings after that which resets the values overriding the o/s default settings. Once that's done, you have to stop the miner, reset the p-state in nvidia config panel  and then run the miner again... but the problem is that oneBash changes the setting back when it loads.  I've tested setting the cards individually to 'prefer max performance' in nvidia's control panel while miner is running.  P-state doesn't change.  As soon as the miner, is closed the p-state will go up to max settings as you'd expect.

One way I think you could avoid this is by not setting power states in oneBash, but rather set them manually in the nvidia xserver settings so they are in place before oneBash runs.  I'm sure there's a way to automate the P-state through nvidia xserver config file, but I'd have to dig around to find out how precisely to do it.  That also means you'd have to remove the p-state config from the oneBash options to avoid it being changed back by the miner script.

In my tests different GPUs had different levels of responsiveness to the power state cmds.  Maybe the new driver will work better overall.

There is no explict power state implementation in oneBash; if it is changed it is due to internal workings of the nvidia api when either OC or powerlimit is applied.  It is possible that adding power state cmds before the launching of the mining client, but after the powerlimit and OC will work.




How tricky would it be to use the nvidia-settings commands from within oneBash to run the following command recursively for each card that's detected when it runs, rather than setting a specific power limit or clock offset?

Enable PowerMizer (Prefer Maximum Performance)

nvidia-settings -a '[gpu:0]/GPUPowerMizerMode=1'


essentially this, but with more efficient code because I never was good at foreach loops and such...

if [ $POWERLIMIT == "NO" ]
then
sudo nvidia-settings -a '[gpu:0]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:1]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:2]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:3]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:4]/GPUPowerMizerMode=1'
sudo nvidia-settings -a '[gpu:5]/GPUPowerMizerMode=1'
fi


On a side note, I updated to the most recent nvidia open source drivers (v. 381) and now my 1080 ti is recognized, so that's a plus.  Before it just listed as 'graphical device'

This is what I was alluding to before.

Have you tried this placing it: after the powerlimit and OC, but before launching the mining client?

fullzero
Legendary
*
Offline Offline

Activity: 1092



View Profile
July 08, 2017, 07:23:25 PM
 #1640

Got my second right up and running, currently tuning the 1060's with Ethminer, genoil fork on VER0017 nvOC.

So far getting around 149MH/s total on 6 cards with these settings. Short term stability is their, long term us currently unknown.

Getting about 24.83MH/s on each.

5 cards:
PL: 125
Core: -100
Mem: 1700

1 card:
PL: 125
Core: -100
Mem: 1700

Anybody else have settings for their 1060s as I am curious to what others have gotten so far.

Besides stability for the OC, going to be stepping down the power to optimize it.

these hashrates are better than the ones I am currently getting with my 1060 rigs; I will have to try these settings.  How long have they been stable?

Pages: « 1 ... 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 [82] 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 ... 287 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!