Bitcoin Forum
May 02, 2024, 06:07:19 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 ... 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 [294] 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 ... 1240 »
  Print  
Author Topic: CCminer(SP-MOD) Modded GPU kernels.  (Read 2347498 times)
bathrobehero
Legendary
*
Offline Offline

Activity: 2002
Merit: 1051


ICO? Not even once.


View Profile
September 14, 2015, 10:29:24 PM
 #5861

Are there more details about this alleged ctrl+c issue?  It's not clear to me what you're doing, let alone what you expect.

Ctrl+C doesn't do anything but signal some loops to break in the control logic.  Which loops depends on if it's the first or second time you've pressed it.  It doesn't actually touch the GPUs at all.

I think you're either observing the intended behavior or a driver bug.

If you press ctrl+c once, then see GPUs drop to power-saving, that's expected behavior.  You've told the GPU worker threads to stop after they're done with current work.  So as the threads complete, you will see GPUs drop to low power modes, since they have nothing to do.  App is just waiting for the rest of the threads to finish before exiting.

If you're running ccminer, killing it with ctrl+c, then starting it again and seeing low performance until a reboot.  That sounds like a driver bug to me.

Do you or sp_ (or anyone else) know what method is the fastest and most reliable way to exit ccminer without:
waiting for the threads to finish and
without ccminer crashing?

Whatever I tried in my fork it either hangs until all the threads have their work finished which takes somewhere between a few seconds to a minute (probably up to diff) or ccminer will crash which would be fine except it sometimes crashes the driver and puts a card or two in different P-state (405 mhz).

Not your keys, not your coins!
1714673239
Hero Member
*
Offline Offline

Posts: 1714673239

View Profile Personal Message (Offline)

Ignore
1714673239
Reply with quote  #2

1714673239
Report to moderator
1714673239
Hero Member
*
Offline Offline

Posts: 1714673239

View Profile Personal Message (Offline)

Ignore
1714673239
Reply with quote  #2

1714673239
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
sp_ (OP)
Legendary
*
Offline Offline

Activity: 2898
Merit: 1087

Team Black developer


View Profile
September 14, 2015, 10:44:11 PM
 #5862

Are there more details about this alleged ctrl+c issue?  It's not clear to me what you're doing, let alone what you expect.
Ctrl+C doesn't do anything but signal some loops to break in the control logic.  Which loops depends on if it's the first or second time you've pressed it.  It doesn't actually touch the GPUs at all.
I think you're either observing the intended behavior or a driver bug.
If you press ctrl+c once, then see GPUs drop to power-saving, that's expected behavior.  You've told the GPU worker threads to stop after they're done with current work.  So as the threads complete, you will see GPUs drop to low power modes, since they have nothing to do.  App is just waiting for the rest of the threads to finish before exiting.
If you're running ccminer, killing it with ctrl+c, then starting it again and seeing low performance until a reboot.  That sounds like a driver bug to me.
Do you or sp_ (or anyone else) know what method is the fastest and most reliable way to exit ccminer without:
waiting for the threads to finish and
without ccminer crashing?
Whatever I tried in my fork it either hangs until all the threads have their work finished which takes somewhere between a few seconds to a minute (probably up to diff) or ccminer will crash which would be fine except it sometimes crashes the driver and puts a card or two in different P-state (405 mhz).

The old ccminer version slept for 10 seconds before they died. After the T nelson fix, they will quit faster.
In order to boost your profit, you need a good miner control software. I can write good miner control software on windows, but not on linux. The last distro I tested was Red Hat in the 90'ties. Had a botnet of university sun sparc stations though. Long time ago... We where mining passwords.

Team Black Miner (ETHB3 ETH ETC VTC KAWPOW FIROPOW MEOWPOW + dual mining + tripple mining.. https://github.com/sp-hash/TeamBlackMiner
joblo
Legendary
*
Offline Offline

Activity: 1470
Merit: 1114


View Profile
September 14, 2015, 10:53:47 PM
 #5863

Are there more details about this alleged ctrl+c issue?  It's not clear to me what you're doing, let alone what you expect.

Ctrl+C doesn't do anything but signal some loops to break in the control logic.  Which loops depends on if it's the first or second time you've pressed it.  It doesn't actually touch the GPUs at all.

I think you're either observing the intended behavior or a driver bug.

If you press ctrl+c once, then see GPUs drop to power-saving, that's expected behavior.  You've told the GPU worker threads to stop after they're done with current work.  So as the threads complete, you will see GPUs drop to low power modes, since they have nothing to do.  App is just waiting for the rest of the threads to finish before exiting.

If you're running ccminer, killing it with ctrl+c, then starting it again and seeing low performance until a reboot.  That sounds like a driver bug to me.

Agreed on the driver issue. The following summary illustrates my experience and I believe the experience
of others. I'm sure I'll be corrected if not.

There are a few scenarios where the GPU will get stuck at low performance. It can happen if the GPU crashes
such as an out of memory error on launch. It can also happen if ccminer is killed with a Ctrl-C. Windows will
also pop up an error window when this occurs. I have seen the degradation on Linux although I'm not sure if
it was ever associated with Ctrl-C.

This seems to be a fairly new problem introduced sometime in the ccminer 1.5 branch (both tpruvot and sp_).
There was some work done around proper_exit when I first started noticing the problem. It's possible that is
just a coincidence and a driver update around the same time could have introduced it.

Ctrl-C is expected to kill the shell process without any side effects such as Windows errors or degraded HW.

For some reason the problem does not seem to occur if the process is killed from another shell
but I can't say it's never occurred. I don't know if any of the recent changes has resolved the Ctrl-C aspect
of the problem because I exclusively use taskkill/pkill.

It's clear that something is messing up the GPU under certain task termination situations. A messy recovery can be
forgiven when the GPU crashes but controlled terminations should be clean.


AKA JayDDee, cpuminer-opt developer. https://github.com/JayDDee/cpuminer-opt
https://bitcointalk.org/index.php?topic=5226770.msg53865575#msg53865575
BTC: 12tdvfF7KmAsihBXQXynT6E6th2c2pByTT,
antonio8
Legendary
*
Offline Offline

Activity: 1386
Merit: 1000


View Profile
September 15, 2015, 01:45:29 AM
 #5864

I am still getting the

JSON call failed Invalid parameter
submit_upstream_work json_rpc_call failed

while trying to solo mine.

No issues solo mining with any other dev's miner.

Using the exact same bat on all dev's miner and same wallet.

If you are going to leave your BTC on an exchange please send it to this address instead 1GH3ub3UUHbU5qDJW5u3E9jZ96ZEmzaXtG, I will at least use the money better than someone who steals it from the exchange. Thanks Wink
flipclip
Member
**
Offline Offline

Activity: 111
Merit: 10


View Profile
September 15, 2015, 03:15:08 AM
 #5865

Are there more details about this alleged ctrl+c issue?  It's not clear to me what you're doing, let alone what you expect.

Ctrl+C doesn't do anything but signal some loops to break in the control logic.  Which loops depends on if it's the first or second time you've pressed it.  It doesn't actually touch the GPUs at all.

I think you're either observing the intended behavior or a driver bug.

If you press ctrl+c once, then see GPUs drop to power-saving, that's expected behavior.  You've told the GPU worker threads to stop after they're done with current work.  So as the threads complete, you will see GPUs drop to low power modes, since they have nothing to do.  App is just waiting for the rest of the threads to finish before exiting.

If you're running ccminer, killing it with ctrl+c, then starting it again and seeing low performance until a reboot.  That sounds like a driver bug to me.

Agreed on the driver issue. The following summary illustrates my experience and I believe the experience
of others. I'm sure I'll be corrected if not.

There are a few scenarios where the GPU will get stuck at low performance. It can happen if the GPU crashes
such as an out of memory error on launch. It can also happen if ccminer is killed with a Ctrl-C. Windows will
also pop up an error window when this occurs. I have seen the degradation on Linux although I'm not sure if
it was ever associated with Ctrl-C.

This seems to be a fairly new problem introduced sometime in the ccminer 1.5 branch (both tpruvot and sp_).
There was some work done around proper_exit when I first started noticing the problem. It's possible that is
just a coincidence and a driver update around the same time could have introduced it.

Ctrl-C is expected to kill the shell process without any side effects such as Windows errors or degraded HW.

For some reason the problem does not seem to occur if the process is killed from another shell
but I can't say it's never occurred. I don't know if any of the recent changes has resolved the Ctrl-C aspect
of the problem because I exclusively use taskkill/pkill.

It's clear that something is messing up the GPU under certain task termination situations. A messy recovery can be
forgiven when the GPU crashes but controlled terminations should be clean.

I'm leaning towards it being a driver issue (linux).  The issue is/was that when closing ccminer with ctrl-c after a few times, one of my cards (I have two different makes of 750Ti's in my box) would hash at a lower rate then normal.  This never happened with my bash auto-switcher script, but that uses "kill" to close ccminer.  (I only noticed the issue when testing changes to my bash auto-switcher, because I would use the ctrl-c command to close ccminer.)  I updated my driver to 355.06 this weekend and then tonight did a bunch of git checkouts / compiles going back to 1.5.50 , then skipping to 1.5.55 then skipping to 1.5.61 and then going up to present.  I would start ccminer, let it hash for a few minutes and then use ctrl-c to close it down.  I would get "Segmentation fault (core dumped)" using the .50 and .55 releases (and probably some of the .6x releases, though not after the nelson-t changes were committed), and with setting the intensity too high I would get "illegal memory" errors.  However, even after all these errors I never encountered slower hashing on the restart of ccminer.  I want to say I was using NVIDIA driver 340 (from ppa "xorg-edgers fresh X crack") before this weekend.  I guess the next step would be to downgrade my drivers and try testing ccminer again... but it is late.  For linux people having the "ctrl-c slower hashing" issue, what version drivers are you running?
sambiohazard
Hero Member
*****
Offline Offline

Activity: 840
Merit: 1000



View Profile
September 15, 2015, 04:06:53 AM
 #5866

@sp_ please fix the choppiness of GPU power/usage on your miner. I get 21.1 Mhash/s on your miner release 66 but this is what i get on nicehash.



I get 19.8 Mhash/s on djm 0.4 and on nicehash i get



I have noticed that djm34 ccminer has more stable Gpu usage & temps.

Everything is same in above two runs in terms of h/w & other s/w running, cpu usage etc. My gpus fluctuate between 0-100% usage every 2-5 secs when using your miner, i dont think its good for them so i am currently using djm's miner. Please look into this. you asked me to set GPU performance state to P0 using NV Inspector. I tried it but it wont change. My 970 runs P2 state all the time while mining. Also the problem is in the miner as djm's miner works fine for me with P2 state but your miner gets all choppy.

I am not sure if i am the only one having this problem.  Sad
impulse2000
Newbie
*
Offline Offline

Activity: 26
Merit: 0


View Profile
September 15, 2015, 05:29:21 AM
 #5867

All systems have fresh Win 8.1 installations.
I am running the 355.60 Driver on all rigs and i am using the 61 build due to best performance for me with that one.

Can you try build 67 please.

Do you have a cudart32_65.dll in the ccminer folder?

Downgrading the driver could increase by 200-300KHASH per 750ti. (Hot tip Smiley)

Don't forget to donate guys. Smiley
Downgrade to what? Now 355.60.
sp_ (OP)
Legendary
*
Offline Offline

Activity: 2898
Merit: 1087

Team Black developer


View Profile
September 15, 2015, 06:36:39 AM
 #5868

@sp_ please fix the choppiness of GPU power/usage on your miner. I get 21.1 Mhash/s on your miner release 66 but this is what i get on nicehash.

Is this the lyra2 algo? Release 67 has an improved performance on the 970.

Team Black Miner (ETHB3 ETH ETC VTC KAWPOW FIROPOW MEOWPOW + dual mining + tripple mining.. https://github.com/sp-hash/TeamBlackMiner
sambiohazard
Hero Member
*****
Offline Offline

Activity: 840
Merit: 1000



View Profile
September 15, 2015, 07:55:10 AM
 #5869

@sp_ please fix the choppiness of GPU power/usage on your miner. I get 21.1 Mhash/s on your miner release 66 but this is what i get on nicehash.

Is this the lyra2 algo? Release 67 has an improved performance on the 970.


EDIT:
I haven't tried all algos but on lyra2v2 & quark its very choppy. I am assuming it will be so on other algos as well. Also in miner hashrate is obviously better for your miner but its obviously not stable enough, so its not about max hashrate but stability of that maximum. I dont think improved performance will solve choppiness but increase it as usage bumps into my temp limits. Please check out these 2 posts to see the difference between your & djm34's miner. I don't know what he is doing but its very stable for me.

https://bitcointalk.org/index.php?topic=826901.msg12111544#msg12111544 (your miner)

https://bitcointalk.org/index.php?topic=826901.msg12111898#msg12111898 (djm34 miner)
fenomenhaa
Sr. Member
****
Offline Offline

Activity: 248
Merit: 250



View Profile
September 15, 2015, 07:58:58 AM
 #5870

@sp_ please fix the choppiness of GPU power/usage on your miner. I get 21.1 Mhash/s on your miner release 66 but this is what i get on nicehash.

Is this the lyra2 algo? Release 67 has an improved performance on the 970.


I haven't tried all algos but on lyra2v2 & quark its very choppy. I am assuming it will be so on other algos as well.
At least 1000mh/s lost in Relase 67(970)


           ▄▀▀▀▄
   ▄▀▀▀▄   █   █   ▄▀▀▀▄
   █   █    ▀█▀    █   █
    ▀▀▀▀▄    ▀    ▄▀▀▀▀
DE]   ██▀▀▀█▄   ▀▀█   █
 ▀▀▀      ██▄▄▄█▀      ▀▀▀
        ▄   ▀ ▀   ▄
   ▄▀▀▀█     █     █▀▀▀▄
   █   █   ▄▀▀▀▄   █   █
    ▀▀▀    █   █    ▀▀▀
            ▀▀▀
          ▄▄█▄█▄[/color]
▄▀▀▀▄     ██   ██     ▄▀▀▀▄
█   █▀▀[color=#2C97
██████
██████
██████
██████
██████  ██████
██████  ██████
██████  ██████
██████  ██████  ██████
██████  ██████  ██████
██████  ██████  ██████
██████  ██████  ██████
██████  ██████  ██████
██████  ██████  ██████
✓  SUPER FAST TRANSACTION
✓  USER-FRIENDLY INTERFACE
✓  FAST & EASY REGISTRATION
▄██████
███▀▀▀▀
███
███
███
███
███
███
███
███
███
███▄▄▄▄
▀██████
JOIN AFFILIATE PROGRAM
UP TO 50% COMMISSIONS
██████▄
▀▀▀▀███
███
███
███
███
███
███
███
███
███
▄▄▄▄███
██████▀
pallas
Legendary
*
Offline Offline

Activity: 2716
Merit: 1094


Black Belt Developer


View Profile
September 15, 2015, 07:59:42 AM
 #5871

Why do you need to break with ctrl-c ? Seems to work fine in windows.


because there is a new commit in git and I wonna compile and run it :-)

pallas
Legendary
*
Offline Offline

Activity: 2716
Merit: 1094


Black Belt Developer


View Profile
September 15, 2015, 08:02:09 AM
 #5872

Are there more details about this alleged ctrl+c issue?  It's not clear to me what you're doing, let alone what you expect.

I think you're either observing the intended behavior or a driver bug.

If you press ctrl+c once, then see GPUs drop to power-saving, that's expected behavior.  You've told the GPU worker threads to stop after they're done with current work.  So as the threads complete, you will see GPUs drop to low power modes, since they have nothing to do.  App is just waiting for the rest of the threads to finish before exiting.

If you're running ccminer, killing it with ctrl+c, then starting it again and seeing low performance until a reboot.  That sounds like a driver bug to me.

I'm just expecting that it exits and that if I run it again afterwords, it works :-)
I always hit ctrl-c only once and wait for it to exit.
I perfectly understand gpu power saving mode: it should not stay while it's mining!

pallas
Legendary
*
Offline Offline

Activity: 2716
Merit: 1094


Black Belt Developer


View Profile
September 15, 2015, 08:06:50 AM
 #5873

I'm leaning towards it being a driver issue (linux).  The issue is/was that when closing ccminer with ctrl-c after a few times, one of my cards (I have two different makes of 750Ti's in my box) would hash at a lower rate then normal.  This never happened with my bash auto-switcher script, but that uses "kill" to close ccminer.  (I only noticed the issue when testing changes to my bash auto-switcher, because I would use the ctrl-c command to close ccminer.)  I updated my driver to 355.06 this weekend and then tonight did a bunch of git checkouts / compiles going back to 1.5.50 , then skipping to 1.5.55 then skipping to 1.5.61 and then going up to present.  I would start ccminer, let it hash for a few minutes and then use ctrl-c to close it down.  I would get "Segmentation fault (core dumped)" using the .50 and .55 releases (and probably some of the .6x releases, though not after the nelson-t changes were committed), and with setting the intensity too high I would get "illegal memory" errors.  However, even after all these errors I never encountered slower hashing on the restart of ccminer.  I want to say I was using NVIDIA driver 340 (from ppa "xorg-edgers fresh X crack") before this weekend.  I guess the next step would be to downgrade my drivers and try testing ccminer again... but it is late.  For linux people having the "ctrl-c slower hashing" issue, what version drivers are you running?

I've always been running 352.21.

sp_ (OP)
Legendary
*
Offline Offline

Activity: 2898
Merit: 1087

Team Black developer


View Profile
September 15, 2015, 09:19:34 AM
 #5874

970,980 user should use this build. I checked in a hashrate fix after build 67 was published

http://cryptomining-blog.com/5730-updated-windows-binary-of-the-ccminer-1-5-67-git-fork-by-sp-for-maxwell/

Team Black Miner (ETHB3 ETH ETC VTC KAWPOW FIROPOW MEOWPOW + dual mining + tripple mining.. https://github.com/sp-hash/TeamBlackMiner
pallas
Legendary
*
Offline Offline

Activity: 2716
Merit: 1094


Black Belt Developer


View Profile
September 15, 2015, 10:52:36 AM
 #5875

I managed to reproduce the performance state bug and that's how I did it.
Since it only happened recently, I thought of what changes I did to the system.
The only one was: creating a xorg.conf for all the cards in order to be able to change the fan speeds using the cool_cpu2.sh script.
That means it starts an X server at boot, while it didn't before.
If you leave the X server running, no problems.
If you stop it by running "sudo service lightdm stop", the hashrate bug starts to happen after the following ccminer ctrl-c.
If you start lightdm again, THE ISSUES IS GONE, without the need to reboot :-)

sp_ (OP)
Legendary
*
Offline Offline

Activity: 2898
Merit: 1087

Team Black developer


View Profile
September 15, 2015, 11:08:13 AM
 #5876

@sp_ please fix the choppiness of GPU power/usage on your miner. I get 21.1 Mhash/s on your miner release 66 but this is what i get on nicehash.
Is this the lyra2 algo? Release 67 has an improved performance on the 970.
EDIT:
I haven't tried all algos but on lyra2v2 & quark its very choppy. I am assuming it will be so on other algos as well. Also in miner hashrate is obviously better for your miner but its obviously not stable enough, so its not about max hashrate but stability of that maximum. I dont think improved performance will solve choppiness but increase it as usage bumps into my temp limits. Please check out these 2 posts to see the difference between your & djm34's miner. I don't know what he is doing but its very stable for me.
https://bitcointalk.org/index.php?topic=826901.msg12111544#msg12111544 (your miner)
https://bitcointalk.org/index.php?topic=826901.msg12111898#msg12111898 (djm34 miner)

Your card is trottle'ing. You can try to reduce the intensity. F.ex -X 16 My default intensities are high.

Start gpuz and check why your card is trottle'ing. there is a graph that explains it for you. The graph have different colors for different symptoms POW, or TMP etc. If you get Vop like I did here in this picture, you need to overclock the card.


Team Black Miner (ETHB3 ETH ETC VTC KAWPOW FIROPOW MEOWPOW + dual mining + tripple mining.. https://github.com/sp-hash/TeamBlackMiner
sambiohazard
Hero Member
*****
Offline Offline

Activity: 840
Merit: 1000



View Profile
September 15, 2015, 11:27:46 AM
 #5877

@sp_ please fix the choppiness of GPU power/usage on your miner. I get 21.1 Mhash/s on your miner release 66 but this is what i get on nicehash.
Is this the lyra2 algo? Release 67 has an improved performance on the 970.
EDIT:
I haven't tried all algos but on lyra2v2 & quark its very choppy. I am assuming it will be so on other algos as well. Also in miner hashrate is obviously better for your miner but its obviously not stable enough, so its not about max hashrate but stability of that maximum. I dont think improved performance will solve choppiness but increase it as usage bumps into my temp limits. Please check out these 2 posts to see the difference between your & djm34's miner. I don't know what he is doing but its very stable for me.
https://bitcointalk.org/index.php?topic=826901.msg12111544#msg12111544 (your miner)
https://bitcointalk.org/index.php?topic=826901.msg12111898#msg12111898 (djm34 miner)

Your card is trottle'ing. You can try to reduce the intensity. F.ex -X 16 My default intensities are high.

Start gpuz and check why your card is trottle'ing. there is a graph that explains it for you. The graph have different colors for different symptoms POW, or TMP etc. If you get Vop like I did here in this picture, you need to overclock the card.

https://ip.bitcointalk.org/?u=http%3A%2F%2Fi58.tinypic.com%2Ff4kaxj.png&t=556&c=iENngrR06DvFLg

I am aware that my card is throttling as i mention i have a temp limit of 77C. My question is  why it throttles so much more on your build as compared to djm34's build? Its the amount of throttling & its frequency on your build that worries me. I sure want to get higher performance but stability as well.

My perfcap reason is mostly thermal limit that i have chosen & power. But i guess even if i allow power upto 110% it will throttle due to temp limit.

I guess i will stick with djm 0.4 until he releases next version.

@djm34 can you release a new version with latest optimizations?
pallas
Legendary
*
Offline Offline

Activity: 2716
Merit: 1094


Black Belt Developer


View Profile
September 15, 2015, 11:32:43 AM
 #5878

more performance = (in most cases) more power usage = more throttling

sp_ (OP)
Legendary
*
Offline Offline

Activity: 2898
Merit: 1087

Team Black developer


View Profile
September 15, 2015, 11:36:12 AM
 #5879

I am aware that my card is throttling as i mention i have a temp limit of 77C. My question is  why it throttles so much more on your build as compared to djm34's build? Its the amount of throttling & its frequency on your build that worries me. I sure want to get higher performance but stability as well.
My perfcap reason is mostly thermal limit that i have chosen & power. But i guess even if i allow power upto 110% it will throttle due to temp limit.
I guess i will stick with djm 0.4 until he releases next version.
@djm34 can you release a new version with latest optimizations?

I told you. I use high default intensities. (the amount of work passed from gpumem to the gpu-core)

My builds use more memory and power.. you can easily avoid this by reducing the intensity levels yourself.

With the -i switch or the -X switch.

Team Black Miner (ETHB3 ETH ETC VTC KAWPOW FIROPOW MEOWPOW + dual mining + tripple mining.. https://github.com/sp-hash/TeamBlackMiner
sp_ (OP)
Legendary
*
Offline Offline

Activity: 2898
Merit: 1087

Team Black developer


View Profile
September 15, 2015, 11:51:37 AM
 #5880

Here is from the DJM-34 code: default lyra2v2 intensites.

        unsigned int intensity = 256*256*8;
   intensity = (device_sm[device_map[thr_id]] == 500) ? 256 * 256 * 4 : intensity;


-X 8 on gtx ,950,960,970, 980,980ti
-X 4 on gtx 750,750ti

While in my modded lyra2v2 I use

-X 18 on 970,980,980ti
-X 16 on 750ti
-X 5 on 750
-x 16 on 960

With my mod (sp-mod 67) lyra2v2 is doing 5 MHASH with overclocking. DJM34's is doing 4,5 with overclocking.

(My version is around 10% faster on the 750ti)

Team Black Miner (ETHB3 ETH ETC VTC KAWPOW FIROPOW MEOWPOW + dual mining + tripple mining.. https://github.com/sp-hash/TeamBlackMiner
Pages: « 1 ... 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 [294] 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 ... 1240 »
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!