Bitcoin Forum
June 20, 2024, 04:51:59 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 [39] 40 41 42 43 44 45 46 47 48 49 50 »
  Print  
Author Topic: BFL board project COINTAMINATION - EU facility - ORDERS OPENED for Chili  (Read 73031 times)
asjfdlksfd
Full Member
***
Offline Offline

Activity: 128
Merit: 100


View Profile
January 05, 2014, 01:53:20 PM
 #761

Yes that is same thing... It can go over 20A per chips at this speed... And power supply is only 160A with cooling...
I guess your boards have better copper plates or so. All first 4 boards I've assembled now have the same issue. It is not relevant how many engines could be started. My board with 126 engines have the same problem as my board with only 109 engines. It looks like not only the problem comes from higher voltages > 1.1. One board with 109 engines allready resets at 1.0V. I can start it only by stopping the cpu fan so the blf will arrive the 70° limit earlier. I will check later.

All these are now limit to the power fets. Maybe there can be replaced by ones with 60 A support?

Cheers...
GrapeApe
Sr. Member
****
Offline Offline

Activity: 476
Merit: 250



View Profile
January 05, 2014, 03:43:32 PM
Last edit: January 05, 2014, 05:24:03 PM by GrapeApe
 #762


Does anybody knows how I can flash with linux without any windows installing? I use kvm but in real usb is not working from it.

Cheers..

I just went through this myself. You need Virtual Box (VB) It has settings for usb support but you have to add your user to the vb group for the usb to work.  Another member helped me a little bit. here.


No worries, you just go down to the bottom right of the Virtual Box in which XP is running and click on the USB icon.  Then choose the one that corresponds to the Chili.  I'm at work, so I don't remember which one that is, but you can lsusb on the linux side or just unplug and see where the difference is in that popup menu choice.  You have to have set up the VirtualBox for XP to allow USB passthrough, but that's an easy configuration choice.  Depending on your port, you might also have to have installed the extension pack to allow USB 3.0 if it's one of those.
.

Lucko (OP)
Hero Member
*****
Offline Offline

Activity: 826
Merit: 1000



View Profile
January 05, 2014, 04:07:26 PM
 #763

Yes that is same thing... It can go over 20A per chips at this speed... And power supply is only 160A with cooling...
I guess your boards have better copper plates or so. All first 4 boards I've assembled now have the same issue. It is not relevant how many engines could be started. My board with 126 engines have the same problem as my board with only 109 engines. It looks like not only the problem comes from higher voltages > 1.1. One board with 109 engines allready resets at 1.0V. I can start it only by stopping the cpu fan so the blf will arrive the 70° limit earlier. I will check later.

All these are now limit to the power fets. Maybe there can be replaced by ones with 60 A support?

Cheers...
I tested every board with 2 restarts. But I did figure out that they act batter if they have there own unshare power supply... If it manage to get to speed(when speeds start drooping) I declare it without problems. Also if it crashed above 38GH. If it chresh but was able to get it going with restarts it was also send out. So that is why your 109 is restarting because of unknown problem that MrTeal and I are looking. You did opted to have them made anyway... Only if I could not make it run I didn't send it out(OK I did one by mistake)...

And I'm not sure what you mean by 60A support...
asjfdlksfd
Full Member
***
Offline Offline

Activity: 128
Merit: 100


View Profile
January 05, 2014, 05:46:25 PM
 #764

I tested every board with 2 restarts. But I did figure out that they act batter if they have there own unshare power supply... If it manage to get to speed(when speeds start drooping) I declare it without problems. Also if it crashed above 38GH. If it chresh but was able to get it going with restarts it was also send out. So that is why your 109 is restarting because of unknown problem that MrTeal and I are looking. You did opted to have them made anyway... Only if I could not make it run I didn't send it out(OK I did one by mistake)...

And I'm not sure what you mean by 60A support...
Replace all 40A psr chips by 60A types? I've not checked that they exists. I guess it's a problem with limit of the psr.

It would be fine we can disable chips manually by Z-Commands before starting bfgminer.
And it would be fine we can limit manually by Z-Commands the voltage and also maybe the voltage-ramp.

I think the fw is not open source to extend them by myself? This is a question which can only answered by mr. teal.

Cheers....
bx8389
Member
**
Offline Offline

Activity: 80
Merit: 10


View Profile
January 05, 2014, 07:32:47 PM
 #765

I tested every board with 2 restarts. But I did figure out that they act batter if they have there own unshare power supply... If it manage to get to speed(when speeds start drooping) I declare it without problems. Also if it crashed above 38GH. If it chresh but was able to get it going with restarts it was also send out. So that is why your 109 is restarting because of unknown problem that MrTeal and I are looking. You did opted to have them made anyway... Only if I could not make it run I didn't send it out(OK I did one by mistake)...

And I'm not sure what you mean by 60A support...
Replace all 40A psr chips by 60A types? I've not checked that they exists. I guess it's a problem with limit of the psr.

It would be fine we can disable chips manually by Z-Commands before starting bfgminer.
And it would be fine we can limit manually by Z-Commands the voltage and also maybe the voltage-ramp.

I think the fw is not open source to extend them by myself? This is a question which can only answered by mr. teal.

Cheers....


You can check on TI site Fet Specifications http://www.ti.com/product/csd96370q5m
They have also a version w/ silighty more output. Chilli have 4 so 40A x 4 = 160 A max, 100 A recomended in long run

For me, don't seem a problem of intensity output, this is rather a symptom. I haven't checked yet ( have a daytime work ) but I belive more on things like saturation on inductors or capacitor rds due the random nature with same components. For instance a saturated inductor on this kind of power supply acts like a short so this can reset a controler.

But your sugestion to have some firmware configuration is good: these units can hash at 23-25 Mhs so if can program firmware to stop at 2 VRM stage and don't got 3th stage VRM at least we can have at least some return during problem investigation time. As is, on this moment, these boards are useless

Regards
MrTeal
Legendary
*
Offline Offline

Activity: 1274
Merit: 1004


View Profile
January 05, 2014, 08:38:45 PM
 #766

Yes that is same thing... It can go over 20A per chips at this speed... And power supply is only 160A with cooling...
I guess your boards have better copper plates or so. All first 4 boards I've assembled now have the same issue. It is not relevant how many engines could be started. My board with 126 engines have the same problem as my board with only 109 engines. It looks like not only the problem comes from higher voltages > 1.1. One board with 109 engines allready resets at 1.0V. I can start it only by stopping the cpu fan so the blf will arrive the 70° limit earlier. I will check later.

All these are now limit to the power fets. Maybe there can be replaced by ones with 60 A support?

Cheers...
Restarting at @1V does not have much to do with the number of engines. I only have one board here so it is difficult to diagnose whether all the boards with those issues are due to the same root cause, but the one I have is due to an actual hardware error. Lucko is looking into whether those other ones exhibit the same symptoms.
As for replacing the FETs, the ones rated to 50A available in stock now, so if you wanted to replace them on some boards you could do that. You'd also need to make changes to the controller itself though as it is what limits the current to 160A. The fets would definitely need more cooling though; the 96370 @ 40A will dissipate ~7W each if you keep the junction temperature to 75C. The 96371 @ 50A will dissipate ~9.5W each at the same conditions, and keeping the junction temperature down will be harder.


You can check on TI site Fet Specifications http://www.ti.com/product/csd96370q5m
They have also a version w/ silighty more output. Chilli have 4 so 40A x 4 = 160 A max, 100 A recomended in long run

For me, don't seem a problem of intensity output, this is rather a symptom. I haven't checked yet ( have a daytime work ) but I belive more on things like saturation on inductors or capacitor rds due the random nature with same components. For instance a saturated inductor on this kind of power supply acts like a short so this can reset a controler.

But your sugestion to have some firmware configuration is good: these units can hash at 23-25 Mhs so if can program firmware to stop at 2 VRM stage and don't got 3th stage VRM at least we can have at least some return during problem investigation time. As is, on this moment, these boards are useless

Regards
100A isn't the recommended for the long run, it's just the current at the rated Ploss spec. The fets will run fine at the 40A rating continuously as long as they're cooled, but efficiency is lower than if they were run at 25A per phase.

I don't believe it is likely a component issue; we have made almost 1000 of these and none have experienced (that I know of) this particular problem. I also replaced the affected phases with parts out of my stock and it did not solve the issue. I will try to find some time this week to feed the control and current feedback signal on the bad phase outside the board to see that can solve the problem.
bx8389
Member
**
Offline Offline

Activity: 80
Merit: 10


View Profile
January 05, 2014, 10:34:13 PM
 #767

Yes that is same thing... It can go over 20A per chips at this speed... And power supply is only 160A with cooling...
I guess your boards have better copper plates or so. All first 4 boards I've assembled now have the same issue. It is not relevant how many engines could be started. My board with 126 engines have the same problem as my board with only 109 engines. It looks like not only the problem comes from higher voltages > 1.1. One board with 109 engines allready resets at 1.0V. I can start it only by stopping the cpu fan so the blf will arrive the 70° limit earlier. I will check later.

All these are now limit to the power fets. Maybe there can be replaced by ones with 60 A support?

Cheers...
Restarting at @1V does not have much to do with the number of engines. I only have one board here so it is difficult to diagnose whether all the boards with those issues are due to the same root cause, but the one I have is due to an actual hardware error. Lucko is looking into whether those other ones exhibit the same symptoms.
As for replacing the FETs, the ones rated to 50A available in stock now, so if you wanted to replace them on some boards you could do that. You'd also need to make changes to the controller itself though as it is what limits the current to 160A. The fets would definitely need more cooling though; the 96370 @ 40A will dissipate ~7W each if you keep the junction temperature to 75C. The 96371 @ 50A will dissipate ~9.5W each at the same conditions, and keeping the junction temperature down will be harder.



100A isn't the recommended for the long run, it's just the current at the rated Ploss spec. The fets will run fine at the 40A rating continuously as long as they're cooled, but efficiency is lower than if they were run at 25A per phase.

I don't believe it is likely a component issue; we have made almost 1000 of these and none have experienced (that I know of) this particular problem. I also replaced the affected phases with parts out of my stock and it did not solve the issue. I will try to find some time this week to feed the control and current feedback signal on the bad phase outside the board to see that can solve the problem.


Many thanks for your attention to this problem and explanations MrTeal.
bcollan
Member
**
Offline Offline

Activity: 69
Merit: 10


View Profile
January 06, 2014, 12:52:22 AM
 #768

Please make mine too Lucko.
I'm not getting any younger... Grin
Mudbankkeith
Hero Member
*****
Offline Offline

Activity: 868
Merit: 1000



View Profile
January 06, 2014, 12:54:19 PM
 #769

Boards are here

https://www.dropbox.com/s/58hnjh3wnijcr2m/2014-01-06%2012.48.40.jpg

Heatsinks are here

https://www.dropbox.com/s/noj49yxyw3k5cbv/2014-01-06%2012.48.52.jpg

Even the nuts and bolts

https://www.dropbox.com/s/0zasjaemgckqkul/2014-01-06%2012.49.25.jpg

Now it gets serious

BTc donations welcome:-  13c2KuzWCaWFTXF171Zn1HrKhMYARPKv97
asjfdlksfd
Full Member
***
Offline Offline

Activity: 128
Merit: 100


View Profile
January 06, 2014, 01:03:32 PM
 #770

Hello,

yesterday I had find a solution of some of my miners by replacing the Enermax 1500 W Platinum (supports 6 rails a 30 A so I tried to use on three different rails one chilly/rail) by an older style Enermax 1350 W Revo power supply. This helps me so the chilly will not restart below 1.160 V.
I find this issue because my old style 700 W Enermax Noisetaker (2 rails with 18A) mostly find not a problem.

But in fact, I'm still not cooling my fets because if the voltage will increase to max short time later every chilly I'm cooling also the chilly will reboot. So my chillies will work ~30 GH/s which is not perfect but ok as long I find a suitable xp version which i can use to flash it. But still it would be welcome if anybody find a solution to flash with linux. I can use also my rasberry pi to program directly with jtag (in same way I do that with jally Smiley

In fact, Mr. Teal, could you add an command to limit the voltage manually e.g. after restarting?
Also can you add in firmware a way so we can disable each chip?
Also it would be fine can we have a command to recheck/reset every chip by hand?

Another problem is, after rebooting the chillies will not further work mostly because all engines mostly are enabled at 0 MHz. bfgminer 3.9.0 will not find the board or cannot refind it. Only to disconnect power brings them back online.

Cheers...
Mudbankkeith
Hero Member
*****
Offline Offline

Activity: 868
Merit: 1000



View Profile
January 06, 2014, 02:00:45 PM
 #771

Hi Lucko

Now that I have unpacked these boards,

The delivery has been rough on them.

9 boards have bent pins on the fan connectors, I am now checking each board for other damage.

BTc donations welcome:-  13c2KuzWCaWFTXF171Zn1HrKhMYARPKv97
ronin4bits
Newbie
*
Offline Offline

Activity: 59
Merit: 0


View Profile
January 06, 2014, 03:24:42 PM
 #772

Hi Lucko. Excellent work - and thanks to MrTeal for assisting.

Please proceed with making my boards, as well.

Thanks!
Lucko (OP)
Hero Member
*****
Offline Offline

Activity: 826
Merit: 1000



View Profile
January 06, 2014, 04:07:55 PM
Last edit: January 06, 2014, 05:10:09 PM by Lucko
 #773

Another problem is, after rebooting the chillies will not further work mostly because all engines mostly are enabled at 0 MHz. bfgminer 3.9.0 will not find the board or cannot refind it. Only to disconnect power brings them back online.

Cheers...
Is only 7 blinking? That means that it is cooling... I have same problem. If I don't use a pipe to get cooled air to it it will not start. They are at 43C and you need to get them to about 38C(a guess) or so to start startup... I guess it is too hot...

Hi Lucko

Now that I have unpacked these boards,

The delivery has been rough on them.

9 boards have bent pins on the fan connectors, I am now checking each board for other damage.
With all that bubble raping? Well as long as it is only bent pins it is not a big deal...

EDIT: now that I think of... Was box damaged? I think it could even be me packing it. I might pressed bubble raping in to hard...
Mudbankkeith
Hero Member
*****
Offline Offline

Activity: 868
Merit: 1000



View Profile
January 06, 2014, 07:23:03 PM
 #774

Another problem is, after rebooting the chillies will not further work mostly because all engines mostly are enabled at 0 MHz. bfgminer 3.9.0 will not find the board or cannot refind it. Only to disconnect power brings them back online.

Cheers...
Is only 7 blinking? That means that it is cooling... I have same problem. If I don't use a pipe to get cooled air to it it will not start. They are at 43C and you need to get them to about 38C(a guess) or so to start startup... I guess it is too hot...

Hi Lucko

Now that I have unpacked these boards,

The delivery has been rough on them.

9 boards have bent pins on the fan connectors, I am now checking each board for other damage.
With all that bubble raping? Well as long as it is only bent pins it is not a big deal...

EDIT: now that I think of... Was box damaged? I think it could even be me packing it. I might pressed bubble raping in to hard...
The box is ok but the contents were a bit loose(moving within the bubble wrap)

As to the boards working....................................
I have assembled 2, both have the same fault....................
eventually they start and hash for about 2 minutes, hash till the power rises to about 140 watts then reset.
BFG tries to do a restart after 60 seconds, sometimes they restart, sometimes they cycle again for another 60 seconds.

BTc donations welcome:-  13c2KuzWCaWFTXF171Zn1HrKhMYARPKv97
asjfdlksfd
Full Member
***
Offline Offline

Activity: 128
Merit: 100


View Profile
January 06, 2014, 08:19:54 PM
 #775

eventually they start and hash for about 2 minutes, hash till the power rises to about 140 watts then reset.
BFG tries to do a restart after 60 seconds, sometimes they restart, sometimes they cycle again for another 60 seconds.
Thats exactly what I have mostly with all boards from Lucko. Mostly I could fixed them by do not cool the psr so the ambient temperature will limit the core voltage. FW 1.1V Fix shoulf fix the problem. But some of them allready resets at 1.0 V. Cooling doesn't helped so I used my soldering heater to warm up the board at starting bfgminer. This looks like a workaround because the limit which is shown in "Manage devices" section rises the 45 °C earlier than the voltage arives 1.1 V. With this Voltage and ~27 + ~29 GH/s the 2 problem boards are mining stable at the moment. If I cool the vrms the boards resets again.
Facts: Preheating will bring the boards to mining status but it's not optimal to hash lower 30 GH/s :-(

Still my suggestion is to limit or set the voltage manually to a point which left the boards been stable.
Maybe it could be integrated in bfgminer/cgminer.
Mudbankkeith
Hero Member
*****
Offline Offline

Activity: 868
Merit: 1000



View Profile
January 06, 2014, 09:30:04 PM
 #776

Another problem is, after rebooting the chillies will not further work mostly because all engines mostly are enabled at 0 MHz. bfgminer 3.9.0 will not find the board or cannot refind it. Only to disconnect power brings them back online.

Cheers...
Is only 7 blinking? That means that it is cooling... I have same problem. If I don't use a pipe to get cooled air to it it will not start. They are at 43C and you need to get them to about 38C(a guess) or so to start startup... I guess it is too hot...

Hi Lucko

Now that I have unpacked these boards,

The delivery has been rough on them.

9 boards have bent pins on the fan connectors, I am now checking each board for other damage.
With all that bubble raping? Well as long as it is only bent pins it is not a big deal...

EDIT: now that I think of... Was box damaged? I think it could even be me packing it. I might pressed bubble raping in to hard...
The box is ok but the contents were a bit loose(moving within the bubble wrap)

As to the boards working....................................
I have assembled 2, both have the same fault....................
eventually they start and hash for about 2 minutes, hash till the power rises to about 140 watts then reset.
BFG tries to do a restart after 60 seconds, sometimes they restart, sometimes they cycle again for another 60 seconds.

Come on Lucko & MrTeal we need a proper fix for this,

For the prices paid for these devices they are not very good.(as hashing goes, a box full of asic erupters would be more reliable)


BTc donations welcome:-  13c2KuzWCaWFTXF171Zn1HrKhMYARPKv97
GrapeApe
Sr. Member
****
Offline Offline

Activity: 476
Merit: 250



View Profile
January 06, 2014, 10:12:54 PM
Last edit: January 07, 2014, 01:11:22 AM by GrapeApe
 #777

From Mr.Teal himself in the chili thread.


As of right now there isn't an option to limit voltage. Below is a version of 14e compiled to limit the voltage to 1.1V instead of 1.15V.

Over the break for the next couple weeks I plan to get some changes to the firmware done to allow limits to be imposed on different variables like frequency setpoint and voltage through command extensions.

BTW- Not super useful for most people since you can't use it while hashing since the mining software has the port open, but the ZlX command (lowercase "L") will return the temperature of all the chips. If you're compiling your own cgminer, you might want to add in a way to read that.


Second I don't have my Lucko boards yet so this may really mean nothing, but I was having this very issue with 2 boards from batch 3 (I've actually posted this several times). I'm a little confused as to why NOBODY is trying the 1.1V firmware. I flashed the modified firmware and it fixed them immediately, but I took a small hit on the hash rate until my MOSFET heat sinks came in a day or 2 later and I stuck them on, put a fan on them and my hash rate is a consistent 39GH/s on both of them holding steady at 1.1V using CGMiner 3.9.0. Before I flashed the modified firmware they would shoot up to 40-41GH/s, the voltage would eventually hit 1.16V and then they would reset.  So I guess using the internal safeguards to limit the boards by heating them up would work because they will throttle themselves back but wouldn't it be easier to just flash the firmware and then if you don't mind report back to us what the results are?

All of you that are having issues with the boards resetting should at least for shits and grins flash the modified firmware

I'll be honest this is the first I'm hearing of low volt resets. I was under the impression that the Lucko boards were reseting on the high side. Ignore the above.

For the Linux households out there from a helpful forum member. For the usb to work you also have to add your user to the virtual box group (I hope I said that right).


BTW, you can actually flash the chili from within a VirtualBox image using XP.  You just have to make sure you allow the USB ID to go through, and then it works just fine.  So: create virtualbox, install xp, add driver for bfgminer for chili, flash the chili, remove virtualbox, and you're not inflicted with windows for very long.  We are mostly windows-free, too - I just keep an image around for just these types of things.

MudBankKeith I was under the impression you had already flashed them with the modified firmware. If so are you still having issues after?

EDIT: BTW I am using the CoolerMaster 212 EVO for coolers on those 2 boards. I have others with the Accelero TT2 and they don't have this issue but they run much hotter than the others and hash at 32-34GH/s.
Mudbankkeith
Hero Member
*****
Offline Offline

Activity: 868
Merit: 1000



View Profile
January 06, 2014, 10:46:14 PM
 #778

From Mr.Teal himself in the chili thread.


As of right now there isn't an option to limit voltage. Below is a version of 14e compiled to limit the voltage to 1.1V instead of 1.15V.

Over the break for the next couple weeks I plan to get some changes to the firmware done to allow limits to be imposed on different variables like frequency setpoint and voltage through command extensions.

BTW- Not super useful for most people since you can't use it while hashing since the mining software has the port open, but the ZlX command (lowercase "L") will return the temperature of all the chips. If you're compiling your own cgminer, you might want to add in a way to read that.


Second I don't have my Lucko boards yet so this may really mean nothing, but I was having this very issue with 2 boards from batch 3 (I've actually posted this several times). I'm a little confused as to why NOBODY is trying the 1.1V firmware. I flashed the modified firmware and it fixed them immediately, but I took a small hit on the hash rate until my MOSFET heat sinks came in a day or 2 later and I stuck them on, put a fan on them and my hash rate is a consistent 39GH/s on both of them holding steady at 1.1V using CGMiner 3.9.0. Before I flashed the modified firmware they would shoot up to 40-41GH/s, the voltage would eventually hit 1.16V and then they would reset.  So I guess using the internal safeguards to limit the boards by heating them up would work because they will throttle themselves back but wouldn't it be easier to just flash the firmware and then if you don't mind report back to us what the results are?

All of you that are having issues with the boards resetting should at least for shits and grins flash the modified firmware

For the Linux households out there from a helpful forum member. For the usb to work you also have to add your user to the virtual box group (I hope I said that right).


BTW, you can actually flash the chili from within a VirtualBox image using XP.  You just have to make sure you allow the USB ID to go through, and then it works just fine.  So: create virtualbox, install xp, add driver for bfgminer for chili, flash the chili, remove virtualbox, and you're not inflicted with windows for very long.  We are mostly windows-free, too - I just keep an image around for just these types of things.

MudBankKeith I was under the impression you had already flashed them with the modified firmware. If so are you still having issues after?

EDIT: BTW I am using the CoolerMaster 212 EVO for coolers on those 2 boards. I have others with the Accelero TT2 and they don't have this issue but they run much hotter than the others and hash at 32-34GH/s.


messed around for a good 2 hours this afternoon, then re-flashed to the 1.1v firmware, NO DIFFERENT after another 8 hours.
my combined hash rate for 2 chili's is still just under 10 Gh since lunchtime.

BTc donations welcome:-  13c2KuzWCaWFTXF171Zn1HrKhMYARPKv97
MrTeal
Legendary
*
Offline Offline

Activity: 1274
Merit: 1004


View Profile
January 06, 2014, 11:08:30 PM
 #779

messed around for a good 2 hours this afternoon, then re-flashed to the 1.1v firmware, NO DIFFERENT after another 8 hours.
my combined hash rate for 2 chili's is still just under 10 Gh since lunchtime.

If they are resetting at low power, it is likely not the same issue that requires using the 1.1V firmware. I would guess that it might be similar to what I am seeing on this one, but I am waiting for some confirmation on tests from Lucko that his other boards have the same cause. If that's the case (and he gives me permission to mutilate this PCB) I will try verifying what the problem is more in depth and it's possible a fix can be done going forward for people haven't gotten their boards made yet.

For those that already have their boards, it will depend on what the root cause of the problem is. If all the boards exhibiting the low power shutdown have the same issues as the one I have, it's entirely possible that it cannot be fixed and still function to its potential. If the problems with the power supply are consistent (IE, it's always the same phase causing problems) it's possible that you could send the boards back to Lucko and he could modify them to run with one phase less and a max current of 120A instead of 160A.
GrapeApe
Sr. Member
****
Offline Offline

Activity: 476
Merit: 250



View Profile
January 06, 2014, 11:53:43 PM
 #780


messed around for a good 2 hours this afternoon, then re-flashed to the 1.1v firmware, NO DIFFERENT after another 8 hours.
my combined hash rate for 2 chili's is still just under 10 Gh since lunchtime.


WOW, I stand corrected. I'm sorry to hear that.
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 [39] 40 41 42 43 44 45 46 47 48 49 50 »
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!