This man speaketh the truth, dont put all eggs in a basket unless you Warren Buffet in which case he says its fine and watch it really carefully but he also says dont touch crypto. Its known as asset allocaiton, never over balance yourself over weighted to one direction in all the various asset types. Helps to stay afloat in rough seas which I think are a reasonable prospect
Pretty much. What kept me going in 2008 was going down to the deposit box and running my hands through the small stash of silver and gold and making noises like a pirate. Aarggh and all that. It helped me to remember that I have things and will be ok. If things go boom this time I will stroke my tools and such knowing that they go with me and can always bring value. That's the important thing in life, be able to create value and you'll be ok. |
|
|
|
At least this is flushing out the stupid altcoins. Bye bye alts, come back to Bitcoin.  |
|
|
|
|
Or people are flocking to dollars to pay maintenance on their Lambos. Those things are pricey, get a used Porsche.
More seriously, I remember in 2008 when EVERYTHING was melting down, even gold. I had some cash stashed and that allowed me not to panic sell. Which turned out to be a good idea since the market tripled. But live within your means now for awhile, don't keep buying Lambos.
|
|
|
|
|
Hey, I cashed in some bitcoin 2 weeks ago to order an Elon Flamethrower. And he threw in a fire extinguisher gratis.
So I'm doing fine. I will use the flamethrower to take chips off boards, so it might be a business expense. Need to ask the IRS about that.
|
|
|
|
|
Right now Bitmainwarranty is the best bet for fixing these. They seem to do a pretty good job, but they don't post anything about how they do it or what the root problem is.
Which is fair enough, I respect that.
That said I operate on a different model: Information is free, work and skills are paid for. Thus I'm posting all of my repair stuff in my "Hacking (whatever it is) for (fun and profit, because it needs to be done, I'm bored, etc) threads.
At the moment I can fix some issues, and am getting closer to fixing all the categories of problems, I just figured out how to isolate and swap bad hashing chips (and there are those). Once I get reliably over that hurdle I should be able to add antminer S9's to my list of things that can be fixed (and anyone else with the skills can do it too, more people fixing these things the better for home miners and such).
Keeps me out of trouble :-)
|
|
|
|
Unfortunately it didn't work. I have just switched pools however, so I am going to see if this may work....
Again each time I reboot it restores hashing
So it runs for a bit then crashes? May be a bad chip that is showing signs of thermal expansion. Want to send it over and I'll look at it in IR? C |
|
|
|
Sure. You can measure the voltage on either of the two choke pins. The choke is the big red thing that looks like a cross between a magnet and a donut, 2 solid wires coming out of it into the board. Below the power plugs, can't miss it. Meantime, been working on a few boards that won't come up. Running the board with no controller attached does pull current, and with an appropriate IR camera you can see some things after awhile. For example:  As you can see, one of the chips is not like the others. Being that it's hotter, it is probably shorted. And being shorted it's not passing control signals and whatnot. So off it comes, to be replaced by another chip.... This seems to happen a fair bit. And it's fun to roll out the bigger tools, kind of makes me feel like a Predator or something...... |
|
|
|
|
Great! Post how you did it (include things like did you remove the old compound from the chips, what you put on, screw torque amounts, etc) so other people can do this as well.
The Ebay coolers are a bargain....
|
|
|
|
|
Good idea: I'm checking up on the current price for my soul.... :-)
|
|
|
|
Dude, just HODL.....  Little miss bitcoin is just going to freshen up, she will be back. Wait..... |
|
|
|
Meantime, what's up with the transaction pool? https://blockchain.info/unconfirmed-transactionsBlockchain is showing under 30k unconfirmed for 620btc in fees at 12-20 tps. This is a massive drop from the 150-200k transactions for a lot less fees earlier. In the old days anything above 3-4 tps would grown the chain, so is this segwit in operation? |
|
|
|
Gotcha Dave. In the meantime here is the predator-vision view of an S7 board that powers up the chips but doesn't hash..... Note the chips glowing normally, and the one chip glowing red. One of these things is not like the others, and in this case it's probably a shorted chip. I've pulled it for review, will swap in another chip this week and see if that fixes it. Note: The orientation of the chips is weird, they alternate 180 degrees as you go from chip to chip on the board. Probably to better line up signal pins, but a bit confusing regardless. |
|
|
|
|
Sure, that would be great, I'll PM you with shipping details.
Been working on the A6 boards, they're weird: If the controller board in the box becomes unplugged from the boards they go into thermal overdrive, you can see at least one chip on each side going low resistance and heating up a *LOT*. The problem with the A6's at the core is that they built the boards without any sort of compensators. For example on the Antminers (with one exception) the boards have a set of FETs, a choke, and a programmed voltage of between 9.6 (Antminer S9) to 10.5 (Antminer S7) then they run the chips off that. Stable, and can handle chips that might go slightly off in terms of resistance.
On the weird S7's with lots of chips, and the Avalon A6's they run them straight off the power supply, 20 chips per string on the A6. Thus each chip normally gets .6 volts, but if a chip gets warmer than its neighbors, the resistance on that chip goes down and very bad things happen. Bad means the chip has a lower resistance meaning it gets more volts+current=more watts=more heat which means more current. Thus they can get hot enough to melt the solder, make balls, and short out/shut down. Bitch.
Moral: Ensure your chips are ballasted.
|
|
|
|
|
Reflow oven is a bitch on wheels. I have been reflowing old broken Titan and Neptune boards, problem is the soft components (caps and plugs) melt. Working on ways to fix that.
Meantime I have something equally cool, predator vision. I'll post a pic in a bit.
C
|
|
|
|
So anyway, time to break out the real fun debugging tools. As we used to say in thermodynamics, heat is the ultimate bullshit generator. Thus if you have unusual amounts of heat or lack thereof somewhere on a board, something is up..... So let's plug our 62 chip board into a power supply with no connection to a controller board (steady voltage, nothing hashing) and then take a look at our board here under the eye of a Predator.....  Look at that. Yes the chips on the inside will be warmer than the outside ones, and yes that big blob of heat is the FETs for the power supply. Normal, but what the hell is that heat blob over on the left there. It looks like one chip is not like the others..... Time to remove that heat sink and see what's going on there. It's one of the chips that doesn't have a second sink on the bottom (they have a delta V of chips without sinks, maybe airflow improvement but very stupid in a series/parallel arrangement) and see what is going on there. Mr. Thermal is your friend. |
|
|
|
Nice work!
I had a board like that with intermittent 0 and not 0 asic.
So if the chips have core vcc (which they do, your number is right) but they don't talk, what's next?
We're assuming they do (well, at least 62 did); my guess is the backplane is series/parallel with all three chips in parallel on the power and ground plane as opposed to three true serial strings tied together at both ends. Nice because the power plane is more stable and uniform, bitch because an open chip would be masked by its' neighbors (although you might see this in heat maps, as the chip would not be running at idle and its partners would be a bit warmer because they are carrying more current through them to the next series of three. Hm, where is my peek....) They need IO vcc, they need clock, and they need an unbroken connection to rx and tx on the header.
Hm. Is each chip wired to rx/tx on the header, or do they daisy chain between the chips? There's advantages to either way, but if they were all in parallel and one chip grounded it would sink the whole line (and rx/tx would read zero). If series any one chip could sink the string if it went open. Hm. And they need to be alive, but since they came up once it's likely they are, and are just suffering an intermittent issue with one of the other items.
Maybe. If one of the 63 put the tx/rx signal to ground or if it broke the chain that would show up as a dead board. The question is which one is doing it? On titans as a comparison, the 4 main dies on each chip are connected to a common signal bus that can be isolated per chip by removing a 0 ohm jumper. However the hotel power and ground cannot, therefore if a die shorts hard the board is junk. If it shorts soft you can isolate the signal, and if it fails open you just have three dies running. Back to the S9, there's also a second supply on this board, looks to be a 14.5 volt supply, I was wondering if that was series shared hotel power for the hashing chips. How good's your scope?
Pretty good, it's an older Tektronix T922. Main problem is it's only a 15mhz scope, I should upgrade it one of these days. |
|
|
|
This is interesting. On one of the boards from last night's test I see that it did come up once briefly.... Check chain[7] PIC fw version=0x03
Fix freq=550 Chain[7] voltage_pic=6 value=940
set_reset_allhashboard = 0x0000ffff
set_reset_allhashboard = 0x00000000
Chain[J8] has 62 asic
set_reset_hashboard = 0x00000080
set_reset_hashboard = 0x00000000
retry Chain[J8] has 62 asic
set_reset_hashboard = 0x00000080
set_reset_hashboard = 0x00000000
retry Chain[J8] has 62 asic
set_reset_hashboard = 0x00000080
set_reset_hashboard = 0x00000000
retry Chain[J8] has 62 asic
set_reset_hashboard = 0x00000080
set_reset_hashboard = 0x00000000
retry Chain[J8] has 62 asic
set_reset_hashboard = 0x00000080
set_reset_hashboard = 0x00000000
retry Chain[J8] has 62 asic
set_reset_hashboard = 0x00000080
set_reset_hashboard = 0x00000000
retry Chain[J8] has 62 asic
Chain[J8] has no freq in PIC, set default freq=550M
Chain[J8] has no core num in PIC
Miner fix freq ...
read PIC voltage=940 on chain[7]
Chain:7 chipnum=62
Asic[ 0]:550
Asic[ 1]:550 Asic[ 2]:550 Asic[ 3]:550 Asic[ 4]:550 Asic[ 5]:550 Asic[ 6]:550 Asic[ 7]:550 Asic[ 8]:550
Asic[ 9]:550 Asic[10]:550 Asic[11]:550 Asic[12]:550 Asic[13]:550 Asic[14]:550 Asic[15]:550 Asic[16]:550
Asic[17]:550 Asic[18]:550 Asic[19]:550 Asic[20]:550 Asic[21]:550 Asic[22]:550 Asic[23]:550 Asic[24]:550
Asic[25]:550 Asic[26]:550 Asic[27]:550 Asic[28]:550 Asic[29]:550 Asic[30]:550 Asic[31]:550 Asic[32]:550
Asic[33]:550 Asic[34]:550 Asic[35]:550 Asic[36]:550 Asic[37]:550 Asic[38]:550 Asic[39]:550 Asic[40]:550
Asic[41]:550 Asic[42]:550 Asic[43]:550 Asic[44]:550 Asic[45]:550 Asic[46]:550 Asic[47]:550 Asic[48]:550
Asic[49]:550 Asic[50]:550 Asic[51]:550 Asic[52]:550 Asic[53]:550 Asic[54]:550 Asic[55]:550 Asic[56]:550
Asic[57]:550 Asic[58]:550 Asic[59]:550 Asic[60]:550 Asic[61]:550
Chain:7 max freq=550
Chain:7 min freq=550
max freq = 550
set baud=2
Chain[J8] set working voltage=940 [6]
setStartTimePoint total_tv_start_sys=167 total_tv_end_sys=168
restartNum = 2 , auto-reinit enabled...
do read_temp_func once...
do check_asic_reg 0x08
get RT hashrate from Chain[7]: (asic index start from 1-63)
Asic[01]=71.4200 Asic[02]=58.5860 Asic[03]=63.6860 Asic[04]=63.3500 Asic[05]=64.3570 Asic[06]=61.4880 Asic[07]=60.3810 Asic[08]=61.5380
Asic[09]=63.5520 Asic[10]=56.8910 Asic[11]=59.5420 Asic[12]=63.0320 Asic[13]=60.8500 Asic[14]=57.7470 Asic[15]=62.7970 Asic[16]=64.5750
Asic[17]=64.4070 Asic[18]=66.4200 Asic[19]=56.5890 Asic[20]=64.8940 Asic[21]=62.4950 Asic[22]=63.6860 Asic[23]=60.8840 Asic[24]=59.5080
Asic[25]=61.9070 Asic[26]=64.7930 Asic[27]=60.9180 Asic[28]=63.6350 Asic[29]=58.8710 Asic[30]=60.3810 Asic[31]=63.7700 Asic[32]=65.3300
Asic[33]=59.5750 Asic[34]=60.9340 Asic[35]=58.5020 Asic[36]=65.6150 Asic[37]=67.2430 Asic[38]=63.7700 Asic[39]=69.1550 Asic[40]=67.3430
Asic[41]=63.2830 Asic[42]=66.5380 Asic[43]=64.1890 Asic[44]=61.3540 Asic[45]=59.8100 Asic[46]=65.2960 Asic[47]=67.3770 Asic[48]=61.2700
Asic[49]=61.7560 Asic[50]=61.7560 Asic[51]=61.7230 Asic[52]=65.2800 Asic[53]=64.1720 Asic[54]=65.3470 Asic[55]=64.3400 Asic[56]=60.3300
Asic[57]=59.3910 Asic[58]=63.2660 Asic[59]=67.0080 Asic[60]=66.5710 Asic[61]=60.9340 Asic[62]=62.4610 Check Chain[J8] ASIC RT error: (asic index start from 1-63)
Done check_asic_reg
do read temp on Chain[7]
Done read temp on Chain[7]
set FAN speed according to: temp_highest=0 temp_top1[PWM_T]=0 temp_top1[TEMP_POS_LOCAL]=0 temp_change=0 fix_fan_steps=0
set full FAN speed...
FAN PWM: 100
read_temp_func Done!
CRC error counter=6567
In other words it came up with 62 Asics briefly, and a high CRC error number. So maybe this is a chip problem. If so, which one....... Hm. |
|
|
|
|
Ok, it's 3 strings of 21. If you follow the chips from the 3 on the left all the way around you see voltages like (all referenced to ground)
8.58, 8.15, 7.75, 7.27, 6.44, 6.02, 5.605, 5.164, 4.74, 4.2, 3.7, 3.2, 2.8, 2.4, 1.6, 1.2 .8, .4, 0.
(9.1 volts at source)
Or each chip pulling about .4 volts. Makes sense.
Likewise it looks like the three chips are run in parallel, so you get .5 ohms from one chip to another in the three chip set. Have to think before I do a chip to chip test, I don't want my multimeter to back-feed voltage and damage anything...
However when we fire up the board we get:
Miner Type = S9
set_reset_allhashboard = 0x0000ffff
set_reset_allhashboard = 0x00000000
set_reset_allhashboard = 0x0000ffff
set_reset_allhashboard = 0x0000ffff
Check chain[5] PIC fw version=0x03
Fix freq=550 Chain[5] voltage_pic=6 value=940
set_reset_allhashboard = 0x0000ffff
set_reset_allhashboard = 0x00000000
Chain[J6] has 0 asic
set_reset_hashboard = 0x00000020
set_reset_hashboard = 0x00000000
retry Chain[J6] has 0 asic
So if the chips are not shorted then they probably are not the source of the problem. A dead shorted chip would also raise the voltages around the string and would probably blow up pretty quickly. An open chip would not be spotted by this test, as the ground planes are probably wired together and would mask an open chip.
I did leave one board powered up for a bit, and the heat sinks eventually warm up. Didn't see a temp differential on the top or bottom heat sinks from sink to sink, so all engines are probably up and idle.
Hm..... Next question is it's either the support chips, or the signal line is cut. But if cut, where.....
|
|
|
|
|
Yep, you're right, I forgot that the bottom (power plane) heat sinks are hot (pulls heat from the chip through the board). I'll make a map, and will post that and the ground fuzzing as a start.
Ultimate question is of course what is shutting down the board? We have three items:
1) hashing chips themselves
2) Power circuitry
3) Signal and support circuitry.
If it's the chip itself then it would fail either open or short. Short can be found using the map technique: Look at the voltages and find the one that reads 0 between adjacent chips. Open is easier, one string will show voltage on the first chip in the chain but no others. Finding the exact chip would then be done by measuring chip to chip resistance, one of them is going to read zero.
A side question in my brain is what's the clock and signaling circuitry like: If they all share a common clock signal then a shorted chip would ground out the clock, which could be measured at the 25mhz crystal. Likewise if they daisy chain the data signal, then a shorted chip will not pass the signal or will ground it.
Back to the drawing board after I get some other stuff done. I'll see if these other two boards have a dead chip.
If it is a chip, then it's possible to remove with air tools and a fair bit of preheat. These look a bit easier than normal QFN chips, as they are thinner (warm up more quickly) and they have those nice big power strips on the side and center which should auto-center them. Hm.
Pulling the heat sinks is not too hard, just warm up the board then use air at a lower temp to soften the glue, then pull sink then clean top of chip to remove. Do you have a pin map of the chip itself, I could hot wire a diode and try it out.
63 chips would be 9*7 or 3*3*3. So we either have 9 strings of 7 (no), 7 strings of 9 (maybe) or 3 strings of 21 (don't know about that). One way to find out....
|
|
|
|
|
Show Posts
