95% is a pretty high threshold. I think there's a high probability we won't make it in time. I would prefer not to subsidize miners making invalid blocks. Perhaps we can lower the threshold, or perhaps fork a parallel p2pool system?

Could someone create a "spam" bitcoin transaction with a message attached requesting p2pool users to all upgrade their p2pool software?

Also, maybe we can make a list of large unupgraded pools and crowdsource an attempt to track down the owners and get them to upgrade? Here's a few:

http://60-249-19-56.hinet-ip.hinet.net:9332/static/ -- 110 TH/s -- Version 13.4 -- Taiwan
http://61-219-120-109.hinet-ip.hinet.net:9332/static/ -- 90.6 TH/s -- Version 13.4 -- Taiwan
http://ec2-54-77-145-190.eu-west-1.compute.amazonaws.com:9332/static/ -- upgrading? -- 83 TH/s
http://ec2-52-24-128-249.us-west-2.compute.amazonaws.com:9332/static/ -- 57.5 TH/s
http://ec2-52-8-153-161.us-west-1.compute.amazonaws.com:9332/static/ -- 61 TH/s
http://cammello.agaland.it:9332/static/ -- 15 TH/s
http://lvps92-51-148-91.dedicated.hosteurope.de:9332/static/ -- 13.5 TH/s -- Germany

I was under the impression that the most important goal was 95%, not 50%. The 50% threshold just means that people will get upgrade notices now. If we don't get to 95%, we will see reductions in mining efficiency once BIP66 takes effect.

Coincadence has about 40 TH/s. That's not exactly what I'd call centralized. Maybe neighborhood-ized?

I've just updated http://toom.im's two nodes, http://74.82.233.205:9334 and http://74.82.233.205:9332. Together, these two nodes provide about 400 TH/s to p2pool, or about 20% of p2pool's total hashrate.

If you need to set backup or replacement nodes during the upgrades, you are welcome to use ours.

I know that at least 10 of these miners exist and are owned by Wissam, since he hosted with us for a month. I know Centerus has those ten, and I've seen strong evidence indicating that they had another 19. I do not believe Wissam to be acting in bad faith. If a scam is occurring, I expect Wissam is going to be the victim of it.

That said, OgNasty, I can respect your position.

bhanu is another one of my customers, and I can confirm his offer is also legitimate.

Just so you know, Oregon Mines is run by the same people who ran Minersource, including Matt Carson and his family. Minersource has done some extremely scammy things, and owes a lot of people money. Minersource owes me and my brother about $34k, for example, and they owe Bitmain and Black Arrow a lot of money too. They disappeared without warning, then created Oregon Mines instead. It might be too late, but I would suggest you do some more research on them before committing to hosting with them.

https://bitcointalk.org/index.php?topic=1065198.0

https://bitcointalk.org/index.php?topic=976172.msg10946707#msg10946707

One of my customers has 28 SP20Es for sale. SP20Es can be as efficient as 0.48 J/GH at the wall if you clock them low enough. PM me if you're interested.

I confirm that this offer is legitimate. These three Antminer S4s are currently hosting in the http://toom.im datacenter in central Washington.

21 Inc's business plan is brilliant.

A common problem with the modern electronics economy is that most transactions are made for a product. Customers pay once, and they pay before they know how useful the product is actually going to be to them. This means that they systematically overpay for crappy products and underpay for solid, useful products with long useful lifetimes. Many companies have tried to deal with this problem by switching over to a subscription-based model instead, but consumers often resist that idea because they don't like it when someone else owns and controls the things they use and rely on. 21's pitch is a way to allow subscription-type revenue streams while still offering consumers complete ownership of their device.

VCs understand this pitch. They see how huge this could be if it works. However, this pitch is not 21's business plan.

21's business plan is to engineer a convincing pitch, which they then present to VCs in order to get a ton of funding, which in turn they then use to design and fab their own ASIC using the world's best microprocessor fabs (Intel's), and then build multi-MW datacenters which they use to self-mine. The VCs and shareholders would not have ordinarily invested in mining, but when they see revenue coming in, they won't think too much about where it's coming from.

So far, it looks like they've been succeeding.

The cell phone mining idea is total nonsense. If they had a 10 GH/s mining chip that ran at 0.25 J/GH, that would be 2.5 watts, enough to drain an iPhone 6 battery (11 Wh) by itself in 4.25 hours. During this time, it would make about 0.00002 BTC, worth about $0.005. In order to not be ludicrous, the mining chips will need to be restricted to devices that are normally plugged in and connected to the internet, like wifi routers.

In their slides they suggest using a "/dev/bitcoin" as a micropayments method. If they sold 1 million of these devices, and each owner made one micropayment per hour (i.e. visited one webpage), that would be 278 transactions per second, or enough to fill up 40 MiB blocks. Given that the current block limit is 1 MiB, I'd say there's an obstacle in their way. Using satoshis as auth tokens would make this even worse.

A 10 GH/s miner currently generates around 00010300 satoshis per day. If a person was using their "/dev/bitcoin" once a day, and if 21 Inc received 0% of their mining revenue, they would be able to make one transaction per day. At that transaction rate, the 0.1 mBTC transaction fee would end up costing 97% of their total daily revenue. Even if you received your money once a day but only spent it once a month, you would still end up spending a high proportion in transaction fees. Since your transaction would have a large number of inputs, it will end up being several kiB in size, and will have a correspondingly large required fee. Using micropayment channels helps this problem, but if you use that, then you're no longer getting freshly mined satoshis, and you might as well not be mining at all. If you use a system like Eligius, where the pool's debt to you is paid through the coinbase transaction once every few days (or maybe weeks?), it might sorta work a little.

Embedding a miner in every device would increase centralization, since there's no way those devices would be useful without mining into 21's specially configured pool. Moreover, this pool would need a large datacenter backing it in order to get acceptably low variance.

I just made an album for you guys of one of our shelf/rack hybrid pods.

http://imgur.com/a/OgAHz


East side of our Gamma pod. The rack on the far right currently draws 70.5 kW, mostly SP35s. We used to have 73.5 kW in there, but one SP30 customer asked to have their machine shipped out.

The "Pull wire" thing is what we use as a door for access to the hot aisle.

These photos were taken at the Toomim Brothers Bitcoin Mining Concern facility in central Washington. You can visit http://toom.im for more information.


East side of Gamma. Thermal photo using a Seek Thermal camera. The camera is miscalibrated, subtract 5°C from all the numbers to get the accurate values. Ambient temperature was about 18°C inside the datacenter when this photo was taken.

Sorry for the rotated image. My pone has its USB port on the side, and Seek's software doesn't like that. Too lazy to fix it in post-production. Rotate clockwise 90° to correspond to the previous image.


We just finished up this section yesterday. Currently only 12 Antminer S5s in there, but we'll be adding about 35 more soon. Since S5s have a low delta T (about 5°C), we've found that we can put two S5s front-to-back while still keeping the intake temperature for the rear S5 well within limits. Thus, this is a 2x2x3 grid.


The plastic we put up for hot air containment is apparently partially transparent to infrared light. Kinda neat. The solid white panels, on the other hand, are not.

Rotate clockwise 90° to correspond to the previous image.

I'd really like to see the successor to the S5 come with a lower CFM fan, or possibly with multiple fan options. The S5 uses up to 260 CFM for 590 W, or 0.44 CFM/W, heating the incoming air by about 4.8°C before spitting it out. In comparison, the SP30 uses up to 300 CFM for 3000 W, or 0.1 CFM/W, heating the incoming air by about 25°C before spitting it out. It's much easier to build a datacenter around 0.1 CFM/W than around 0.44 CFM/W. If you're using air conditioning, the higher SP30 exhaust temperature improves the efficiency of the A/C unit. If you're using free air cooling, it reduces the amount of air you need to filter and pre-cool and improves the efficiency of any heat exchangers you're using.

We're trying to scrape enough money together to do a 1.5 MW expansion. Our current capacity is basically full. Spending money on miners means we have less money available for expansion and less capacity for other customers.

Sorry for the thread hijack; I just replied where btcgnome made the post. Marketplace -> Goods -> Computer Hardware would definitely be the more appropriate forum.

I confirm the above. In case you're wondering, Toomim Bros can serve as an escrow for this transaction.

In other news, ASICSPACE fired their CEO, Damir Kalinkin, last week. I believe the exact process was that the shareholders held a vote of no confidence, then cleared out the Board of Directors (of which Damir was a member), and then the new Board fired Damir. In my (biased?) opinion, Damir was the worst part of ASICSPACE, and responsible for the most egregious misdeeds I have heard attributed to them. With Damir gone, I expect ASICSPACE to improve considerably. Robert is a nice guy, and I like him. From what I have seen, Robert wants to treat his customers properly, unlike Damir, who treats customers as a resource to be exploited.

However, I do not see how Damir can be responsible for the heat, airflow, and networking problems that ASICSPACE has had. He is not a technical person at all. Those issues were due to the decisions made by the engineer and contractors who built the facility and the person who oversaw them, Robert.

When I was at ASICSPACE earlier today, the conditions were considerably improved. The nine flexible ducts that were previously just neutral pressure outside-air intakes have now been connected to a 125 ton portable AC unit on a trailer, powered by what appeared to be a 400 amp 480V 3-phase connection. I didn't go into the cold aisles, but they appeared to be only slightly negatively pressurized relative to the hot aisles, maybe around 10 or 20% of what they had been at before. The general building interior (contiguous with the hot aisles) was strongly positively pressurized relative to the outside air, and the airflow out the open front door was very strong, maybe 50,000 to 100,000 CFM. The exhaust air from S4s felt like about 40 to 45°C, suggesting cold aisle temps around 30 to 35°C. Still high, but no longer unsafe. Most of the miners in their facility appeared to be on and hashing, although I did hear S4 beeps coming from somewhere.

ASICSPACE appears to be making a good faith effort to maintain proper operating conditions in their facility. They are putting a lot of money behind that effort. Unfortunately, they are spending it on the wrong things, like the air conditioning unit instead of a bunch of high-throughput fans. They don't need colder air, they just need more fresh air. However, the AC unit they are renting did come with a large fan inside, so there's that.

Yes, I have a conflict of interest. Sorry if that wasn't very clear. I'll edit a note of that into my initial analysis.

Edit: It's also worth mentioning that I used to be business partners with Robert and Damir, and my brother and I originally planned to design and fund most of their datacenter for them in exchange for free hosting for our SP30s and partial ownership of the business. My brother and I decided that we did not want to be involved with them (mostly because of Damir), so we broke off and forged out on our own. They claim we acted inappropriately and dishonestly, and that our original intent was just to deceive them and use them to find a location to build our own facility. I claim that they were not competent enough to continue collaboration with, and that we didn't realize this until after we were heavily involved with them.
You can read about that history here.

It helps with forensics to have multiple independent sources of information in order to cross-check facts. I first had an estimate of 55°C peak cold aisle temps based on my in-person observations. I later heard a report of ~57°C intake temps on an SP10 from another party who may have an axe to grind with ASICSPACE and might be lying. I then saw a screenshot of a KNCMiner device in their facility showing temps that would indicate intake temps around 50°C. Finally, I observed plastic deformation patterns that could be explained by hot aisle temps in the vicinity of 60°C-90°C.


Someone else might read this wondering how hot things need to be to damage an S5. I don't want people to think that they magically start to burn up as soon as the intake temperature exceeds 40°C. Accuracy is useful even when it's not necessary for the immediate task.


I agree, mostly. I think that runaway temperature was mostly restricted to the hashboards and the intra-heatsink space. It's worth mentioning that all of these S5s were of the older variety that do not have any heatsinks between the hashboards and the side panels. With minimal surface area, very little heat is transferred from the hashboards to the air that passes through that space. Typically, I see a delta-T of about 1 or 2°C for that air when airflow is not obstructed. In the ASICSPACE case, that may have been 4x higher, but that would still only have been around 8°C. Furthermore, the pattern of deformation suggests that the deformation was greatest where airflow was highest, not where it was most stagnant. This implicates ambient temperatures more strongly as the dominant factor in the plastic deformation.

(Minus the expansion of steel and aluminum, which is much smaller.)

Most of the plastic deformation indicates a shear stress, not compressive stress. The only exception to this is the buckling that was observed in a small number of the most severely deformed plastic. The buckling of the plastic occurred entirely along the short axis of the plastic shields. If thermal expansion were the culprit, I would expect to see buckling along the long axis as well, especially along the line in between the front and back screws. That line is the least deformed part of the side panels. The bottom edge is more securely attached than the top edge as well, so if the main cause of deformation was thermal expansion, one would expect the bottom edge to show more buckling and deformation. The opposite is true. The greatest deformation occurred on edges which I expect had the greatest airflow (evenly along the top edge, plus the back and front edge near the holes for the tail exhaust and fan).

Much of the deformation occurred on the top edge, above the highest screw attachment point. That edge typically is relatively straight, but it sags outward and downward. Since this edge was not being squeezed, and if it were it wouldn't cause the edge to sag like that, I don't think thermal expansion is a satisfactory explanation. Thermal softening combined with airflow is.

The apparent buckling I observed on a few panels might actually have been fluid dynamic effects similar to ocean waves or sand dunes being created by wind rather than actual buckling.

Something just occurred to me. If the airflow was anterograde but slow through the whole intra-heatsink space, the air coming out the exhaust port would be very hot. The side spaces would not maintain their pressure as well, so airflow there would be more likely to turn retrograde. This would pull the hot exhaust air around 180° back in the side ports before flowing either out the top of the miner or through the gap in the panel between the screws. This could explain the pattern of deformation pretty well.


I think high pressures alone might have been enough to cause the ASIC damage observed, but I doubt it. I think high temperatures alone might have been enough to cause both the ASIC damage and the plastic deformation, but I doubt it. I think the evidence is clear that both existed. Indeed, it's difficult to have a positively pressurized hot aisle without getting hot air recirculation and overheating everything. I don't think there's any reason to try to pin the blame on one factor.

I observed cold aisle temps around 35°C. My estimate of cold aisle temps around 55°C in the weeks before I visited was based on several factors:

1. There were network problems when I visited, resulting in less heat generated. It looked like about 50% of ASICSPACE's miners were off while I was there due to the network problems. If the 50% estimate is correct, then the delta T versus outside might be sqrt(2) to 2 times higher, depending on how much higher inside temperatures assist the stack effect. As the cold aisle temps were already 20°C hotter than outside, this alone might be enough to produce cold aisle temps of 55°C even on a somewhat cool day. However, my 50% estimate could be off. Also, many of the machines were S4s, which may also produce heat while the network is disconnected. For my estimate, I think I was generous and presumed that the network problems reduced temps by 8°C.

2. The nine flexible ducts supplementing the cold aisle with outside air at ambient pressure looked to be hastily added. I presumed they were added within the last week before my visit, and were not present when the damage occurred. I estimate they added about 20% to the airflow into the cold aisles. Roughly speaking, that should have decreased the cold-aisle/outdoor delta T by 20%, which would be about 4°C on the day I visited, or possibly as much as 8°C on other days. There may have been other similar quick fixes that I did not notice. I used 2°C in my quick estimate.

3. When I visited (6pm to 11pm), the outside temp was 16°C or lower. The high the day before was 26°C. That's 10°C. I think there was a day or two in the week or two prior that was even hotter.

8°C + 2°C + 10°C = 20°C
35°C + 20°C = 55°C

Other effects: as the temperatures rose, many machines may have turned off. On the other hand, reliance on the stack effect may have worsened the effects of a high outside temperature.

My math might not be right, of course. It's just an educated guess.


The damage we observed was not linear. In terms of deformation, pretty much all of the plastic panels were deformed. Most of the panels were only slightly deformed, maybe 1 or 2 mm per panel. Other panels were very heavily deformed, with about 20 mm of deformation at several different points on each panel. If the maximum temperature seen by each miner was normally distributed, and the amount of deformation as a function of temperature is something like D(T) = e^T, that would fit our observed distribution of deformation pretty closely.

We saw something similar with the amount of electrical damage on each hashboard. Most of the hashboards had either zero bad stages or more than two bad stages. This could also be caused by a domino effect, though.

It's also worth mentioning that the damage to the two hashboards in a single miner were correlated. Most of the miners we received either had two bad hashboards or zero bad hashboards. The number of machines with exactly one bad hashboard was less than I had expected.


I disagree. I do not see how air pressure alone could have caused this deformation. I just went over and tested, and using my finger, it takes about 20 pounds of force to permanently deform a S5 side panel at room temperature by about 3 mm. That would mean you'd need close to 1 atmosphere of pressure difference across the panel to deform it from pressure alone. Typical HVAC system pressure differences are on the order of 100 Pa (0.015 PSI). Axial fans typically have static pressure capabilities around 300 Pa. If we are very pessimistic and say that the hot aisle was 600 Pa higher pressure than the cold aisle due to the two S4 fans (i.e. the S4s had zero net airflow, 100% static pressure), then that would mean about 0.06 newtons per square centimeter, or 0.08 PSI. I think temperature must have been a large factor.

Polylactic acid has a glass transition temperature of 60°C to 65°C. What are the panels actually made of? I don't know. Also, even below or above the glass transition temperature, plastics will become softer and more plastic as temperature increases. The Vicat softening temperature for HDPE is about 70°C, for example. The 0.08 PSI might not be enough to deform it at all at 20°C, but it might be enough to deform it by 1 mm at 70°C, and enough to deform by 10 mm at 80°C. I do not feel inclined to test this right now, but if this goes to court, it would be simple enough to verify.