I'm mildly confused - there is an announcement on ZenCloud but not HashTalk, in fact the whole "Announcements" category seems to be missing on HT.
Josh from ZenCloud
Hi Everyone,
You have probably noticed my increased presence in the forum this week. I’m learning that it’s always better to over-communicate with you guys when it comes to updates and announcements and this is another example of my commitment to practice what I preach.
First off, let me start out by thanking all of you here for your patience while we worked through important ZenCloud updates and bugfixes in the past few days. Our development team is still hard at work tonight pushing more updates and manually reviewing the few remaining Hashlets that were affected this week. As I mentioned before, no accounts have been compromised and everyone’s payouts are running normally.
So grab a cup of coffee as we go through the new improvements our team has implemented to deal with the few bad apples earlier this week. What you’ll find below is a detailed issue/resolution status update and it also represents our commitment to transparency and open communications.
For those just tuning in this week for the first time simply know this: we take transparency, security, and customer protection extremely seriously in this company. You are not only the lifeblood of this dedicated community here but also our company as a whole. So with that, here’s tonight’s update which is split into a few key sections:
Communication
Issue: Sometimes hard to follow GAW communications
Description: There are often multiple channels that an important communication from us is delivered on and it is sometimes difficult to find which place to check for important updates (i.e. HashTalk Announcements, ZenCloud Messages, Email Newsletters).
Resolution: We have heard your feedback in our [communication poll] and will be streamlining all of our key updates and announcements right into the ZenCloud Messaging system within your account. This means that every time you log into your ZenCloud account you will have the latest information and update notes made available to you.
Issue: Long Ticket Resolution Times
Description: Support Ticket Resolution Times were going unanswered or unresolved
Resolution: I’m working directly with our customer support team and am commiting that by Friday October 24 2014 all of our unassigned and backlogged tickets as well as all new tickets received up to Friday morning will be resolved.
Also, new internal processes are being built to change the internal reporting structure of our customer service team as well as the coordination of customer service with our development, sales, financial and leadership teams. As a result of this revamped process GAW will be introducing the following ticketing service level agreements:
New tickets will be assigned and responded to within 2 Hours.
Tier 1 support will resolve all tickets within 4 hours of assignment or escalate to tier 2.
Tier 2 support will resolve all tickets within 48 hours of assignment or escalate to tier 3.
Tier 3 support will consist of Development, Sales or Financial issues and will be resolved within 96 hours from assignment (4 days).
All tickets not resolved within 5 days total time will be escalated to senior management for resolution and to track needed training’s or systems changes.
Hashlet Orders
Issue: Glitches Created Ordering Issues with Hashlets (Incorrect MH/s)
Description: Starting with the 10/13 release of ZenCloud 1.12.X, glitches in Hashlet orders were creating improper MH/s and GH/s amounts for a few customer’s orders. There were a few different ways these improper scenarios could have taken place and we have outlined them below as well as the resolution we are taking for each scenario.
Scenario:
A customer orders multiple Hashlets with different MH/s power levels in a single order and then sells them on the marketplace:
John Doe places this order for new Prime Hashlets (from us or from a reseller)
1 x Prime Hashlet 1MH/s
1 x Prime Hashlet 5MH/s
1 x Prime Hashlet 25MH/s
Starting with the release on Monday night, an order bug was present that ignored the MH/s power levels of all Hashlets in one order except for the final Hashlet ordered. The above example would have actually activated to ZenCloud as the following
1 x Prime Hashlet 25MH/s
1 x Prime Hashlet 25MH/s
1 x Prime Hashlet 25MH/s
Although only a small amount of orders were affected by this glitch, it presented an issue when combined with the open marketplace where Hashlet Primes could now be sold and BTC could be then withdrawn.
There was a small group of unscrupulous users who discovered this issue on Monday and were able to abuse the system by splitting and selling their glitched Hashlets on the marketplace and then using the proceeds to buy more glitched Hashlets and repeat the process. Using the above example, the original 31MH/s total in Primes is now 75MH/s when activated and could be split into 1MH/s units and quickly sold on the market at below-market prices.
This issue was discovered shortly after the marketplace opened for Hashlet Primes on 10/13 9pm ET. Once the issue was identified, the marketplace for Primes was taken offline and we brought ZenCloud into maintenance mode while the issues were resolved (see below resolution steps).
Glitched Hashlet Resolution
The ultimate resolution for Hashlets that had improper power levels was to restore those devices back to their intended MH/s or GH/s that was originally purchased. There were three different resolution paths that we took:
Scenario A: The Hashlet code was activated in ZenCloud and then the Hashlet was split and sold on the marketplace to a new owner
Resolution: To resolve this issue, we first identified the original glitched Hashlet that was activated and then traced any splitting and sales on the marketplace to other accounts that became involved. In order to correct the power level on the original Hashlet we had to first reverse the sale of the Hashlet in the marketplace by debiting the sellers ZenCloud account and crediting the buyers ZenCloud account. We then updated the Hashlet MH/s or GH/s to the correct power level.
The small group of sellers of these glitched Hashlets had their accounts locked and put under manual review. If it was determined that any intentional marketplace abuse took place because of the glitched orders, these users had their accounts revoked and have sinced been banned.
Overall only 0.2% of Hashlet orders since the Monday release were affected by this bug.
Scenario B: The Hashlet code was activated in ZenCloud but the Hashlet was not sold on the open marketplace
Resolution: These Hashlets have since had their power levels MH/s or GH/s updated to the correct level based on the original purchase price.
Scenario C: The Hashlet code was not activated yet in ZenCloud
Resolution: We proactively corrected the MH/s or GH/s on all remaining unactivated ZenCloud devices that were affected by the glitch so that, when activated, the correct Hashlet will be brought online
Issue: Crash of Hashlet Prime pricing on the market
Description: As a result of a glitch in Hashlet Prime orders (see above issues) there were accounts that received more Hashlet Primes then they had legitimately paid for. A small group of users would then split these Primes and sell them on the open market for very low prices which, once sold, they could use the funding to purchase more glitched Hashlet Primes and repeat the process. The overall impact of this was the drive all Hashlet Prime prices down on the marketplace — including the legitimate sales.
Resolution: We have sinced fixed the Hashlet glitch and are 90% through reconciling accounts with glitched Hashlet Primes back to their original state. It may take the overall market up to a week to recover from the downward price pressure that the glitched Hashlet Primes caused. We expect the Hashlet Prime market to return to equilibrium shortly.
Issue: Hashlet Prime Primary and Secondary Pool Updating
Description: Hashlet Primes that were bought from the marketplace that were previously using the Double Dip Amp were then, at the new owner’s account, unable to change the pools from the ones that the old owner had set. One temporary workaround was to change the name of the miner which would free up the pool assignments.
Resolution: As of 1pm ET today we have fixed this issue and now Prime pool assignment is working as it should be.
Issue: 10GH/s Genesis Hashlets that were split into 1GH/s are still being charged 10GH/s fees.
Description: A few customers discovered that 10GH/s Genesis Hashlets could be split into 10 x 1GH/s Hashlets. Unfortunately, these 1GH/s Hashlets still carried the 10GH/s maintenance fee.
Resolution: We are correcting the maintenance fees in a release tonight 10/17 for the 1GH/s Hashlet Genesis.
Issue: Certain Resellers were marking up Hashlets to higher-than-market prices
Resolution: We are now proactively controlling and monitoring the prices that our resellers set. Our approved resellers list may be found here: link
https://hashtalk.org/topic/11029/official-gaw-resellers-listIssue: Hashlets in the marketplace were listed at mistakenly low prices by customers
Description: Since the launch of the marketplace, customers are now able to set their own prices for their Hashlets for sale. In rare cases, a few customers have sold high power Hashlets at lower-than-market prices.
Resolution: We have added additional precautions when customers go to list their Hashlets for sale. First, a popup confirmation box will appear which will confirm that they do indeed want to sell their Hashlet. This box will confirm the sale price and the specifications of the Hashlet for sale (MH/s or GH/s).
As an additional layer of protection, we will be adding a new notification on Monday 10/20 which will warn any prospective Hashlet seller when their listed sale price is lower than the lowest price for that Hashlet type on the market before the Hashlet is posted for sale.
Deposits and Withdrawals
Issue: Withdrawals from your ZenCloud account
Description: Earlier this week we were resolving orders glitches that were creating illegitimate Hashlets being bought and sold. As a result, to protect all customers, we paused withdrawls from accounts until we identified what was behind the glitches and which accounts were affected.
Resolution: Going forward, we have implemented steps to only target and pause withdrawls on those specific ZenCloud accounts that have been affected with bugs and other issues. This will greatly reduce the likelihood that withdrawls from your account will be affected and will speed up the resolution process for new issues going forward.
In addition, we have setup a brand new email hotline,
fraudprevention@gawminers.com, that you can contact directly anytime if you ever are having issues depositing or withdrawing from your account.
Issue: Deposits Not Showing Up in Account
Description: We put the ZenCloud system into maintenance mode this week from 10/14 11:30pm - 10/15 10:30am ET to resolve glitches in our ordering system that were creating Hashlets improperly. While the system was in maintenance mode, any deposits made during this time were not credited automatically.
Resolution: All deposits made while ZenCloud was in maintenance mode have now been manually reconciled and credited to those accounts affected. If you have made a deposit that has still not cleared please email
fraudprevention@gawminers.com for an additional review.
Issue: Customer Accounts Locked
Description: There were a few customers who unknowingly bought Hashlets on the marketplace that were created through bugs. In order to reconcile these bugs, we had to lock any accounts involved in the selling or buying of this small group of Hashlets until everything was resolved. This was an issue because often accounts were locked without the steps provided on how to unlock the account and/or resolve any open account issues.
Resolution: We are actively reconciling all accounts affected by the glitched Hashlets earlier this week. We expect the manual review of all Hashlet Primes that were affected to be finished later tonight.
To expedite any account unlocking issues, we have created a specific hotline,
fraudprevention@gawminers.com, that anyone can contact and have their account reviewed by our team and their issues resolved.
Issue: Dashboard Line Graphs missing Points
Description: As a result of ZenCloud going into maintenance mode to resolve order glitches, the Dashboard hashing power line graphs were missing data points when ZenCloud was brought back online.
Resolution: Although the graph was missing data points, all Hashlets were still hashing and earning payouts during the ZenCloud maintenance mode. The graph will continue to update itself as time progresses.
Quality Assurance Testing
Issue: Quality Assurance Testing on ZenCloud before release
Description: The 10/13 release of ZenCloud had glitches that affected Hashlet orders which we are determined to prevent from happening again in future releases by adopting a more robust QA and review process.
Resolution:
Calling all ZenCloud Beta Testers
We are launching a brand new ZenCloud Beta testing program that a selected group of qualified community members will be able to participate in. This group will be added to our quality assurance and pre-release review team and they will help review, test, identify, document, and resolve any ZenCloud bugs before each release. To submit your application to become a Beta Tester please fill out this form by Monday 10/20.
https://www.surveymonkey.com/s/WCDPLY7Please Welcome Matt Eden our new QA Team Lead!
We have brought a new QA leader on board, Matthew Eden, who has deep experience in quality assurance testing. Matt has over ten years as a Lead Application Developer and Platform Architect. He has designed, tested, implemented, and monitored scalable web applications with over 400,000 active users. Matt comes from a long and storied career working for as a project lead at Dell and AMD which make him a great fit here at GAW Miners. Please join us in welcoming Matt to the development team.
Alright that’s all of it for tonight.
As I stated above, we are committed to open communications and to being transparent in our decision-making process. Please continue to be patient with our team as we work through open tickets from this week. Rest assured that we will continue to make enhancements to our system and continually provide you updates.
Hash On
GAW_CEO
