Show Posts
My name is Mike. My team has been developing and promoting MoonBot trading terminal and related resources for 3 years: chat rooms, Telegram channels, signal channel, trading statistics website, forum and so on. The full list of our resources can be found at the end of the article.
In the course of our work, we have successfully overcome many difficulties and solved many interesting problems, but we cannot solve one. It is related to the work of Binance API, the best (despite these API problems) cryptocurrency exchange to date.
This article is intended primarily for our users who trade on the MoonBot terminal. The article may also be interesting for those who use other scalper terminals for trading on binaries and have faced the problems described below. The main purpose of the article is to explain to the users what is going on and to inform Binance representatives that the API can be improved and should be.
The problem with Binance’s API is — lags
This problem is divided into 2:
The first one is a lag when receiving exchange data. This is the delay between the time when the price was established at the exchange (a transaction took place) and the time when the information about the price of this transaction reached your terminal via web socket. More details follow.
The second is the so-called “Order Does Not Exist” problem, synchronization of the exchange database. If you see such entries in the log, it means you have been affected as well.
The essence of the problem: when you place a new order, it is not yet in the database of the Binance, and gets there only after some time. According to Binance, this time does not exceed 1 second normally, which is quite acceptable if it would be true for 100% cases. Actually, this time can exceed 5–10 minutes sometimes, which is absolutely unacceptable!
When we write about it to their tech support, Binance says, “Use the web socket.” Gentlemen, it’s 2020, everybody’s been using the web socket for a long time. The problem is that the information on the web socket is sent only once, and if at the moment of sending, some lag (see paragraph 1), or even a disconnect happens, it does not matter on whose side — information about the order will not come! And it will not be sent again.
Some people will say that the connection is very reliable, and how can the information not arrive? It’s very simple. The most obvious example is to pull the network cable out of the computer for a second and you will see a break on the tick chart. The gap is the period during which the data has not arrived. More complex and more real cases are the various failures and overloads of network equipment all the way from Tokyo (location of BInance servers) to your PC. The result is that the connection falls off after a timeout. MoonBot restores it immediately, but some data is already irrecoverably lost.
Please note that we are talking about trading information that is vital for saving and multiplying your deposit. It is not a problem if, for example, you do not get the weather forecast information, but if you lose important information about an order, you will not be able to close it in time and will lose your funds.
Therefore, in addition to the web socket, MoonBot uses GET /api/v3/order API function to monitor the order status. If for some reason the order update has not been received via the web socket, a secondary function of GET /api/v3/order will be used, but it does not work due to the database de-synchronization! To be more precise, it works, but after an indefinite period of time.
We have collected statistics on how long this function may not work. The chart below:
This chart contains statistics from all trading bots per day. We can see that the API function to check the order status did not work between 200 and 600 seconds!
You can check actual real-time charts for the last 24 hours here
Let’s talk about the lags in their pure form
i.e. about delays between some events on the exchange and receiving information about them via the API.
Please note that we are talking about scalper terminals and bots, such as MoonBot, where the trader cares about every fraction of a second to analyze the market situation and manage orders, while in auto trading, milliseconds are important. If you trade in the medium term, you may not notice lags at all. But if you are scalping, if you want perfect software responsiveness, perfect tick chart accuracy and order management — lags are very important for you! Many MoonBot users have found times when the exchange was operating without lags, so they have something to compare with.
The lag we are talking about is a delay between an event and receiving information about it more than 500 ms later. Many of our users believe that even lower values of 200–300 ms are acceptable for manual trading, while in auto trading at values above 50 ms are a sign that bots should be turned off.
Here’s an example of what lag looks like on the chart:
If you are not a MoonBot user, this chart might scare you, but it’s simple — different lines are BTC/USDT price values on Binance, obtained from different sources (transaction price, ASK price in the order book, mark price, spot market price). Obviously, at each moment the BTC price was actually about the same value, so all lines must match or be close to each other. But they do not coincide due to the fact that the trade data came with a delay of 10–20 seconds compared to the orderbook data, and the chart shows the outdated price also with a huge delay!
We have collected accurate data on lags from all working MoonBots on the chart.
On this chart below is an example of how the delays in receiving transactions from Binance were distributed during the day 23.08.2020
Online lag statistics for the last 24 hours can be found here: https://api.moon-bot.com/latency/
the data is updated every 20 seconds.
A week ago Binance created a new web socket fstream3.binance.com, we were asked to test it in work. It turned out that this socket has no support for data compression (permessage-deflate), so the situation with the lags has become even more complicated: now in addition to depending on Binance’s servers, you also depend on the quality of your Internet: the amount of uncompressed data is too large, and additional lag is added by your ISP.
We immediately reported this issue with compression to Binance, but to our greatest surprise today this socket was published in the news … still without compression correction.
It seems Binance Exchange is in such a hurry to increase trading volumes and attract as many users as possible, that there is no time left for quality control issues.
Brief summary with technical details on how MoonBot works with Binance API
The tick chart is drawn by the following web socket flows:
- Futures: wss://fstream.binance.com/ws/<symbol>@aggTrade
- Spot: wss://stream.binance.com:9443/ws/<symbol>@trade
A lag on the chart means that data from these streams came with delays. If you see on the chart that trades are lagging by N seconds, it means that your PC receives data from Binance with a lag of N seconds.
If you see gaps on the chart, it means that your PC does not physically receive data from Binance at the above addresses. If you set the logging level in MoonBot to 5, you will most likely see connection timeouts in the log.
You can compare your lag indicators with other users and see the general picture of Binance’s lags for the current day here:
https://api.moon-bot.com/latency/
Our resources:
Website: instructions, downloads, blog https://moon-bot.com
Telegram chats and channels:
- MoonBot news channel https://t.me/MoonBotNews
- English-speaking MoonBot audience https://t.me/Moon_Bot_Public
- general issues, discussions, Moon booth proposals https://t.me/moon_bot_crypto
- Chat to discuss the special channel Moon Bot with trading signals in Telegram https://t.me/crypto_moon_mafia
- Trading Theory and General Philosophical Questions https://t.me/moon_bot_kurilka
Trading statistics on the MoonBot terminal (manual trading and algorithmic trading) contains data on volumes and transactions from all users who have agreed to send deals.
It is updated online at https://stat.moon-bot.com/
Our forum http://forum.moon-bot.com/
Our thread on bitcointalk forum: https://bitcointalk.org/index.php?topic=2234450.msg22522662
