Bitcoin Forum

Bitcoin => Project Development => Topic started by: Dwynr on October 21, 2019, 06:05:02 PM



Title: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 21, 2019, 06:05:02 PM
Hello!

I've been working on a website for a while now that shows trading volume and prices in realtime excluding wash trading.

My main incentive for creating this site was me getting sick of all the fake volume exchanges listed on CoinMarketCap.
I'm using a list of exchanges created by BTI (http://www.bti.live/exchanges/) and presented to the US SEC when they applied for the BTC ETF.
Some of the approved exchanges still include (+-5%) wash trading on a few pairs and thats why I also wrote an algorithm that mostly detects strange behaviour and filters it out.

The main difference between this site and all other price trackers is that it saves all trades made on the exchange in a Database and then calculates price and volume rather than just trusting the exchanges API.
The site also comes with a REST and WebSocket API for free which you can use to build your own tools (mostly trading bots, I guess).

Feel free to join the Discord (https://discord.gg/eEQq5PF) if you find a bug, have a question or want to submit a feature request! :)

The website is in active development so expect things to change.

Link: https://marketcap.cc


__________________________________________________________________________

Semi-huge update: You can now see liquidity of every asset (collected in realtime from all exchanges' orderbooks), sort tables by clicking their headers and see high volume transactions on selected blockchains (whale alert).
Order book viewing will be enabled soon (for every market).
__________________________________________________________________________

Update: The RESTful & Socket.io APIs are now fully documented (https://marketcap.cc/api). Both are completely for free and have no limits, because data should be free Smiley If you need a specific RESTful endpoint for your project don't hesitate to contact me, I'll try my best to include it into the API.
__________________________________________________________________________

Update: You can now see Liqudity of every market, exchange etc. instead of asset-based liqudity only. Ranking for Liquidity also added.
__________________________________________________________________________

"Update teaser": I am currently working on a full port of the website to an App. The App will be available for Android & iOS, starting with an Android Beta release first. Here is a screenshot of the work in progress:

https://i.imgur.com/YhdEq5j.jpg
__________________________________________________________________________

Great news! Google Play just approved the official App for MarketCap.cc. You can find it here: https://play.google.com/store/apps/details?id=cc.marketcap.app
__________________________________________________________________________


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: rijaljun on October 22, 2019, 07:51:23 AM
I have a question, it's excluding fake volume? So, what will happen if exchanges use a fake volume? are they will have different volume between your site and the exchange site or... please give me some explanation about this.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: OmegaStarScream on October 22, 2019, 08:23:08 AM
If you are retrieving the volume/orders using the exchange's API, how can you 100% distinguish the real orders from the fake ones?


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 22, 2019, 10:09:06 AM
I have a question, it's excluding fake volume? So, what will happen if exchanges use a fake volume? are they will have different volume between your site and the exchange site or... please give me some explanation about this.

Simply: yes.

If you are retrieving the volume/orders using the exchange's API, how can you 100% distinguish the real orders from the fake ones?

It is pretty easy to see if a market pair is getting wash traded by looking at the frequency and amount and quantitiy of each trade.
Please refer to this article I wrote: https://medium.com/@Dwynr/building-a-website-that-shows-real-prices-and-trading-volumes-of-cryptocurrencies-aa4d9a2f4772 if you want to learn more about it.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: elda34b on October 22, 2019, 11:48:13 AM
Seems like you'll require a lot of storage to maintain the wash trading activity.

So, at what point your algorithm considers a wash trading occurs? Suppose there is an X market trade that was done by a user, will this consider as wash trading per se because the quantity exactly matches the sell/buy order?


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: DiamondCardz on October 22, 2019, 12:29:51 PM
So you're storing every single trade that happens on every exchange for every cryptocurrency pair? That sounds like it's going to take a fairly rapidly increasing amount of storage. Perhaps a better idea would be to store the prices and volumes for each day, and then store only the trades which are flagged as a wash trade - that allows you to calculate the real volume values without having to store every single transaction.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 22, 2019, 02:25:01 PM
So you're storing every single trade that happens on every exchange for every cryptocurrency pair? That sounds like it's going to take a fairly rapidly increasing amount of storage. Perhaps a better idea would be to store the prices and volumes for each day, and then store only the trades which are flagged as a wash trade - that allows you to calculate the real volume values without having to store every single transaction.

The main reason for storing all trades is that I will open an API endpoint to fetch historical trades for the public. This can be REALLY useful for people doing TA, coding trading bots etc.
Data is stored in Redis (KeyValue in Memory Database) for 48h and then offloaded to a BigData database after that, where storage capacity isnt really a problem. Data growth per day is about 10GB.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: rijaljun on October 23, 2019, 04:09:57 AM
The main reason for storing all trades is that I will open an API endpoint to fetch historical trades for the public. This can be REALLY useful for people doing TA, coding trading bots etc.
Data is stored in Redis (KeyValue in Memory Database) for 48h and then offloaded to a BigData database after that, where storage capacity isnt really a problem. Data growth per day is about 10GB.
So, data storage must be no problem for you. How about the whole data fetching process? Doesn't it will take longer time than taking from exchange API?


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 23, 2019, 01:58:29 PM
The main reason for storing all trades is that I will open an API endpoint to fetch historical trades for the public. This can be REALLY useful for people doing TA, coding trading bots etc.
Data is stored in Redis (KeyValue in Memory Database) for 48h and then offloaded to a BigData database after that, where storage capacity isnt really a problem. Data growth per day is about 10GB.
So, data storage must be no problem for you. How about the whole data fetching process? Doesn't it will take longer time than taking from exchange API?

No, fetching data for the statistics (24h change, volume, price etc.) is pretty fast since the data is in memory storage (Redis). After it has been processed and calculated its offloaded to the other Database.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: BrewMaster on October 23, 2019, 03:48:37 PM
so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.

in addition to that you are putting too much efford and focus in something that doesn't really matter for sites such as yours that list coins. you are still making the same mistake as coinmarketcap.com and rank coins based on their market capitalization which is a completely meaningless and wrong approach.
not to mention that your site lacks the option to list prices against bitcoin whcih in my opinion is the only thing that matters, you only have USD option.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 23, 2019, 05:46:12 PM
so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.

in addition to that you are putting too much efford and focus in something that doesn't really matter for sites such as yours that list coins. you are still making the same mistake as coinmarketcap.com and rank coins based on their market capitalization which is a completely meaningless and wrong approach.
not to mention that your site lacks the option to list prices against bitcoin whcih in my opinion is the only thing that matters, you only have USD option.

Ranking based on Liquid Market Cap will come in the next update. I will also include pricing against Bitcoin. I am taking any suggestions and will try to make the site better everyday. Not worth talking the site down since it's only been online for roughly a week now.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Theb on October 23, 2019, 07:00:05 PM
I've seen an article about you which I saw in the press section (https://bitcointalk.org/index.php?topic=5195335.msg52853553#msg52853553) of the forum and I'm still a bit skeptical on how accurate you will calculate and segregate the fake volume from crypto exchanges, even removing the -/+ 5% won't be enough for you to have the true volume on each exchange. So can you explain a little bit more on how or what you will do to remove those "strange volume" in their data and what you will do to identify them? How do you even consider one to be a strange behavior in the first place?


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: johndoughsmith2019 on October 23, 2019, 08:50:14 PM
The idea of being able to "see" true volume of different coins is very valuable. Looking at the website i real like the UI, it's clean, easy to navigate, and responsive. Haven't tried out the API so cannot speak to that.

The fundamental core of this project is the accuracy of the algorithm used for filtering out wash trade, of which I still have no clue on how it is done after reading you article (https://medium.com/@Dwynr/building-a-website-that-shows-real-prices-and-trading-volumes-of-cryptocurrencies-aa4d9a2f4772). If the algorithm are just detecting certain trading patterns, then I don't see how it can be used to detect wash trade with sufficient accuracy since exchange might have or come up with sophisticated wash trading patterns.

That being said, I do believe idea is still valuable even if the algorithm only filters out a majority of the fake volumes rather than 100%.

Great job OP.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 23, 2019, 10:07:31 PM
The idea of being able to "see" true volume of different coins is very valuable. Looking at the website i real like the UI, it's clean, easy to navigate, and responsive. Haven't tried out the API so cannot speak to that.

The fundamental core of this project is the accuracy of the algorithm used for filtering out wash trade, of which I still have no clue on how it is done after reading you article (https://medium.com/@Dwynr/building-a-website-that-shows-real-prices-and-trading-volumes-of-cryptocurrencies-aa4d9a2f4772). If the algorithm are just detecting certain trading patterns, then I don't see how it can be used to detect wash trade with sufficient accuracy since exchange might have or come up with sophisticated wash trading patterns.

That being said, I do believe idea is still valuable even if the algorithm only filters out a majority of the fake volumes rather than 100%.

Great job OP.

The RESTful API will be documented in the coming days, the WebSocket API already is. The algorithm cannot truly filter out 100% of all fake volume, but I'll try to improve it over time. As you mentioned, it currently only looks for pre-determined patterns and filters them out. The next version of the algo will be self learning, which will cover almost all patterns an exchange might use (I hope, it's just a theory for now :P).
Thanks for your feedback!


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: rijaljun on October 24, 2019, 07:21:44 AM
so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.
We can't really make exchanges to stop faking volumes anyway, it will be always happening and I don't think this project is aiming to stop it directly. All I get from OP is to remove those fake volumes from price tracker site and creating a place where people can find a more genuine prices.

Although it might only obvious fake volumem at start and volume fakers won't stop, at least OP has a big efforts to avoid tracking fake volumes and of course the site and algo will improve if OP really commit to develop this.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Slow death on October 24, 2019, 10:10:12 AM
https://i.imgur.com/l5LMc9i.png

https://coinmarketcap.com

https://i.imgur.com/wOkByXV.png

https://marketcap.cc

The difference is very big, your site is excellent. I have a suggestion:

can you add the ann threads of the coins? something like that:

https://i.imgur.com/sup6nb1.png


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 24, 2019, 12:46:26 PM
https://i.imgur.com/l5LMc9i.png

https://coinmarketcap.com

https://i.imgur.com/wOkByXV.png

https://marketcap.cc

The difference is very big, your site is excellent. I have a suggestion:

can you add the ann threads of the coins? something like that:

https://i.imgur.com/sup6nb1.png

Thanks for your feedback and yes, more information like website, block explorer, ANN thread etc. for coins will be added in the next update.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: BrewMaster on October 26, 2019, 03:19:00 PM
Not worth talking the site down since it's only been online for roughly a week now.

sorry, i didn't mean to belittle your efforts, i was just criticising the site. as long as you work on the "ranking" mechanism that you use and don't rank coins based on their market capitalization, it will be a lot better than the alternative.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 27, 2019, 03:27:08 AM
so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.
We can't really make exchanges to stop faking volumes anyway, it will be always happening and I don't think this project is aiming to stop it directly. All I get from OP is to remove those fake volumes from price tracker site and creating a place where people can find a more genuine prices.

Although it might only obvious fake volumem at start and volume fakers won't stop, at least OP has a big efforts to avoid tracking fake volumes and of course the site and algo will improve if OP really commit to develop this.

Thanks for your feedback and yes, I am comitted to continue the development. I am adding new features everyday, just check it out once in a while (or maybe use it full time).


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: hello_good_sir on October 27, 2019, 03:50:11 AM
It's a good attempt by a smaller company/single person to combat washed trading/market making/market manipulation, whatever new word they are using to call fake trade volumes - It seems like you are making a valid attempt to not counted the millions in faked volume, and even though I'm sure there will be some exchanges that can slip through, I congratulate you on your effort!

I do enjoy the design aspects of your listing site as well. Although I do get the feeling that text is hard to read, especially in dark mode and when you click a coin - the 4 boxes that are all black (black text, black background, etc) is really distracting and I don't enjoy it at all, maybe try it with white text?

Good luck mate, as long as this keeps being updated you'll have a banger on your side!


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 27, 2019, 03:06:03 PM
It's a good attempt by a smaller company/single person to combat washed trading/market making/market manipulation, whatever new word they are using to call fake trade volumes - It seems like you are making a valid attempt to not counted the millions in faked volume, and even though I'm sure there will be some exchanges that can slip through, I congratulate you on your effort!

I do enjoy the design aspects of your listing site as well. Although I do get the feeling that text is hard to read, especially in dark mode and when you click a coin - the 4 boxes that are all black (black text, black background, etc) is really distracting and I don't enjoy it at all, maybe try it with white text?

Good luck mate, as long as this keeps being updated you'll have a banger on your side!

Some design elements are still temporary and will change once I found a more fitting attempt! I will also start developing a "day mode" with a white theme soon. Thanks a lot for your feedback.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on October 29, 2019, 07:44:54 PM
Semi-huge update: You can now see liquidity of every asset (collected in realtime from all exchanges' orderbooks), sort tables by clicking their headers and see high volume transactions on selected blockchains (whale alert).
Order book viewing will be enabled soon (for every market).


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on November 06, 2019, 08:33:46 AM
Update: The RESTful & Socket.io APIs are now fully documented (https://marketcap.cc/api). Both are completely for free and have no limits, because data should be free :) If you need a specific RESTful endpoint for your project don't hesitate to contact me, I'll try my best to include it into the API.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on November 17, 2019, 09:37:58 PM
Update: You can now see Liqudity of every market, exchange etc. instead of asset-based liqudity only. Ranking for Liquidity also added.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on November 21, 2019, 04:14:18 PM
"Update teaser": I am currently working on a full port of the website to an App. The App will be available for Android & iOS, starting with an Android Beta release first. Here is a screenshot of the work in progress:

https://i.imgur.com/YhdEq5j.jpg


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: aa7356 on November 24, 2019, 11:08:44 AM
I never really trust the volume indicator even on exchanges ...

I prefer to do correlations on order book depth ...

to better explain my thoughts there is a good article here

Quote
But pundits know that trading volume is an unreliable metric. Almost every cryptocurrency exchange reports larger trading volumes than what they actually have.

On the other hand, order book data is significantly more trustworthy.


Source: https://www.hodlbot.io/blog/analysis-of-order-book-depth-on-binance


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: BrewMaster on November 24, 2019, 04:08:45 PM
Quote
But pundits know that trading volume is an unreliable metric. Almost every cryptocurrency exchange reports larger trading volumes than what they actually have.

On the other hand, order book data is significantly more trustworthy.


with that logic orderbook data is also equally untrustworthy since exchanges can easily fake that too. you can check out Yobit exchange for a live manipulation of literary everything. their orderbooks are also filled with orders that you can never fill even if you succeed in reaching them!

the thing is when it comes to analysis, you should use all these data together and also analyze the changes. then in the end know that your analysis is only a guideline to the most possible outcome not a prediction of future.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: unbanked_peon on November 24, 2019, 10:48:13 PM
A familiar look and feel. Congrats on executing your idea. That's what counts.

I think another member suggested that this would absolutely destroy your I/O if you were to take into concern your site blowing up. I can see niche/featured functions like this in something more private.

Always down to help. Good luck!


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: tinyteapot on November 25, 2019, 07:47:03 AM
I like the theme of your website, the titan dark color will not make looking for at it for so long to affect the eyes.
You website appear like www.bitcoinwisdom.io though you have more coins listed and you quote all their prices in USD which makes it easily understandable to non btc background visitor.
I will also appreciate if you can remove from your topic
Quote
excluding fake volume
because you only take your feeds from trading exchanges and there is no specific method for you to know which volume is real or fake since the trades are not been done directly on your website.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on November 27, 2019, 12:42:45 AM
A familiar look and feel. Congrats on executing your idea. That's what counts.

I think another member suggested that this would absolutely destroy your I/O if you were to take into concern your site blowing up. I can see niche/featured functions like this in something more private.

Always down to help. Good luck!

Thanks for your feedback.

Yes, I already have the infrastructure ready for when there is big traffic coming, don't worry.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: examplens on November 27, 2019, 10:05:48 AM
this is great, it's funny how some exchanges report high volume.
But you still have only 15 exchanges in your algorithm which again does not provide accurate information.
Any idea of how to add more exchanges, although small, they still have a certain amount of trading


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on November 27, 2019, 03:56:20 PM
this is great, it's funny how some exchanges report high volume.
But you still have only 15 exchanges in your algorithm which again does not provide accurate information.
Any idea of how to add more exchanges, although small, they still have a certain amount of trading

The Data on the site is pretty much the most accurate you will get out there. We calculate Liquidity between real established markets with real volume. Exchanges on the list are picked for a reason. We don't inflate our stats with no name exchanges nobody ever heard of which are reporting billions of volume per day. Our list of exchanges is based on the BTI report to the US SEC. We might include new exchanges in the future tho.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on November 27, 2019, 08:47:36 PM
Great news! Google Play just approved the official App for MarketCap.cc. You can find it here: https://play.google.com/store/apps/details?id=cc.marketcap.app


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: IconFirm on December 09, 2019, 11:28:49 AM
I'm impressed. Your figures highlight just how much fake volume there is on CMC - something I've been telling peeps for years now - well done!

I'm still not sure about some of those exchanges figures though.... but good work nonetheless.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: Dwynr on December 12, 2019, 08:50:33 PM
I'm impressed. Your figures highlight just how much fake volume there is on CMC - something I've been telling peeps for years now - well done!

I'm still not sure about some of those exchanges figures though.... but good work nonetheless.

I will tweak the algo in the future - mainly for Huobi and HitBTC, but thanks for your feedback!


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: IconFirm on December 13, 2019, 08:45:26 PM
I'm impressed. Your figures highlight just how much fake volume there is on CMC - something I've been telling peeps for years now - well done!

I'm still not sure about some of those exchanges figures though.... but good work nonetheless.

I will tweak the algo in the future - mainly for Huobi and HitBTC, but thanks for your feedback!

Those were the two that I thought were still inflated, well spotted.....

Good work indeed.


Title: Re: Crypto prices, trades & volume in realtime excluding fake volume
Post by: walerikus on December 17, 2019, 06:17:58 PM
Hello!

I've been working on a website for a while now that shows trading volume and prices in realtime excluding wash trading.

My main incentive for creating this site was me getting sick of all the fake volume exchanges listed on CoinMarketCap.
I'm using a list of exchanges created by BTI (http://www.bti.live/exchanges/) and presented to the US SEC when they applied for the BTC ETF.
Some of the approved exchanges still include (+-5%) wash trading on a few pairs and thats why I also wrote an algorithm that mostly detects strange behaviour and filters it out.

The main difference between this site and all other price trackers is that it saves all trades made on the exchange in a Database and then calculates price and volume rather than just trusting the exchanges API.
The site also comes with a REST and WebSocket API for free which you can use to build your own tools (mostly trading bots, I guess).

Feel free to join the Discord (https://discord.gg/eEQq5PF) if you find a bug, have a question or want to submit a feature request! :)

The website is in active development so expect things to change.

Link: https://marketcap.cc


It would be nice if you could exclude fake cryptocurrencies also.
__________________________________________________________________________

Semi-huge update: You can now see liquidity of every asset (collected in realtime from all exchanges' orderbooks), sort tables by clicking their headers and see high volume transactions on selected blockchains (whale alert).
Order book viewing will be enabled soon (for every market).
__________________________________________________________________________

Update: The RESTful & Socket.io APIs are now fully documented (https://marketcap.cc/api). Both are completely for free and have no limits, because data should be free Smiley If you need a specific RESTful endpoint for your project don't hesitate to contact me, I'll try my best to include it into the API.
__________________________________________________________________________

Update: You can now see Liqudity of every market, exchange etc. instead of asset-based liqudity only. Ranking for Liquidity also added.
__________________________________________________________________________

"Update teaser": I am currently working on a full port of the website to an App. The App will be available for Android & iOS, starting with an Android Beta release first. Here is a screenshot of the work in progress:

https://i.imgur.com/YhdEq5j.jpg
__________________________________________________________________________

Great news! Google Play just approved the official App for MarketCap.cc. You can find it here: https://play.google.com/store/apps/details?id=cc.marketcap.app
__________________________________________________________________________