Bitcoin Forum
December 08, 2019, 06:58:15 PM *
News: Latest Bitcoin Core release: 0.19.0.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: Crypto prices, trades & volume in realtime excluding fake volume  (Read 497 times)
Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 21, 2019, 06:05:02 PM
Last edit: November 27, 2019, 08:47:54 PM by Dwynr
Merited by examplens (1)
 #1

Hello!

I've been working on a website for a while now that shows trading volume and prices in realtime excluding wash trading.

My main incentive for creating this site was me getting sick of all the fake volume exchanges listed on CoinMarketCap.
I'm using a list of exchanges created by BTI (http://www.bti.live/exchanges/) and presented to the US SEC when they applied for the BTC ETF.
Some of the approved exchanges still include (+-5%) wash trading on a few pairs and thats why I also wrote an algorithm that mostly detects strange behaviour and filters it out.

The main difference between this site and all other price trackers is that it saves all trades made on the exchange in a Database and then calculates price and volume rather than just trusting the exchanges API.
The site also comes with a REST and WebSocket API for free which you can use to build your own tools (mostly trading bots, I guess).

Feel free to join the Discord (https://discord.gg/eEQq5PF) if you find a bug, have a question or want to submit a feature request! Smiley

The website is in active development so expect things to change.

Link: https://marketcap.cc


__________________________________________________________________________

Semi-huge update: You can now see liquidity of every asset (collected in realtime from all exchanges' orderbooks), sort tables by clicking their headers and see high volume transactions on selected blockchains (whale alert).
Order book viewing will be enabled soon (for every market).
__________________________________________________________________________

Update: The RESTful & Socket.io APIs are now fully documented (https://marketcap.cc/api). Both are completely for free and have no limits, because data should be free Smiley If you need a specific RESTful endpoint for your project don't hesitate to contact me, I'll try my best to include it into the API.
__________________________________________________________________________

Update: You can now see Liqudity of every market, exchange etc. instead of asset-based liqudity only. Ranking for Liquidity also added.
__________________________________________________________________________

"Update teaser": I am currently working on a full port of the website to an App. The App will be available for Android & iOS, starting with an Android Beta release first. Here is a screenshot of the work in progress:

https://i.imgur.com/YhdEq5j.jpg
__________________________________________________________________________

Great news! Google Play just approved the official App for MarketCap.cc. You can find it here: https://play.google.com/store/apps/details?id=cc.marketcap.app
__________________________________________________________________________
1575831495
Hero Member
*
Offline Offline

Posts: 1575831495

View Profile Personal Message (Offline)

Ignore
1575831495
Reply with quote  #2

1575831495
Report to moderator
1575831495
Hero Member
*
Offline Offline

Posts: 1575831495

View Profile Personal Message (Offline)

Ignore
1575831495
Reply with quote  #2

1575831495
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1575831495
Hero Member
*
Offline Offline

Posts: 1575831495

View Profile Personal Message (Offline)

Ignore
1575831495
Reply with quote  #2

1575831495
Report to moderator
rijaljun
Sr. Member
****
Offline Offline

Activity: 798
Merit: 263

CryptoTalk.Org - Get Paid for every Post!


View Profile WWW
October 22, 2019, 07:51:23 AM
 #2

I have a question, it's excluding fake volume? So, what will happen if exchanges use a fake volume? are they will have different volume between your site and the exchange site or... please give me some explanation about this.

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
CryptoTalk.org| 
MAKE POSTS AND EARN BTC!
🏆
OmegaStarScream
Staff
Legendary
*
Offline Offline

Activity: 1862
Merit: 1544


Exchange Bitcoin quicky--https://blockchain.com.do


View Profile
October 22, 2019, 08:23:08 AM
 #3

If you are retrieving the volume/orders using the exchange's API, how can you 100% distinguish the real orders from the fake ones?

Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 22, 2019, 10:09:06 AM
Last edit: October 22, 2019, 10:46:29 AM by Dwynr
Merited by OmegaStarScream (2)
 #4

I have a question, it's excluding fake volume? So, what will happen if exchanges use a fake volume? are they will have different volume between your site and the exchange site or... please give me some explanation about this.

Simply: yes.

If you are retrieving the volume/orders using the exchange's API, how can you 100% distinguish the real orders from the fake ones?

It is pretty easy to see if a market pair is getting wash traded by looking at the frequency and amount and quantitiy of each trade.
Please refer to this article I wrote: https://medium.com/@Dwynr/building-a-website-that-shows-real-prices-and-trading-volumes-of-cryptocurrencies-aa4d9a2f4772 if you want to learn more about it.
elda34b
Sr. Member
****
Offline Offline

Activity: 602
Merit: 334


View Profile
October 22, 2019, 11:48:13 AM
 #5

Seems like you'll require a lot of storage to maintain the wash trading activity.

So, at what point your algorithm considers a wash trading occurs? Suppose there is an X market trade that was done by a user, will this consider as wash trading per se because the quantity exactly matches the sell/buy order?
DiamondCardz
Legendary
*
Offline Offline

Activity: 1134
Merit: 1088


CryptoTalk.Org - Get Paid for every Post!


View Profile WWW
October 22, 2019, 12:29:51 PM
 #6

So you're storing every single trade that happens on every exchange for every cryptocurrency pair? That sounds like it's going to take a fairly rapidly increasing amount of storage. Perhaps a better idea would be to store the prices and volumes for each day, and then store only the trades which are flagged as a wash trade - that allows you to calculate the real volume values without having to store every single transaction.

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
.CryptoTalk.org.|.MAKE POSTS AND EARN BTC!.🏆
Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 22, 2019, 02:25:01 PM
 #7

So you're storing every single trade that happens on every exchange for every cryptocurrency pair? That sounds like it's going to take a fairly rapidly increasing amount of storage. Perhaps a better idea would be to store the prices and volumes for each day, and then store only the trades which are flagged as a wash trade - that allows you to calculate the real volume values without having to store every single transaction.

The main reason for storing all trades is that I will open an API endpoint to fetch historical trades for the public. This can be REALLY useful for people doing TA, coding trading bots etc.
Data is stored in Redis (KeyValue in Memory Database) for 48h and then offloaded to a BigData database after that, where storage capacity isnt really a problem. Data growth per day is about 10GB.
rijaljun
Sr. Member
****
Offline Offline

Activity: 798
Merit: 263

CryptoTalk.Org - Get Paid for every Post!


View Profile WWW
October 23, 2019, 04:09:57 AM
 #8

The main reason for storing all trades is that I will open an API endpoint to fetch historical trades for the public. This can be REALLY useful for people doing TA, coding trading bots etc.
Data is stored in Redis (KeyValue in Memory Database) for 48h and then offloaded to a BigData database after that, where storage capacity isnt really a problem. Data growth per day is about 10GB.
So, data storage must be no problem for you. How about the whole data fetching process? Doesn't it will take longer time than taking from exchange API?

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
CryptoTalk.org| 
MAKE POSTS AND EARN BTC!
🏆
Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 23, 2019, 01:58:29 PM
 #9

The main reason for storing all trades is that I will open an API endpoint to fetch historical trades for the public. This can be REALLY useful for people doing TA, coding trading bots etc.
Data is stored in Redis (KeyValue in Memory Database) for 48h and then offloaded to a BigData database after that, where storage capacity isnt really a problem. Data growth per day is about 10GB.
So, data storage must be no problem for you. How about the whole data fetching process? Doesn't it will take longer time than taking from exchange API?

No, fetching data for the statistics (24h change, volume, price etc.) is pretty fast since the data is in memory storage (Redis). After it has been processed and calculated its offloaded to the other Database.
BrewMaster
Hero Member
*****
Offline Offline

Activity: 1386
Merit: 856


There is trouble abrewing


View Profile
October 23, 2019, 03:48:37 PM
 #10

so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.

in addition to that you are putting too much efford and focus in something that doesn't really matter for sites such as yours that list coins. you are still making the same mistake as coinmarketcap.com and rank coins based on their market capitalization which is a completely meaningless and wrong approach.
not to mention that your site lacks the option to list prices against bitcoin whcih in my opinion is the only thing that matters, you only have USD option.

Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 23, 2019, 05:46:12 PM
Merited by BrewMaster (1)
 #11

so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.

in addition to that you are putting too much efford and focus in something that doesn't really matter for sites such as yours that list coins. you are still making the same mistake as coinmarketcap.com and rank coins based on their market capitalization which is a completely meaningless and wrong approach.
not to mention that your site lacks the option to list prices against bitcoin whcih in my opinion is the only thing that matters, you only have USD option.

Ranking based on Liquid Market Cap will come in the next update. I will also include pricing against Bitcoin. I am taking any suggestions and will try to make the site better everyday. Not worth talking the site down since it's only been online for roughly a week now.
Theb
Sr. Member
****
Online Online

Activity: 1134
Merit: 460



View Profile
October 23, 2019, 07:00:05 PM
 #12

I've seen an article about you which I saw in the press section of the forum and I'm still a bit skeptical on how accurate you will calculate and segregate the fake volume from crypto exchanges, even removing the -/+ 5% won't be enough for you to have the true volume on each exchange. So can you explain a little bit more on how or what you will do to remove those "strange volume" in their data and what you will do to identify them? How do you even consider one to be a strange behavior in the first place?

      ▄ ▄█▄ ▄█ ▄
     ▄▐██▀▀▀▀▀▀
      ▀▄▄████▄ █▄
   ▄ ██▄█▀▀   ▀▀ ▀
  ▄██▄██▄ ▀██▄▀ ▀█▄
 ▀███████▄▄▄▄▄█▄▄▄██
▐███████████▀▀  ▀█▀ █
█▀███████████  ▄▄█▄ ██
 ▐█████████████▀   ███
  ████▀██████████▄███
  ▐█▀  ████████████ ▀
   ▀  ▐███████████
     ▄██████▀▀ █▀
.
JACKMATE'S
MAJESTIC

  ███████████████████
 ███████████████████
███████████████████
         █████████
        █████████
       █████████
      █████████
     █████████
    █████████
   █████████
  █████████
 █████████
████████
██
██
██
██
██
██
██
██
██
██
██
████████
██████████████████████████████████████████████████████████████████████████████████████████████     ███
.
WIN 1 BITCOIN ON EVERY PREMIER LEAGUE MATCHDAY
.
███     ██████████████████████████████████████████████████████████████████████████████████████████████
████████
██
██
██
██
██
██
██
██
██
██
██
████████
████████████████████████████████
████████████▀█▀ ▀█▀█▀███████████
███████████▄ ▄▄▄▄▄▄▄████████████
███████████▀▀▄▄▄▄▄▄▄▄███████████
█████████▀▄ ██▀▄▄▄ ▀ ▄▀█████████
███████▀ ▀█████▄▄▄█▄▄▄██████████
███████▀▄████████▀  ▀█ █▐███████
███████ ▀█████████▄█▀▀██ ███████
████████ ███▀██████ ▄ ██ ███████
████████▌▐▀▄ ██████████ ▄███████
█████████▄██▌▐█████▀██ █████████
████████████▄▀▀▀▀▀▄ ▀▄██████████
████████████████████████████████
██████
██
██
██
██
██
██
██
██
██
██
██
██████
.
JOIN US - IT'S FREE!
██████
██
██
██
██
██
██
██
██
██
██
██
██████
johndoughsmith2019
Newbie
*
Offline Offline

Activity: 6
Merit: 0


View Profile
October 23, 2019, 08:50:14 PM
 #13

The idea of being able to "see" true volume of different coins is very valuable. Looking at the website i real like the UI, it's clean, easy to navigate, and responsive. Haven't tried out the API so cannot speak to that.

The fundamental core of this project is the accuracy of the algorithm used for filtering out wash trade, of which I still have no clue on how it is done after reading you article (https://medium.com/@Dwynr/building-a-website-that-shows-real-prices-and-trading-volumes-of-cryptocurrencies-aa4d9a2f4772). If the algorithm are just detecting certain trading patterns, then I don't see how it can be used to detect wash trade with sufficient accuracy since exchange might have or come up with sophisticated wash trading patterns.

That being said, I do believe idea is still valuable even if the algorithm only filters out a majority of the fake volumes rather than 100%.

Great job OP.
Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 23, 2019, 10:07:31 PM
 #14

The idea of being able to "see" true volume of different coins is very valuable. Looking at the website i real like the UI, it's clean, easy to navigate, and responsive. Haven't tried out the API so cannot speak to that.

The fundamental core of this project is the accuracy of the algorithm used for filtering out wash trade, of which I still have no clue on how it is done after reading you article (https://medium.com/@Dwynr/building-a-website-that-shows-real-prices-and-trading-volumes-of-cryptocurrencies-aa4d9a2f4772). If the algorithm are just detecting certain trading patterns, then I don't see how it can be used to detect wash trade with sufficient accuracy since exchange might have or come up with sophisticated wash trading patterns.

That being said, I do believe idea is still valuable even if the algorithm only filters out a majority of the fake volumes rather than 100%.

Great job OP.

The RESTful API will be documented in the coming days, the WebSocket API already is. The algorithm cannot truly filter out 100% of all fake volume, but I'll try to improve it over time. As you mentioned, it currently only looks for pre-determined patterns and filters them out. The next version of the algo will be self learning, which will cover almost all patterns an exchange might use (I hope, it's just a theory for now Tongue).
Thanks for your feedback!
rijaljun
Sr. Member
****
Offline Offline

Activity: 798
Merit: 263

CryptoTalk.Org - Get Paid for every Post!


View Profile WWW
October 24, 2019, 07:21:44 AM
 #15

so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.
We can't really make exchanges to stop faking volumes anyway, it will be always happening and I don't think this project is aiming to stop it directly. All I get from OP is to remove those fake volumes from price tracker site and creating a place where people can find a more genuine prices.

Although it might only obvious fake volumem at start and volume fakers won't stop, at least OP has a big efforts to avoid tracking fake volumes and of course the site and algo will improve if OP really commit to develop this.

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
CryptoTalk.org| 
MAKE POSTS AND EARN BTC!
🏆
Slow death
Hero Member
*****
Offline Offline

Activity: 1400
Merit: 755


blackjack.fun- place your bet now


View Profile
October 24, 2019, 10:10:12 AM
 #16



https://coinmarketcap.com



https://marketcap.cc

The difference is very big, your site is excellent. I have a suggestion:

can you add the ann threads of the coins? something like that:


.

 ▀
   ▀
.

 ▀
   ▀

█▀▄
▄▄▄
█████
█████
█████
█████
█████
█████
█████
█████
█████
█████
██████████████
█▀▄██████████████
▄▄▄██████████████
████▄▄▄▄▄▄▄███████
████████████▄█████
████████▀█████████
█████▀██▀██▀██████
████████▄█████████
███▄▄▄███▌██████
███▄██████████████
██▄███████▌███████
██▀█████▀█████████
██████████████████
.blackjack.fun..

  ▀
   ▄
.

  ▀
   ▄
Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 24, 2019, 12:46:26 PM
 #17

https://i.imgur.com/l5LMc9i.png

https://coinmarketcap.com

https://i.imgur.com/wOkByXV.png

https://marketcap.cc

The difference is very big, your site is excellent. I have a suggestion:

can you add the ann threads of the coins? something like that:

https://i.imgur.com/sup6nb1.png

Thanks for your feedback and yes, more information like website, block explorer, ANN thread etc. for coins will be added in the next update.
BrewMaster
Hero Member
*****
Offline Offline

Activity: 1386
Merit: 856


There is trouble abrewing


View Profile
October 26, 2019, 03:19:00 PM
 #18

Not worth talking the site down since it's only been online for roughly a week now.

sorry, i didn't mean to belittle your efforts, i was just criticising the site. as long as you work on the "ranking" mechanism that you use and don't rank coins based on their market capitalization, it will be a lot better than the alternative.

Dwynr
Newbie
*
Offline Offline

Activity: 27
Merit: 4


View Profile
October 27, 2019, 03:27:08 AM
 #19

so you are excluding the "obvious" fake volume not fake volume in general. the exchanges can still continue producing fake volume and if someday your website (or the method in general) became popular they can easily mask it.
We can't really make exchanges to stop faking volumes anyway, it will be always happening and I don't think this project is aiming to stop it directly. All I get from OP is to remove those fake volumes from price tracker site and creating a place where people can find a more genuine prices.

Although it might only obvious fake volumem at start and volume fakers won't stop, at least OP has a big efforts to avoid tracking fake volumes and of course the site and algo will improve if OP really commit to develop this.

Thanks for your feedback and yes, I am comitted to continue the development. I am adding new features everyday, just check it out once in a while (or maybe use it full time).
hello_good_sir
Hero Member
*****
Offline Offline

Activity: 742
Merit: 510

CryptoTalk.Org - Get Paid for every Post!


View Profile
October 27, 2019, 03:50:11 AM
 #20

It's a good attempt by a smaller company/single person to combat washed trading/market making/market manipulation, whatever new word they are using to call fake trade volumes - It seems like you are making a valid attempt to not counted the millions in faked volume, and even though I'm sure there will be some exchanges that can slip through, I congratulate you on your effort!

I do enjoy the design aspects of your listing site as well. Although I do get the feeling that text is hard to read, especially in dark mode and when you click a coin - the 4 boxes that are all black (black text, black background, etc) is really distracting and I don't enjoy it at all, maybe try it with white text?

Good luck mate, as long as this keeps being updated you'll have a banger on your side!

 
                                . ██████████.
                              .████████████████.
                           .██████████████████████.
                        -█████████████████████████████
                     .██████████████████████████████████.
                  -█████████████████████████████████████████
               -███████████████████████████████████████████████
           .-█████████████████████████████████████████████████████.
        .████████████████████████████████████████████████████████████
       .██████████████████████████████████████████████████████████████.
       .██████████████████████████████████████████████████████████████.
       ..████████████████████████████████████████████████████████████..
       .   .██████████████████████████████████████████████████████.
       .      .████████████████████████████████████████████████.

       .       .██████████████████████████████████████████████
       .    ██████████████████████████████████████████████████████
       .█████████████████████████████████████████████████████████████.
        .███████████████████████████████████████████████████████████
           .█████████████████████████████████████████████████████
              .████████████████████████████████████████████████
                   ████████████████████████████████████████
                      ██████████████████████████████████
                          ██████████████████████████
                             ████████████████████
                               ████████████████
                                   █████████
.CryptoTalk.org.|.MAKE POSTS AND EARN BTC!.🏆
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!