Bitcoin Forum
March 19, 2024, 02:56:46 AM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Design behind cryptocurrency exchanges' payment processing  (Read 325 times)
NorwayFish (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 4


View Profile
May 19, 2019, 03:01:27 PM
Merited by dbshck (2), ABCbits (1), o_e_l_e_o (1)
 #1

I'm working on a small project for a while now and I still can't get my head around this topic. In my project users would have an internal balance for bitcoin and some other altcoins. They would be given a unique deposit address during registration and they would be able to deposit any amount of currency at any given moment, which would appear in their internal balance after 6 confirmations. They would also be able to enter their own address and withdraw funds. The same kind of system every cryptocurrency exchange has in place. I genuinely can not find any info on the internet on how to implement these systems, every single payment processor is invoice based.

It's pretty possible to implement something like this by yourself with the official Bitcoin Core and Full Node Softwares for bitcoin's clones, like litecoin and dogecoin. There has to be a mechanism in place that would be catching transactions on the network. However, I'm concerned about bitcoin core's walletnotify. I was told it creates a lot of memory problems for the machine. ZMQ, on the other hand, catches all the transactions, even the irrelevant ones.

So how do all of these cryptocurrency exchanges operate? Do they simply write these wrappers around the node software that collects needed data and saves in the database? Some of them have hundreds of coins! Do they have an instance of every coin running on their servers and separate software around these instances? Is there a common name for such payment processing? Is there any more information that I can look at?
1710817006
Hero Member
*
Offline Offline

Posts: 1710817006

View Profile Personal Message (Offline)

Ignore
1710817006
Reply with quote  #2

1710817006
Report to moderator
1710817006
Hero Member
*
Offline Offline

Posts: 1710817006

View Profile Personal Message (Offline)

Ignore
1710817006
Reply with quote  #2

1710817006
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1710817006
Hero Member
*
Offline Offline

Posts: 1710817006

View Profile Personal Message (Offline)

Ignore
1710817006
Reply with quote  #2

1710817006
Report to moderator
ABCbits
Legendary
*
Offline Offline

Activity: 2805
Merit: 7298


Non-custodial BTC Wallet


View Profile
May 19, 2019, 05:26:23 PM
Merited by dbshck (4)
 #2

It's pretty possible to implement something like this by yourself with the official Bitcoin Core and Full Node Softwares for bitcoin's clones, like litecoin and dogecoin.

Probably no, AFAIK Bitcoin core isn't designed for such tasks.

So how do all of these cryptocurrency exchanges operate? Do they simply write these wrappers around the node software that collects needed data and saves in the database?

There's no way to know that since most exchange keep it "closed-source", but i doubt it's that simple.

Some of them have hundreds of coins! Do they have an instance of every coin running on their servers and separate software around these instances? Is there a common name for such payment processing? Is there any more information that I can look at?

No idea, but few known common standard are :
1. Use cold and hot wallet. Hot wallet only refilled when it's empty.
2. Withdraw request with small amount are verified and processed automatically, but it's done manually for withdraw request with big amount.
3. Using multi-signature address for cold wallet.

On a side note, there are few open-source software for cryptocurrency such as https://github.com/peatio/peatio

█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

█          ▄         ▄      ▄▄▄▄▄
█       ▄███      ▄███      █████
█        ████      ████     ▀▀▀▀▀
█         ████      ████
█          ████▄▄▄▄▄▄████▄▄▄▄▄▄▄▄
█           █████████████████████
█            ▀█████▄   ▀█████▄
█              ▀█████▀   ▀█████▀
█                 ▀▀        ▀▀

█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
.....Your private Bitcoin wallet for desktop.....█▀▀▀▀▀▀











█▄▄▄▄▄▄
▀▀▀▀▀▀█











▄▄▄▄▄▄█
Heisenberg_Hunter
Legendary
*
Offline Offline

Activity: 1580
Merit: 1265


Heisenberg Design Services


View Profile WWW
May 20, 2019, 05:27:35 AM
Merited by dbshck (5)
 #3

Cryptocurrency exchanges operate in a very much different way than they are intended to do so. The updation of the balances and other transfers are all limited to the front end of the exchanges and there isn't any real transfers taking place unless you withdraw or deposit your coins. I don't use centralized exchanges much, but based on my understanding they do move our deposited coins to their cold storage wallet once they receive the coins.

They have a cold storage wallet which isn't really exposed all the time to the public traders. Apart from that, they do have a wallet which can generate thousands of address for many traders. So when a trader creates an account in an exchange, a getnewaddress rpc is called and the generated address is shown to the trader. They deposit their bitcoins to the generated address. Here in the back end code, the exchanges move our coins from the hot wallet to their cold storage for security purposes but there isn't any real balance change taking place in the front end of the exchange. So the trader's balance is same as what he deposited in the exchange. Perhaps, this is more commonly being followed in most of the exchanges but some exchanges keep their balances in the hot wallet itself.

Once a trader sells his coins in the exchange, the balance gets updated in his account as 0 but there wouldn't be any real blockchain transaction taking place since the coins are present in the cold storage. So here the buyer account is filled with new balance. You could even check this out by manually tracing the transaction in a centralized exchange. The exchanges code uses the listaddressgroupings command and get a list of addresses which have been generated and used and keeps those addresses aside.

When the buyer of your coins initiates a withdrawal, a request is sent to the cold storage and that's the time when the real transaction takes place. The exchange cold wallet probably uses the bitcoin's coin selection algorithm to check for the UTXO and spend them in one of the transaction inputs. By this way, transaction fees are reduced to a much larger extent and the UTXO set has been reduced.

More info on coin selection algorithm : Murch's coin selection Strategies and gmaxwell's coin selection theory

Hope this helps.
NorwayFish (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 4


View Profile
May 20, 2019, 08:59:48 AM
 #4

Well, just looked at my NEO GAS deposit address on Binance. I've sent some money to it a year ago and you are right, they indeed send it to their cold storage 2 months later. My main concern and the main reason why I've created this post is that building something like this with the official bitcoin core felt really awkward. It feels like bitcoin core is more of a tool for a single person to use. It's possible to write a wrapper around bitcoin core, that would make sure bitcoin core always alive and that would be processing all the transactions that bitcoin core cathes and store them. I just have no idea about how it's done in reality with all these big exchanges. There will always be differences between node software for different coins, does that mean that big exchanges run all of them behind different wrappers? That seems a bit absurd. Do they write a separate software that easily adapts and scales for many coins or do they actually have hundreds of official nodes and wrappers around them? I really want to meet a person here who ever had experience with building these systems in production.
turndealer
Newbie
*
Offline Offline

Activity: 16
Merit: 3


View Profile
May 21, 2019, 12:32:33 PM
 #5

I'm working on a small project for a while now and I still can't get my head around this topic. In my project users would have an internal balance for bitcoin and some other altcoins. They would be given a unique deposit address during registration and they would be able to deposit any amount of currency at any given moment, which would appear in their internal balance after 6 confirmations. They would also be able to enter their own address and withdraw funds. The same kind of system every cryptocurrency exchange has in place. I genuinely can not find any info on the internet on how to implement these systems, every single payment processor is invoice based.

It's pretty possible to implement something like this by yourself with the official Bitcoin Core and Full Node Softwares for bitcoin's clones, like litecoin and dogecoin. There has to be a mechanism in place that would be catching transactions on the network. However, I'm concerned about bitcoin core's walletnotify. I was told it creates a lot of memory problems for the machine. ZMQ, on the other hand, catches all the transactions, even the irrelevant ones.

So how do all of these cryptocurrency exchanges operate? Do they simply write these wrappers around the node software that collects needed data and saves in the database? Some of them have hundreds of coins! Do they have an instance of every coin running on their servers and separate software around these instances? Is there a common name for such payment processing? Is there any more information that I can look at?

Since you have saved each address in mapping of your customer.

You can run listtransactions command like this

Sample listtransactions('*', 100, 0)

here * means get transaction for all accounts,
100 here is last number of transactions ,
0 here is skip how many transactions .

This technique will work just fine for BTC, LTC , Doge .

Ethereum transaction reading technique is quite tough, You go to read block by block.
Waves platforms technique is also similar to ethereum , However they have another method to read individual transactions .

So far bitcoin detection is Quite good!
Source:I have been using an exchange script

Where
linenoise
Sr. Member
****
Offline Offline

Activity: 300
Merit: 290


View Profile WWW
May 22, 2019, 07:08:09 AM
 #6

Exchanges work as a large database, they don't truly move funds around with each transaction. Users send funds, the funds stay in the exchange account until they withdraw.  The wallets themselves can give you new addresses, you as the developer are the one responsible for how the wallets talk to your database and vice versa. As in any project the number of users will change how to do things best.

Relying on notifywallet is risky as if there is ever a problem with your notify script you could miss a transaction. Also there are scaling issues. Best to use the calls of the various wallets to get recent transactions, or if you are running a full node you could even poll every address you are watching and see if there is something new.

Normally you'd want to know which transactions aren't completed, or are in enough blocks. Make a pending transactions list and check on a regular basis if those have reached the required block count. Most exchanges have a different number of completed blocks per coin so this could be quite variable.

Exchanges will need to be running the actual coin daemon, there is indeed a real cost to adding new coins.

The best way to do this depends on the the user base, transaction volume, number of machines, database, and even design. Your devs should all be able to help you design a system that matches your needs.. good luck.

I can't think of many examples of an exchange but plenty of examples of how to do a notifywallet in many many ways which is really what you're after.





RHavar
Legendary
*
Offline Offline

Activity: 2555
Merit: 1886



View Profile
May 25, 2019, 09:44:34 PM
Last edit: May 26, 2019, 05:04:56 PM by RHavar
Merited by dbshck (5), ABCbits (1)
 #7

I've implemented this sort of thing probably now like half a dozen times for different businesses, so I'm probably pretty qualified to answer this one.


The way I'd strongly recommend:

Get a trezor, and copy the master pubkey for deriving addresses. Use this to generate deposit addresses.

Create a table in your database with like 3 columns:  (derive_index, user_id, address)  which basically is "We derived X address for user Y by using deriv index Z". Use indexes for all columns.


Now the simplest deposit system you can get, would be to:
* Ask bitcoin core for the best block height
* Subtract N from that, and download that block   (where most exchanges might use N=6)
* Create a list of all addresses that that block "paid"  and then ask your database which of them are deposit addresses.
* Credit everyone who made a deposit.

---

So now basically your deposits should be working, and going into cold storage. Now to process withdrawals automatically, you can use the same core instance as a "withdraw only" wallet and keep topping it up, by sending from your trezor into it (ideally with super low fee transactions).

---

Note: this way is not the best, but it's the simplest. It has pretty crappy privacy properties, but is operationally the most simple and you don't need to worry about dusting attacking or coin-selection gone wild.

Check out gamblingsitefinder.com for a decent list/rankings of crypto casinos. Note: I have no affiliation or interest in it, and don't even agree with all the rankings ... but it's the only uncorrupted review site I'm aware of.
yegor256
Member
**
Offline Offline

Activity: 224
Merit: 15

Creator of Zold


View Profile WWW
May 26, 2019, 10:48:27 AM
 #8

I don't think you need to have your own bitcoin core. You can just use Blockchain.com API and call it a day.

Creator of Zold / click here to buy a few!
NorwayFish (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 4


View Profile
May 26, 2019, 05:46:15 PM
 #9

What's the notification system that has to be in place for this? How do I make sure I go through every single block, 6 below the tip. Do I need some kind of record to keep track of the blocks that I've processed?

Also, what if I want my users to know that their transaction is seen by my system all the way from mempool to 6 confirmations? Should I be catching transactions in any way?
Maybe walletnotify or zmq?

Having a withdraw only core means to have a core with a few addresses that I can top up at any time and simply use sendtoaddress rpc? How can I possibly improve the privacy of this? Any way to make withdrawal transactions more anonymous for customer's security?

Are there any rules of thumb when working with walletnotify and blocknotify? Having an express instance running to process curls from it feels reeeeally awkward...

Any recommendations on how to control the bitcoin core? Should there be more than one core running simultaneously? Should there be a program that bitcoind can be run behind? Any research or guides on this topic?
RHavar
Legendary
*
Offline Offline

Activity: 2555
Merit: 1886



View Profile
May 26, 2019, 07:22:09 PM
Merited by ABCbits (1)
 #10

What's the notification system that has to be in place for this? How do I make sure I go through every single block, 6 below the tip. Do I need some kind of record to keep track of the blocks that I've processed?

You always need to design the system to work without notifications (e.g. sometimes you might miss one). So they're kind of always an added luxury. It's pretty easy to build a robust system with no notifications at all (e.g. just poll "getbestblock" every N seconds).



You don't really *need* a notification system. The since you *have* to already cover the case where you miss notifications (i.e. downtime or w/e), the notification system is just a little luxury to speed things up.


Quote
Also, what if I want my users to know that their transaction is seen by my system all the way from mempool to 6 confirmations? Should I be catching transactions in any way?
Maybe walletnotify or zmq?

What I've done is scan the mempool (hardest) and top blocks to find deposits, and put it in the database was `credited=false` and then only when it gets N depth, set `credited=true` while crediting them.  If you want to get fancy, you should also support *uncrediting* deposits (e.g. in the case of reorgs) which makes it safe to reasonably safe to accept deposits with very small amount of confirmations.

Quote
Having a withdraw only core means to have a core with a few addresses that I can top up at any time and simply use sendtoaddress rpc? How can I possibly improve the privacy of this? Any way to make withdrawal transactions more anonymous for customer's security?

If you want way better privacy properties, use a core wallet for both deposits and withdrawals. Bitcoin Core's coinselection however is really not designed for this usecase, so you'll end up paying a lot more in fees than the 2-wallet approach (since you can consolidate from receive-wallet to send-wallet with super tiny fees) but you'll require less float.

Quote
Are there any rules of thumb when working with walletnotify and blocknotify? Having an express instance running to process curls from it feels reeeeally awkward...
I'd recommend adding them as a luxury later on. You need to make sure your system works without notifications (i.e. incase it misses one). So you can start with polling every 10 seconds or something, and then use notifications to speed it up.

Quote
Any recommendations on how to control the bitcoin core? Should there be more than one core running simultaneously? Should there be a program that bitcoind can be run behind? Any research or guides on this topic?

Core has an rpc api you can use

Check out gamblingsitefinder.com for a decent list/rankings of crypto casinos. Note: I have no affiliation or interest in it, and don't even agree with all the rankings ... but it's the only uncorrupted review site I'm aware of.
NorwayFish (OP)
Newbie
*
Offline Offline

Activity: 4
Merit: 4


View Profile
May 28, 2019, 04:03:48 AM
 #11

So what's is my core was offline for a bit? It would have to download many blocks in a very short period of time, which would make my cron job to skip many blocks as a result.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!