Checking Address Balance with Bitcoin Core

[Crypto-Collector]

I have an online store and already accept CC payments and I want to start accepting crypto. I'm using the regtest to test the application and so I can't rely on third party APIs plus I feel doing that will hurt at scaling. I'm trying to set it up where each new order will have its own address and I'll have a cronjob periodically check if the address received the payment then send money to cold storage wallet as follows:

Generate address -> receive payment -> wait for confirmations -> send coins to cold storage and show confirmed payment

Currently I save the address in a database associated with the order and generate it using getnewaddress(). I'm stuck on checking the balance of said address and how many confirmations the payment has. I saw accounts looked kind of like what I was looking for but its deprecated so I cant use that. I looked into using blockchain.info JSON API but then with scaling I have to worry about request limits and if I'm trying to request information on too many addresses at once I think it will result in an error. For example, say I have like 4000 orders and the cronjob needs to check all 4000 addresses during the cronjob to get it done efficiently then the addresses that received enough confirmation switch order to paid in database and send coins to cold storage wallet off of server. Plus I'm trying to test it in regtest mode so obviously I cant use a 4rd party API. I saw listunspent() after manually adding addresses and rescanning the chain but I feel if you have thousands of orders and addresses it will be very inefficient every time the job is run and slow the server down.

What is a solution to what I'm trying to accomplish?

I'm very tech savvy and am interested in the be my own bank aspect so I'm not trying to use third party processors or use blockchain.info's wallet API.

[bob123]

Generate address -> receive payment -> wait for confirmations -> send coins to cold storage and show confirmed payment

Why don't you use the master public key to derive public keys and addresses while the private keys are in cold storage ?
That way, you would directly accept payments into your cold wallet and would save on transaction fees.

Currently I save the address in a database associated with the order and generate it using getnewaddress(). I'm stuck on checking the balance of said address and how many confirmations the payment has.

You could use walletnotify to get notified each time the state of your wallet changes (i.e. transaction received, transaction sent, first confirmation on a TX).
That'd be more efficient than having a cronjob checking the addresses each X seconds/minutes.

If you require more than 1 confirmation, you can combine this with blocknotify, to get notified each time core receives a block.
If walletnotify says that you received 1 confirmation on transaction X, you can be sure the next time blocknotify is called, this TX has one more confirmation.

[Crypto-Collector]

Why don't you use the master public key to derive public keys and addresses while the private keys are in cold storage ?
That way, you would directly accept payments into your cold wallet and would save on transaction fees.

Was most likely going to do this in deployment environment. Was just going to add extra functionality to send to single address if I wanted it stored in one address for whatever reason.

You could use walletnotify to get notified each time the state of your wallet changes (i.e. transaction received, transaction sent, first confirmation on a TX).
That'd be more efficient than having a cronjob checking the addresses each X seconds/minutes.

If I have thousands of transactions couldnt that cause performance issues if theres payments in many addresses in very short periods of time? I'm trying to build it to scale from the beginning.

[bob123]

If I have thousands of transactions couldnt that cause performance issues if theres payments in many addresses in very short periods of time? I'm trying to build it to scale from the beginning.

While i can't speak out of experience, i don't think you'll have performance issues.

walletnotify and blocknotify execute a shell script upon receiving a transaction / block.

You probably won't be able to process thousands of transactions each minute with a raspberry pi, but since the network can't handle so much itself you shouldn't get any problem with a medicore VPS / whatever you are using.

The current amount of transactions the bitcoin network can handle is at roughly 200 per minute.
Besides the fact that i doubt you alone will fill 50% of all blocks, it definitely should be possible without performance issues.

[Crypto-Collector]

So how do I check if that specific address had a transaction? After walletnotify tells the bash script to go, what methods do I use to check specific addresses vs the whole wallets balance?

[bob123]

Well, you'd call the script with the transaction id as a parameter:

Code:

walletnotify=/path/to/your/script %s

And inside of your script (shell, php, whatever..), you could call

Code:

bitcoin-cli gettransaction $TX

to get all necessary data from the transaction.

Afterwards process the data as you need and/or fill your database.


You wouldn'd directly check whether address X received a transaction, but you'd call that script upon receiving any transaction.
Then using gettransaction, you'd get all necessary information (receiving address, amount, etc.. ) for further processing.

[Crypto-Collector]

Well, you'd call the script with the transaction id as a parameter:

Code:

walletnotify=/path/to/your/script %s

And inside of your script (shell, php, whatever..), you could call

Code:

bitcoin-cli gettransaction $TX

to get all necessary data from the transaction.

Afterwards process the data as you need and/or fill your database.


You wouldn'd directly check whether address X received a transaction, but you'd call that script upon receiving any transaction.
Then using gettransaction, you'd get all necessary information (receiving address, amount, etc.. ) for further processing.

So when walletnotify runs the script where do I get the transaction ID from to use $TX in gettransaction? What happens if theres multiple transactions at once? Will it just run multiple instances or will $tx be returned as an array?

[bob123]

So when walletnotify runs the script where do I get the transaction ID from to use $TX in gettransaction? What happens if theres multiple transactions at once? Will it just run multiple instances or will $tx be returned as an array?

It is being handed over to the script as an argument when calling it (%s is the TXID):

Code:

walletnotify=/path/to/your/script %s

Then it depends on the scripting language how to access it.

For a shell script, you would use:

Code:

$txid = $1

For a php script:

Code:

$txid = $argv[1];

walletnotify will run each time:

• you receive a transaction associated with your wallet
• you send a transaction
• an incoming transaction receives its first confirmation

For X incoming transactions, it will be run 2*X times in total.

Note that you can set up core to be run on the testnet, to see how it exactly behaves and how to make use of it most efficiently.



P.s.
I appreciate that you don't want to use some 3rd party services and want to be in full control over the funds - basically how it is supposed to be.
If you have any concerns or questions, even going further than just setting this up, feel free to ask.
There are a lot of people here who appreciate that engagement and would help you with your questions / concerns.

[legendster]

Why don't you use the master public key to derive public keys and addresses while the private keys are in cold storage ?
That way, you would directly accept payments into your cold wallet and would save on transaction fees.

From a privacy standpoint, is that the best thing to do? If it were my business I'd keep my cold storage out of the public eye and perhaps only reveal it to authorities for taxation requirements.

Otherwise, there can be phishing attempts on the admin email with the cold storage address included in an official-looking email. People have fallen victim to much simpler fraud attempts than this.

[bob123]

From a privacy standpoint, is that the best thing to do? If it were my business I'd keep my cold storage out of the public eye and perhaps only reveal it to authorities for taxation requirements.

The xpub would be stored on the web server handling the payment requests.
It wouldn't be publicly available to customer / visitors of the site.

You'd be generating a new receiving address for each customer using this xpub. While the private keys are in cold storage.

This does not have any implications on the privacy compared to receiving coins to a hot wallet and afterwards sending it to the cold storage.


Or what exactly was your concern ? Maybe i misunderstood it ?

Otherwise, there can be phishing attempts on the admin email with the cold storage address included in an official-looking email. People have fallen victim to much simpler fraud attempts than this.

The xpub is not an address, but is used to derive public keys (and therefore addresses).
However, i doubt the admint of a shop which accepts crypto could even fall for such a phishing attempt. I mean.. who would be the sender ?
There is no 'official authority' who could request anything. If you are the admin, you are the admin. No one else should ask you for anything.

[legendster]

Or what exactly was your concern ? Maybe i misunderstood it ?

No, I get it that the public keys can be known publicly without giving away anything. But can't the sender figure out the address from the tx id? That would expose the cold wallet address itself and make it prone to being monitored.

I can't remember when exactly but I have come across multiple wallet watching websites where anyone can track a wallet's transactions. Hell, if someone's that desperate they can do with any blockchain explorer.

All I am saying is perhaps accepting payments directly into the cold wallet could not be the wisest thing to do.

[bob123]

But can't the sender figure out the address from the tx id? That would expose the cold wallet address itself and make it prone to being monitored.

He could.

But the address is only used for one single transaction.

Note that a public key is not an address. The address is being 'derived' from the public key.
And with a master public key (xpub) you are able to derive multiple public keys using a counter (and therefore multiple addresses).

While the master private key (xpriv) would be stored in cold storage, you could derive a new address for each transaction on the online webserver using the xpub.


A customer would not be able to determine the full balance, simply because he only knows that one address which is used for this single transaction.

All I am saying is perhaps accepting payments directly into the cold wallet could not be the wisest thing to do.

From a privacy point of view it doesn't matter where your private keys are stored. There is absolutely no difference.
But from a security point of view, it definitely is favorable to store them in cold storage.

I don't see a single disadvantage in directly receiving coins to the cold storage. Except maybe for the fact that it takes a bit longer to access them in case you want to spend them.

[btctaipei]

I have an online store and already accept CC payments and I want to start accepting crypto. I'm using the regtest to test the application and so I can't rely on third party APIs plus I feel doing that will hurt at scaling. I'm trying to set it up where each new order will have its own address and I'll have a cronjob periodically check if the address received the payment then send money to cold storage wallet as follows:

Generate address -> receive payment -> wait for confirmations -> send coins to cold storage and show confirmed payment

Currently I save the address in a database associated with the order and generate it using getnewaddress(). I'm stuck on checking the balance of said address and how many confirmations the payment has. I saw accounts looked kind of like what I was looking for but its deprecated so I cant use that. I looked into using blockchain.info JSON API but then with scaling I have to worry about request limits and if I'm trying to request information on too many addresses at once I think it will result in an error. For example, say I have like 4000 orders and the cronjob needs to check all 4000 addresses during the cronjob to get it done efficiently then the addresses that received enough confirmation switch order to paid in database and send coins to cold storage wallet off of server. Plus I'm trying to test it in regtest mode so obviously I cant use a 4rd party API. I saw listunspent() after manually adding addresses and rescanning the chain but I feel if you have thousands of orders and addresses it will be very inefficient every time the job is run and slow the server down.

What is a solution to what I'm trying to accomplish?

I'm very tech savvy and am interested in the be my own bank aspect so I'm not trying to use third party processors or use blockchain.info's wallet API.

I wish there were more people like you (not using bitpay) to protect the privacy of your customers.  You are doing similar to How I take VPN monthly payments (see a recent jump in number of user using this type of payment option), and just to share my experience - (correct me please if any of those steps were deemed sub-optimal):

1) with offline and isolated PHYSICAL, non virtualized (Zen, vmware, etc) computer (with much older hardware, say Pentium III no IPMI/other build in BMI chips to scrape your memory during run-time) pre-generate pool of cold private keys address, using dumpprivkey(getnewaddress()). I keep a changing pool of address on live (or more exposed) system and keep no logs for privacy, and those address are only good for receive 1 payment for 1 instance only.
    - example:  getnewaddress "" bech32 to create bc1 prefixed native-segwit receive address to let your customer save fees.. assume you have core running.
    - Also create like 15% of those payment address in 3xx p2sh-segwit format simple getnewaddress() "" p2sh-segwit, then dumpprivkey() those said address and guard it / safe keeping those off-line.
    - Note current block count with getblockcount() for faster load for preparing your bitcoin core wallet for manual coinjoin transaction in step 5
    - After generated btc payment address/private key in safe place (see step 5)    
2) call API to generate QR bitmaps (zxing, etc.. a little off topic so I can't say more)
3) check those pre-generated balance using getreceivedbyaddress bc1xxxxxxxxxx etc.
4) No need to send to cold storage.  you pre-generated those private keys they are in floppy disks (or what ever else you choose.. automated Type 316 stainless steel plate tele-type would be my choice)
5) when you need to pay hosting or in my case co-location fees and need fiat, send it to chipmixer if its large (not feasible in my case because payments are normally 8 - 100 usd and chipmixer only do 0.001 btc). Or, if you want to do it yourself,
   a) on a different computer, with freshly install OS importprivkey() those address where in step 3 have getreceivedbyaddress(addr)>0. make sure to importprivkey "privatekey" false to keep your system away from rescanning private key.
   b) then rescanblockchain (Chain height from Step 1) to ready your wallet for manual coin-join / chipmix hybrid mixing.
6) create larger coinjoin transactions (I'm still writing software to automate RPC call with createrawtransaction() to a pool of chipmixer inputs.  You can tor and than hit chipmixer and get a pool say 32 session keys,
    a) now you have 32 output address - Note: Don't wait long between this step and complete your coin join transaction, because chipmixer only good for limited time.. I usually drag it out to 24 to save on tx fees (core is good at estimate this or to be sure visit bitcoinfees.earn.com i.e.,)
    b) each one with base 2 power of kbtc and have say around 64 inputs.  So you have output to each of those 32 as 0.001, 0.002 ... 0.00x; where x^2 represents the chip size you want
    c) enjoy lower fees with bc1 / p2sh-segwit enabled receive addresses
I manually create and select mgmt enabled).  I found p2sh-segwit address receives payment confirmation than legacy. bc1 comes in at about the same speed as legacy but with much reduced size and higher satoshi/byte it generally arrives as quick as legacy payment input address.
7) face2face and cash in those chips... I usually keep like 100 or so chipmixer private keys so I can cash out exact amount during LBC face-2-face meetups to cash out NTD fiats.  I usually PGP a reputable bitcoin dealer from HK/Korea/ in LBC and they will stop regularly to help give me cash at spot price.

Note: Help on 6 on createrawtransaction() bc1 segwit-enabled coin-join transaction is much appreciated. Anyone could provide me with example / sample script would save me about 2 hrs each week to make this transaction bulletproof manually.

[Crypto-Collector]

So I just thought about it, what would I do in the case that my node went down and wasnt running for all the new transactions or blocks? Would it just skip over those notifications or handle them when the node restarts?

When I use listtransactions does it list all transactions in the block or just transactions in that block for my addresses associated with my xpub?

[bob123]

So I just thought about it, what would I do in the case that my node went down and wasnt running for all the new transactions or blocks? Would it just skip over those notifications or handle them when the node restarts?

Those scripts would be run whenever your node receive a block / transaction.

If your node is down, it doesn't receive anything.
When you start it up again, and it starts catching up those missing blocks, all received blocks will call blocknotify and all transactions to your wallet received will call walletnotify.

It does not skip anything. It resumes operation after the node is up again.

Also, when I use listtransactions does it list all transactions in the block or just transactions in that block for my addresses associated with my xpub?

listtransactions gives you the last X transaction sent to your wallet.

[Crypto-Collector]

With walletnotify I'd just get a new transaction and I can use the tx info to determine which address it goes towards and update in mysql the number of transactions for that order, however I want to handle cases where users need to send multiple transactions to the address for example if the first transaction wasnt enough to pay the whole tab. II was thinking use blocknotify then search my database for all orders with less than required confirmations and have orders associated with the address for the order. With multiple transactions, using one confirmations field in the database doesnt work so whats a strategy I could use to keep track of all the different transactions for each address/order? I'd have to keep track of all the transactions confirmations in a separate table and associate them with the order ID in the table keeping track of orders wouldnt I?

I'm kind of having trouble coming up with how to handle block notify since it sends block hash instead of tx hash.

[bob123]

With walletnotify I'd just get a new transaction and I can use the tx info to determine which address it goes towards and update in mysql the number of transactions for that order, however I want to handle cases where users need to send multiple transactions to the address for example if the first transaction wasnt enough to pay the whole tab. II was thinking use blocknotify then search my database for all orders with less than required confirmations and have orders associated with the address for the order. With multiple transactions, using one confirmations field in the database doesnt work so whats a strategy I could use to keep track of all the different transactions for each address/order? I'd have to keep track of all the transactions confirmations in a separate table and associate them with the order ID in the table keeping track of orders wouldnt I?

You could associate one order to one address and then keep track of all transactions to this order. You could have tables like this:
order <-> address (1:1)
address <-> transactions (1:n)
transaction <-> confirmation (1:n)

I think this should work (not absolutely sure about any edge cases).

I'm kind of having trouble coming up with how to handle block notify since it sends block hash instead of tx hash.

You should use blocknotify only to count confirmations.

Better use walletnotify for all incoming transactions -> assigning in your database etc. and for the first confirmation.
Then increase the count of the confirmations the next time blocknotify is called. I don't think you need blocknotify for anything else.

Or am i missing something ?

[Crypto-Collector]

Ok, so I wrote PHP scripts for blocknotify and wallet notify along with the needed tables. So when I generate new blocks and send bitcoin to a generated address I can see in the regtest error log that the scripts arent working:

Code:

AddToWallet 3449dc56136cd9d5cc74a639f2a08ab7141ee0fdcabf41672fcb3e41975eb5cf  new
runCommand error: system(/home/bitcoin/blocknotify.php 40ecf86e83735ca8d1b8d95cdcd54bdd94f885efcd291374bb666d369ccdcf01) returned 65280

I see that for every new block generated and also the same for the script for walletnotify. My table is empty so I dont think the script is even running. I set walletnotify and blocknotify in bitcoin.conf and just start the daemon using ./bitcoind -regtest -daemon

Is there a way that I'm supposed to type it differently because I'm using PHP? I'm assuming its because .php files are not executables. I could write an executable but right now I'm basically just making a proof of concept in PHP because its faster to do so. Also, how would I check for error logs for my PHP scripts if theres syntax errors and such? I'm assuming they wont be in the same place as PHP error logs for site code because its being run from bitcoind rather than my server software. I'm using Debian Linux.

This is my bitcoin.conf

Code:

rpcuser=user
rpcpassword=password
regtest=1
blocknotify=/home/bitcoin/blocknotify.php %s
walletnotify=/home/bitcoin/walletnotify.php %s

I've tried adding php -f before the location of the file but still same error.

[bob123]

Your config looks right to me.

You have basically two options to call your php script.

1)
Like you are trying it to do:

Code:

walletnotify=/home/bitcoin/walletnotify.php %s

This requires your php file to be executable (chmod +x) and a shebang pointing to the php executable in the first line of your script (e.g.  #!/usr/bin/php)


2)

Code:

walletnotify=/usr/bin/php /home/bitcoin/walletnotify.php %s

This does not require the file to be executable or to contain a shebang in the first line.


Note that you have to replace /usr/bin/php with your actual path to the php executable, which can be found using this command:

Code:

which php

[Crypto-Collector]

So if I have errors thrown in my script where will I find the error logs? It should be somewhere different because bitcoind is running it and not my web server right?

I added the php executable to my config file and still getting an error

Code:

runCommand error: system(/usr/bin/php /home/bitcoin/script.php d5cab12159ebf0f9df963b972e6c9323477ee822c68e44dc346c20ffa) returned 65280

My table is still empty in my database so the script didnt run (or work). What behavior will it do if the script throws an error?