As a developer, what's the best way to accept BTC without using third-parties

[sidhujag]

You cAn port any shopping cart software that uses bitcoin to talk to a local daemon instead infact that baron one does this but its not a shopping cart plugin. Depends on the use cases

[sidhujag]

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.

Code:

1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?

its not trustless

[gweedo]

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.

Code:

1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?

its not trustless

Bitcoinj isn't trustless either

[contactlight]

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.

Code:

1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?

its not trustless

Bitcoinj isn't trustless either

Yes, it is not. It isn't unreasonable to trust some full-nodes you are running, though.

[sidhujag]

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.

Code:

1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?

its not trustless

Bitcoinj isn't trustless either

How? Does it force u to connect to an online api? If not it is trustless in the context we care about

[gweedo]

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.

Code:

1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?

its not trustless

Bitcoinj isn't trustless either

How? Does it force u to connect to an online api? If not it is trustless in the context we care about

SPV clients are not suppose to be trustless so it isn't trustless in any context, it is convenience. Security is given up for convenience.

Unless you want to run a full node in front of the bitcoinj library and force bitcoinj to only connect to that. Then it is trustless.

Yes, it is not. It isn't unreasonable to trust some full-nodes you are running, though.

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.

[gmaxwell]

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.

"Scalability" is ultimately limited by the Bitcoin network, which-- by definition-- a Bitcoin full node can handle.  Interposing a 3rd party does not necessarily improve scalability-- it likely reduces it, especially since you're talking about unpaid services, so there is nothing to fund your utilization-- and using a third party service necessarily reduces security, more so than SPV, and necessarily adds additional points of failure: the reliability of third party blockchain services has been very low.

I'm sad to see that no one seems to have noticed the Baron link I gave above. It's a full order processing solution based on a local full node. It deserves more attention.

[contactlight]

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.

"Scalability" is ultimately limited by the Bitcoin network, which-- by definition-- a Bitcoin full node can handle.  Interposing a 3rd party does not necessarily improve scalability-- it likely reduces it, especially since you're talking about unpaid services, so there is nothing to fund your utilization-- and using a third party service necessarily reduces security, more so than SPV, and necessarily adds additional points of failure: the reliability of third party blockchain services has been very low.

I'm sad to see that no one seems to have noticed the Baron link I gave above. It's a full order processing solution based on a local full node. It deserves more attention.

Baron looks very interesting. Looking into it right now.

[gweedo]

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.

"Scalability" is ultimately limited by the Bitcoin network, which-- by definition-- a Bitcoin full node can handle.  Interposing a 3rd party does not necessarily improve scalability-- it likely reduces it, especially since you're talking about unpaid services, so there is nothing to fund your utilization-- and using a third party service necessarily reduces security, more so than SPV, and necessarily adds additional points of failure: the reliability of third party blockchain services has been very low.

I'm sad to see that no one seems to have noticed the Baron link I gave above. It's a full order processing solution based on a local full node. It deserves more attention.

Your right in one aspect and you are wrong in another aspect. Scalability is not ultimately limited by the bitcoin network, but also by connections/users. Bitcoin-core I think is becoming less and less used by companies just because it is so heavy and not scalable. It really tops out at only a few 1000 connections, even raising the file limit from 1024 (default for most linux systems) to higher and higher numbers doesn't help anymore. Then you probably need to run a bitcoin proxy as I have been using, that manages connections and limits. This causes another problem which we have seen time and time again, that if you do a withdraw and don't say it could take 12-24 hours to process, and it doesn't show up on the network in a few seconds. The idea that company is now a scammer, is what consumes the mind. So now you are not able to use the the programical instant withdraw that only the bitcoin network as a bootstrap startup without risking your reputation.

I believe we are failing as a community in many ways about this, I recommend 3rd party apis because that is the only solution currently. There are no other full nodes that can handle this unless you enter the enterprise stage. I don't think anyone wants to do that unless they have to. I think as btcd, tosh.io, and bitcoin-core grow we will see many forks addressing this problem. Until then there is really no way to be trustless and scale.

[gmaxwell]

Scalability is not ultimately limited by the bitcoin network, but also by connections/users. Bitcoin-core I think is becoming less and less used by companies just because it is so heavy and not scalable. It really tops out at only a few 1000 connections, even raising the file limit from 1024 (default for most linux systems) to higher and higher numbers doesn't help anymore.

It intentionally doesn't support more concurrent connections at this time. While it would be easy to make it do so I am not aware of a single use case for which supporting more would make sense.  Nothing of relevance to this thread should result in running many concurrent connections to the daemon. If you'd care to state an application, I'd be interested in hearing it.

(Considering that the entire network can only support on the other of 10tx per second, I am at a complete loss as to what aspect of customer invoicing would require you to run a thousand concurrent connections to a single Bitcoin daemon.)

This causes another problem which we have seen time and time again, that if you do a withdraw and don't say it could take 12-24 hours to process, and it doesn't show up on the network in a few seconds. The idea that company is now a scammer, is what consumes the mind. So now you are not able to use the the programical instant withdraw that only the bitcoin network as a bootstrap startup without risking your reputation.

Can you retry saying this, I'm unable to parse your English.

I believe we are failing as a community in many ways about this, I recommend 3rd party apis because that is the only solution currently. There are no other full nodes that can handle this unless you enter the enterprise stage.

You've failed to state an actual reason here. Can you try to clearly specify particular problems with concrete details. Simply saying "it doesn't scale" is not helpful.

[sidhujag]

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.

Code:

1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?

its not trustless

Bitcoinj isn't trustless either

How? Does it force u to connect to an online api? If not it is trustless in the context we care about

SPV clients are not suppose to be trustless so it isn't trustless in any context, it is convenience. Security is given up for convenience.

Unless you want to run a full node in front of the bitcoinj library and force bitcoinj to only connect to that. Then it is trustless.

Yes, it is not. It isn't unreasonable to trust some full-nodes you are running, though.

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

[gweedo]

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.

[contactlight]

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.

Bitcoinj actually supports full verification at this point given you use the PostgresFullPrunedBlockStore or MySQLFullPrunedBlockStore as your BlockStore.

[gweedo]

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.

Bitcoinj actually supports full verification at this point given you use the PostgresFullPrunedBlockStore or MySQLFullPrunedBlockStore as your BlockStore.

Bitcoinj has two modes it has SPV and full verification. I clearly stated the SPV mode in all my post.

To be honest though I don't think bitcoinj's full verification passes all the test. I don't know haven't really looked at the project much in the last year.

[contactlight]

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.

Bitcoinj actually supports full verification at this point given you use the PostgresFullPrunedBlockStore or MySQLFullPrunedBlockStore as your BlockStore.

Bitcoinj has two modes it has SPV and full verification. I clearly stated the SPV mode in all my post.

To be honest though I don't think bitcoinj's full verification passes all the test. I don't know haven't really looked at the project much in the last year.

Yeah, I understand that the SPV mode is different and that you were referring to that. No problems there.

As far as I know, BitcoinJ's full verification passes all the tests. I know it used to not but now it does. The initial syncing process is very slow, though. After reaching the Block #200000, it can only process less than two blocks per second, which is disconcerting.

[myusuario]

if you dont want to trust to anyone, you can have a full node and database to handle your payment system, the bitcoin core api have all you need, https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list, getnewaddress, getreceivedbyaddress, listreceivedbyaddress, listsinceblock etc

[Nicolas Dorier]

Sometimes you have to balance best practice and being pragmatic.

Use one address per customer, and sweep money to cold storage times to times. (With one cold storage per customer)
When you spend your coins, try to use only coins you got from one customer, and do not mix with others. (the manual coin selection of bitcoinqt is really good for that)
It stays relatively simple, and good enough privacy wise. (For me)