Bitcoin Forum
May 13, 2024, 11:56:06 AM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: 1 2 [All]
  Print  
Author Topic: As a developer, what's the best way to accept BTC without using third-parties  (Read 3917 times)
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 17, 2014, 09:11:23 PM
 #1

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user
1715601366
Hero Member
*
Offline Offline

Posts: 1715601366

View Profile Personal Message (Offline)

Ignore
1715601366
Reply with quote  #2

1715601366
Report to moderator
1715601366
Hero Member
*
Offline Offline

Posts: 1715601366

View Profile Personal Message (Offline)

Ignore
1715601366
Reply with quote  #2

1715601366
Report to moderator
1715601366
Hero Member
*
Offline Offline

Posts: 1715601366

View Profile Personal Message (Offline)

Ignore
1715601366
Reply with quote  #2

1715601366
Report to moderator
Each block is stacked on top of the previous one. Adding another block to the top makes all lower blocks more difficult to remove: there is more "weight" above each block. A transaction in a block 6 blocks deep (6 confirmations) will be very difficult to remove.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1715601366
Hero Member
*
Offline Offline

Posts: 1715601366

View Profile Personal Message (Offline)

Ignore
1715601366
Reply with quote  #2

1715601366
Report to moderator
jonald_fyookball
Legendary
*
Offline Offline

Activity: 1302
Merit: 1004


Core dev leaves me neg feedback #abuse #political


View Profile
November 17, 2014, 09:24:58 PM
 #2

I dont think you need to worry about peers, blocks, unspent outputs, etc.

Just generate a key/address pair , give the public address to your customer.
What's wrong with that?

contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 17, 2014, 09:41:41 PM
 #3

I dont think you need to worry about peers, blocks, unspent outputs, etc.

Just generate a key/address pair , give the public address to your customer.
What's wrong with that?

I need to be able to programmatically check for the incoming transactions to that address and sweep all the unspent outputs. I am already planning on generating a key/address pair and give the public address to the customer.

I just need to have a reliable way to be notified of incoming transactions to that address. My current Bitcoinj setup notifies me and sweeps the transactions when notified. However, this only happens if my service is up and running when the user makes the transaction. If the service is down, the transactions go unnoticed and I don't get notified during the blockchain sync process.

I was using BlockChain and PostgresFullPrunedBlockStore, and I noticed that it wasn't populating the unspent outputs table. This was probably due to me using BlockChain instead of FullPrunedBlockChain. I refactored that into my code and now I am syncing again. Let's see what happens with that.
jonald_fyookball
Legendary
*
Offline Offline

Activity: 1302
Merit: 1004


Core dev leaves me neg feedback #abuse #political


View Profile
November 17, 2014, 11:29:45 PM
 #4

best practice is not to re use addresses.  You get one transaction per address, and then you can sweep it.

contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 18, 2014, 01:46:46 AM
 #5

best practice is not to re use addresses.  You get one transaction per address, and then you can sweep it.

Thanks but that's what I will be doing anyway. After sweeping, the address will be taken out of the "active_addresses" list and won't be checked ever again. This detail however doesn't make downtime any better. When the Bitcoin payment processing daemon is down, I won't be notified of the new transactions. BitcoinJ, when catching up with the blockchain, refuses to notify me of my transactions.
jonald_fyookball
Legendary
*
Offline Offline

Activity: 1302
Merit: 1004


Core dev leaves me neg feedback #abuse #political


View Profile
November 18, 2014, 01:53:40 AM
 #6

so you do need to worry about outputs then.  Will you need to write a "balance check" function from scratch?
I assume you are running a full node.

contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 18, 2014, 01:56:40 AM
 #7

so you do need to worry about outputs then.  Will you need to write a "balance check" function from scratch?
I assume you are running a full node.

I'm debating if I should run a full node or if I should code a "pretend" full node. Either way, you can assume that I can store the entire blockchain. I just don't want to store all the unspent outputs, but only the ones for my addresses. Because indexing unspent outputs for all addresses can get tedious and require up to 320GB of disk space.

I will likely need to write a balance check function from scratch. I was planning on using "PostgresFullPrunedBlockStore" from BitcoinJ and call calculateBalanceForAddress(Address) but they don't seem to have implemented this class well.
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 20, 2014, 03:57:49 AM
 #8

Generate keys offline, put them in database table, check using an api, then you don't have worry about sweeping them.

Can't be any easier.
jonald_fyookball
Legendary
*
Offline Offline

Activity: 1302
Merit: 1004


Core dev leaves me neg feedback #abuse #political


View Profile
November 20, 2014, 06:34:05 AM
 #9

Generate keys offline, put them in database table, check using an api, then you don't have worry about sweeping them.

Can't be any easier.

I think he wants to check without relying on an api to some third party service.
He want to validate the balances on his own by parsing the blockchain.

justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1009



View Profile
November 20, 2014, 07:33:02 AM
 #10

Generate keys offline, put them in database table, check using an api, then you don't have worry about sweeping them.

Can't be any easier.

I think he wants to check without relying on an api to some third party service.
He want to validate the balances on his own by parsing the blockchain.
All the decent full node implementations have a good API that you can use locally.

API does not imply a third party service.
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 20, 2014, 03:35:06 PM
 #11

Generate keys offline, put them in database table, check using an api, then you don't have worry about sweeping them.

Can't be any easier.

I think he wants to check without relying on an api to some third party service.
He want to validate the balances on his own by parsing the blockchain.
All the decent full node implementations have a good API that you can use locally.

API does not imply a third party service.

I do not want to rely on a separate process. I want my daemon to be self reliant. That's why I have been focusing on BitcoinJ.

My implementation is now a SPV. Instead of keeping track of all unspent outputs, it checks for transactions relevant to my addresses on each block and sweeps them. It seems to be working now but I need to test it further.
justusranvier
Legendary
*
Offline Offline

Activity: 1400
Merit: 1009



View Profile
November 20, 2014, 03:50:25 PM
 #12

I do not want to rely on a separate process. I want my daemon to be self reliant.
You should have mentioned up front that intentionally getting hax0red was part of the design requirements.
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 20, 2014, 04:04:15 PM
 #13

I do not want to rely on a separate process. I want my daemon to be self reliant.
You should have mentioned up front that intentionally getting hax0red was part of the design requirements.

Why do you think that this design would lead to a breach? I'm not going to be running everything else on it. It will just be a payment processor connected to the network and the database, keeping track of balances and transactions.

One issue I can see with it would be that we wouldn't run validation on the inputs. Receiving the new block from multiple nodes should fix that, though. They wouldn't have relayed the blocks if the transactions in it weren't valid. So I need 1 confirmation.
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 20, 2014, 09:28:28 PM
 #14

I do not want to rely on a separate process. I want my daemon to be self reliant.
You should have mentioned up front that intentionally getting hax0red was part of the design requirements.

Some people don't have any idea how bitcoin works but they use their programming certificates as "I know everything yet I am still asking". Now I know why I left this forum for while.
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 20, 2014, 09:31:37 PM
 #15

I do not want to rely on a separate process. I want my daemon to be self reliant.
You should have mentioned up front that intentionally getting hax0red was part of the design requirements.

Some people don't have any idea how bitcoin works but they use their programming certificates as "I know everything yet I am still asking". Now I know why I left this forum for while.

Not sure if that is directed towards me but if it is, that's very uncalled for and childish. You can avoid this drama by not looking at threads with questions and moving on your way on your high horse.
gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 4172
Merit: 8420



View Profile WWW
November 21, 2014, 10:44:14 PM
 #16

Have you looked into Baron? https://github.com/slickage/baron
nuno12345
Sr. Member
****
Offline Offline

Activity: 276
Merit: 284


View Profile
November 21, 2014, 10:55:26 PM
 #17

If you can run a daemon you dont need to rely on any third partys.
You can create an account for each address in your daemon and use getbalance account to check that specific address balance.
Addresses need to be created on that daemon or imported and an account needs to be assign to that address.
dabura667
Sr. Member
****
Offline Offline

Activity: 475
Merit: 252


View Profile
November 22, 2014, 08:54:49 AM
 #18

If you can run a daemon you dont need to rely on any third partys.
You can create an account for each address in your daemon and use getbalance account to check that specific address balance.
Addresses need to be created on that daemon or imported and an account needs to be assign to that address.

Then once you have 100000000000000 keys on your wallet, you'll need 30 GB for the blockchain, and 500TB for your wallet.dat file... lol

My Tip Address:
1DXcHTJS2DJ3xDoxw22wCt11FeAsgfzdBU
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 22, 2014, 05:44:09 PM
 #19

If you can run a daemon you dont need to rely on any third partys.
You can create an account for each address in your daemon and use getbalance account to check that specific address balance.
Addresses need to be created on that daemon or imported and an account needs to be assign to that address.

Then once you have 100000000000000 keys on your wallet, you'll need 30 GB for the blockchain, and 500TB for your wallet.dat file... lol

Someone here that actually has a wallet with over 2 million addresses generated. It actually doesn't take up that much space. Plus you can use tools to remove private keys from your wallet.
BitcoinExchangeIndia.com
Sr. Member
****
Offline Offline

Activity: 311
Merit: 264


View Profile
November 22, 2014, 05:52:48 PM
 #20

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?

sidhujag
Legendary
*
Offline Offline

Activity: 2044
Merit: 1005


View Profile
November 22, 2014, 05:55:57 PM
 #21

You cAn port any shopping cart software that uses bitcoin to talk to a local daemon instead infact that baron one does this but its not a shopping cart plugin. Depends on the use cases
sidhujag
Legendary
*
Offline Offline

Activity: 2044
Merit: 1005


View Profile
November 22, 2014, 05:58:23 PM
 #22

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?
its not trustless
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 22, 2014, 06:23:26 PM
 #23

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?
its not trustless

Bitcoinj isn't trustless either Wink
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 22, 2014, 11:35:20 PM
 #24

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?
its not trustless

Bitcoinj isn't trustless either Wink

Yes, it is not. It isn't unreasonable to trust some full-nodes you are running, though.
sidhujag
Legendary
*
Offline Offline

Activity: 2044
Merit: 1005


View Profile
November 23, 2014, 06:20:11 AM
 #25

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?
its not trustless

Bitcoinj isn't trustless either Wink
How? Does it force u to connect to an online api? If not it is trustless in the context we care about
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 23, 2014, 08:58:55 PM
 #26

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?
its not trustless

Bitcoinj isn't trustless either Wink
How? Does it force u to connect to an online api? If not it is trustless in the context we care about

SPV clients are not suppose to be trustless so it isn't trustless in any context, it is convenience. Security is given up for convenience.

Unless you want to run a full node in front of the bitcoinj library and force bitcoinj to only connect to that. Then it is trustless.

Yes, it is not. It isn't unreasonable to trust some full-nodes you are running, though.

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.
gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 4172
Merit: 8420



View Profile WWW
November 23, 2014, 11:12:34 PM
 #27

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.
"Scalability" is ultimately limited by the Bitcoin network, which-- by definition-- a Bitcoin full node can handle.  Interposing a 3rd party does not necessarily improve scalability-- it likely reduces it, especially since you're talking about unpaid services, so there is nothing to fund your utilization-- and using a third party service necessarily reduces security, more so than SPV, and necessarily adds additional points of failure: the reliability of third party blockchain services has been very low.

I'm sad to see that no one seems to have noticed the Baron link I gave above. It's a full order processing solution based on a local full node. It deserves more attention.
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 24, 2014, 12:15:28 AM
 #28

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.
"Scalability" is ultimately limited by the Bitcoin network, which-- by definition-- a Bitcoin full node can handle.  Interposing a 3rd party does not necessarily improve scalability-- it likely reduces it, especially since you're talking about unpaid services, so there is nothing to fund your utilization-- and using a third party service necessarily reduces security, more so than SPV, and necessarily adds additional points of failure: the reliability of third party blockchain services has been very low.

I'm sad to see that no one seems to have noticed the Baron link I gave above. It's a full order processing solution based on a local full node. It deserves more attention.

Baron looks very interesting. Looking into it right now.
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 24, 2014, 02:31:18 AM
 #29

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.
"Scalability" is ultimately limited by the Bitcoin network, which-- by definition-- a Bitcoin full node can handle.  Interposing a 3rd party does not necessarily improve scalability-- it likely reduces it, especially since you're talking about unpaid services, so there is nothing to fund your utilization-- and using a third party service necessarily reduces security, more so than SPV, and necessarily adds additional points of failure: the reliability of third party blockchain services has been very low.

I'm sad to see that no one seems to have noticed the Baron link I gave above. It's a full order processing solution based on a local full node. It deserves more attention.

Your right in one aspect and you are wrong in another aspect. Scalability is not ultimately limited by the bitcoin network, but also by connections/users. Bitcoin-core I think is becoming less and less used by companies just because it is so heavy and not scalable. It really tops out at only a few 1000 connections, even raising the file limit from 1024 (default for most linux systems) to higher and higher numbers doesn't help anymore. Then you probably need to run a bitcoin proxy as I have been using, that manages connections and limits. This causes another problem which we have seen time and time again, that if you do a withdraw and don't say it could take 12-24 hours to process, and it doesn't show up on the network in a few seconds. The idea that company is now a scammer, is what consumes the mind. So now you are not able to use the the programical instant withdraw that only the bitcoin network as a bootstrap startup without risking your reputation.

I believe we are failing as a community in many ways about this, I recommend 3rd party apis because that is the only solution currently. There are no other full nodes that can handle this unless you enter the enterprise stage. I don't think anyone wants to do that unless they have to. I think as btcd, tosh.io, and bitcoin-core grow we will see many forks addressing this problem. Until then there is really no way to be trustless and scale.
gmaxwell
Moderator
Legendary
*
expert
Offline Offline

Activity: 4172
Merit: 8420



View Profile WWW
November 24, 2014, 03:03:36 AM
 #30

Scalability is not ultimately limited by the bitcoin network, but also by connections/users. Bitcoin-core I think is becoming less and less used by companies just because it is so heavy and not scalable. It really tops out at only a few 1000 connections, even raising the file limit from 1024 (default for most linux systems) to higher and higher numbers doesn't help anymore.
It intentionally doesn't support more concurrent connections at this time. While it would be easy to make it do so I am not aware of a single use case for which supporting more would make sense.  Nothing of relevance to this thread should result in running many concurrent connections to the daemon. If you'd care to state an application, I'd be interested in hearing it.

(Considering that the entire network can only support on the other of 10tx per second, I am at a complete loss as to what aspect of customer invoicing would require you to run a thousand concurrent connections to a single Bitcoin daemon.)

Quote
This causes another problem which we have seen time and time again, that if you do a withdraw and don't say it could take 12-24 hours to process, and it doesn't show up on the network in a few seconds. The idea that company is now a scammer, is what consumes the mind. So now you are not able to use the the programical instant withdraw that only the bitcoin network as a bootstrap startup without risking your reputation.
Can you retry saying this, I'm unable to parse your English.

Quote
I believe we are failing as a community in many ways about this, I recommend 3rd party apis because that is the only solution currently. There are no other full nodes that can handle this unless you enter the enterprise stage.
You've failed to state an actual reason here. Can you try to clearly specify particular problems with concrete details. Simply saying "it doesn't scale" is not helpful.
sidhujag
Legendary
*
Offline Offline

Activity: 2044
Merit: 1005


View Profile
November 24, 2014, 05:11:21 AM
 #31

Hi,

I am a developer with a lot of experience in C++, Java and Python. I am familiar with BitcoinJ and similar frameworks that help you interact with the Bitcoin network.

I am trying to accept Bitcoin payments for a project and I would like to avoid using third-party services such as Blockchain's Payment API, Coinbase and BitPay.

I'm thinking of working with "btcd" or "Bitcoinj". I have a prototype that uses the PostgresFullPrunedBlockStore. It works perfectly except when the service is down for a while. It misses transactions and syncing the blockchain doesn't seem to help. I end up not getting notified of "relevant" transactions that occurred when my service was down, despite syncing the blockchain.

I do realize that I can easily interact with "bitcoind" using RPC and add new addresses but I don't want to rely on bitcoind for that. I'm thinking of potential scalability issues.

What do you think would be the right approach here? I'm hoping to come up with a generic solution so that I can open-source it and other people can integrate it into their own systems.

I have one potential solution in mind. Let me know if you see any problems with this.
Code:
1- Join the network, wait for new blocks from peers
2- Loop indefinitely and wait for new blocks from peers. Receive block N:
  2a- Look at the block, check the transactions included in the block, are they relevant?
  2b- If they are relevant, note the unspent output and store it (making the assumption that its inputs are valid, if they weren't, it wouldn't be included in the block and propagated? Is this a safe assumption to make?)
  2c- Sweep the address and send it to cold wallet
  2d- Increment the balance of the user

What's wrong in using a 3rd party API ?
its not trustless

Bitcoinj isn't trustless either Wink
How? Does it force u to connect to an online api? If not it is trustless in the context we care about

SPV clients are not suppose to be trustless so it isn't trustless in any context, it is convenience. Security is given up for convenience.

Unless you want to run a full node in front of the bitcoinj library and force bitcoinj to only connect to that. Then it is trustless.

Yes, it is not. It isn't unreasonable to trust some full-nodes you are running, though.

No, but you are concerned about scalability while I don't know what you are doing, that is the key word that I would use a 3rd party api. Lowest overhead and easiest to be scaling with.
Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 24, 2014, 06:59:56 AM
 #32

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 24, 2014, 07:08:36 AM
 #33

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.

Bitcoinj actually supports full verification at this point given you use the PostgresFullPrunedBlockStore or MySQLFullPrunedBlockStore as your BlockStore.
gweedo
Legendary
*
Offline Offline

Activity: 1498
Merit: 1000


View Profile
November 24, 2014, 07:15:18 AM
 #34

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.

Bitcoinj actually supports full verification at this point given you use the PostgresFullPrunedBlockStore or MySQLFullPrunedBlockStore as your BlockStore.

Bitcoinj has two modes it has SPV and full verification. I clearly stated the SPV mode in all my post.

To be honest though I don't think bitcoinj's full verification passes all the test. I don't know haven't really looked at the project much in the last year.
contactlight (OP)
Full Member
***
Offline Offline

Activity: 168
Merit: 100


View Profile
November 24, 2014, 07:22:09 AM
 #35

Not meant to be trustless?  Bitcoinj was trustless i thought it connects to peers and not some random api on the net unless its for a price feed which is irrelevent. What do u mean by this?

It connects to peers but it doesn't verify blocks like bitocoin-core does, as well as bitcoinj doesn't have the entire blockchain. Without those two things you can't be trustless. SPV are just meant to get your unspent outputs, sign transactions and broadcast your transactions. Bitcoin-core actually keeps the network healthy by verifying blocks.

Now you can make it trustless by running your own bitcoin-core and have bitcoinj only connect that.


@gmaxwell This shows how much work we work we have to do as a community as people don't understand the important differences. I know you are probably going to say "People don't need to know how it works" but having the different levels of security laid out can't hurt.

Bitcoinj actually supports full verification at this point given you use the PostgresFullPrunedBlockStore or MySQLFullPrunedBlockStore as your BlockStore.

Bitcoinj has two modes it has SPV and full verification. I clearly stated the SPV mode in all my post.

To be honest though I don't think bitcoinj's full verification passes all the test. I don't know haven't really looked at the project much in the last year.

Yeah, I understand that the SPV mode is different and that you were referring to that. No problems there. Smiley

As far as I know, BitcoinJ's full verification passes all the tests. I know it used to not but now it does. The initial syncing process is very slow, though. After reaching the Block #200000, it can only process less than two blocks per second, which is disconcerting.
myusuario
Newbie
*
Offline Offline

Activity: 7
Merit: 0


View Profile
November 25, 2014, 11:27:21 AM
 #36

if you dont want to trust to anyone, you can have a full node and database to handle your payment system, the bitcoin core api have all you need, https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list, getnewaddress, getreceivedbyaddress, listreceivedbyaddress, listsinceblock etc
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 714
Merit: 621


View Profile
November 25, 2014, 03:07:15 PM
 #37

Sometimes you have to balance best practice and being pragmatic.

Use one address per customer, and sweep money to cold storage times to times. (With one cold storage per customer)
When you spend your coins, try to use only coins you got from one customer, and do not mix with others. (the manual coin selection of bitcoinqt is really good for that)
It stays relatively simple, and good enough privacy wise. (For me)

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
Pages: 1 2 [All]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!