Bitcoin Forum
September 04, 2024, 07:52:18 PM *
News: Latest Bitcoin Core release: 27.1 [Torrent]
 
  Home Help Search Login Register More  
  Show Posts
Pages: « 1 ... 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 [54] 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 ... 315 »
1061  Economy / Web Wallets / Re: i have a tx that is being rebroadcasted over and over and over.. on: February 25, 2016, 06:03:00 AM
im stuck at importing the key.. :/

blockchain is giving me a key that starts with a 2

i need a key that has
5Hxxxx or 5Kxxxx or 5Lxxxxx



i dunno.. do i have to convert the key they give me?

is the key that starts with 2, by any chance 66 bytes long?
if so, it is the pubkey

you need the wif privkey, did you contact their support?
1062  Alternate cryptocurrencies / Announcements (Altcoins) / Re: [ANN] BitcoinDark (BTCD)--Financial_Privacy/SuperNET_Core/InstantDEX/PAX/Divs on: February 25, 2016, 05:24:54 AM
BTW, I forgot to thank James for finally getting back to BCT, I know where you hang around, but as you know for sure there would be no bitcoin without bitcointalk.
Thank you.
Most of the time now I am in slack and in the Bitcoin tech section: https://bitcointalk.org/index.php?board=6.0

https://bitcointalk.org/index.php?topic=1364951 is the atomic swap protocol which has a few practical issues, but we are refining it to solve all identified problems

With that as a base, the the roadmap in #tradebots (inside supernet slack) can be achieved.

James
1063  Economy / Web Wallets / Re: i have a tx that is being rebroadcasted over and over and over.. on: February 25, 2016, 05:16:53 AM
When and if you import the correct private key into electrum, I am thinking it might still show it as pending until the transaction gets forgotten by other nodes, which might not happen if someone is re-broadcasting it continuously, cmiiw.
given two different tx in the unconfirmed pool using the same utxo, not sure if the miners will pick the one with the higher fee using default miner. I think they have pretty liberal discretion to pick and choose among the valid unconfirmeds, but not sure the exact logic the big miners are using

some are just ignoring the txfees completely and just going for the blockreward. So it might have to wait until a miner that gives preference to the higher paying tx mines a block.

Would be interested to know how this turns out.

James

My point was, Electrum won't let you create another transaction with same outputs until the previous transaction is dropped and it might not happen until it is being re-broadcasted by anyone. So to create the same tx with higher fee, OP may have to use another method/wallet-client.
createrawtransaction?
or just 0.12 bitcoind using coincontrol
1064  Bitcoin / Development & Technical Discussion / Re: Any Javascript/JQuery/YUI/Dojo hackers in the house? on: February 25, 2016, 04:50:29 AM
I'd love to see a pure-Javascript front-end GUI for bitcoind developed.

I'm tempted to write one myself, but I've got a lot of other things on my TODO list right now.  Here's what I'm imagining:

+ Open source, pure JavaScript interface to bitcoin that communicates with a running bitcoin/bitcoind using the JSON-RPC api.

+ Open up the index.html page that is the GUI and you'd be asked for the host:port (default: localhost:8332), username and password.

+ From there, you'd have a nice Javascript/HTML GUI showing all your wallet transactions (using RPC listtransactions).

+ And it'd show your default receiving address, have a Send Bitcoins button, etc.

+ And it'd poll bitcoin/bitcoind every, oh, minute or so to look for new transactions.

I'm imagining shipping a webGUI/index.html (plus associated CSS/javascript/etc) as part of the bitcoin(d) source package.

We are developing something pretty close to this based on a stripped down copay wallet. It is self-contained JS/HTML with no external website dependencies and it is can be setup to use only the bitcoind rpc. It is designed for use with the iguana chrome app which will be a oneclick install that gets the JS GUI along with pnacl pexe that implements the bitcoind rpc, among other things

My feeling is that having something that can be run with a oneclick install will reduce the adoption barrier among the non-technical population. I know a lot of people wont want to touch a chrome app with a 10 foot pole, but the codebase is portable C, so there are also native versions.

Regardless of whether the chrome app version or native version is run, or whether the iguana is run at all, the same JS/HTML would work. I would be happy to have a version specific to what you want made so you wont have to worry about including iguana and the 50,000 lines of C code that is compiled into JS bytecodes.

James
1065  Economy / Web Wallets / Re: i have a tx that is being rebroadcasted over and over and over.. on: February 25, 2016, 04:36:50 AM
When and if you import the correct private key into electrum, I am thinking it might still show it as pending until the transaction gets forgotten by other nodes, which might not happen if someone is re-broadcasting it continuously, cmiiw.
given two different tx in the unconfirmed pool using the same utxo, not sure if the miners will pick the one with the higher fee using default miner. I think they have pretty liberal discretion to pick and choose among the valid unconfirmeds, but not sure the exact logic the big miners are using

some are just ignoring the txfees completely and just going for the blockreward. So it might have to wait until a miner that gives preference to the higher paying tx mines a block.

Would be interested to know how this turns out.

James
1066  Economy / Web Wallets / Re: i have a tx that is being rebroadcasted over and over and over.. on: February 25, 2016, 04:02:47 AM
some node is rebroadcasting it when it fails. the app just notifies me the coins are being sent.

Once you broadcast a transaction, anybody on the network can keep a copy of that transaction and re-broadcast it anytime they like until it either confirms or becomes invalid.  Generally there isn't any reason that a random user would want to do this, it doesn't benefit them at all.

While it is possible that some random user might be re-broadcasting the transaction just because they want to annoy random bitcoin users that have un-confirming transactions, it is FAR more likely that the recipient of the transaction is re-broadcasting it in order to make sure that they get the payment that they are expecting.

As such, it might be worthwhile to contact whatever person or business you sent the payment to, and ask them if they are re-broadcasting the transaction.  If they are, you might be able to arrange for them to stop so you can re-send a payment that will actually confirm.



it could be this.. i didnt know this is something they could setup.. figured it was just automatic rebroadcast from nodes.

im going to try to do with blockchain said and import my keys in to electrum and try to send the bitcoin to another wallet.. with like a .0003 fee.


yes. use coin control and pick the exact same unspent that is circling around.
make sure it has plenty of txfee
when that confirms, the old tx becomes invalid and wont be relayed


i tried to import the private key blockchain gives me but electrum said its invalid..


sorry cant help you there. you need to make sure you are getting the privkey from blockchain.info  that is compatible with electrum.

usually wif format for interchange, but I am not familiar with what blockchain.info sends

1067  Bitcoin / Development & Technical Discussion / Re: Atomic swaps using cut and choose on: February 25, 2016, 03:55:46 AM
Some details have changed, but the original spec from the other thread were this came from:

1) Cut and choose

Bob generates 1000 keypairs and sends hash(bob_priv_n) and bob_pub_n to Alice for each n (1 to 1000).

Alice picks n from 1 to 1000 and Bob sends bob_priv_n for the 999 other n's.

If they are ok, then Alice assumes the pair she didn't see is valid.

Similarly, Alice generates 1000 key pairs and has Bob choose.

This gives the following public info.

  hash(bob_priv_n) matches bob_pub_n
  hash(alice_priv_m) matches alice_pub_m

There is a 0.1% chance of fraud for each attempt.  This means that fees must be at least 0.1% of the transaction value.  In that way, there is no incentive to cheat at the cut-and-choose.

2) Bob pays D to

Code:
OP_IF
  <now + 2 days> OP_CLTV OP_DROP <alice_pub_1001> OP_CHECKSIG
OP_ELSE
  OP_HASH160 <hash(bob_priv_n)> OP_EQUALVERIFY <bob_pub_1001> OP_CHECKSIG
OP_ENDIF

D should be more valuable than the value of the Bitcoins or altcoins. 

This is a promise by Bob to release bob_priv_n within 2 days (or lose his deposit).

3) Alice pays a altcoins to

Code:
OP_2 <bob_pub_n> <alice_pub_m> OP_2 OP_CHECKMULTISIG

4) Bob pays b Bitcoins to

Code:
OP_IF
  <now + 1 day> OP_CLTV OP_DROP <bob_pub_1002> OP_CHECKSIG
OP_ELSE
   OP_HASH160 <hash(alice_priv_m)> OP_EQUALVERIFY <alice_pub_key_1001> OP_CHECKSIG
OP_ENDIF

Alice has 1 day to claim her output, but that means she must provide alice_priv_m.

5) Alice spends the output (and provides alice_priv_m).

6) Bob can spend the altcoin output (since he has both keys)

7) Bob reclaims his deposit

This is atomic and it is recoverable if either party refuses to complete a step.

1) Either party refuses to perform cut-and-choose.

Nothing has been sent to either chain, so no harm done.

2) Bob refuses to pay deposit.

Nothing has been sent to either chain, so no harm done.

3) Alice refuses to bail-in.

Bob reclaims his D after 2 days.
Releasing bob_priv_n has no effect since there was no bail-in by Alice, so the key is never used.

4) Bob refuses to bail-in

If Bob reclaims his deposit, then Alice will have both private keys and can reclaim her bail-in.

If Bob fails to claim his deposit, then Alice gets that as compensation instead of her bail-in.  In that case, Bob loses his deposit and can't claim the altcoins.  This means he should reclaim his output.

5) Alice refuses to claim her Bitcoins

1 day later Bob can reclaim his Bitcoins

Alice gets her altcoins back or she gets D instead (if Bob doesn't reclaim his deposit).

6) Bob refuses to claim his altcoins

He loses his altcoins when he reclaims his deposit.  Alice already has his Bitcoins by this point.

He has an incentive to complete this step.

7) Bob refuses to claim his deposit

He loses his deposit and Alice gets it.

He has an incentive to complete this step.

For each step, there is either an incentive to complete the step or failure to complete the step causes the protocol to abort.  In an abort, the incentives are to perform a clean abort.

###
There is plain english description
FSM embodiment of the english description
and the scripts

Not sure what part you feel is not a "straight answer"

Maybe you can zoom in to a specific step (state) with a specific scenario? The problem is that this solution depends on exact behavior of the scripts, so without detailed understanding of them it is only possible to get a general idea. You probably dont want to see the 2000 lines of C code that implements this either.

I am trying to find an FSM visualizer so it can convert my FSM description into pretty circles and boxes with lines connecting them, but graphics apps are not my speciality.

the FSM spec is just to define the name of the state and the name of the state if things timeout using the createstate function. Once a state is created the addevent function adds an event to that state that it looks for and the next state it goes to along with the event message it sends to the other side.

James
1068  Bitcoin / Development & Technical Discussion / Re: Atomic swaps using cut and choose on: February 25, 2016, 03:36:35 AM
I would be grateful if you or TierNolan could actually answer my question?

Stated in another way, what prevents Bob from finalizing the transaction on the block chain with Bitcoin CLTV op code, taking the funds from Alice, while also not issuing the reciprocal transaction paying to Alice on the other block chain?

I seem to never get a straight answer.

In order to spend the funds from Alice, bob has to disclose the random number Alice can use to spend the bitcoins

Afaics, you have not answered my question. Please read my question again. I didn't ask how Bob finalizes the payment from Alice to Bob. I asked, "while also not issuing the reciprocal transaction paying to Alice on the other block chain?". Alice can't spend Bitcoins, because Bob is issuing a transaction on a block chain that is not Bitcoin (and doesn't support CLVT).

Are you implying that Bob will issue his transaction first (paying to Alice) on the block chain without a CLVT op code? Then how does Bob get a refund if Alice doesn't complete her side of the transaction and how can Bob's transaction be verified against a hash of a random number when the CLVT op code isn't supported.
The simple act of spending the payment requires to disclose the information the other party needs to do their spend.

Ignoring fees, at the high level:
1. bob sends in a deposit that alice verifies she can spend after a certain amount of time. However if bob follows the protocol, he can reclaim this refund before alice is eligible to.

You've apparently swapped the block chains of Alice and Bob relative to scenario I described. But that is okay, I will follow your choice, where Bob is transacting on the block chain with CLTV and Alice is not.

The transaction above signed by Bob I assume requires CLTV (Check-Lock-Time-Verify) since it must be refunded if it is not verified before the timeout of the lock.

2. alice verifies the above is in the blockchain so is assured that worst case she can get the deposit, which is 113% of the transaction amount, so she might actually prefer that. when assured she sends bob a payment that requires both secrets to be spent.

I don't understand how Alice can receive the CLTV output unless she already knows how to verify it. But if she knows how to verify it, then she can steal it at any time.

How can Alice send a payment that requires verification of two secrets if the other block chain doesn't support pay contingent on verification of hash preimage?

3. bob sees this on the altcoin chain and sends in a payment to alice that requires a secret from alice that would allow him to spend the payment from 2.

Bob issued the deposit in #1 and now he also issues another payment to Alice, but which is spendable by Alice contingent on knowing the preimage (secret) of a hash. Only Alice knows this secret preimage at this point until she reveals it. I thus understand this step #3.

4. alice sees this real payment and then cashes in the bitcoins, but in doing so has to divulge the secret bob needs to spend the altcoins

I am disputing whether Alice can make such a transaction available in step #2 if her altcoin doesn't support pay contingent on verification of hash preimage.

Edit: I understand you want to use a forfeitable deposit to ensure that Bob does step #3, since Alice has no way to refund step #2 (because her block chain doesn't support CLTV). I presume you are assuming that the altcoin supports pay contingent on verification of hash preimage. But I still don't see how the mechanics of the forfeitable deposit work? "I don't understand how Alice can receive the CLTV output unless she already knows how to verify it. But if she knows how to verify it, then she can steal it at any time."
OK, I think we take one issue at a time.

The altcoin payment on the network that doesnt support CLTV is an ordinary multisig that uses the pubkeys from the temp privkeys. This is the key that allows the single pull double throw, the fact that on the BTC blockchain it is just a random number that hashes to the previously specified value.

It is actually the privkey that makes pubkey that is put into a 2of2 multisig.

I know you dont like scripts, but please bear with me:

OP_2 <alice_pubM> <bob_pubN> OP_2 OP_CHECKMULTISIG

The above is an original standard multisig, not even the p2sh form, so basically all altcoins using bitcoin 0.7 (or even 0.5?) support it.

the alice_pubM is the pubkey from the temporary privkey that was selected by bob during the cut and choose.

similarily bob_pubN is the pubkey from the temporary privkey that was selected by alice during the cut and choose.

In order to spend the 2 of 2 multisig, both privkeys are needed. Alice has the privkey for the alice_pubM and bob has the privkey for bob_pubN. So if during the protocol, either party gets the other privkey, then and only then can they spend this.

The protocol (and FSM based on it) follow blockchain verifiable steps to allow each side to progress incrementally to the next state where eventually either both sides can get their corresponding payments, or it unwinds and they get their funds back.

This is impossible at first sight, but this is where the evil genius of TierNolan comes in. The scripts for the deposit and payment that are timelocked to prevent the wrong side from being able to spend it, even if they have the required keys. So it does require both sides to actively protect their interests and do the spends that they are able to. If they dont, after the timer expires, the other side would.

But this is all automated in the protocol.

So if at the high level you can agree that it is theoretically possible that if we can setup the corresponding payment to the 2of2 multisig that enables the opposite party to spend the corresponding payment, we get atomicity. And that is what the atomic protocol purportedly does and thus requires scrutiny of the FSM and scripts to verify.

OP_IF
    <now + INSTANTDEX_LOCKTIME> OP_CLTV OP_DROP <bob_pubB1> OP_CHECKSIG
OP_ELSE
    OP_HASH160 <hash(alice_privM)> OP_EQUALVERIFY <alice_pubA0> OP_CHECKSIG
OP_ENDIF

I know you dont like scripts, but hopefull the possibility of non-impossibilty is conveyed. In the above, the first case is only spendable by bob as it does a CHECKSIG against his pubkey, but it cannot be done until after the locktime passes.

the other case allows alice and only alice to spend it as her signature is verified, but it also requires alice to divulge the privM key, which once it is divulged can be used to sign the 2of2 multisig, along with verify that the cut and choose wasnt cheated on.

Granted, this is the point where alice can refuse, but in such case, time passes and bob can spend it just by waiting.

So this protocol barely works as if any script is wrong or cltv isnt existing or multisig doesnt work or ..., but assuming we didnt miss any case, it works, is atomic and blockchain verified, with the cut and choose offering a probabilistic penalty to make cheating at cut and choose uneconomical.

The intricacies of using conditional scripts that allow different parties to spend the same tx and requiring part of it to have a delay, is what is at the same time confusing, but also enables atomicity. Without the time delay, then yes the other party can spend right away. And getting it setup so the altcoin side uses only standard multisig is a thing of beauty.

I dont quite have it fully debugged, but so far I am only finding minor issues in the protocol itself. It has gone from an impractical "both coins must have CLTV" protocol that would allow trading just a few coins, to a protocol that should work with 100+ variety

James
1069  Economy / Web Wallets / Re: i have a tx that is being rebroadcasted over and over and over.. on: February 25, 2016, 03:17:44 AM
some node is rebroadcasting it when it fails. the app just notifies me the coins are being sent.

Once you broadcast a transaction, anybody on the network can keep a copy of that transaction and re-broadcast it anytime they like until it either confirms or becomes invalid.  Generally there isn't any reason that a random user would want to do this, it doesn't benefit them at all.

While it is possible that some random user might be re-broadcasting the transaction just because they want to annoy random bitcoin users that have un-confirming transactions, it is FAR more likely that the recipient of the transaction is re-broadcasting it in order to make sure that they get the payment that they are expecting.

As such, it might be worthwhile to contact whatever person or business you sent the payment to, and ask them if they are re-broadcasting the transaction.  If they are, you might be able to arrange for them to stop so you can re-send a payment that will actually confirm.



it could be this.. i didnt know this is something they could setup.. figured it was just automatic rebroadcast from nodes.

im going to try to do with blockchain said and import my keys in to electrum and try to send the bitcoin to another wallet.. with like a .0003 fee.


yes. use coin control and pick the exact same unspent that is circling around.
make sure it has plenty of txfee
when that confirms, the old tx becomes invalid and wont be relayed
1070  Economy / Web Wallets / Re: i have a tx that is being rebroadcasted over and over and over.. on: February 25, 2016, 12:37:35 AM
why not use coin control to spend the same inputs (use enough fee!)
when that one is confirmed the floating tx will be rejected as invalid and wont be rebroadcast
1071  Alternate cryptocurrencies / Announcements (Altcoins) / Re: Declaration of Independence - Atomic Cross Chain Asset Standard on: February 25, 2016, 12:24:20 AM

Could you please explain this in laymen tearms for asset holder of supernet, nxtventure, jl777hodl, etc.
;Most of them are down 70-90%. Are we gonna loose the last 20% of value left?
Will they stay on these awkward child chain?
I will make a way to move to the BTC chain, so that seems not an awkward child chain.
do not worry, I will do what it takes to protect the assetholders.

James
1072  Bitcoin / Development & Technical Discussion / Re: Atomic swaps using cut and choose on: February 25, 2016, 12:21:08 AM
I would be grateful if you or TierNolan could actually answer my question?

Stated in another way, what prevents Bob from finalizing the transaction on the block chain with Bitcoin CLTV op code, taking the funds from Alice, while also not issuing the reciprocal transaction paying to Alice on the other block chain?

I seem to never get a straight answer.

In order to spend the funds from Alice, bob has to disclose the random number Alice can use to spend the bitcoins

Afaics, you have not answered my question. Please read my question again. I didn't ask how Bob finalizes the payment from Alice to Bob. I asked, "while also not issuing the reciprocal transaction paying to Alice on the other block chain?". Alice can't spend Bitcoins, because Bob is issuing a transaction on a block chain that is not Bitcoin (and doesn't support CLVT).

Are you implying that Bob will issue his transaction first (paying to Alice) on the block chain without a CLVT op code? Then how does Bob get a refund if Alice doesn't complete her side of the transaction and how can Bob's transaction be verified against a hash of a random number when the CLVT op code isn't supported.
The simple act of spending the payment requires to disclose the information the other party needs to do their spend.

Ignoring fees, at the high level:
1. bob sends in a deposit that alice verifies she can spend after a certain amount of time. However if bob follows the protocol, he can reclaim this refund before alice is eligible to.

2. alice verifies the above is in the blockchain so is assured that worst case she can get the deposit, which is 113% of the transaction amount, so she might actually prefer that. when assured she sends bob a payment that requires both secrets to be spent.

3. bob sees this on the altcoin chain and sends in a payment to alice that requires a secret from alice that would allow him to spend the payment from 2.

4. alice sees this real payment and then cashes in the bitcoins, but in doing so has to divulge the secret bob needs to spend the altcoins

5. bob gets the required secret from alice and spends the altcoins

So the high level concepts are to advance to each stage only when you are assured that the other side is following the protocol and the FSM is setup so that if either party bails at any point, the funds in limbo can be reclaimed.

To verify this, the bitcoin scripts must be understood in detail. I see no other way. Conceptually, it is a single pull, double throw switch where a single random number releases two transactions

James
1073  Bitcoin / Development & Technical Discussion / Re: Atomic swaps using cut and choose on: February 25, 2016, 12:04:32 AM
I would be grateful if you or TierNolan could actually answer my question?

Stated in another way, what prevents Bob from finalizing the transaction on the block chain with Bitcoin CLTV op code, taking the funds from Alice, while also not issuing the reciprocal transaction paying to Alice on the other block chain?

I seem to never get a straight answer.
In order to spend the funds from Alice, bob has to disclose the random number Alice can use to spend the bitcoins
1074  Bitcoin / Development & Technical Discussion / Re: Atomic swaps using cut and choose on: February 24, 2016, 10:45:49 PM
Thus the only way I can see that Bob can be required to issue that said transaction paying to Alice (when employing cut & choose on a block chain without Bitcoin op codes), is if the private key for signing the transaction for Bob paying to Alice (on the block chain without Bitcoin op codes) is revealed in order to finalize the already confirmed transaction for Alice paying to Bob (on the block chain with the CLTV Bitcoin op code).
I believe the red part is what causes the confusion.

What is being revealed is just a random number, NOT the privkey for signing the transaction.
This tx is made so this random number must be revealed for the tx to be valid.

I think this point needs to be understood first. The random number has both algebraic properties itself and it is part of the 1000 cut and choose dataset, but that really has little to do with the point you are complaining about.

James

1075  Bitcoin / Development & Technical Discussion / Re: Bitcoin Core 0.12 full initial sync: results and analysis on: February 24, 2016, 04:43:56 PM
Also I think I only need to validate from the latest checkpoint, so that reduces significantly the sheer number of sigs that need to be verified.

Actually, I was running rc5 as it turns out, which had the checkpoint way back at block #295000.  That's been updated in the final release, which of course will make things go much faster.
majority of tx since 295000
It doesnt take too long to process that far, but it starts really slowing down at around 350,000. I think that is when p2sh started being used a lot more and tx just got bigger and more complicated
1076  Bitcoin / Development & Technical Discussion / Re: Bitcoin Core 0.12 full initial sync: results and analysis on: February 24, 2016, 04:29:00 PM
while you can verify sigs a lot of times from the recent blocks, there are many times where you just dont know what the txid the vin refers to is, since you havent seen it yet during a parallel sync.

Which places rather tight constraints on parallelism.

I have the blockchain validating serially all throughout the parallel sync, so all the sig validations can take 30 minutes using N cores without affecting the final completion time. Also I think I only need to validate from the latest checkpoint, so that reduces significantly the sheer number of sigs that need to be verified.

As long as users can see balances, even if the validation isnt complete yet, just need to prevent any spends until all the inputs for the tx are validated. My guess is for most of the people most of the time, it can be done so the time for sig validation is not noticeable. Also for user tx before fully validated, I could select the oldest inputs and specificallly validate the things needed to make sure those are valid.

James
1077  Bitcoin / Development & Technical Discussion / Re: --bench (?) on: February 24, 2016, 04:22:12 PM
Perhaps the benchmark methodology could be like

a) online (measuring network transfers as well)
b) offline (measuring validation of disks in the data)

further broken down to

a) real-time (getting RL metrics of blocks as they are added in terms of network speeds, cpu time required, I/O time required), including "catchup mode" where you open the -qt client, you are 1 day behind, and you can measure that as well in terms of how much cpu or I/O is used per mb of block data etc - although that is not 100% accurate due to differences in the way txs are included in blocks.

b) non-real time where you could have something like --bench=height 350000-351000 where you'd measure I/O and CPU speeds for validating a pre-selected range of 1000 blocks that are already stored in your hard disk.

...Any alterations on metrics that are useful could also be included.

I made a custom build with march=native flags and I don't really have any serious tool to measure whether my binary is faster than the official one, or my distribution's binary. I also want to experiment with various GCC versions, Intel and AMD C compilers, clang etc etc to see which gets the best performance, but I'm lacking benchmarking tools.
Using -O2 seems to get pretty close to the best variant with most things, but maybe some specific vector optimizations could make a noticeable difference.

And without a benchmark we'll never know Tongue

Quote
From what I have seen RAM, DB slowness and serialization issues are the main bottlenecks now.
+
The serial nature creates a lot of dropouts  from the saturated bandwidth ideal, so definitely parallel sync is needed for fastest performance.

RAM can be traded with CPU with something like ZRAM where you increase the data that can fit into it, by compressing them in real time. It's pretty handy. With LZ4 I'm getting 2.5-3x RAM compression ratios and can easily avoid swapping to disk which is very expensive in terms of I/O times.

In theory Bitcoin could use its own ram compression scheme with an off the shelf algorithm for things like its caching system or other subsystems.

Same with disk compression which is an I/O tradeoff with CPU. I tried installing the blockchain in a BTRFS compressed partition, it was slightly faster in terms of throughput and also saved 10gb+ in size. From what I remember Windows 7 also have compressed folders support, so it should be doable in windows too.

Serialization is indeed an issue but if you can get a serial process to get on with it 10-20% faster due to custom compilation, then it's worth it - until these things are fixed in the code.

Quote
For sync,  I make sure network bandwidth is as saturated as possible for as long as possible. This is a shortcut, but practical. If the code cant process the data at full speed, then it can be optimized. Well at least there is  a chance to.

Makes sense.
squashfs cut the size of the "DB" files from 25GB to ~15GB

so that automatically takes advantage of disk compression without slowing down high bandwidth disk usage where the HDD is the bottleneck.

When streaming at 100MB/sec, there is no time for a compression stage. There isnt even time for mallocs. The compression needs to be after all the final files are created and it takes about the same half hour it takes to do the sync

James
1078  Bitcoin / Development & Technical Discussion / Re: sendrawtransaction wire protocol? on: February 24, 2016, 03:52:59 PM
The are two ways to send a transaction. The node can send an 'inv' message with the txid of the transaction. Then the peers respond with a 'getdata' message for that transaction and your node responds with a 'tx' message which contains the transaction. Alternatively the node can simply send 'tx' messages to its peers. Both methods require sending unsolicited messages to the peers.
makes sense.

So with 3 nodes, A, B and C. A can simply send a tx message to B and then I wait to see if B sends it to C. If it does, that indicates B was happy with the tx, if not, then rummage through the debug log to see why it rejected it.

And this will work even if A is a non-relaying node? Of course B has to be a relaying node.

James
1079  Bitcoin / Development & Technical Discussion / Re: --bench (?) on: February 24, 2016, 03:47:11 PM
Perhaps the benchmark methodology could be like

a) online (measuring network transfers as well)
b) offline (measuring validation of disks in the data)

further broken down to

a) real-time (getting RL metrics of blocks as they are added in terms of network speeds, cpu time required, I/O time required), including "catchup mode" where you open the -qt client, you are 1 day behind, and you can measure that as well in terms of how much cpu or I/O is used per mb of block data etc - although that is not 100% accurate due to differences in the way txs are included in blocks.

b) non-real time where you could have something like --bench=height 350000-351000 where you'd measure I/O and CPU speeds for validating a pre-selected range of 1000 blocks that are already stored in your hard disk.

...Any alterations on metrics that are useful could also be included.

I made a custom build with march=native flags and I don't really have any serious tool to measure whether my binary is faster than the official one, or my distribution's binary. I also want to experiment with various GCC versions, Intel and AMD C compilers, clang etc etc to see which gets the best performance, but I'm lacking benchmarking tools.
Using -O2 seems to get pretty close to the best variant with most things, but maybe some specific vector optimizations could make a noticeable difference. Though without any compiler optimations, truly horrible performance is possible.

From what I have seen RAM, DB slowness and serialization issues are the main bottlenecks now.

For sync,  I make sure network bandwidth is as saturated as possible for as long as possible. This is a shortcut, but practical. If the code cant process the data at full speed, then it can be optimized. Well at least there is  a chance to.

The serial nature creates a lot of dropouts  from the saturated bandwidth ideal, so definitely parallel sync is needed for fastest performance.

James
1080  Bitcoin / Development & Technical Discussion / Re: Bitcoin Core 0.12 full initial sync: results and analysis on: February 24, 2016, 02:09:49 PM
Those pauses aren't pauses. You're going by log entries, which only note when verification has completed.

Of course that explains it.  Otherwise there would be out-of-order entries in the log, which there never are.  It doesn't explain the lack of CPU activity though during the pauses.  Sigs could be checked in the order received, couldn't they?

If you were to run with a larger dbcache setting you would likely see vastly improve performance

I may try that -- and post the results for comparison purposes.

It can't really set it automatically, because, unless you had much more ram than you do, bitcoin would end up hogging most of your memory when you wanted to use your computer for other things

A dynamic dbcache setting might be OK if the limit were set conservatively. A default setting should never be seriously sub-optimal, in my opinion.
while you can verify sigs a lot of times from the recent blocks, there are many times where you just dont know what the txid the vin refers to is, since you havent seen it yet during a parallel sync.

So you could verify a significant subset as it comes in during the first pass, but until the blockchain is fully there you cant verify all the sigs.

as the highest block is advanced, whatever sigs that are not verified yet could be verified. But with the checkpointing, there isnt the sig verification needed until the last checkpoint anyway
Pages: « 1 ... 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 [54] 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 ... 315 »
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!