Bitcoin Forum
May 23, 2024, 05:03:12 AM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
  Home Help Search Login Register More  
  Show Posts
Pages: « 1 ... 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 [161] 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 ... 233 »
3201  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 19, 2015, 12:49:52 PM
How long do you let this part run until you conclude it is hanging?

5-10 mins. Could there be something intended happening for longer than that length of time that consumes ~1% CPU on all cores? It's not consistent with the behaviour of your first couple of 0.94 commits, they took seconds to handle that stage.

Glancing at the code, it's pretty old and inefficient, but that wouldn't warrant that long a wait. This is probably some sort of dead lock. Unfortunately I haven't managed to reproduce it in this version (which is why I thought it was fixed), so it's gonna take me some time to figure it out. I expect this is the last hurdle left for this code to be stable and functional.
3202  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 19, 2015, 12:16:06 PM
How long do you let this part run until you conclude it is hanging?
3203  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 19, 2015, 10:15:50 AM
That's quite interesting considering I added verbose at this stage for this specific reason (was curious how deep it would get before failing). I guess it doesn't even get that far.
3204  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 19, 2015, 02:26:08 AM
New version. This one should fix the segfaults mid scan. Still working on the others ones. Pull away =)
3205  Bitcoin / Armory / Re: Armory stuck in offline mode... on: June 12, 2015, 08:19:42 AM
You need to give the user you're logged in as write access to E:\Bitcoin\core or turn off bitcoind auto management.
3206  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 10, 2015, 03:48:37 PM
4. At block 360206, and armoryd says "BDM thread failed: The scanning process interrupted unexpectedly" and asks me to rebuild and rescan. I killed armoryd by Control-C.

That's consistent with the bug I'm going after currently. That error is too aggressive, half the time you can ignore it and restart without rescanning.

Quote
5. I started armory-qt. I didn't request a rescan but it seems doing that automatically.

There will be an empty file named either rescan.flag or rebuild .flag in your datadir when the previous instance of Armory wants to signal the next one to rebuild/rescan. Delete it and it won't.

Quote
7. I restored the previous databased folder by last ffreeze. It was incomplete and I stopped it at about block 300,000 after more than a week of scanning
8. I started armory-qt. No re-scan.

That's odd, it should try to catch up from 300,000 onwards

Quote
9. I imported this private key: 5JdeC9P7Pbd1uGdFVEsJ41EkEnADbbHGq6p1BwFxm6txNBsQnsw (SHA256 of "1") for the address 12AKRNHpFhDSBDD9rSn74VAzZSL3774PxQ. It only shows the 2 txs in 2014 but not the latest ones of 2015.

Expectable if the scan ended around block 300,000

Quote
So I guess both databases are somehow corrupted?

Shouldn't be, but I guess they are. Actually what's off is that they don't pick up where they left. I'd say hang on to your 300k DB, I intent to add some more checks to allow the DB to fix itself (instead of rebuild/rescan).
3207  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 10, 2015, 02:42:03 AM
Could well be just chance since the behaviour before was clearly somewhat nondeterministric.  But I did the above and it synced my wallets perfectly first time (and I hadn't managed to get a successfully sync at all so far up till now).  If I get the chance tomorrow I'll nuke the databases and try again to confirm whether this is repeatable.

Well yes and no. I expect the issue is object life span: one type of threads creates some data that gets cleaned up before the next type of threads is done with it. If you reduce the amount of threads per groups to the bare minimum, you also reduce the "surface area" for the bug and thus the chance it will occur.

Quote
Just to confirm, only change I made was as you suggested:

That's the proper change to force thread count for all groups down to 1.
3208  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 09, 2015, 09:40:07 PM
This isn't the thread adjustment, it's failing to grab the block data or the data has expired before it got to process it. It's consistent with the symptoms of the issue I'm going after ATM so I'll have you wait for the fix to try this again. Right now I don't think there is anything else you can do about it.

If you want to manually set your thread counts regardless, go inside BlockWriteBatcher.cpp and do the following:

1) comment out line 2004 to prevent thread adjustment
2) hardcode values in the BlockWriteBatcher ctor (line 112 for fullnode, 99 for supernode)
3209  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 09, 2015, 09:01:45 PM
Gimme the part of the log where it posts the timings after the scan is over. I want to see the last block it scanned.
3210  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 09, 2015, 08:02:48 PM
(On OS X)

No errors building the database - but doesn't find any transactions and all wallet balances are 0.0

EDIT: Ah, OK, database appears to build OK judging by progress, but then scanning transactions seems to complete almost instantly (so evidently it actually failed - but didn't give an error).

roy

Lemme see the last few lines of your log file. Also, what's the top block displayed in Armory?
3211  Bitcoin / Armory / Re: Armory stuck in offline mode... on: June 09, 2015, 06:58:34 PM
In Files -> Settings, uncheck the first checkbox, then restart Armory.
3212  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 09, 2015, 06:47:08 PM
New version out in ffreeze, please test away. There's a wealth of robustness changes and it should also be a bit faster. However no one at Armory has experienced the issues Carlton Banks and btcchris ran into, so most of these changes were speculative. It's possible these bugs will still be there partially or in full.

As usual, looking forward to your bug reports and logs, and again thank you for the time and effort you put in helping us to test things out =)

Have you made any big change re supernode? In the previous ffreeze it did not complete even with a week. In this version it took only a few hours to complete.

Quote

I made changes to some stuff that makes supernode faster (some weird opts I should have rolled but figured it could wait for fear it would be unstable). The other issue, with pulling block data and the object lifespan remains. It's a rare occurrence but common enough to trip certain setup quite often (2 out of my 3 testers T_T). I ended rolled the optimization with the stability changes (it didn't cost that much more to add the opts at that stage)

Now I'm also convinced that the code speeding up supernode is not responsible for the instability. That gives me room to add some more optimizations (again supernode only) without fear of making bugs more convoluted (and thus harder to isolated).

Not having a clue what the issue was, I ended up fixing what you were suffering from (which was a lot more obvious than the remaining issue) in an attempt to fix the bug they are experiencing. In grand scheme of things it doesn't matter what order I fix these, since I have to go after both anyways. In the current scope, it doesn't speak so good of me since I thought I was fixing one thing and got the other instead =P

Quote
I can now get the details of a random tx with "armoryd.py gettransaction txid". I guess this means the supernode in running properly?

Not really, that just means you are using supernode indeed (which is the only the DB format now that support random txhash resolution), but that doesn't mean supernode is working per se. What you need to do to test it is to add a random bitcoin address to a wallet and getledger on the wallet, which will display that random address history along with the rest.

You can also verify this in the UI by importing some publicly known private keys to a wallet (kind of an oxymoron I know) and verify they get imported quasi instantly and that their history shows up in the main ledger.
3213  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 08, 2015, 11:30:29 PM
Ok this really is totally consistent behaviour. Tried not rebuilding the Db, just let it carry on from where it hits the (infinite?) loop. The headers get read again (implying the data structure for the headers wasn't committed to disk in the failed run), then it hits the same loop. I could try this with testnet I guess, but that's a fair amount of setup to confirm something so fundamental and consistent.

I've reproduced this issue and I'm on it. I've also got a clue on the other one in the process so I may get both by the next commit.
3214  Bitcoin / Development & Technical Discussion / Re: Elastic block cap with rollover penalties on: June 08, 2015, 08:16:04 PM
I'm getting slammed at work so I won't be able to come up with a properly crafted response to your new model this week.
3215  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 08, 2015, 12:42:43 AM
There is a difference to my setup, in that I'm using jessie instead of wheezy, but I don't think that's the source of the problem. I know, better to keep the setup the same, but it's a long story, and 93.1 works identically on jessie as it does on wheezy.

It should work on any setup as long as the compiler is doing its job really.

Can you get through with testnet?
3216  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 08, 2015, 12:14:43 AM
Having trouble getting past this stage in the DB build:

Code:
-INFO  - 1433717171: (BlockUtils.cpp:1337) Looking for first unrecognized block

CPU cores hit the floor, that block never gets found.

Fresh DB folder? Is it maxing one core or all cores?

Yes, fresh DB. CPU activity cuts out altogether (occasional 1% blip for python). Logs and gdb output all look uninteresting.

This is weird I didn't touch that part of the code. I'll get back to you when it's fixed (hopefully I can reproduce it)

What's the last line in the terminal?
3217  Bitcoin / Armory / Re: Armory 0.93 testing release! (with 0.05 BTC bug bounty) on: June 07, 2015, 11:46:18 PM
No x86 for 0.93

Go back to 0.92 or wait for 0.94
3218  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 07, 2015, 11:45:30 PM
Having trouble getting past this stage in the DB build:

Code:
-INFO  - 1433717171: (BlockUtils.cpp:1337) Looking for first unrecognized block

CPU cores hit the floor, that block never gets found.

Fresh DB folder? Is it maxing one core or all cores?
3219  Bitcoin / Armory / Re: Starting preliminary 0.94 testing - "Headless fullnode" on: June 07, 2015, 09:31:57 PM
New version out in ffreeze, please test away. There's a wealth of robustness changes and it should also be a bit faster. However no one at Armory has experienced the issues Carlton Banks and btcchris ran into, so most of these changes were speculative. It's possible these bugs will still be there partially or in full.

As usual, looking forward to your bug reports and logs, and again thank you for the time and effort you put in helping us to test things out =)
3220  Bitcoin / Development & Technical Discussion / Re: Elastic block cap with rollover penalties on: June 05, 2015, 02:31:20 PM
firstly, what we're talking about here is not, as DumbFruit generalizes, a "rollover fee". It's a disporportional penalty on mining large blocks. I'm not sure wether this changes his argument or its validity.

Unless I missed something huge, the proposal is not only to penalize large blocks, but to redistribute the penalties collected from these blocks back to other miners. In that sense there is a rollover of mining rewards (since penalized miners stand to earn their own penalties back), just not "fees rollover" per say.

Quote
Do the 2 25% miners (or 2 of the 10% miners) have a higher-than-in-current-system incentive to collude?

I don't think they would collude, rather I expect a large miner can deplete the mempool of high fee transactions (and only these) while staying under the soft cap (thus paying no penalties), and leave other miners to pick up the slack. This behavior is incentivized by the fact that the other pools, if they do pick up the slack (and try to deplete the mempool with less regard for fees) stand to pay penalties for the extra service rendered, and the large miner in return stands to gain from that.

This results in one of 2 scenarii:

1) Other miners decide to pick the slack from the large miner, they naturally loose in profitability as the large miner is vampirizing their rewards by collecting penatlies. As a result, a lot of users from the other pools will point their hardware to the large miner, which will get an ever increasing share of the network hash rate, which enables his scheme even further and so on.

2) As a reaction to the large miner's behavior, every other miner adopts his policies, which is to at least avoid the penalties, by always emitting blocks below the softcap, and this change never fixed what it was meant to.

Quote
Is Menis proposal making it easier for the 2 25% miners to try to drive out small (as in bandwidth) miners by mining disproportionally large blocks?

Other pools need to verify a block before they start to mine on top of it, so very large blocks, that propagate slower and take longer to validate could supposedly give a head start to the miner who emitted it, and be a disadvantage to smaller pools which stand at a risk of having their block orphaned the longer it takes for other miners to receive and validate their work.

I expect any centralized pool, however small they are, can afford the bandwidth and processing power to deal with much larger blocks than we have at the moment. This could hurt p2pool miners though.

Here are example scenarios, with made up values for the penalty function. I assume for simplicity (not a necessary assumption) that there is endless demand for transactions paying 1mBTC fee,  that typical blocks are around 2K txs, and that there are no minted coins. The pool clears at 1% per block.

Scenario 1: The network has 100 1% miners.
Every 1% miner knows he's not going to claim much of any penalty he pays, so he includes a number of transactions that maximize fees-penalty for the block. This happens to be 2K txs, with a total fee of 2 BTC and penalty of 1 BTC.

The equilibrium for the pool size is 100 BTC.
Miners get 2 BTC per block (2 fees + 1 pool collection - 1 penalty).
There are 2K txs per block.

Scenario 2: The network has 1 90% miner, and 10 1% miners.
The 1% miners build blocks with 2K txs, fee 2 BTC, penalty 1 BTC, like before.
The 90% miner knows that if he includes more txs, he'll be able to reclaim most of the penalty, so the marginal effective penalty exceeds the marginal fee only with larger blocks - say, which have 4K txs, 4 BTC fees, 4 BTC penalty.

The average penalty per block is 3.7 BTC. The equilibrium pool size is 370 BTC.
There are on average 3.8K txs per block.
A 1% miner gets, per block he finds, (2 + 3.7 - 1) = 4.7 BTC - more than in scenario 1!
The 90% miner gets, per block he finds, (4 + 3.7 - 4) = 3.7 BTC - less than small miners get in this scenario, but more than miners get in scenario 1!

Those examples do not stand. They hinge on the premise that there is endless demand for transactions paying 1mBTC fee. I understand the need to simplify these demonstrations but that defeats the underlying premise of this entire discussion. You example assumes there is no competition over fees, which is the premise of a "no block limit + hard fees" system. Your system sets both soft and hard caps to the block size, so there is no reason to believe people will sit at a 1mBTC fee when there is an endless demand for transactions

Model your demonstration with a fee structure using the Pareto principle i.e. 20% of the transactions pay for the 80% of the total fees available in the mempool (which is a lot closer to the current network's fee distribution than your examples), and the system falls apart. Anyone building blocks large enough to get penalized is just giving his rewards away to miners that prioritize big fee transactions and make a point of staying under the soft cap.

The issue with your proposal is not the penalty per say, it's the reward: there is a point where it is more profitable to let others get penalized. The existence of this point creates a threat that keeps all miners functioning below the soft cap. The threat is that they lose profitability in comparison to other pools, and those pools start siphoning away their hash rate as miners migrate.

If you were to take away the reward from the system a few things would be smoother:

1) No opportunities to game the system anymore. It all comes down to where the acceptable margin of fee vs penalty stands for the given mempool.
2) Very simple to implement.

The drawback is that since there is no reward, obviously the penalties are just destroyed. I'm not sure that's a drawback per say, for the following reasons:

1) It's trivial to blackhole bitcoins and it's been agreed that this is not damaging to the system. So this method isn't introducing some new DOS attacks on the system.
2) By destroying the penalty, the value of every other BTC just went up. As opposed to your system where you want to reward other miners from the penalties, this time everyone is getting rewarded, albeit in a much smaller magnitude. This means both miners, but everyone else holding coins is rewarded when a miner builds a block above the soft cap. Incidentally, that also means people running nodes (as long as they hold BTC, which is expectable).

This is perhaps the only proposition so far that has some sort of reward mechanism for node maintainers (granted it's tiny) which take equal part in the cost of node propagation and validation as miners.
Pages: « 1 ... 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 [161] 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 ... 233 »
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!