I've been toying around with a 384 bit quadratic extension field curve that supports a four dimensional endomorphism (GLV-GLS).  The use of the quadratic extension avoids much of the bad scaling, but I'm not to a point where I can benchmark anything yet.

yes, they're actually quoting pieter and I from #bitcoin-dev (telling the miner in advance that the transaction he was creating would take a _LONG_ time to verify). They created a huge non-standard 1MB transaction and part of the verification time is quadratic (in the number of inputs).

It's actually possible to create a block that would take many minutes to verify, though not with standard transactions-- only something contrived.

No clue, no node I have access to is seeing that much-- they may have turned off the minfee rules (not unreasonable for a metrics thing)...

Even given that, again, 37MB doesn't explain your swap.

It wouldn't expect the miner latency part to be size dependant: the miner can't even tell how big the prior block was.  I expect your function relating them to have a big constant term in it! (thats why I asked if you tried other regression approaches. )

I suppose there may be some dependance that is introduced by virtue of what percentage of the miners got the dummy work.  Would be pretty interesting to try to seperate that.

Another trap of empirical analysis in this kind of discussion is that we can only measure how the system is-- but then we use that to project the future;  e.g.  say we didn't have ECDSA caching today, you might then measure that it was taking >2 minutes to verify a maximum size block... and yet 100 lines of code and that cost vanishes; which is bad news if you were counting on it to maintain incentives.

Gavin was insufficently precise. There is a reddit thread is full of people calling gavin a fool ( ) for saying "memory" when he should have been saying fast storage.  https://twitter.com/petertoddbtc/status/596710423094788097

Why do you think it's prudent to argue this with me?

Okay, lets take a bet. Since you're so confident; surely you'll grant me 1000:1 odds?-- I'll give my side away to a public cause.

The question is "Is the entire UTXO set kept in ram in Bitcoin Core ever released?"

I will bet 3 BTC and, with the 1000:1 odds, if you lose you'll pay 3000 BTC (which I will to the hashfast liquidators, to return it to the forum members that it was taken from; which will also save you some money in ongoing lawsuit against you).

Sounds good?  How will we adjudicate?  If not, what is your counter-offer for the terms?

The highest number of unconfirmed transactions I've seen ever is about 8MB. Even if we assume the real max was 3x that this is not explaining your hundreds of megabytes of swap.   We just had half the hashpower of the network mining without validating creating multiple large forks and large reorginizations, but you don't see any destabilization. Okay.


Here I suspect you're suffering from an excess of empiracisism without adequately devling into the mechenism.   You can directly measure that time time from input to minable on an actual node under your control and will observe the time is hundreds of times faster than your estimate. Why?   Miners don't magically know when their pool has new work, they'll get work in the first milliseconds and then grind on it some time before submitting returning work.  Even if the pool long polls them, it takes time to replace work. So what I suspect you're actually measuring there is the latency of the mining process...  which is consistent with what we've expirenced with P2Pool (5-20 second latencies from ASIC miners are common).

I noted you posted a result of a classification, did you run the same data through a simple logistic regression with prior size as the treatment? The intercept in the model would be interesting.

But indeed, these conversations have been conflating several seperate issues (latency vs throughput, etc.). Tricky to avoid that since they're all relevant.

I have, they'd previously cranked it down, and were producing small blocks and were flamed in public.  They've since turned it back up.

As PeterR points out, they only need to wait for verification to actually verify (which they're not doing today), though they may have to wait longer to include transactions---- though I point out thats not fundimental e.g. no matter how big the backlog is you can produce a template sufficient to completely fill a block while doing no more work than handling a mempool of twice the maximum block size.  (by using a tiered mempool, though no one has bothered to implement this yet-- no one has even been complaining about how long createnewblock takes, due to the ability to produce empty blocks without skipping transactions).

The blockchain itself constain substantial counter-eficidence. Any block over 750k is running with changed settings; as are a substantial chunk of the transactions.  I think this is all well and good, but it's not the case that its all consistent.

It has never relayed a _single_ block, not in a lab, not anywhere. It does _not_ exist. It certantly can and will exist-- though it's not yet clear how useful it will be over the relay network-- Gavin, for example, doesn't believe it will be useful "until blocks are hundreds of megabytes".

But don't you think that I'm saying anything bad about it-- I'm not. Cypherdoc was arguing that mempools were (and had) to be the same, and cited IBLT as a reason---- but it cannot currently be a reason, because it doesn't exist.  Be careful about assigning virtue to the common fate aspect of it-- as it can make censorship much worse. (OTOH, rusty's latest optimizations reduce the need for consistency; and my network block coding idea-- which is what insired IBLT, but is more complex-- basically eliminates consistency pressure entirely)

It depends on what you're talking about, if you're talking about throughput it's at best a 2x improvement, if your'e talking about latency it's more.  But keep in mind that the existing, widely deployed block relay network protocol reduces the data sent per already known transaction _two bytes_.

GAH! I'm not saying it's a good setting-- I'm just giving a concrete example that nodes (and miners) can control their mempool sizes, as this was at odds with cypherdoc's expectations-- instead he thought miners might be suffering because of large mempools-- and I pointed out that if their mempool was too big they could simply reduce it and he said he didn't believe me. I don't know how I could have made it more clear, but I hope its clear now.

No idea, I'm not aware of anything that tracks that-- also what does "typical mean", do you mean stock unmodified Bitcoin Core?

I expect correlation between empty blocks and mempool size-- though not for the reason you were expecting here: Createnewblock takes a long time, easily as much as 100ms,  as it sorts the mempool multiple times-- and no one has bothered optimizing this at all becuase the standard mining software will mine empty blocks while it waits for the new transaction list. So work generated in the first hundred milliseconds or so after a new block will usually be empty. (Of course miners stay on the initial work they got for a much loonger time than 100ms).

This is, however, unrelated to SPV mining-- in that case everything is still verified. As many people have pointed out (even in this thread) the interesting thing here isn't empty blocks, its the mining on an invalid chain.

And before someone runs off with an argument that aspect of the behavior, instead defines some kind of upper limit-- optimizing the mempool behavior would be trivial if anyone cared to, presumably people will care to when the fees they lose are non-negligible.  Beyond elimiating the inefficient copying and such, the simple of expident of running a two stage pool where the block creation is done against a smaller pool that constains only enough transactions for 2 blocks (which is refilled from a bigger one), would eliminate virtually all the cost. Likewise, as I pointed out up-thread incrementing your minfee can make your mempool as small as you like (the data I captured before was at a time when nodes with a default fee policy had 2.5 MB mempools).

Uh. I don't care what the consensus of the "Gold collapsing" thread is, the UTXO set is not stored in memory. It's stored in disk,  it's in the .bitcoin/chainstate directory.  (And as you may note, a full node at initial startup uses much less memory than the current size of the UTXO). Certantly the UTXO size is a major concern for the viability of the system, since it sets a lower bound on the resource requirements (amount of online storage) for a full node... but it is not held in memory and has no risk of running hosts out of ram as you claim.

In your message to me you argued that f2pool was SPV mining becuase "the" mempool was big. I retored that their mempool has nothing to do with it, and besides they can make their mempool as small as they want. You argued that the mempools were the same, I pointed out that they were not. You responded claiming my responses was inconsistent with the points about verification delay; and I then responsed that no-- those comments were about verification delay, not mempool. The two are unrelated.  You seem to have taken as axiomatic that mempool == verification delay, a position which is technically unjustified but supports your preordaned conclusions; then you claim I'm being inconsistent when I simply point out that these things are very different and not generally related.

I think your phone made your response too short here, I'm not sure where you're going with that.

When you're back on a real computer, I'd also like to hear your response to my thought, that It is "Super weird that you're arguing that the Bitcoin network is overloaded with average of space usage in blocks, while you're calling your system "under utilized" when you're using a similar proportion of your disk and enough of your ram to push you deeply into swap."

Most of the thought has just been of the forum "The utxo set size needs to be kept down" with an emphasis on the minimum resources to run a full node over the long term.  The database itself has n log n behavior, though if the working set is too large the performance falls off--and the fall of is only enormous for non-SSD drives.  Maybe the working set size is owed more attention, but my thinking there is that user tolerance for resource consumption kicks in long before thats a serious issue.

When you talk about "would it be possible" do you mean an attack?  It's possible to construct a contrived block today that takes many minutes to verify, even within the 1MB limit; though a miner that did that would mostly be hurting themselve unless they had some arrangement with most of the hashpower to accept their block.

Again, you're wrong on the technology. The UTXO set is not held in ram. (There is caching, but its arbritary in size, controlled by the dbcache argument).

"They" in that case is sketchy nutballs advocating these "stress tests", and _you_ arguing that unconfirmed transactions are the real danger.

Super weird that you're arguing that the Bitcoin network is overloaded with average of space usage in blocks, while you're calling your system "under utilized" when you're using a similar proportion of your disk and enough of your ram to push you deeply into swap.

PeterR wasn't saying anything about mempools, and-- in fact-- he responded expressing doubt about your claim that mempool size had anything to do with this.  Moreover, I gave instructions that allow _anyone_ to measure verification times for themselves.  Your argument was that miners would be burned by unconfirmed transactions, I responded that this isn't true-- in part because they can keep whatever mempool size they want.

To further make the point about mempools, here is what the mempool looks like on a node with mintxfee=0.0005 / minrelaytxfee=0.0005 set:


$ ~/bitcoin/src/bitcoin-cli  getmempoolinfo
{
    "size" : 301,
    "bytes" : 271464
}


Their response was not to use smaller blocks, their response was to stop validating entirely.  (And, as I pointed out-- other miners are apparently mining without validating and still including transactions).

If we're going to accept that every correlation means causation;  what should we say about the correlation between finding out that you've taken hundreds of thousands of dollars in payments for paid shilling and finding out loud and opinionated you are on this blocksize subject?

In this case, these forks are only visible by someone mining an invalid block, which no one had previously done for over a year.

They are no longer seeing any orphans at all, they "solved" them by skipping validation entirely. They opposed that initial proposal, in fact, and suggested they could at most handle 8MB, which brought about a new proposal which used 8MB instead of 20MB though only for a limited time. Even there the 8MB was predicated on their ability to do verification free mining, which they may be rethinking now.

I am glad to explain things to people who don't understand, but you've been so dogmatically grinding your view that it's clear that every piece of data you see will only "confirm" things for you; in light of that I don't really have unbounded time to waste trying. Perhaps someone else will.

Basic misunderstanding there--- Being a larger miner has two effects: One is throughput not latency related: Being larger creates a greater revenue stream which can be used to pay for better resources.   E.g. if the income levels support one i7 CPU per 10TH/s of mining, then a 10x larger pool can afford 10x more cpus to keep up with the overall throughput of the network, which they share perfectly (relay network is about latency not so much about throughput-- its at best a 2x throughput improvement, assuming you were bandwidth limited);    the other is latency related,   imagine you have a small amount of hashpower-- say 0.01% of the network-- and are a lightsecond away on the moon.  Any time there is a block race, you will lose because all of the earth is mining against you because they all heard your block 1+ seconds later.  Now imagine you have 60% of the hashpower on the moon, in that case you will usually win because even though the earth will be mining another chain, you have more hashpower. For latency, the size of miner matters a lot, and the size of the block only matters  to the extent that it adds delay.

When it comes to orphaning races miner sizes matters, in some amount that is related to the product of the size-of-the-miner and time it takes to validate a block.

There is no requirement that mempools be in sync, -- in fact, they're not and the whole purpose of the blockchain is to synchronize nodes.  The mempools of nodes with identical fee and filtering policies and whom are similarly positioned on the network will be similar, but any change in their policies will make them quite different.

IBLT doesn't currently exist, and other mechenisms like the relay network protocol don't care about mempool synchronization levels.


Almost none of the blocks have been 1MB, the issues arise before then. _Consistent_ 1MB blocks wouldn't have been supportable on the network at the time that limit was put in place-- back in the 0.5.x-ish days we were getting up to 2minutes for a 100k block to reach the whole network; the 1MB was either forward-looking, set too high, or only concenred about the peak (and assuming the average would be much lower) ... or a mixture of these cases.


F2Pool reported that as block sizes grew they saw increased orphaning rates and that they were seeing an orphan rate of 4% though this was at a time before the relay network and when GHash in europe had ~50% of the hashpower under them.  Excluding the recent issues they've had almost no orphans since, they report.

Ouch,  the latency related issues issues are made much worse by smaller interblock gaps once they are 'too small' relative to the network radius. When a another block shows up on the network faster than you can communicate about your last you get orphaned.  And for throughput related bottlenecks it doesn't matter if X transactions come in the form of a 10mb block or 10 1mb blocks.

Because it greatly reduces the time it takes to transmit blocks but does not completely eliminate it-- nothing can (due to the speed of light).  In particular, something I didn't know until my conversation with them on July 4th:  the nearest relay network hub to F2Pool is still 200ms away due to insane routing that sends traffic between some networks in china and singapore via the US (thanks NSA?).

No.

I don't know what you're talking about there.  AFAICT F2Pool would also consider e.g. 750k "large".

Do you mean why was 1MB selected as the particular hard limit in the protocol?   ::shrugs:: It happens to be the the highest value you could sync over a modem and stay up with the network (though not for mining, due to latency), though that could be by chance.

Thats quite fair and true.

As an aside, today's 3 block invalid chain reorg included a 'v3' block on the invalid fork which contained a lot of transactions. Which may suggest someone is SPV mining while including transactions (something I'd pointed out was possible previously).

He seems to have representation now (its in the legal docs but PM me if you need contact information for his attornies for some reason).

Makes sense, after all-- 3000 BTC pay for a lot of representation.  I assume part of the point of getting the attachment is so that he can't spend it all on his legal defense and then claim to be insolvent after losing.

From the complaint, the agreement was 10%, and yes-- more than 30,000 BTC it appears.

The audio recording in the response is halarious, Cypherdoc's attorney describes him as the  "the LeBron James of the Bitcoin world"-- presumably that was the least redicilous claim they could deploy to provide any other rationale the amounts paid in an effort to avoid the conclusions people actually in the Bitcoin world drawing in this thread.

Where I expicitly pointed out to you in many places, in excruciating detail, that this was not at all the case??  https://www.reddit.com/r/Bitcoin/comments/39tgno/letting_miners_vote_on_the_maximum_block_size_is/cs6rek5?context=3   You seemed so happy to argue with it before, has your memory vanished now that you don't think it would be convient for you?


The incohearence in some of these posts is so foaming so thick that it's oozing out and making the floor slick; careful-- you might slip and mess up your future as "the LeBron James of the Bitcoin world" (as your attorney decribed you (18:30), under oath, to a federal judge as part of litigation related to your possession of 3000 BTC taken primarly from members of this forum.).

As miners have created larger blocks F2Pool expirenced high orphaning (>4% according to them); they responded by adding software to mine without transfering or verifying blocks to avoid delays related to transfering and processing block data. Contrary to your claim-- the blocksize limit stems the bleeding here. Their issue is that large blocks take more time to transfer/handle and that they're falling behind as a result. Making blocks _bigger_ would not help this problem, it would do the _opposite_. If a miner wanted to avoid any processing of transaction backlog they'd simply set their minimum fee high and they'd never even mempool the large backlog.

Reasonable minds can differ on the relative importance of difference considerations, but when you're falling all over yourself to describe evidence against your position as support of it-- redefining F2pools crystal clear and plain descption of "large blocks" as their source of problems with the technically inexplicable "full" that you think supports your position, it really burns up whatever credibility you had left. That you can get away with it in this thread without a loud wall of "WTF" just shows what a strange echochamber it has become.

The database is intended to handed unclean shutdowns and I tested it with _thousands_ of unclean shutdowns previously.  There may be system specific reasons or hardware problems that are causing you to see corruption on power loss, they're certantly not expected or intended (or expirenced by everyone else). If it is the case that your system is outright corrupting recently written blocks on unclean shutdown then there may be no real remedy possible.

Neither of these two things improve the time it takes to download blocks.

There is no BIP62 in the Bitcoin system today. It's an incomplete proposal; so there is no support for BIP62 at all.

Well you called out the reason there-- not having an SSD.  I run bitcoin core on a battery life optimized ultra light laptop and never notice it running.  That its a fine desktop application doesn't mean that it's super awesome on a 4 year old non-ssd enabled laptop. The much larger blocks these days do require more work to catch up on, but the software is also quite a bit faster.

Assuming a user will use all of their bandwidth is probably not good, You probably need to assign a marginal utlity to running a node and then scaling their tolerance based on their marginal utility and the available bandwidth.

I haven't personally chased this kind of approach because there will be so many judgements that the result won't really be interesting to anyone but the person who created it... but perhaps I'm wrong.

This isn't fair or correct.

Bitcoin Core should work fine for Decksperiment, it shouldn't even mind his poor attitude.

But it's not-- even though it works fine for many many other people--, unfortunately Decksperiment has not provided enough useful information to begin troubleshooting, and his approach doesn't exactly encourage anyone competent to try to help.

If you're on windows and running the latest version, then I suggest running memtest86 and some disk check-- defective hardware is a common cause of issues. If you have antivirus software some of it is known to corrupt Bitcoin, you should disable it or except the Bitcoin directories.