How is the target value (PoW) determined?

[bill gator]

I'm trying to get a more in-depth understanding of exactly what is happening within our protocol. I've mostly taken it for granted over the years, and lately I've been really attempting to understand what is under the hood. Understanding how mining works has been particularly difficult, but it seems to have boiled down to where does the target value come from? The "answer" to the "question" all of the miners are rushing to "solve", I cannot figure it out. I did attempt to use the search function here, read through the whitepaper again and can't seem to find my answer. I'm assuming it is in the source code, but I'm a bit incapable of scouring through it.

I think it has something to do with combining all of the transaction data (headers?) that is to be included within the block, and then the miner is solving for the data (nonce) that needs to be added to the block (transaction data) in order to create an appropriate sha-256 hash (X # of leading 0's, according to difficulty) to be accepted, is that right? The hash is unique and unpredictable, because of the information being added to a block constantly being generated (in a decentralized manner), through new transactions and timestamps; still right?

Would someone be able to create a block of exclusively transactions and information they have control over and have already generated a nonce for? Effectively solving an entirely self-created block (in advance) that would be accepted by the blockchain and allow you to reap the block reward. Is this prevented by accepting the "longest" valid chain? This may all be wrong, and foolish.

P.S. I know I can easily see the hash of a block, once solved, but is there any way or any record of the "nonce" or answer that lead to the hash?

[goddog]

I have some problem to understand your question. target is calculated every 2016 (using last 2016blocks-last timestamps)
all nodes calculates the same target from the chain with more accumulated pow(the longest)
a block to be valid must have hash(header)<=target.
header must contain: https://bitcoin.org/en/developer-reference#block-headers
here is where target is calculated in bitcoin core client: https://github.com/bitcoin/bitcoin/blob/5961b23898ee7c0af2626c46d5d70e80136578d3/src/pow.cpp#L49

thats the code used by electrum to check the target.

Code:

def get_target(self, index):
        # compute target from chunk x, used in chunk x+1
        if constants.net.TESTNET:
            return 0
        if index == -1:
            return MAX_TARGET
        if index < len(self.checkpoints):
            h, t = self.checkpoints[index]
            return t
        # new target
        first = self.read_header(index * 2016)
        last = self.read_header(index * 2016 + 2015)
        bits = last.get('bits')
        target = self.bits_to_target(bits)
        nActualTimespan = last.get('timestamp') - first.get('timestamp')
        nTargetTimespan = 14 * 24 * 60 * 60
        nActualTimespan = max(nActualTimespan, nTargetTimespan // 4)
        nActualTimespan = min(nActualTimespan, nTargetTimespan * 4)
        new_target = min(MAX_TARGET, (target * nActualTimespan) // nTargetTimespan)
        return new_target

Would someone be able to create a block of exclusively transactions and information they have control over and have already generated a nonce for? Effectively solving an entirely self-created block (in advance) that would be accepted by the blockchain and allow you to reap the block reward. Is this prevented by accepting the "longest" valid chain? This may all be wrong, and foolish.

block header must contain merkle root for included transactions
and previous block hash to build the chain

the nonce itself is not the solution to pow, the solution is the whole header
thats the code used by electrum to check the header.
hash(header)<=target

Code:

def verify_header(self, header, prev_hash, target):
        _hash = hash_header(header)
        if prev_hash != header.get('prev_block_hash'):
            raise Exception("prev hash mismatch: %s vs %s" % (prev_hash, header.get('prev_block_hash')))
        if constants.net.TESTNET:
            return
        bits = self.target_to_bits(target)
        if bits != header.get('bits'):
            raise Exception("bits mismatch: %s vs %s" % (bits, header.get('bits')))
        if int('0x' + _hash, 16) > target:
            raise Exception("insufficient proof of work: %s vs target %s" % (int('0x' + _hash, 16), target))

[bill gator]

I have to do some research on what a "merkle root" is. I am not asking about difficulty, I am asking about each individual blocks solution. Are there multiple solutions to a block?

[goddog]

I have to do some research on what a "merkle root" is. I am not asking about difficulty, I am asking about each individual blocks solution.

merkle root is a merkle tree of all transactions it is calculated hashing all transactions:
https://bitcoin.org/en/developer-reference#merkle-trees

Are there multiple solutions to a block?

yes, but when you find the first solution you wil start mining a new block. You don't have to find another solution.

[bill gator]

I understand that you begin mining a new block, I was simply wondering if miners were racing towards a single solution or if there were multiple possibilities. Very interesting, I hadn't considered that before.

Seems Merkle root was just a term I hadn't known, but I knew the information that constituted it.

I have a clearer picture now.

I just need to do more reading and research it seems like. Even my questions are malformed, and probably impossible to answer; I'm using the wrong terms and don't understand basic concepts.

[achow101]

The process of mining uses hashing. It really isn't solving a problem or answering a question. It is just doing work. The data that is hashed is the block header. The block header is 80 bytes and contains multiple fields: a version number, the previous block hash, the merkle root (hash of all of the transactions in the block), a timestamp, a nonce, and a compact representation of the target value. A miner's goal is to modify this block header until they are able to find a hash that, when interpreted as a 256 bit integer, is less than some target value.

In order to do this, miners will modify as much data in the block header as possible. You can really imagine this process as simply constructing a block (complete with block header), computing the hash of the block header, and comparing it with the target. Then doing this over and over with different blocks (done by tweaking some values within the block itself) until one is found that has a block header hash that is less than the target.

Miners can change a lot of things in the block header in order to get different hashes. First and foremost is the nonce. The nonce is specifically designed to be changed in order to get different block headers. However the nonce is fairly small, only 4 billion possible values. Thus something else needs to be changed. Miners can then change the timestamp and the merkle root. Miners change the merkle root by including different transactions in their trial blocks. They can also permute the order of transactions. Miners can also change the data within the coinbase transaction such as adding extra data as a extraNonce.

The target value that miners compare their block hashes to is calculated from the original target value from the genesis block. The target value changes every 2016 blocks. So the target value can be known and calculated simply by looking at the blockchain and computing the targets for every 2016 blocks. Since the target uses the timestamp, all data necessary to compute the target is available to anyone who has the complete blockchain. It is important to note that the difficulty adjustment is actually the target adjustment. The difficulty is simply the inverse of the target and is not actually part of the blockchain or consensus rules. Only the target is.

[bill gator]

I am confused about miners being able to modify timestamp data within the header. I thought the only modifiable piece of data within the header was the nonce. I understand changing the Merkle root, through different transactions being included in a trial block, but how are timestamps modifiable? I thought the timestamps where reliant upon the sender, and when they pushed their transaction, is this incorrect?

I understand where the target value comes from now, much more so than last night. Now I'm wondering what happened if/when SHA-256 becomes compromised to some degree. Thanks for the input, goddog and achow101.

I'd still argue that it is doing work, in order to create a solution to a problem/question, but I do understand why that would be confusing terminology. Plus, it's clear that I understand very little about our protocol, so my arguments hold little weight.

[goddog]

I am confused about miners being able to modify timestamp data within the header. I thought the only modifiable piece of data within the header was the nonce. I understand changing the Merkle root, through different transactions being included in a trial block, but how are timestamps modifiable? I thought the timestamps where reliant upon the sender, and when they pushed their transaction, is this incorrect?

I understand where the target value comes from now, much more so than last night. Now I'm wondering what happened if/when SHA-256 becomes compromised to some degree. Thanks for the input, goddog and achow101.

I'd still argue that it is doing work, in order to create a solution to a problem/question, but I do understand why that would be confusing terminology. Plus, it's clear that I understand very little about our protocol, so my arguments hold little weight.

the timestamp is choosen by the miner, it is another arbitrary data the miner can use. But there is a fixed range of values. It can not bee too much in the future or too much in the past.

The block time is a Unix epoch time when the miner started hashing the header (according to the miner). Must be strictly greater than the median time of the previous 11 blocks. Full nodes will not accept blocks with headers more than two hours in the future according to their clock.

[achow101]

I am confused about miners being able to modify timestamp data within the header. I thought the only modifiable piece of data within the header was the nonce. I understand changing the Merkle root, through different transactions being included in a trial block, but how are timestamps modifiable?

It is impossible to enforce that the timestamp be exactly a specific time due to network latency and the imprecision of clocks in computers. So the timestamp in a block just has to be within a range of times that goes from the median time of the last 11 blocks to a few hours ahead of current time. A miner can modify the timestamp as long as it fits within this range of time.

I thought the timestamps where reliant upon the sender, and when they pushed their transaction, is this incorrect?

No. Transactions do not have timestamps and their times have no effect on the block's timestamp.

Now I'm wondering what happened if/when SHA-256 becomes compromised to some degree.

There are two forms of "compromised" for hash functions: a preimage attack and a collision attack. A preimage attack allows the attacker to determine the data that a given hash came from. Basically a preimage attack allows you to reverse a hash. This kind of attack is difficult, and AFAIK, no known preimage attack has been found for any hash function, even ones considered broken. If a preimage attack were found on SHA256, it may allow miners to simply mine blocks faster. Depending on how the attack works. this could mean that blocks are found at a constant time regardless of the difficulty. But such attacks are unlikely, and even if one existed, it may not even result in finding a valid block header since, technically, there are infinitely many pieces of data that hash to a given hash.

Collision attacks are a different story. Collision attacks are much easier to find and possibly more problematic than a preimage attack. All broken hash functions are broken because of collision attacks. A collision attack is where two pieces of data can be found that hash to the same thing. Although technically there exists infinitely many pieces of data that hash to the same thing, finding those pieces of data is incredibly hard and there are so many possible hashes that finding a collision through brute force would take a very very long time. However, a collision attack allows someone to find two colliding pieces of data in a practical time frame (e.g. several years). If someone were able to find another valid block which has a hash that matches one of another existing block (so a collision), then there would be a lot of problems since the hashes would match but the transaction history could be different.