Technical question(where and how to get a block header to hash)

[andnan]

So, I've read the articles and this is interesting stuff.  I do however have some technical questions.  How and where do you get a block header?  I'm not looking for a "from the pool" answer but a more technical "connect to bla.bla.bla port blabla and send a Jason formatted packet as defined by protocol bla bla bla and parse the response as define by protocol bla bla bla.", you know technical stuff?  

[DumbFruit]

"getwork"
JSON-RPC over http, without a data parameter gives you the block header you're looking for.

https://en.bitcoin.it/wiki/Getwork
https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list

Edit: Woops, ya getblocktemplate supercedes that method (below), that's what I get for skimming over the information..
https://en.bitcoin.it/wiki/Getblocktemplate

[achow101]

The two most commonly used mining protocols are stratum and getblocktemplate. They use JSON formatted requests submitted to a server through an HTTP Post request. You can read the stratum specifications at https://mining.bitcoin.cz/help/#!/manual/stratum-protocol and the getblocktemplate specifications at https://github.com/bitcoin/bips/blob/master/bip-0022.mediawiki

[andnan]

Thanks, it's exactly what I was looking for.

[DannyHamilton]

So, I've read the articles and this is interesting stuff.  I do however have some technical questions.  How and where do you get a block header?  I'm not looking for a "from the pool" answer but a more technical "connect to bla.bla.bla port blabla and send a Jason formatted packet as defined by protocol bla bla bla and parse the response as define by protocol bla bla bla.", you know technical stuff?  

Technical stuff?

Ok...

Edit:  Ok, so perhaps I went too far?  I suppose this isn't the level of technical details you were looking for?

Start by building software that connects to peers on the network.

You'll need to implement your own peer discovery system. The default port that most peers listen on is port 8333.

Implement the bitcoins communications protocol to request blocks from your connected peers.

Verify the entire contents of every block since the genesis block to make sure you are on the correct chain.

Once you've downloaded and verified the entire blockchain, you'll need to take note of the sha256d hash of the most recently solved block.

Acquire a set of valid transactions that you intend to confirm (don't forget to include the coin generation transaction that will pay you your block reward).  Organize those transactions and build a merkle tree.  Take note of the merkle root.

Now you are ready to build your own block header.

The first 4 bytes will be the block version number represented as a signed integer.
The next 32 bytes are the 32 byte shad256d result of the most recently solved block.
The next 32 bytes are the merkle root calculated from your transaction list.
The next 4 bytes are an unsigned integer timestamp represented as the number of seconds since 1970-01-01T00:00 UTC
The next 4 bytes are a representation of the current difficulty target
The last 4 bytes are available for you to use as a nonce in your hashing algorithm

Sha256d hash this header and if the resulting value is lower than the current difficulty target, then broadcast the block to all your connected peers.
If the resulting value is NOT lower than the current difficulty target, then you'll need to modify one of the values and try again.

The easiest value to quickly modify is the nonce.  If you've used up all the available nonce values and still haven't found a solution, then you can modify the timestamp.  If you have run out of valid timestamp values and you still haven't found a solution, then you'll need to modify your transaction list and re-calculate your merkle root.

You can find more resources here:
https://en.bitcoin.it/wiki/Block_hashing_algorithm
https://en.bitcoin.it/wiki/Protocol_documentation
https://en.bitcoin.it/wiki/Difficulty

[andnan]

My reply was not sarcastic, it was accurate as is, I truly appreciated you taking the time to answer a noobs question.  Once again with a humble heart, thanks. 

The additional details are useful and it points that I may not have a full understanding of how this is working.  Can I connect to an existing server and get a block header without being part of a pool?  If I start my own pool with freeware (with just myself in it) will it know how to get block headers and my app simply sends a JSON request to a pool server on the same machine?  I'm just trying to write some code in my spare time to keep my java skills sharp.  I'm a hardware engineer and most of my code is embedded firmware.  I haven't written any JSON code since I finished my masters.  My goal is to write a miner in java(yea, you can start laughing now) just for the technical challenge.  I'm sure all those ASIC guys a shaking in their boots.

Anything is hard when you don't know how to do it.

[achow101]

My reply was not sarcastic, it was accurate as is, I truly appreciated you taking the time to answer a noobs question.  Once again with a humble heart, thanks. 

The additional details are useful and it points that I may not have a full understanding of how this is working.  Can I connect to an existing server and get a block header without being part of a pool?  If I start my own pool with freeware (with just myself in it) will it know how to get block headers and my app simply sends a JSON request to a pool server on the same machine?  I'm just trying to write some code in my spare time to keep my java skills sharp.  I'm a hardware engineer and most of my code is embedded firmware.  I haven't written any JSON code since I finished my masters.  My goal is to write a miner in java(yea, you can start laughing now) just for the technical challenge.  I'm sure all those ASIC guys a shaking in their boots.

Anything is hard when you don't know how to do it.

You can get all of the data needed for putting together a header by running a local Bitcoind and then sending it getblocktemplate requests through the JSON-RPC server run by Bitcoind. It will return all of the information you need in order to create a block.

[DannyHamilton]

Can I connect to an existing server and get a block header without being part of a pool?

Bitcoin operates as a peer-to-peer network. There aren't "clients" and "servers".  Of course someone operating a "peer" could create a service for others to use. They could then set up their own "server" and allow others to connect to their server to acquire whatever information they wish to provide.  This is what the blockchain.info website has done.  They run many peers that operate on the network. Those peers collect information about what is happening on the bitcoin network and that information is then stored in a database.  They then provide an API that people can use if they want to trust blockchain.info to provide valid information.

So, you can create software that generates your own block headers, or you can use existing software such as Bitcoin Core to generate the headers, or you can find a service (or pool) that will provide you with pre-built headers.

[andnan]

I'm learning as I go....

Blockchain provides a list of unconfirmed transactions(https://blockchain.info/unconfirmed-transactions?format=json).  So I add a coin generation transaction(still lost on how to do this) to the top of the list and create a merkle tree from this list.  This is how I get the Merkle root for my newly minted header.  Now I perform the SHA256(SHA256(Block_Header)).  Now through my coding wizardry I find a hash with the appropriate number of leading zero's in the first round(yea, I'm that good, lucky or whatever....).   Any idea on how to send this winning header to the collective?  I can't see (or don't know where to look for) documentation on this from Blockchain.

Once again, thanks everyone for the help.

[achow101]

I'm learning as I go....

Blockchain provides a list of unconfirmed transactions(https://blockchain.info/unconfirmed-transactions?format=json).  So I add a coin generation transaction(still lost on how to do this) to the top of the list and create a merkle tree from this list.  This is how I get the Merkle root for my newly minted header.  Now I perform the SHA256(SHA256(Block_Header)).  Now through my coding wizardry I find a hash with the appropriate number of leading zero's in the first round(yea, I'm that good, lucky or whatever....).   Any idea on how to send this winning header to the collective?  I can't see (or don't know where to look for) documentation on this from Blockchain.

Once again, thanks everyone for the help.

Blockchain.info probably doesn't have information on this. For specific information on Bitcoin protocol stuff, you should read the documentation at https://bitcoin.org/en/developer-documentation. The guide and reference explain things very well.

Everything that you want to do is supported by Bitcoin Core. Bitcoin Core has a console command which will allow you to submit a block to the network. It is aptly named submitBlock. The full documentation is at https://bitcoin.org/en/developer-reference#submitblock. To use it, start up Bitcoin Core and open the console in Help > Debug Window and then the console tab. Then just type

Code:

submitblock <block hex>

where <block hex> is the hex string of the block's data. It will submit the block when you hit enter.

[Kefkius]

...
So I add a coin generation transaction(still lost on how to do this)
...

The coin generation ("coinbase") transaction is a transaction with a null input - an input that appears to spend output 0xffffffff (the maximum output number) of transaction 0000000000000000000000000000000000000000000000000000000000000000. Each block is allowed to have only one. The transaction pays X coins to you, where X is the block reward. What separates it from any other transaction is that those X coins aren't actually being spent, they're being generated.