R value parse from signature on Blockchain input transactions

[mamuu]

Hi!

Can I export all signatures from blockchain and parse R values of signatures? Can you suggest a method?

https://gz.blockchair.com/bitcoin/inputs/
from the files at this address
I did it with Python but it's very slow
Do you know a code with C?

Thank you.

[MixMAx123]

If you're looking for double r-values, forget about it!
You are definitely not the first to search the blockchain for it.

I wrote a blockchain explorer that you can use to search for anything in the blockchain. But it's not finished yet and needs a lot of memory and RAM.

[pooya87]

Can I export all signatures from blockchain and parse R values of signatures?

There are billions of signatures on the bitcoin blockchain, why do you need to export "all" of them?

I did it with Python but it's very slow
Do you know a code with C?

Things aren't "very slow" because of the programming language, they are usually slow because of bad code or simply slow process.
As for other code, there are general blockchain parsers but there is no point in extracting signatures alone so there isn't any "good" code for it.

[mamuu]

If you're looking for double r-values, forget about it!
You are definitely not the first to search the blockchain for it.

I wrote a blockchain explorer that you can use to search for anything in the blockchain. But it's not finished yet and needs a lot of memory and RAM.

Thank you

I will not be looking at repeating R values. I will use it in some tests and at the same time I am trying to learn and develop the parse operation in the blockchain.

Can I export all signatures from blockchain and parse R values of signatures?

There are billions of signatures on the bitcoin blockchain, why do you need to export "all" of them?

I did it with Python but it's very slow
Do you know a code with C?

Things aren't "very slow" because of the programming language, they are usually slow because of bad code or simply slow process.
As for other code, there are general blockchain parsers but there is no point in extracting signatures alone so there isn't any "good" code for it.

Thank you for the explanation pooya87.

I had such an idea after reviewing the video here.
https://www.youtube.com/watch?v=C6zrPMnyWSw

[PawGo]

Download the whole blockchain (bitcoin-core).
Use https://github.com/gcarq/rusty-blockparser to parse blockchain. As you are developer you will manage to export only sigscript from each transaction input. (hint: change file https://github.com/gcarq/rusty-blockparser/blob/master/src/callbacks/csvdump.rs)
Then you may write your program to extract R from each sigscript.

[pooya87]

I had such an idea after reviewing the video here.
https://www.youtube.com/watch?v=C6zrPMnyWSw

That's a pretty bad example.
First of all that is not the correct way of benchmarking a code, as you can see the repeated execution of the same command reports different times (totals) that have a spread with a difference as big as 68% to my calculation.
The correct way would be a warm-up, then execution of the same method (aka a function) in a loop with an accurate timer attached that measures the time it takes to run that chunk of code then reports the mean with deviance discarding any value that is way off the mark. This also eliminates the time it takes for the first execution of the program.
I don't know much about python but I can write a C# program that takes a longer time to open for the first time (eg. self-contained without ReadyToRun) but then runs just as fast C.

Secondly the speed of an application as a whole depends on the quality of the code and the amount of optimization that has gone into it. A bad C application runs just as slowly as A bad python application. It comes down to how familiar that developer is with the language and how correctly they can benchmark it and how effectively they can improve their code. For example there are certain cases where the following runs faster:

Code:

for (int i=0; i<n; i+=4)
{
  sum1 += data[i+0];
  sum2 += data[i+1];
  sum3 += data[i+2];
  sum4 += data[i+3];
}
sum = sum1 + sum2 + sum3 + sum4;

Finally C is faster than most other languages in certain cases and only if you have squeezed every ounce of performance from codes written in each language.

Take a look at these two posts, different people write the same code in different languages some even rewrite the same code in python and get faster results than the starter:
https://codegolf.stackexchange.com/questions/26323/how-slow-is-python-really-or-how-fast-is-your-language
https://codegolf.stackexchange.com/questions/26371/how-slow-is-python-really-part-ii

[NotATether]

Bear in mind that any program that extracts R values will need to use different procedures for segwit and non-segwit transactions, because segwit transactions have the signatures in the witness data while normal transactions (including "wrapped" segwit transactions) have them in the scriptSig.

Secondly the speed of an application as a whole depends on the quality of the code and the amount of optimization that has gone into it. A bad C application runs just as slowly as A bad python application. It comes down to how familiar that developer is with the language and how correctly they can benchmark it and how effectively they can improve their code. For example there are certain cases where the following runs faster:i

C# is actually much faster than Python because the files are compiled down to byte code that .NET runtime understands.  Same with Java and other languages with a runtime. Python has an not well documented feature where it can compile the python files down into its own bytecode separately, and you can get similar performance as C# from that since the burden of parsing statements is removed.

C is just compiling all its statements down to assembly code for x86 processors which is just a tiny bit faster than C# for all code that isn't doing some scientific for math work because nearly all programs for the most part spend most of their time making system calls or altering data structures for which native assembly code cannot optimize (there are SSE registers, AVX registers, the L1 and L2 cache but they are all too small to store complex data structures and are useless for shortening the overhead programs transition between user land and OS land).

And of course all performance advantages of C (and C++ which for speed purposes is just C with classes) are lost the moment you use a big general-purpose library inside your program.

There are bitcoin-related programs written in C and the reason they are so fast is because they use their own hand-written functions, and this works because they know their application requirements so they only write the functionality that will actually be used, unlike a general-purpose library (such as Boost) which cannot make this assumption.

[LoyceV]

https://gz.blockchair.com/bitcoin/inputs/
from the files at this address
I did it with Python but it's very slow

If it helps, I made [~500 GB work in progress] Bitcoin block data: inputs, outputs and transactions:

See blockdata.loyce.club
blockdata.loyce.club/inputs/ (159 GB - currently incomplete, growing 8 GB per day)
blockdata.loyce.club/outputs/ (117 GB)
blockdata.loyce.club/transactions/ (43 GB)
Sample: blockchair_bitcoin_addresses_and_balance_DATE.tsv.gz (sorted by balance, highest first (in satoshis)):

It's not complete yet, and unfortunately I only now noticed my download got stuck 7 days ago. Currently, I have 220-ish GB of data from "inputs" available at much better download speed than Blockchair.com itself offers.

[mamuu]

Hi

My question went like this

I understand asserts in python. but I didn't understand the first '48' part.
my aim is to separate the "signature" and separate it as in the link below, of course together with the PubKey

Can you give me information about conditions (assert) that will help me?
Thanks.


information that helps me
https://bitcoin.stackexchange.com/questions/58853/how-do-you-figure-out-the-r-and-s-out-of-a-signature-using-python


type :
pubkeyhash

spending_signature_hex :
483045022100cbee7b355c737bccdaaaf566b52e07c6e560fa33861b3035d37feffad94f66e6022 053114a529ab13eba906f9e08288bcf02dd5142ae0ecb117f10d54f56868443af014104d720973d f5c090aa1dd17adefd9f575baa2f0547dd2913fb4982bf423f1cf82623b700e0d1b7f2b7b5dcfa9 301f8c197dd0b601204d00e02b544251c5d9ac45a

information I want to parse
r = 00cbee7b355c737bccdaaaf566b52e07c6e560fa33861b3035d37feffad94f66e6
s = 53114a529ab13eba906f9e08288bcf02dd5142ae0ecb117f10d54f56868443af
PubKey= 04d720973df5c090aa1dd17adefd9f575baa2f0547dd2913fb4982bf423f1cf82623b700e0d1b7f 2b7b5dcfa9301f8c197dd0b601204d00e02b544251c5d9ac45a

Code:

def parse_element(hex_str, offset, element_size):
    """
    :param hex_str: string to parse the element from.
    :type hex_str: hex str
    :param offset: initial position of the object inside the hex_str.
    :type offset: int
    :param element_size: size of the element to extract.
    :type element_size: int
    :return: The extracted element from the provided string, and the updated offset after extracting it.
    :rtype tuple(str, int)
    """

    return hex_str[offset:offset+element_size], offset+element_size


def dissect_signature(hex_sig):
    """
    Extracts the r, s and ht components from a Bitcoin ECDSA signature.
    :param hex_sig: Signature in  hex format.
    :type hex_sig: hex str
    :return: r, s, t as a tuple.
    :rtype: tuple(str, str, str)
    """

    offset = 0
    # Check the sig contains at least the size and sequence marker
    assert len(hex_sig) > 4, "Wrong signature format."
    sequence, offset = parse_element(hex_sig, offset, 2)
    # Check sequence marker is correct
    assert sequence == '30', "Wrong sequence marker."
    signature_length, offset = parse_element(hex_sig, offset, 2)
    # Check the length of the remaining part matches the length of the signature + the length of the hashflag (1 byte)
    assert len(hex_sig[offset:])/2 == int(signature_length, 16) + 1, "Wrong length."
    # Get r
    marker, offset = parse_element(hex_sig, offset, 2)
    assert marker == '02', "Wrong r marker."
    len_r, offset = parse_element(hex_sig, offset, 2)
    len_r_int = int(len_r, 16) * 2   # Each byte represents 2 characters
    r, offset = parse_element(hex_sig, offset, len_r_int)
    # Get s
    marker, offset = parse_element(hex_sig, offset, 2)
    assert marker == '02', "Wrong s marker."
    len_s, offset = parse_element(hex_sig, offset, 2)
    len_s_int = int(len_s, 16) * 2  # Each byte represents 2 characters
    s, offset = parse_element(hex_sig, offset, len_s_int)
    # Get ht
    ht, offset = parse_element(hex_sig, offset, 2)
    assert offset == len(hex_sig), "Wrong parsing."

    return r, s, ht


example_sig = None  # insert_sig_here
r, s, ht = dissect_signature(example_sig)
print "r: %s\ns: %s\nht: %s\n" % (r, s, ht)

[alexeyneu]

There are bitcoin-related programs written in C and the reason they are so fast is because they use their own hand-written functions, and this works because they know their application requirements so they only write the functionality that will actually be used, unlike a general-purpose library (such as Boost) which cannot make this assumption.

C has sse support. almost no one use it.  
say

Code:

	std::cout <<"\nCoinbase: "<< txScriptSig <<"\nPubkeyScript: "<< pubScriptSig <<"\nMerkle Hash: "<< merkleHash <<"\nByteswapped: "<< merkleHashSw <<"\nGenerating block...\n\n";
	unsigned char  block_hash1[32];
#ifdef _MSC_VER
		_declspec(align(16))	unsigned char  block_hashfp[32];
#else
		__attribute__((aligned(16))) unsigned char  block_hashfp[32];
#endif
		unsigned int counter=0, start = time(NULL);
		int drift = 4;
	__m128i r, m, b, camp;
	__m128i zero = _mm_setzero_si128();

	unsigned short turt;
	while(1) {

		SHA256((unsigned char*)&block_header, 80, block_hash1);
		SHA256(block_hash1, 32, block_hashfp);

		r = _mm_load_si128((__m128i *)block_hashfp);
		m = _mm_load_si128((__m128i *)(block_hashfp+16));
		switch (drift)
		{
			case 0:
			    break;
			case 1:
				b = _mm_alignr_epi8(m, r, 31);

			    break;
			case 2:
				b = _mm_alignr_epi8(m, r, 30);
		
			    break;
			case 3:
				b = _mm_alignr_epi8(m, r, 29);
			    
				break;
			case 4:
				b = _mm_alignr_epi8(m, r, 28);
		
			    break;
			case 5:
				b = _mm_alignr_epi8(m, r, 27);
		
			    break;
			case 6:
				b = _mm_alignr_epi8(m, r, 26);
		
			    break;
			case 7:
				b = _mm_alignr_epi8(m, r, 25);
		
			    break;
			case 8:
				b = _mm_alignr_epi8(m, r, 24);
	
			    break;
			case 9:
				b = _mm_alignr_epi8(m, r, 23);

			    break;
			case 10:

				b = _mm_alignr_epi8(m, r, 22);
			    break;
			case 11:
				b = _mm_alignr_epi8(m, r, 21);

			    break;
			case 12:
				b = _mm_alignr_epi8(m, r, 20);
		
			    break;
			case 13:
		
			    break;
			default:

			    break;
		}
/*now it turns out like:
* Hashes 6018c18fcd4bfabb523f2ee1f6c18880f38135f3eb0ff1aa76acbc2fa94eb8ce
* And 
* b		0000000000000000 000000006018c18f
*/

		camp = _mm_cmpeq_epi8(b ,zero);
		turt = _mm_movemask_epi8(camp);
		if (turt == 0xffff)
		{
			std::reverse(block_hashfp,block_hashfp +32);
			std::cout << "\nBlock found!\nHash: " << bin2hex(block_hashfp, 32) <<"\nNonce: " << block_header.startNonce << "\nUnix time: "<< block_header.unixtime << std::endl;
			break;
		}
		if(time(NULL) - start > 0) {
			std::cout << counter <<" Hashes/s, Nonce "<< block_header.startNonce<<'\r';
			start = time((time_t *)(counter = 0));
		}
		if( ++block_header.startNonce == 0 ) block_header.unixtime++ ; //trick is that to change pre-start time to find a block(really it's smth else) faster then nonce wraps
		counter++;
	}

 
48 stuff looks like some hex'ed ascii phase(readable)

[COBRAS]

Hi!

Can I export all signatures from blockchain and parse R values of signatures? Can you suggest a method?

https://gz.blockchair.com/bitcoin/inputs/
from the files at this address
I did it with Python but it's very slow
Do you know a code with C?

Thank you.

I use this - rscan + rsz scrypt

https://cryptoxploit.com/rsz-key/

But data need recheck, because public key not for rsz ome time

[mamuu]

There are bitcoin-related programs written in C and the reason they are so fast is because they use their own hand-written functions, and this works because they know their application requirements so they only write the functionality that will actually be used, unlike a general-purpose library (such as Boost) which cannot make this assumption.

C has sse support. almost no one use it.  
say

Code:

	std::cout <<"\nCoinbase: "<< txScriptSig <<"\nPubkeyScript: "<< pubScriptSig <<"\nMerkle Hash: "<< merkleHash <<"\nByteswapped: "<< merkleHashSw <<"\nGenerating block...\n\n";
	unsigned char  block_hash1[32];
#ifdef _MSC_VER
		_declspec(align(16))	unsigned char  block_hashfp[32];
#else
		__attribute__((aligned(16))) unsigned char  block_hashfp[32];
#endif
		unsigned int counter=0, start = time(NULL);
		int drift = 4;
	__m128i r, m, b, camp;
	__m128i zero = _mm_setzero_si128();

	unsigned short turt;
	while(1) {

		SHA256((unsigned char*)&block_header, 80, block_hash1);
		SHA256(block_hash1, 32, block_hashfp);

		r = _mm_load_si128((__m128i *)block_hashfp);
		m = _mm_load_si128((__m128i *)(block_hashfp+16));
		switch (drift)
		{
			case 0:
			    break;
			case 1:
				b = _mm_alignr_epi8(m, r, 31);

			    break;
			case 2:
				b = _mm_alignr_epi8(m, r, 30);
		
			    break;
			case 3:
				b = _mm_alignr_epi8(m, r, 29);
			    
				break;
			case 4:
				b = _mm_alignr_epi8(m, r, 28);
		
			    break;
			case 5:
				b = _mm_alignr_epi8(m, r, 27);
		
			    break;
			case 6:
				b = _mm_alignr_epi8(m, r, 26);
		
			    break;
			case 7:
				b = _mm_alignr_epi8(m, r, 25);
		
			    break;
			case 8:
				b = _mm_alignr_epi8(m, r, 24);
	
			    break;
			case 9:
				b = _mm_alignr_epi8(m, r, 23);

			    break;
			case 10:

				b = _mm_alignr_epi8(m, r, 22);
			    break;
			case 11:
				b = _mm_alignr_epi8(m, r, 21);

			    break;
			case 12:
				b = _mm_alignr_epi8(m, r, 20);
		
			    break;
			case 13:
		
			    break;
			default:

			    break;
		}
/*now it turns out like:
* Hashes 6018c18fcd4bfabb523f2ee1f6c18880f38135f3eb0ff1aa76acbc2fa94eb8ce
* And 
* b		0000000000000000 000000006018c18f
*/

		camp = _mm_cmpeq_epi8(b ,zero);
		turt = _mm_movemask_epi8(camp);
		if (turt == 0xffff)
		{
			std::reverse(block_hashfp,block_hashfp +32);
			std::cout << "\nBlock found!\nHash: " << bin2hex(block_hashfp, 32) <<"\nNonce: " << block_header.startNonce << "\nUnix time: "<< block_header.unixtime << std::endl;
			break;
		}
		if(time(NULL) - start > 0) {
			std::cout << counter <<" Hashes/s, Nonce "<< block_header.startNonce<<'\r';
			start = time((time_t *)(counter = 0));
		}
		if( ++block_header.startNonce == 0 ) block_header.unixtime++ ; //trick is that to change pre-start time to find a block(really it's smth else) faster then nonce wraps
		counter++;
	}

 
48 stuff looks like some hex'ed ascii phase(readable)

i will review

Thanks alexeyneu

[gmaxwell]

anyone want to take bets on the above 'precompiled binary' from the newbie account being malware?

[mamuu]

anyone want to take bets on the above 'precompiled binary' from the newbie account being malware?

We are not being optimistic anymore, I agree and increase the bet.