[1. Scalability]

We're trying to address some of the problems and shortcoming with the existing mnemonic algorithms namely BIP-39 and Electrum. The idea is to add the information that the wallet needs to derive child keys to final string that the user writes down and more importantly we see mnemonic as a simple encoding and avoid modifying the entropy when deriving the BIP-32 seed (see #4).

The C# implementation can be found here: https://github.com/Autarkysoft/Denovo/blob/master/Src/Autarkysoft.Bitcoin/Experimental/BetterMnemonic.cs
Like any experimental class under Experimental namespace this is a rough idea and the code is untested.

Code:

4 bit version | 4 bit depth | 32 bit index * depth | CompactInt entropy length | entropy | checksum (pad)

1. Scalability
To make the algorithm scalable a small 4 bit version is added to the beginning of the encoding. This way we can have different definitions using the same scheme.

2. Derivation path/script type
In order to let the wallet know which BIP-32 derivation path to use, that path has to be encoded into the mnemonic. This path could also act as the script (address) type of the child keys. For instance m/84'/0'/0'/0 is for P2WPKH addresses.
The downside is increasing the length of the final mnemonic. This could be mitigated by introducing version 2 where instead of encoding the entire path a 4 bit integer is used to indicate the predefined paths. 16 hard-coded paths can be defined this way and for anything new next version could be used.
version 1: 4-bit version | 4 bit derivation path depth | each index as 32-bit unsigned integers
version 2: 4-bit version | 4 bit hard-coded path

BIP-39 doesn't have this feature.
Electrum has a partial solution by first increasing the entropy size from 128 to 132 bits then brute forcing a version that indicates the derivation paths and script types

3. Creation time
This can be useful for the full nodes to avoid re-scanning the entire blockchain when the user imports their mnemonic. The creation time (as a LockTime object) can quickly tell the client from when in the history to begin the scan. For example a mnemonic created today doesn't have to scan the 300 GB of existing history.
Since this is LockTime as used in transactions (instead of simple DateTime) it could be used by both online and offline clients generating the mnemonic as the online synced client knows the block height and can use that and the offline (cold storage) can use the DateTime (values as Unix timestamp and above 500,000,000).

The rest is implementation detail, for example the wallet UI can show the LockTime (block height or DateTime) and still give the user the option to do a full rescan.

4. No modification of the initial entropy
This is the main and biggest difference. Both BIP-39[2] and Electrum[3] are modifying the mnemonic:
A. Modifying by normalization
BIP-39 uses a Unicode normalization with full compatibility decomposition (form KD) but Electrum takes a step further and modifies the input mnemonic more by removing accents, changing Chinese words, removing spaces for word lists other than English, etc. Then the byte array representation of the string is used to derive the keys.
This can create incompatibility between different implementations and lead to losses specially for any word list other than English.

If the words are only used to look up index of each word in the word list to get the entropy out and nothing else, this problem is eliminated.
In other words if user enters a word like "aliener" instead of "aliéner" (from French word list) it still can be searched inside the word list and if the search fails (due to bad normalization) the import process fails right away instead of going ahead. While normalizing this word can give different values:

Code:

0x616c6965cc816e6572  No normalization
0x616c6965cc816e6572  NFKD
0x616c69656e6572      Electrum
0x616c69656e6572      No accent

The problem is more palpable when using Japanese word list for example (the start of the difference is marked by ^, also note the lengths):

Code:

そつう　れきだい　ほんやく　わかす　りくつ　ばいか　ろせん　やちん　そつう　れきだい　ほんやく　わかめ
e3819de381a4e38186e38080e3828ce3818de381a0e38184e38080e381bbe38293e38284e3818fe38080e3828fe3818be38199e38080e3828ae3818fe381a4e38080e381b0e38184e3818be38080e3828de3819be38293e38080e38284e381a1e38293e38080e3819de381a4e38186e38080e3828ce3818de381a0e38184e38080e381bbe38293e38284e3818fe38080e3828fe3818be38281
e3819de381a4e38186e3828ce3818de3819fe38184e381bbe38293e38284e3818fe3828fe3818be38199e3828ae3818fe381a4e381afe38184e3818be3828de3819be38293e38284e381a1e38293e3819de381a4e38186e3828ce3818de3819fe38184e381bbe38293e38284e3818fe3828fe3818be38281
e3819de381a4e3818620e3828ce3818de3819fe38299e3818420e381bbe38293e38284e3818f20e3828fe3818be3819920e3828ae3818fe381a420e381afe38299e38184e3818b20e3828de3819be3829320e38284e381a1e3829320e3819de381a4e3818620e3828ce3818de3819fe38299e3818420e381bbe38293e38284e3818f20e3828fe3818be38281
                  ^

B. Use the UTF-8 decoded bytes of the mnemonic instead of the entropy
When creating a mnemonic a random entropy with a good distribution of bits is chosen. For example in a 12 word mnemonic the entropy is an octet string with length of 16 (ie. 128 bits). Each octet can have a value between 0 and 255 inclusive which is 256 different values.
When this is converted to 12 words using English word list then UT8 decoded to be used to derive BIP-32 seed, it is a longer octet string but each octet can only have a value between 97 and 122 inclusive which is only 26 different values.
In other words a big bias is introduced in the input of PBKDF2 while the initial entropy had no bias at all (assuming it was chosen using a strong RNG).

By using the entropy itself we can eliminate this bias and also avoid the normalization bugs that could occur in different implementations while greatly simplifying the implementation.

Downside
The only disadvantage of this proposal is the longer mnemonic. For example a 12-word BIP-39/Electrum mnemonic would turn into a 26-word version 1 BetterMnemonic using BIP-84 derivation path (without locktime) or 14-word version 2+ (without locktime).

[1] Experimental-1: https://bitcointalk.org/index.php?topic=5245712.0
[2] https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki#from-mnemonic-to-seed
[3] https://github.com/spesmilo/electrum/blob/392a648de5f6edf2e06620763bf0685854d0d56d/electrum/mnemonic.py

I understand what it does and why, I just don't get how it does it.

Code:

SECP256K1_INLINE static int secp256k1_scalar_check_overflow(const secp256k1_scalar *a) {
    int yes = 0;
    int no = 0;
    no |= (a->d[7] < SECP256K1_N_7); /* No need for a > check. */
    no |= (a->d[6] < SECP256K1_N_6); /* No need for a > check. */
    no |= (a->d[5] < SECP256K1_N_5); /* No need for a > check. */
    no |= (a->d[4] < SECP256K1_N_4);
    yes |= (a->d[4] > SECP256K1_N_4) & ~no;
    no |= (a->d[3] < SECP256K1_N_3) & ~yes;
    yes |= (a->d[3] > SECP256K1_N_3) & ~no;
    no |= (a->d[2] < SECP256K1_N_2) & ~yes;
    yes |= (a->d[2] > SECP256K1_N_2) & ~no;
    no |= (a->d[1] < SECP256K1_N_1) & ~yes;
    yes |= (a->d[1] > SECP256K1_N_1) & ~no;
    yes |= (a->d[0] >= SECP256K1_N_0) & ~no;
    return yes;
}

Link

I'm adding a new recovery option for Armory's paper backup to FinderOuter but I need some test vectors from the project itself which I couldn't find. It also doesn't help that the repository is a fork and GitHub doesn't search within forks.

Could you help me find some, specially the process for deriving addresses from the "recovery phrase"?

Using bitcoin core 0.21.0 on RegTest.
Block #101 with the default RegTest target of 0x207fffff
Full header is

Code:

00000020c21aa3b43d104b5db7eb46d90430308b668b4edebdd23d72ba6cfc685601f1666769ae0d507824c9b1220f02db880618240a6b5dff17c1319a443eadd4b218e7a76c3360ffff7f2000000000

Hash versus target:

Code:

6f295fbd976fd8dd1668bf4ba82b8e7b2089c3497cf3f9dc4a61beb519992958
7fffff0000000000000000000000000000000000000000000000000000000000

50279827699679515341806937817482573096355300064251967888806579886749916932440
57896037716911750921221705069588091649609539881711309849342236841432341020672

This block is being rejected by core with a short message only saying "high-hash".

I'm trying to figure out what validations are being performed on block size and so far found this line:
https://github.com/bitcoin/bitcoin/blob/828bb776d29cbdfad3937ba100c428e3244c652f/src/validation.cpp#L3348

Code:

if (block.vtx.empty() ||
    block.vtx.size() * WITNESS_SCALE_FACTOR > MAX_BLOCK_WEIGHT ||
    ::GetSerializeSize(block, PROTOCOL_VERSION | SERIALIZE_TRANSACTION_NO_WITNESS) * WITNESS_SCALE_FACTOR > MAX_BLOCK_WEIGHT)

Google tells me that in c++ the size() method in std::vector (which is block.vtx type) returns item count which makes me wonder why is transaction count being multiplied by 4 and compared with max block weight?

As far as I can tell the only way to download blocks from another peer is through using getdata messages (is there any other messages?) which have a limit of 50000 items; but there is also MAX_BLOCKS_IN_TRANSIT_PER_PEER (=16) which I'm not sure what it's used for.

My getdata requests with 17 inventories are ignored sometimes which has left me confused. I'm not sure if it is the limit on number of blocks I can request or if it is fPauseSend which will then erase the m_getdata_requests completely here which means the request is ignored entirely.

I'm starting to look into key-value stores and I'm curious whether there has been any research and comparison between different options from a bitcoin usage point of view.
I'm looking into MongoDB, Redis, PostGres, Cassandra and of course BerkeleyDB since bitcoin core uses it. There are a lot more options which make things even harder.

The information I've found so far are for purposes different than bitcoin, for instance PostGres is used by Reddit and they claim it is the fastest based on their benchmarks.

Hopefully a comparison into their performance (speed) and scalability but also I'm also interested in concurrency and whether it could be taken advantage of using these DBs. I know that Cassandra and Redis support concurrency but BerkeleyDB doesn't seem to.

1. Why or when would a node set its ServiceFlags to NONE?
I'm probably testing "fake" nodes again but they seem to be providing me with headers without an issue.

2. Are NetworkLimited nodes capable of supplying all block headers even though they don't have most of the blocks?

I just added the recovery option for Electrum mnemonics to The FinderOuter and even though the derivation path could be set manually but it would be simpler if it were set automatically based on mnemonic type selected (or the address type entered).


However I couldn't find what derivation path is used for each mnemonic type above (eg. m/0 for standard) and need help.
I'd appreciate it if you could also include the link to code where it is defined.

It is my understanding that when the relay (the last byte of a version message) is set to 0 (false) the other node is supposed to not send any inv messages containing transactions but they seem to ignore this and at the very beginning of the connection (after handshake) start sending inv messages of type MSG_TX.
Why is that?

Haven't tried all but from those that tested so far: new bitcoin core versions (0.20.1 and 0.18.0) are like this while old ones (0.12.1) aren't. In other words the new versions seem to not respect the relay flag anymore.

The ServiceFlags enum is jumping from the previously defined 5^th bit (1 << 4 ie XThin) to 7^th bit (1 << 6 ie CompactFilters). I couldn't find the 6^th bit ever being set in the history of protocol.h file so is there any reason for this jump?

https://github.com/bitcoin/bitcoin/blob/af22322dab1a2277483b2512723491a5fad1a606/src/protocol.h#L268-L302

Although the code seems to be sending addr message to outbound connections (ref) but in my experience it is sending it to inbound connections too.
And in both cases I can't think of why it is doing that. When the connection is made specially if it is initiated by the other node, they should know our address and sending it again seems pointless to me.
On top of that the addr message is sent after version message (hand-shake) which already contains connection information (ie. the addr message).

I remember a while ago there was a topic on this forum from someone who had a large amount of bitcoin stuck in a P2SH-P2WPKH output that used an uncompressed public key and since it is non-standard it wasn't being relayed or mined and miners they contacted were also worried about their block being rejected if they included this transaction.
I can't find the topic though.

Well here is an example block with such a transaction on TestNet on block #1836615 (https://tbtc.bitaps.com/1836615) which I just mined. Maybe this can help.
The transaction is the only tx in the block apart from coinbase with hash= 134112254a1d0d687af0c4aa7185e62bbacf191165369d0150206f6c369ca417
You can see the uncompressed public key in second witness item (the second PushData):

Code:

{
  "Version": 1,
  "TxInList": [
    {
      "Outpoint": {
        "TxHash": "6b16690cdc1f4f8d6a07dd241cc8526bb075b9ad62f3d9d9a9fcc710ba94db16",
        "Index": 0
      },
      "SigScript": "PushData<0014ae456b5f88b76d858802da6996e05d5c4f084314>",
      "Sequence": 4294967295
    }
  ],
  "TxOutList": [
    {
      "Amount": 999750,
      "PubScript": "OP_DUP OP_HASH160 PushData<db1c22d11851f82e3e463f3b33c83400b15e6ab4> OP_EqualVerify OP_CheckSig"
    }
  ],
  "WitnessList": [
    "PushData<3044022009c9dca57c6e4212f471169c9c89c6d5e2ef570aa9a48b55dc2dfe4f13c9f0c202205cce77d15a383b29c45deb307cd60bcc848cfd98c52778a9c0c6699390f6889401>
     PushData<04576e65bd864ff1dca1116901833d432e1bc096e0a0f037f57cd0a09e9d3f96b3a88c3a627d3f9a4dfab5fff1d0dfcf03a68411722aed8e05cb3850652b4571ee>"
  ],
  "LockTime": "0"
}

Past couple of days have been very interesting, ever since I opened my listening socket to test my code I've been somewhat flooded with many inbound connections some of which seem to be only gathering information and nothing else (what I like to call "hit and run" nodes), a rare malicious node and only a handful of real nodes that behaved normally.

The malicious node was a node that I ended up playing "ping-pong" with until I manually cut it off, which was interesting as it kept sending me ping messages which I obviously replied with a pong to no end!

Another case which doesn't seem malicious but it is not normal either is a fixed IP range that has about 6 different user agents (satoshi:0.15 satpshi:0.18,... bitcoinj and nodesmulti). They only send a getaddr message and disconnect right away just to repeat it again later.
These "hit and run" nodes seem to only care about gathering information and nothing else and there are many of them.

This makes me wonder what are the cases that bitcoin core bans other nodes for "misbehaving" apart from obvious ones such as invalid block/tx/pow/chain?
If you also have any other information regarding P2P network I would love to study it.

[Ping]

I though Ping message was to check the connection when some time was passed and to check the connection is still alive. But I'm receiving it right at the beginning. The sequence of messages that I receive is this (without delay):
(send Version)
Version Verack
(send Verack)
SendHeaders
SendCmpct (2 times one for each version!)
Ping
Addr
GetHeaders

[100]

I'm trying to add Electrum mnemonic recovery option to The FinderOuter and looking at the test_mnemonic.py file I'm having trouble understanding the tests even though I think I understand how the mnemonic.py file works.

The English test vectors in this file and also random keys I created using Electrum work fine meaning after computing HmachSha512(data=words, key="Seed version") I get a digest that starts with SEED_PREFIX (01) or SEED_PREFIX_SW (100). eg. the first digest is 1001bc7d1ea.... which is all Electrum looks for in a valid mnemonic (there is no checksum).

But the rest don't.
Take the Chinese case for example, the digest is

Code:

0f5c4c9ff66e87bcbdde59f06ad540eba48fe06f7bdfa16b1248cb868ae3cd3fe7f6ea08e50bc77fdec1f08d8b5710bd25a9d76427e636feed23cbcb3ec8b8cb

Which is incorrect.
It is worth mentioning that the words are normalized using form KD as is with BIP-39 and return the same bytes as the "words_hex" in test vector.

So what is the problem here?

I'm adding BIP-39 recovery option to The FinderOuter and even though adding the option to also check uncompressed public keys or the addresses created using those is trivial, I'd rather not add it if it were never used in the history since sometimes having too many options is creating more confusion and my focus is simplicity of the tool as much as possible.

I've seen some reasons here and there but looking for full arguments against this BIP.

The dev. doc.[1] keeps referring to MAX_SIZE in serialize.h file[2] which is 32 MiB when it comes to message sizes but this strange* comparison in net.cpp[3] seems to be rejecting anything above 4 MB[4].
Am I reading it correctly or am I missing something here?

* Strange because it is basically saying if size > 33,554,432 OR size > 4,000,000 and obviously the first check is redundant since both values are constant.

[1] https://developer.bitcoin.org/reference/p2p_networking.html
[2] https://github.com/bitcoin/bitcoin/blob/60abd463ac2eaa8bc1d616d8c07880dc53d97211/src/serialize.h#L23
[3] https://github.com/bitcoin/bitcoin/blob/9bc7751cadbd038faf8ac1d62cda23fcf00d4cc2/src/net.cpp#L660
[4] https://github.com/bitcoin/bitcoin/blob/9bc7751cadbd038faf8ac1d62cda23fcf00d4cc2/src/net.h#L56

I've traced it all the way to this line:
https://github.com/bitcoin/bitcoin/blob/24f70290642c9c5108d3dc62dbe055f5d1bcff9d/src/script/script.cpp#L278
that is called by GetSigOpCount a bunch of lines above this which is called by GetTransactionSigOpCost from tx_verify.cpp
And to me it looks like a broken script such as

Code:

OP_2 OP_DIV 0xff OP_CheckMultiSig

is counted as 20 SigOps. Is my interpretation correct? (Obviously OP_DIV is disabled and 0xff is not an OP at all).

PS. SigOpCount tests of core needs a lot more cases, right now they seem to only test the elementary standard scripts only.