Transactions in mempool

[TheWolf666]

I am doing some experiment with Bitcoin source code.
I am doing a new blockchain.
I am mining correctly.

But all my transactions are staying in mempool

Now here my question.

How can I debug this? Because there is no error, no warning, even in debug mode.
How I can know what the problem with my transactions

I use -fallbackfee to insure that the fees are enough (0.0002).

Is there a methodology to trace and debug why the blockassembler refuses a mempool transaction?

[Heisenberg_Hunter]

Which version of the bitcoin code have you forked? The new codes and the whole bitcoin network is very much complex and it would be better for you to fork a older 2010 or 2011 code to learn about them completely. Did you change any of the code in miner.cpp? AFAIK, miner.cpp might be responsible for including transactions to the block. So, if you have changed the code and build them wrongly such issue may exist. There are varying problems for a transaction not to be included in a mempool first. They can be

• If you have changed the total coin circulation, the transaction which you have sent should have lesser coins than the total coins present.
• The size of a transaction which you are going to include in a block should be less than MAX_BLOCK_SIZE.
• I guess the nLockTime also plays a role in inclusion of transactions in a mempool. The Lock Time should be less than the maximum values int can have.
• Segwit soft-forks can also pose a problem in the inclusion. I am not really used to the newer codes and would stick to very old codes from 2011-2012 era for easy understanding of the network.

How can I debug this? Because there is no error, no warning, even in debug mode.

Is there a methodology to trace and debug why the blockassembler refuses a mempool transaction?

Check debug.log and see to it weather there is anything present which might give you a positive result. Filter the debug categories to mempool and see to it if you can find anything. The OP is pretty much empty to answer anything favourable, kindly try to provide information on what you have changed in the code?

Some other questions to be answered by you :
Do you mine empty blocks? Do you receive block rewards by mining empty blocks? The new version doesn't have -setgenerate since the mining difficulty has risen too high in the original implementation. So how do you mine? -generate command was available only in regtest and in the upcoming 0.19 release they will be removed completely.


P.S Correct me if I am wrong!

EDIT :

Which coin did you clone (including the release)?

I am doing some experiment with Bitcoin source code.

[mocacinno]

Which coin did you clone (including the release)? I remember cloning litecoin a while ago to do some tests, and at that point, getblocktemplate never included any transactions... Using the cli "generate 1 1000000" resulted in blocks including transactions from the mempool tough...

In order to find out what's happening, it would be usefull if we knew which coin you forked, when you created the fork, how you configured your node, and how you are mining...

the command

Code:

yourcoin.cli getblocktemplate 

might be a nice place to start (depending on which method you're using to mine)

[TheWolf666]

Which version of the bitcoin code have you forked? The new codes and the whole bitcoin network is very much complex and it would be better for you to fork a older 2010 or 2011 code to learn about them completely. Did you change any of the code in miner.cpp? AFAIK, miner.cpp might be responsible for including transactions to the block. So, if you have changed the code and build them wrongly such issue may exist. There are varying problems for a transaction not to be included in a mempool first. They can be

• If you have changed the total coin circulation, the transaction which you have sent should have lesser coins than the total coins present.
• The size of a transaction which you are going to include in a block should be less than MAX_BLOCK_SIZE.
• I guess the nLockTime also plays a role in inclusion of transactions in a mempool. The Lock Time should be less than the maximum values int can have.
• Segwit soft-forks can also pose a problem in the inclusion. I am not really used to the newer codes and would stick to very old codes from 2011-2012 era for easy understanding of the network.

How can I debug this? Because there is no error, no warning, even in debug mode.

Is there a methodology to trace and debug why the blockassembler refuses a mempool transaction?

Check debug.log and see to it weather there is anything present which might give you a positive result. Filter the debug categories to mempool and see to it if you can find anything. The OP is pretty much empty to answer anything favourable, kindly try to provide information on what you have changed in the code?

Some other questions to be answered by you :
Do you mine empty blocks? Do you receive block rewards by mining empty blocks? The new version doesn't have -setgenerate since the mining difficulty has risen too high in the original implementation. So how do you mine? -generate command was available only in regtest and in the upcoming 0.19 release they will be removed completely.


P.S Correct me if I am wrong!

EDIT :

Which coin did you clone (including the release)?

I am doing some experiment with Bitcoin source code.

I cloned 0.18.1, the latest from this month.
I didn't mined empty blocks.
I did changed the CalculateNextWorkRequired in pow.cpp, I added these lines:

Code:

    if (bnNew.GetCompact()<0x1d00ffff && pindexLast->nHeight>500) bnNew=0x1d00ffff;
    if (bnNew<=1.0 && pindexLast->nHeight>500) bnNew=1.0;
    if (bnNew<=0.0 && pindexLast->nHeight<=500) bnNew=1.0;

The idea was to limit the difficulty to 1 and not go in negative which can f*ck up the blockchain definitively. Also do some experiment with negative diff.
I will investigate nLockTime thanks for the info.
Segwit is posing problems with new blockchains, but only for the first blocks after the genesis, after it seems to work fine.
Fees calculation can also be a problem like if the fees are not calculated correctly, then the transactions are not included in the next block... I am investigating that too. I use fallbackfee.

The weird thing is that I have no warning (even in debug mode) or errors... both on the mining wallet that send the transactions, than in the receiving wallets that are catching the transactions.

[TheWolf666]

I have another question. How many computer I need to get a consensus over a transaction?
For example let's say I have 10 computer running a wallet.

2 computer are working together one send a transaction to another. The transaction arrives but stay in mempool.

If 1 computer reject the transaction, is that is going to keep it in mempool? How many computer need to say ok so that the assembler include the mempool transaction into the blockchain?

I am talking about forks and upgrade. Imagine that you fork your coin, not everyone is going to upgrade to the new version, so it might do a mess with some wallet accepting the transactions, some not accepting and the blockchain will be fragmented?

[mocacinno]

I tried to distill the 3 "main" questions from your post, and tried to answer them individually

If 1 computer reject the transaction, is that is going to keep it in mempool?

no... If a node rejects a transaction, the default behaviour is not to store it in it's mempool... Theoretically, i guess you can change the sourcecode and store rejected transactions in your mempool or relay them, but not select them to get included in a block... But i don't see any reason for such a patch (i guess you'd have to rewrite a lot of sourcecode to implement this behaviour)

How many computer need to say ok so that the assembler include the mempool transaction into the blockchain?

only one. If a node accepts a transaction, stores it in it's mempool, is used as a mining node, inserts the transaction in the block the miner is trying to solve, and if this miner finds a block header whose hash is under the current target, the transaction is included in a valid block... However, you're talking about forking... If your fork has different consensus rules, and the tx is accepted by one node, but rejected by a different node because this second node has different rules (because of a hardfork), the second node will not accept blocks including the offending tx, while the first node accepts this block without a problem

Imagine that you fork your coin, not everyone is going to upgrade to the new version, so it might do a mess with some wallet accepting the transactions, some not accepting and the blockchain will be fragmented?

Not fragmented, but there might be a fork... It might be a good idear to use checkpoints when implementing new rules...

[TheWolf666]

I tried to distill the 3 "main" questions from your post, and tried to answer them individually

If 1 computer reject the transaction, is that is going to keep it in mempool?

no... If a node rejects a transaction, the default behaviour is not to store it in it's mempool... Theoretically, i guess you can change the sourcecode and store rejected transactions in your mempool or relay them, but not select them to get included in a block... But i don't see any reason for such a patch (i guess you'd have to rewrite a lot of sourcecode to implement this behaviour)

How many computer need to say ok so that the assembler include the mempool transaction into the blockchain?

only one. If a node accepts a transaction, stores it in it's mempool, is used as a mining node, inserts the transaction in the block the miner is trying to solve, and if this miner finds a block header whose hash is under the current target, the transaction is included in a valid block... However, you're talking about forking... If your fork has different consensus rules, and the tx is accepted by one node, but rejected by a different node because this second node has different rules (because of a hardfork), the second node will not accept blocks including the offending tx, while the first node accepts this block without a problem

Imagine that you fork your coin, not everyone is going to upgrade to the new version, so it might do a mess with some wallet accepting the transactions, some not accepting and the blockchain will be fragmented?

Not fragmented, but there might be a fork... It might be a good idear to use checkpoints when implementing new rules...

Thanks for this great answer, very useful and I guess that could be useful for others too. This part of the code is really tricky to understand.

I am using checkpoints, to insure that the main fork is the one that stay, and avoid a majority of nodes taking over.

About the rejection, what you say is that the nodes are not rejecting the transactions, because they are storing them in mempool, but then why not adding them in the next mined block, and why not trowing an error message or a warning.
I am going to put lot of printf in mining.cpp and go to where the problem is. Once I find it, I will publish my results.

[TheWolf666]

I have found the problem. This might be useful to anyone who is trying to start a blockchain from zero by forking the latest version of Bitcoin core (now 0.18.1)

Transactions will stay in Mempool because SEGWIT is set on by default from the genesis, but this code is returning false for some reason:

Code:

validation.cpp

bool IsWitnessEnabled(const CBlockIndex* pindexPrev, const Consensus::Params& params)
{
    LOCK(cs_main);
    return (VersionBitsState(pindexPrev, params, Consensus::DEPLOYMENT_SEGWIT, versionbitscache) == ThresholdState::ACTIVE);   
}

The thing is that all the consensus rules in Bitcoin are specific at the history of Bitcoin (date starting etc...) but if you are forking Bitcoin to create your own version, you do not need to check for SEGWIT since it should be enabled by default from the start.
Be sure that your pool is adding the SEGWIT witness information or if you cannot, fork a pre-SEGWIT version of Bitcoin or litecoin.

Anyway by returning TRUE in IsWitnessEnabled, the transaction are then accepted because miner get this info at line 137

Code:

 fIncludeWitness = IsWitnessEnabled(pindexPrev, chainparams.GetConsensus());

And then this part of the code will stop rejecting the inclusion of all the transactions based on the missleading IsWitnessEnabled flag.

Code:

bool BlockAssembler::TestPackageTransactions(const CTxMemPool::setEntries& package)
{
    for (CTxMemPool::txiter it : package) {
        if (!IsFinalTx(it->GetTx(), nHeight, nLockTimeCutoff)) {
            return false;
        }
        if (!fIncludeWitness && it->GetTx().HasWitness()) {
            // Here is where the problem is, transaction rejected! I have added this debug code.
            LogPrintf("TestPackageTransactions !fIncludeWitness %i\n",nHeight);
            return false;
        }
    }
    return true;
}

I hope that this will help others. There might be a better solution than just returning true, you are welcome to improve.

Special thanks to the smart answers in this forums that helped me to better understand how this working, I have given some merit points to who deserved.

[Heisenberg_Hunter]

As I have already said, forking the latest version to start our own chain is really cumbersome, time consuming process and there is a possibility we won't be arriving at any positive conditions. When satoshi created the code back in 2007 and announced them in 2008, they were not really very difficult for normal people like me to understand the flow working behind them. But after some of the major implementations like P2SH and Segwit, the code has become complex and it was not really easy for us to either fork them or to jump-in the code for contributing having a very minimal knowledge on satoshi code.

...snip...

As mocacinno pointed out, each full node in the blockchain is responsible for the validation of new blocks in the network and adding them to the already growing chain. But whereas the mining nodes are responsible for doing work and finding out the nounce which must be less than the target and thereby producing new virgin bitcoin in the process. The mining node further broadcasts the block and the full node validates them for adding it to the chain of blocks.

Apart from that, the rest of your questions have been answered by mocacinno.

About the rejection, what you say is that the nodes are not rejecting the transactions, because they are storing them in mempool, but then why not adding them in the next mined block, and why not trowing an error message or a warning.

Each nodes have their own mempool and consider if the node broadcasts an invalid transaction and when these invalid transactions are relayed over other nodes and if they do find that the transaction is invalid, they would automatically ban the peer/node which sent the invalid transaction. The ban happens in net_processing.cpp code somewhere around here

Code:

void Misbehaving(NodeId pnode, int howmuch, const std::string& message) EXCLUSIVE_LOCKS_REQUIRED(cs_main)
{
    if (howmuch == 0)
        return;

    CNodeState *state = State(pnode);
    if (state == nullptr)
        return;

    state->nMisbehavior += howmuch;
    int banscore = gArgs.GetArg("-banscore", DEFAULT_BANSCORE_THRESHOLD);
    std::string message_prefixed = message.empty() ? "" : (": " + message);
    if (state->nMisbehavior >= banscore && state->nMisbehavior - howmuch < banscore)
    {
        LogPrint(BCLog::NET, "%s: %s peer=%d (%d -> %d) BAN THRESHOLD EXCEEDED%s\n", __func__, state->name, pnode, state->nMisbehavior-howmuch, state->nMisbehavior, message_prefixed);
        state->fShouldBan = true;
    } else
        LogPrint(BCLog::NET, "%s: %s peer=%d (%d -> %d)%s\n", __func__, state->name, pnode, state->nMisbehavior-howmuch, state->nMisbehavior, message_prefixed);
}

Also check #L1838 in net_processing.cpp

I am going to put lot of printf in mining.cpp and go to where the problem is. Once I find it, I will publish my results

But why? 

[TheWolf666]

As I have already said, forking the latest version to start our own chain is really cumbersome, time consuming process and there is a possibility we won't be arriving at any positive conditions. When satoshi created the code back in 2007 and announced them in 2008, they were not really very difficult for normal people like me to understand the flow working behind them. But after some of the major implementations like P2SH and Segwit, the code has become complex and it was not really easy for us to either fork them or to jump-in the code for contributing having a very minimal knowledge on satoshi code.

...snip...

As mocacinno pointed out, each full node in the blockchain is responsible for the validation of new blocks in the network and adding them to the already growing chain. But whereas the mining nodes are responsible for doing work and finding out the nounce which must be less than the target and thereby producing new virgin bitcoin in the process. The mining node further broadcasts the block and the full node validates them for adding it to the chain of blocks.

Apart from that, the rest of your questions have been answered by mocacinno.

About the rejection, what you say is that the nodes are not rejecting the transactions, because they are storing them in mempool, but then why not adding them in the next mined block, and why not trowing an error message or a warning.

Each nodes have their own mempool and consider if the node broadcasts an invalid transaction and when these invalid transactions are relayed over other nodes and if they do find that the transaction is invalid, they would automatically ban the peer/node which sent the invalid transaction. The ban happens in net_processing.cpp code somewhere around here

Code:

void Misbehaving(NodeId pnode, int howmuch, const std::string& message) EXCLUSIVE_LOCKS_REQUIRED(cs_main)
{
    if (howmuch == 0)
        return;

    CNodeState *state = State(pnode);
    if (state == nullptr)
        return;

    state->nMisbehavior += howmuch;
    int banscore = gArgs.GetArg("-banscore", DEFAULT_BANSCORE_THRESHOLD);
    std::string message_prefixed = message.empty() ? "" : (": " + message);
    if (state->nMisbehavior >= banscore && state->nMisbehavior - howmuch < banscore)
    {
        LogPrint(BCLog::NET, "%s: %s peer=%d (%d -> %d) BAN THRESHOLD EXCEEDED%s\n", __func__, state->name, pnode, state->nMisbehavior-howmuch, state->nMisbehavior, message_prefixed);
        state->fShouldBan = true;
    } else
        LogPrint(BCLog::NET, "%s: %s peer=%d (%d -> %d)%s\n", __func__, state->name, pnode, state->nMisbehavior-howmuch, state->nMisbehavior, message_prefixed);
}

Also check #L1838 in net_processing.cpp

I am going to put lot of printf in mining.cpp and go to where the problem is. Once I find it, I will publish my results

But why? 

Mainly because the cause of the problem is not a misbehaving issue, no node reject the transactions because of misbehaving. I have found the problem (read my post just before).

I am forking the latest Bitcoin Core because I want to use the lighting network with my coin, also I like SEGWIT, it is an elegant way to adjust the number of transactions to blocks without changing the maximum size of the block. So far it is working and does not need a blockchain fork.
Finally, I think that the Bitcoin Core team, is doing a good job at fixing security bugs, so I won't fork a 2-3 years old Bitcoin core version like others are doing and get the risk to be hacked.

Actually the latest Bitcoin core can be forked, it is just a bit more complicated because of all the security tests added, the SEGWIT tests, and the new block assembler that is not allowing solo mining. For a new coin without solo mining you have to setup your own pool and stratum which considerably add work in the developer's plate.

But I think that all these extra complexity is what make Bitcoin secured. Having anyone able to solo mine is an open door to an early 51% attack. When we announced Kryptofranc (KYF) we got attacks the same day!

I think that in 2009, that was not the case.

Now there are a lot of nasty people with good knowledge of how the code is working that can buy a lot of hashing power at Nicehash and steal your blockchain or include unwanted transactions. It happened to some major crypto.
So having a stratum, pool, and controlling the mining at the early stage, is one solution to secure your blockchain until it is mature enough to let the big miners coming with their Asics.
The extra complexity is worth compared with the added security the Bitcoin Core team have been working on.

My 2cts 

[DaCryptoRaccoon]

Thank you for this post!

I had some issues when testing with the newer code and had similar issues I never did find out what the cause was but I did think it was somehow segwit related.
I think a lot of people who used the older code bases struggled to work out all the changes that came in later versions this is some good insight to these.