Multiprocessing Mining Accurate Hashrate and Repeated Process Stability

[Loid Cypherpunk]

I am testing a CPU miner using Python multiprocessing.

Each worker searches a separate nonce range. The parent process collects the reported hash count and calculates the displayed hashrate.

A simplified part of the implementation:

hashes += 1

if hashes % 100 == 0:
    result["hashes"] = result.get("hashes", 0) + 100

total_hashes = result.get("hashes", 0)
hps = total_hashes / elapsed

Source code:
https://github.com/loidprotagonist/Queen/blob/main/Loid_miner.py

I have two practical questions:

1. Is this a reliable way to measure the actual aggregate hashrate across multiple processes?
2. Can repeatedly creating and terminating the worker processes eventually cause resource leaks, instability, or crashes?

I am looking for implementation-level feedback and reproducible tests, not theoretical discussion.

[Antidote47k]

I'd separate the live display from the benchmark for the hashrate measurement. Since each worker only reports its counter every 100 hashes, so the shared value might be slightly behind the progress of the worker. If you wish to have a more reproducible benchmark, then each worker would have their own local count for a set period of time or for a duration, and then aggregate the counters after all workers are finished. If that is the case, then I believe that should be able to avoid frequent IPC and synchronization during the hashing loop, which makes the measurement less intrusive.

On the process stability, the repeated creation and termination of workers won’t necessarily cause a memory or file descriptor to leak by itself. It is important to join the processes properly and release the resources associated with the processes. One good test to run would be hundreds or thousands of start/stop cycles, while keeping track of the RSS, open file descriptors and process count. If those are still growing after the cleanup, then you have something you can investigate.
For a miner that continuously receives work, I’d also consider persistent worker processes or a multiprocessing.Pool. That eliminates repeatedly paying process creation/teardown costs and allows workers to receive new tasks without the need to recreate them everytime.

[Loid Cypherpunk]

If you're interested, I'd appreciate your help reproducing the issue. It may be faster to investigate together than alone.

Of course, please don't trust me or my code blindly. You should review or scan the source code with security tools before running it on your device.

GitHub:
https://github.com/loidprotagonist/Queen

If you don't have time, no problem. I'll continue investigating it myself.

[Antidote47k]

If you're interested, I'd appreciate your help reproducing the issue. It may be faster to investigate together than alone.

Of course, please don't trust me or my code blindly. You should review or scan the source code with security tools before running it on your device.

GitHub source code:
https://github.com/loidprotagonist/Queen

If you don't have time, no problem. I'll continue investigating it myself.

I’d be interested to see how the investigation goes, I’m not that deep into python multiprocessing to be of much help with the implementation itself but yeah, feel free to tag me if you get stuck on something and I’ll be happy to look into it with you.

[Loid Cypherpunk]

If you're interested, I'd appreciate your help reproducing the issue. It may be faster to investigate together than alone.

Of course, please don't trust me or my code blindly. You should review or scan the source code with security tools before running it on your device.

GitHub source code:
https://github.com/loidprotagonist/Queen

If you don't have time, no problem. I'll continue investigating it myself.

I’d be interested to see how the investigation goes, I’m not that deep into python multiprocessing to be of much help with the implementation itself but yeah, feel free to tag me if you get stuck on something and I’ll be happy to look into it with you.

I ran a small repeated start/stop test on my device.

During mining, the process count and resource usage increased as expected because the worker processes were active. After the block was found and the workers stopped, the values returned close to the baseline.

Example:

Baseline:
Processes: 2
RAM: 26.98 MB
Threads: 2
FD: 30

During mining:
Processes: 3
RAM: 40.66 MB
Threads: 5
FD: 44

After block completion:
Processes: 2
RAM: 24.02 MB
Threads: 2
FD: 30

The monitor also recorded these peaks during the run:

Peak processes: 11
Peak RAM: 169.28 MB
Peak threads: 21
Peak FD: 132

So far I haven't reproduced a persistent increase after the workers finish. I'll continue with a larger number of repeated cycles to see whether the baseline gradually increases.

Hardware: Android phone

This is only an initial observation, not a conclusion that there is no leak.

[BlackHatCoiner]

1. It might be accurate, but not efficient.
2. Constantly killing and respawning Python worker processes creates massive overhead. Spin up your workers once and just feed them new nonce ranges.

Also, mandatory reality check: writing a CPU miner in Python is a fun learning project, but your CPU is silently crying right now.

[Loid Cypherpunk]

1. It might be accurate, but not efficient.
2. Constantly killing and respawning Python worker processes creates massive overhead. Spin up your workers once and just feed them new nonce ranges.

Also, mandatory reality check: writing a CPU miner in Python is a fun learning project, but your CPU is silently crying right now.

I also enjoy experimenting with my weird little design.

Now, do you have any test results from actually running the code?

I'm not looking for “maybe". I'm looking for something reproducible.