Large Bitcoin Collider Thread 2.0

[rico666]

Rico, did you see my suggestion above? Please reply, I really want to know if it's applicable.

No, it's not. It's a variation of what arulbero has suggested further above. To all the problems of your previous suggestion, this adds the problem of scalability to the server (We have - in principle - a 1:many relation between server and clients, and we cannot afford to let the server perform key generation computations).

Moreover:

For the time being, I consider all security-related issues resolved and am as of now not willing to participate in any discussion regarding "LBC security", novel proof-of-work suggestions and the like. If you must stretch this topic, please do so in v1 thread or show me some code. Else: 404

I have addressed every valid security concern in the LBC client. So far, you have brought nothing to the table of value. No working code, no proof of concept. You have no projects of your own, no track of record , no nothing. You only brought stir to the LBC project. But you consider it somehow (I honestly do not know where you take that self-confidence) legit to demand my attention and even answers. You are in no position for that. Am I being clear?

You are this close ----> <------ to my ignore list. Please read and understand in the 1st post of this thread what that means (key phrase is: retro-active). The only thing keeping you from there is, that it could be perceived as martyrdom if I simply kicked you there. But the time nears where I do not care. Your constant gnat buzzing is like a developer DoS and I will not let you swamp me with that.

More than anyone else, your contribution to the LBC meritocracy is negative so far. Before I even consider looking at any of your output ever again, you will have to provide some Gods own code or concept of value for the LBC. Including a prototype implementation. Until then: Try to learn as much as you can and should your fingers tickle and urge you to do a writeup: DON'T.

[1715011158]

1715011158

[1715011158]

1715011158

[rico666]

alternate connection and follow your instructions for the "Clubbed to Death" method.

Yep - that "clubbed to death" method works always - because actually a human is doing what the computer should do. 

Now that I will have more time to spend on usability and infrastructure, I certainly hope to get that automated install more robust and user-friendly.
(e.g. biggest problem of FTP updates was, that the LBC client on some installations just sees an empty directory and I have no idea why.)
With moving the updates of client and generator to HTTPS, I also am using JSON output from the Nginx server for directory listings
(see e.g. https://lbc.cryptoguru.org/static/generators/) which should also help to get the update/install more robust as no brittle output parsing is necessary anymore.

This is actually the authoritative source. FTP will be phased out as soon as the last sub-1.138 client is in use.

   The CPU computes 4096 uncompressed public keys and moves them to GPU
    The GPU computes 4096 hash160 of this and 4096 hash160 of the compressed equivalents
    The 8192 hashes are moved back to the CPU which performs a bloom filter search on them.

This process is repeated 4096 times before you see a 'o' on your screen, which takes around 8 seconds on modern CPU/GPU combinations. Depending on the speed of the GPU, its load may be low - e.g. just 10% per CPU core.

This is not entirely true anymore. The 8192 hashes moving back is not done anymore, the GPU does the bloom filter search now. Also, using a faster ECC (done by arulbero) than what libsecp256k1 provided, allowed the CPU to feed the GPU faster. All in all, the load one CPU core does on GPU is about twice what it was when the quoted text above was written.

As I understand it, the use of GPU processing capacity may be unused depending on the number of cores the CPU has. Does this mean that running LBC on a multi-GPU rig with only an Intel Celeron Dual Core would not necessarily use all the GPUs capacity?

As of now, this is exactly what it means. I am thrilled to say, that this is actually something I will be working on next and GPU usage/saturation will raise at least by a factor of 2 again in the next couple of (days/few weeks).

[SopaXT]

Rico, did you see my suggestion above? Please reply, I really want to know if it's applicable.

No, it's not. It's a variation of what arulbero has suggested further above. To all the problems of your previous suggestion, this adds the problem of scalability to the server (We have - in principle - a 1:many relation between server and clients, and we cannot afford to let the server perform key generation computations).

Moreover:

For the time being, I consider all security-related issues resolved and am as of now not willing to participate in any discussion regarding "LBC security", novel proof-of-work suggestions and the like. If you must stretch this topic, please do so in v1 thread or show me some code. Else: 404

I have addressed every valid security concern in the LBC client. So far, you have brought nothing to the table of value. No working code, no proof of concept. You have no projects of your own, no track of record , no nothing. You only brought stir to the LBC project. But you consider it somehow (I honestly do not know where you take that self-confidence) legit to demand my attention and even answers. You are in no position for that. Am I being clear?

You are this close ----> <------ to my ignore list. Please read and understand in the 1st post of this thread what that means (key phrase is: retro-active). The only thing keeping you from there is, that it could be perceived as martyrdom if I simply kicked you there. But the time nears where I do not care. Your constant gnat buzzing is like a developer DoS and I will not let you swamp me with that.

More than anyone else, your contribution to the LBC meritocracy is negative so far. Before I even consider looking at any of your output ever again, you will have to provide some Gods own code or concept of value for the LBC. Including a prototype implementation. Until then: Try to learn as much as you can and should your fingers tickle and urge you to do a writeup: DON'T.

I *do* have projects of my own -  SopaXorzTaker on GitHub.
If you find it appropriate to take this post to your attention, I should argue with you on the above.
Key generation for the challenge is not that expensive. Assuming that your server has ~200 regular clients, and you get a work requests from all of them (which is exaggerated) every second and there's 16 challenge keys per work request, you'd only need 200*16 = 3200 keys per second to be generated, while my machine does ~800 kkey/sec. I feel it necessary to argue, as I think some of your claims are incorrect, such as this one. I'd love to hear your feedback and arguments on that.

Additionally, you could potentially reuse the challenge keys for less security and more performance, and then you'd need only 16 keys per second to keep up with the clients.
EDIT: that's not possible as every client has a different work from the server, so the challenge has to be different too.
Am I wrong?

[rico666]

I *do* have projects of my own -  SopaXorzTaker on GitHub.

Hm. I see.

This is really my final take on it. All subsequent writeups of you in this thread will be deleted unseen. You may as well copy them to /dev/null
I'm answering under the theoretical premises your proposed PoW would actually be an unfakeable  PoW - which it isn't. Which renders all of this discussion moot.
But hey - why not:

you'd only need 200*16 = 3200 keys per second to be generated, while my machine does ~800 kkey/sec.

You are comparing peas to apples. 800 Kkeys/s incremental key generation compared to 3200 starting keys.
For an incremental key, you simply have to do 1/2 of a G addition. For a starting key, you have to do n G multiplications (or additions if you have a precomputed table) where n is the number of 1-bits in your privkey. "your 800k-machine" can therefore do about 6666 starting keys/s in our current entropy range. Nice number, but too low.

Also, the LBC server is a VM and there is no chance to get a GPU in there. As of now, the server would be able to cope with around 50 Tkeys/s key generation capacity with no code changes whatsoever.

Please build YOUR OWN "LBC Server" project. SHOW me, how it's done by doing it, not by babbling about it.

I feel it necessary to argue, as I think some of your claims are incorrect, such as this one. I'd love to hear your feedback and arguments on that.

Again, your feeling is based on your ignorance (in this case about the key generation process). I did way more than I should to nurture your attention needs. Now begone.

[Real-Duke]

Hi rico,

have done the update to the new generator yesterday evening and all went fine over night.
Just now I have shutdown the LBC and make the following run:

Code:

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -u
Finished update run - system up to date.

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -x
GPU authorized: yes
Will use 4 CPUs.
Testing mode. Using page 0, turning off looping.
Benchmark info not found - benchmarking... Can't exec "gen-hrdcore-avx2+gpu-linux64": Datei oder Verzeichnis nicht gefunden at ./LBC line 2030.
Can't benchmark generator: No such file or directory at ./LBC line 2030.
real-duke@C1-Ubuntu:~/gcollider$ ./LBC
GPU authorized: yes
Will use 4 CPUs.
Benchmark info not found - benchmarking... Can't exec "gen-hrdcore-avx2+gpu-linux64": Datei oder Verzeichnis nicht gefunden at ./LBC line 2030.
Can't benchmark generator: No such file or directory at ./LBC line 2030.
real-duke@C1-Ubuntu:~/gcollider$

The file "gen-hrdcore-avx2+gpu-linux64" is of course in the LBC directory with rights 777 but now you can see Im unable to start the collider again ( Have a working backup somewhere )

[rico666]

have done the update to the new generator yesterday evening and all went fine over night.
Just now I have shutdown the LBC and make the following run:

Code:

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -u
Finished update run - system up to date.

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -x
GPU authorized: yes
Will use 4 CPUs.
Testing mode. Using page 0, turning off looping.
Benchmark info not found - benchmarking... Can't exec "gen-hrdcore-avx2+gpu-linux64": Datei oder Verzeichnis nicht gefunden at ./LBC line 2030.
Can't benchmark generator: No such file or directory at ./LBC line 2030.

The file "gen-hrdcore-avx2+gpu-linux64" is of course in the LBC directory with rights 777 but now you can see Im unable to start the collider again ( Have a working backup somewhere )

Hm. Nevertheless:

Code:

> ls -al

so I can see what is there and what not.

[mrgiacalone]

As I understand it, the use of GPU processing capacity may be unused depending on the number of cores the CPU has. Does this mean that running LBC on a multi-GPU rig with only an Intel Celeron Dual Core would not necessarily use all the GPUs capacity?

As of now, this is exactly what it means. I am thrilled to say, that this is actually something I will be working on next and GPU usage/saturation will raise at least by a factor of 2 again in the next couple of (days/few weeks).

Thanks for the reply.
At the moment I have a 6xGPU rig with a dead mobo/cpu that I will be splitting into two 3xGPU rigs. I intend on using one of them for new mining projects (of the like of LBC), so I'll be checking your updates (for the next days/few weeks) to do some testing on multi-GPU rigs. (Hopefully I'll get my GPU auth flag ready as well with my CPU). My main interest is on the performance with this low CPU high GPU combination, which are very common in GPU rigs at the moment.

As you can see in the stats The pool is slowing down, which is because our top-contributor https://lbc.cryptoguru.org/stats/Unknownhostname is shutting down capacities. Without him, the pool has around 200-250 Mkeys/s.  (Maybe 400 if _KULME_ starts his monster machine  Wink)

Therefore my next focus and priority will be again on the generator to squeeze some more speed out of it and also to make it available on more machines.

Perhaps this will also be a good time to increase incentives to a broader range of contributors to reduce dependence on fewer of them.

MRG

[shortcircuit]

have done the update to the new generator yesterday evening and all went fine over night.
Just now I have shutdown the LBC and make the following run:

Code:

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -u
Finished update run - system up to date.

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -x
GPU authorized: yes
Will use 4 CPUs.
Testing mode. Using page 0, turning off looping.
Benchmark info not found - benchmarking... Can't exec "gen-hrdcore-avx2+gpu-linux64": Datei oder Verzeichnis nicht gefunden at ./LBC line 2030.
Can't benchmark generator: No such file or directory at ./LBC line 2030.

The file "gen-hrdcore-avx2+gpu-linux64" is of course in the LBC directory with rights 777 but now you can see Im unable to start the collider again ( Have a working backup somewhere )

Hm. Nevertheless:

Code:

> ls -al

so I can see what is there and what not.

Already posted in the german section. Here's the ouput of ls -al

drwxr-xr-x 2 bert bert      4096 May  1 14:40 .
drwxr-xr-x 4 bert bert      4096 May  1 12:25 ..
-rw-r--r-- 1 root root     48076 May  1 14:40 diagnostics-LBC.txt
-rw-r--r-- 1 root root 536870912 May  1 14:23 funds_h160.blf
-r-xr-xr-- 1 root root    199200 May  1 14:19 gen-hrdcore-generic-linux64
-rwxr-xr-x 1 root root     63819 May  1 13:50 LBC

[rico666]

Can't exec "gen-hrdcore-avx2+gpu-linux64": File not found yadda

Temporary fix:

Code:

export PATH=$PATH:.

edit: Version 1.140 on Server which does not need this workaround. Problem was only in "LBC -x" - not in normal operation.

[SopaXT]

Rico, he's running it on a machine named "Ubuntu-C2", which suggests of it being an ODROID-C2. That's an ARM single board computer and LBC is not going to run on it.

[rico666]

Rico, he's running it on a machine named "Ubuntu-C2", which suggests of it being an ODROID-C2. That's an ARM single board computer and LBC is not going to run on it.

I see "C1-Ubuntu", which I would now intuitively translate to "Collider1-Ubuntu".
Given the fact that he is running a GTX1060, I seriously doubt he has an ARM board below that.
Given another fact, that the LBC client has recognized AVX2, I would be very interested to see an ARM with avx2 features.

As mentioned, the problem was that "LBC -x" did not start the generator explicitely with the "./" current path prefix. It's fixed in 1.140

[Real-Duke]

LOL nice rumours about my hardware
C1 is my Intel Standalone PC and C2 is my little AMD Notebook...both are no ARM Systems as far as I know  
"C" means nothing more than "Computer" in this case.

[Real-Duke]

edit: Version 1.140 on Server which does not need this workaround. Problem was only in "LBC -x" - not in normal operation.

Hmm but ./LBC -x still gives an error here!
Starting with ./LBC seems to work...but correct way?

Code:

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -u
New client '1.140-LBC.bz2' found.
Finished update run - system up to date.

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -x
GPU authorized: yes
Will use 4 CPUs.
Testing mode. Using page 0, turning off looping.
Benchmark info not found - benchmarking... done.
Your maximum speed is 5246127 keys/s per CPU core.
Generator validity check failed.
real-duke@C1-Ubuntu:~/gcollider$ ./LBC
GPU authorized: yes
Will use 4 CPUs.
Ask for work... got blocks [4941330847-4941353694] (23957 Mkeys)
oo
real-duke@C1-Ubuntu:~/gcollider$ ls -all
insgesamt 526140
drwxrwxr-x  2 real-duke real-duke      4096 Mai  1 21:38 .
drwxr-xr-x 28 real-duke real-duke      4096 Mai  1 21:31 ..
-rwxrwxrwx  1 real-duke real-duke     55018 Feb 19 13:41 1.021-LBC
-rwxrwxrwx  1 real-duke real-duke     55108 Mär  8 22:47 1.031-LBC
-r--r--r--  1 real-duke real-duke     63819 Mai  1 00:20 1.067-LBC
-r--r--r--  1 real-duke real-duke     63819 Mai  1 00:20 1.138-LBC
-rw-rw-r--  1 real-duke real-duke        28 Mai  1 21:34 bench.pst
-rwxrwxrwx  1 real-duke real-duke      4488 Mär  9 18:16 diagnostics-OpenCL.txt
-rw-rw-r--  1 real-duke real-duke 536870912 Apr 24 23:41 funds_h160.blf
-rwxrwxrwx  1 real-duke real-duke    395888 Apr 16 20:59 gen-hrdcore-avx2+gpu-linux64
-rwxrwxrwx  1 real-duke real-duke   1131720 Apr 16 20:31 gen-hrdcore-avx2-linux64
-rwxrwxrwx  1 real-duke real-duke     20814 Mär 26 13:45 hash160_deparsed.cl
-r-xr-xr--  1 real-duke real-duke     63826 Mai  1 21:33 LBC
-rwxrwxrwx  1 real-duke real-duke       124 Apr 16 22:56 lbc.json
real-duke@C1-Ubuntu:~/gcollider$ 

[rico666]

or mail: bots@cryptoguru.org

There's something weird going on with your 1.067 client
edit: server put your client in blacklist, contact me for clarification.

[MetaLynx]

or mail: bots@cryptoguru.org

There's something weird going on with your 1.067 client
edit: server put your client in blacklist, contact me for clarification.

Yeah, I noticed I was blacklisted, but was heading out the door on the way to work and didn't have a forum account or time to start playing fiddlesticks trying to figure out why, also I was hoping it would just fix itself by the time I got home.
So what happened? I was a mere few hundred G/Keys from top 30 and then bam rico shells into my computer and sabotages me so that I have to start all over! I knew it! lol.
But in all fairness it could have possibly also been several other factors. I have it running in the background on an arch antergos box. At the time I was doing an update that affected like 300+ items, while also finding random new programs, and while the LBC was running in the background. So maybe  a dependency was updated while it was running, I think some nvdia things got changed, maybe those? idk.

[Real-Duke]

All sorted out, tonight Im back ON 

Time = 7
for testing...

Code:

real-duke@C1-Ubuntu:~/gcollider$ ./LBC -x
GPU authorized: yes
OpenCL program written.
Will use 4 CPUs.
New generator found. (DL-size: 0.25MB)
Testing mode. Using page 0, turning off looping.
Benchmark info not found - benchmarking... done.
Your maximum speed is 3969272 keys/s per CPU core.
o
Test ok. Your test results were stored in FOUND.txt.
Have a look and then you may want to remove the file.
2d17543d32448acc7a1c43c5f72cd5be459ab302:u:priv:0000000000000000000000000000000000000000000000000000000000000001+0x5e
02e62151191a931d51cdc513a86d4bf5694f4e51:c:priv:0000000000000000000000000000000000000000000000000000000000000001+0x65
9d74ffdb31068ca2a1feb8e34830635c0647d714:u:priv:00000000000000000000000000000000000000000000000000000000000f9001+0xf8c
3d6871076780446bd46fc564b0c443e1fd415beb:c:priv:00000000000000000000000000000000000000000000000000000000000f9001+0xf8c
Ending test run.
real-duke@C1-Ubuntu:~/gcollider$ ./LBC
GPU authorized: yes
Will use 4 CPUs.
Ask for work... got blocks [4974781375-4974787774] (6710 Mkeys)
ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

[rico666]

Yeah, I noticed I was blacklisted, but was heading out the door on the way to work and didn't have a forum account or time to start playing fiddlesticks trying to figure out why, also I was hoping it would just fix itself by the time I got home.
So what happened? I was a mere few hundred G/Keys from top 30 and then bam rico shells into my computer and sabotages me so that I have to start all over! I knew it! lol.

I have no interest in sabotaging you. Why should I? Your client is in blacklist, because there were some weird things going on. I would like to find out what. Blacklist doesn't necessarily mean you would have to start all over. But I would like to discuss what went on in private 1st - for analysis. I have never seen a client behave like that before, so naturally I'd like to find out what happened.

PLEASE -> bots@cryptoguru.org

It may very well be that it's all my fault and the program just misbehaved because of some unforeseen hardware error or whatever. I just don't know yet.

But in all fairness it could have possibly also been several other factors. I have it running in the background on an arch antergos box. At the time I was doing an update that affected like 300+ items, while also finding random new programs, and while the LBC was running in the background. So maybe  a dependency was updated while it was running, I think some nvdia things got changed, maybe those? idk.

Ah! Please let's discuss this via mail, I'll put in a summary here for the interested - of course.

edit: So with the little information you provided (and no further discussion), I solved this on my own...

After your client got the interval 4942213231-4942216430 to work on - and while it ran - you started a system update, probably including update of the drivers for the GPU, system libraries and what not. This is a plausible explanation for what I saw on my end: This made the client go haywire and it answered to validation challenges with a bunch of errors (namely tampered BLF file, execution time, etc.)

I removed the last two intervals your client worked on from your delivered valid Gkeys (6400 Mkeys = 6.4Gkeys) and janitored (= recomputed) them with my client. I will move your Id from blacklist now, but please

and that is @ALL:

Do not perform extensive system upgrades while the client is running. I mean if you exchange the GPU-driver while the GPU is computing - what do you expect to happen?

[rico666]

So for those who do not monitor the News link in my signature, here the "more important" news.


Claiming Finds:

Tomorrow it will be 6 months since our 1st find (https://lbc.cryptoguru.org/trophies) went into custody.

2016-10-11 03:00:34 GMT

The pool found a private key to f6cc30532dba44efe592733887f4f74c589c9602 (1PVwqUXrD5phy6gWrqJUrhpsPiBkTnftGg) as 0x22306e3f1a72. At the time of the find, there were 0.007899 BTC on that address.The funds were transferred to custody at 16cFBcM27DGriyvz5i8SLmd8n5ai8hCTEE. See the announcement and the modalities of the return of the funds to their rightful owner here.

As no one came along to reclaim these funds, they will be moved to the LBC Pot adding some $13 to the  currently present ~$195. *crowd cheers*


New generators: Kardashev

In the next few days, I will roll out our next generation of generators (sounds nice - eh?) called "Kardashev". The predecessor "HRD-core" served us well, but in order to break new ground, we needed to make extensive changes which made the resulting generator not just a newer version of HRD-core:

Unified binary

Mainly, the new generators will already have support for GPU computing, it will be on the client to evaluate "GPU-Auth" and start the generator with the appropriate options.
This will reduce the number of binaries we have to maintain/juggle (no +gpu versions anymore). You can see the generators already appearing on our update URL.

In order to have a seamless upgrade, please make sure you run at least LBC client version 1.140.  Because the new generators are all linked against libOpenCL, you have to make sure you do install OpenCL on your machine - even if you use only the CPU version or have a VM. If you were already running a GPU client, you do not need to do anything. If you are on a VM or even intend to run only CPU clients, you still have to install OpenCL. Luckily, you can install some dummy OpenCL package like https://apps.ubuntu.com/cat/applications/ocl-icd-libopencl1/ or the Mesa OpenCL or whatever (feel free to install the Nvidia, AMD ones if you intend to use that Hardware later on) - just make sure a libOpenCL.so is on your system.

HW support:

Reducing the number of binaries we have to juggle while improving the release cycle (testing & automation), allows us to extend the support for various other nice devices out there. Take for example Intel Goldmont a.k.a. Apollo Lake a.k.a. Pentium/Celeron WTF. These nifty little CPUs do have hardware support for SHA256! And some even niftier ARM CPUs do also! So yeah - we intend to support that. Your little settop box may actually perform a formidable search for BTC keys while you are watching a movie (of course without disturbing the movie).

Speed:

As of now, Kardashev generators are only slightly faster than HRD-core. While there is nothing spectacular as of now, make no mistake: We just left a local optimum, went through some performance valley and are releasing the new generators while climbing up the next performance mountain, just about the height of the last peak. Understood?
In terms of efficiency, already the current versions of HRD-core are the best key generation (and checking) software out there. With Kardashev, we intend to take the game to a whole new level. Doubling the speed seems realistic now. We do already have faster prototypes running here...

About:

Oh - and about the name. Kardashev is of course some tribute to our anglo-saxonian friends. I wouldn't mind to call the binary Kardašov, or Kardaschow or Кapдaшёв, but some people might have trouble with their keyboard.
As for why this name has been chosen: It was an idea sketched out by Ryan Castellucci some months (~10) ago, in reference to the nice but pathetic picture of the sun. Other than that, it should be self-explanatory why the name has been chosen. 

[rico666]

Your client may update to version 1.165 on the next startup

Code:

GPU authorized: yes
New client '1.165-LBC.bz2' found.
Some files were updated - please restart LBC.

and then fetch a "kardashev-xxx" generator, where xxx stands for a CPU architecture
which can be seen in ascending order of new instruction support:

'generic' < 'nehalem' < 'westmere' < 'sandybridge' < 'haswell' < 'skylake'

these correlate somewhat with specific instruction sets the compiler has optimized to. (Like AVX is supported by sandybridge, AVX2 by haswell etc.)

Don't worry if you have an AMD CPU: the right binary for your CPU should be chosen and in case it shouldn't, the 1.165 client has a new option:

    --override <gentype>|?
      Override the LBC generator choice. You get a list of valid
      generators when giving '?' as argument.


So just do a --override generic if you get a "illegal instruction" error or just try the "next lower" binary.


As for speed, the kardashev-generators should be about 5% faster than the old HRD-core generators. While that's not spectacular, it can be 1 Mkey/s (or more) if you had 20 Mkeys/s (or more) - which is pleasant nevertheless.

[Real-Duke]

Thanks for your effort, going to give it a try now!