Just a quick response for now, will discuss more later:
Here is the problem, it has to remain closed source or people running the program could change a few lines of code and steal the bitcoin that everyone in the pool is working hard for. That is the only reason why it is closed source.
Wait, so the client controls the 'payout' or how could one's modification of the client on their side change how other contributors are paid out?
I think it should be possible to code it in a way that the client, which people run, is open source, so e.g. it can be compiled by users themselves and malware for example can be ruled out etc., but the server side software remains closed (similar to when mining with a pool where the pool software is closed and the client software like cgminer is open).
Or are you worried of someone modifying the client and distributing that version so that benefits go to them?
In that case, it should be the same way as when downloading Bitcoin core: you have a GitHub page where people are always pointed to, always recommended not to get it from anywhere else and on the GitHub they can either read and compile the source themselves or download and verify signed binaries that have been compiled for them by you.
None of the software does an auto pay out. That will all be done manually.
If you know what the pool is about and understand how the client works in its original form and how the actual cracking software works in its original form, you can conclude how easy it would be to manipulate work done, i.e. your percentage of work done to receive more than your fair share or flat out take the private key and claim all the BTC for yourself.
Some people have already tested the programs. I can see how many people use a different cracking software to try and run with the client software. But without the version on github, they can run their program all day but receive zero credit because it is "fake" work/not validated. That is because there are 2 fail safes built into the software that detect bad actors and does not give them credit. One on the client side and one on the cracking side. If either are exposed, then the pool is worthless.
If you have ever ran TDs 64 bit pool and actually watch what is going on when your machine is running the program, which by the way, it is also closed course (hmmmm, I wonder why), you can see the vulnerabilities in it. But you have to pay attention...
I will discuss options with NotATether in private to see if their is a viable solution.