Bitcoin Forum

Bitcoin => Project Development => Topic started by: bitmover on January 24, 2023, 07:44:24 PM



Title: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 24, 2023, 07:44:24 PM
Based on suggestions from many users in this post (https://bitcointalk.org/index.php?topic=5435940.msg61617066#msg61617066) I created this simple tool to give more transparency and credibility to giveaways here, specially from honest newbies

https://i.imgur.com/tJz5BfC.png
https://bitcoindata.science/giveaway-manager/ (https://bitcoindata.science/giveaway-manager/)


Giveaways can now have their results easily verified.

It is also possible to save and share the results in a unique URL.



Provably fair giveaway manager

As the blockhash is just a number, its last 6 digits is converted to decimal using this function:

Code:
var decimal = parseInt(blockhash.slice(-6), 16);

Now we have an integer (0 to 16777215) from the blockhash.

After dividing this decimal by the number of participants, we use the modulo operator (%) to get the division remainder becomes the index_number.

This index_number is applied in the participants list, to get the position of the winner.

Code:
var index_number = decimal % competitors.length;
var winner = competitors[index_number];

For additional winners, the past winners are removed from the list and one more digit is added from the blockhash. A maximum 30 was added to avoid working with big numbers.



If you find this useful  , please refer this tool in upcoming giveaways


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: LoyceMobile on January 24, 2023, 07:59:14 PM
What if there's more than 1 winner?
What if the hash ends on 007 and there are 8 participants?
I'm wondering if all candidates have the same odds: what if the number of participants is large: my gut feeling tells me the first one on the list is more likely to be picked than the last one (LoyceV do the math).


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 24, 2023, 08:26:50 PM
What if there's more than 1 winner?

This is an additional feature. I can think about a solution for this. I can just add one more hex digit for each new winner to be rolled.

Quote
What if the hash ends on 007 and there are 8 participants?

The result to this operation is 7
You can check in the jsfiddle
https://jsfiddle.net/ruxqjoLt/

Quote
I'm wondering if all candidates have the same odds: what if the number of participants is large: my gut feeling tells me the first one on the list is more likely to be picked than the last one (LoyceV do the math).

I don't know if that makes sense, maybe someone who knows more statistic can help us. DdmrDdmr?
The number is a 3 digit hex, maximum 4095 in decimal.

If the division has no remainder,  it is zero so the first in list.
It cannot has  more remainders than the number it is divided by...

It looks OK to me. But I am no math specialist


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: hosseinimr93 on January 24, 2023, 08:35:09 PM
LoyceV is right about different chances of winning.

Assume that there are 10 participants.

The participant number 1 wins the game if the outcome is 0, 10, 20, 30, ... 4090.
The participant number 2 wins the game if the outcome is 1, 11, 21, 31, ... 4091.
The participant number 3 wins the game if the outcome is 2, 12, 22, 32, ... 4092.
........
........
The participant number 10 wins the game if the outcome is 9, 19, 29, 39, ...., 4089

The participants 1-6 will have slightly bigger chance than participants  7-10.
The chance of winning for participants 1-6 will be  410/4096 and the chance of winning for participants 7-10 will be 409/4096


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: LoyceMobile on January 24, 2023, 08:36:11 PM
Simple math: you use 1 digit, 0-9, and have 6 participants. That means the first 4 participants have 2 out of 10 chance, the last 2 have 1 out of 10 change of winning.
Easy fix: use 6 digits. Or even more. That makes the difference between positions in the list very small.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 24, 2023, 08:53:57 PM
Simple math: you use 1 digit, 0-9, and have 6 participants. That means the first 4 participants have 2 out of 10 chance, the last 2 have 1 out of 10 change of winning.
Easy fix: use 6 digits. Or even more. That makes the difference between positions in the list very small.

With already 4095 chances (fff), I believe the changes are OK, but I will add 3 more (ffffff)

That is a nice suggestion for the problem.

Then I can roll again with 1 more digit for each subsequent winner.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 25, 2023, 04:11:41 AM
Simple math: you use 1 digit, 0-9, and have 6 participants. That means the first 4 participants have 2 out of 10 chance, the last 2 have 1 out of 10 change of winning.
Easy fix: use 6 digits. Or even more. That makes the difference between positions in the list very small.

With already 4095 chances (fff), I believe the changes are OK, but I will add 3 more (ffffff)

That is a nice suggestion for the problem.

Then I can roll again with 1 more digit for each subsequent winner.

Added all the changes!

Now you can add up to 50 winners (i put those limits to avoid unnecessary  loops with big numbers).

Also, it has now 6 digits to find the first winner.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: examplens on January 25, 2023, 12:10:46 PM
here, the giveaway initiator just randomly adds participants? how does each participant know his winning number?
Are you thinking about the possibility that everyone chooses their own "random" number, like the previous way by an entry in the thread? I know that many people have their "lucky" numbers and always choose them if they are available.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: hosseinimr93 on January 25, 2023, 12:32:58 PM
Now you can add up to 50 winners (i put those limits to avoid unnecessary  loops with big numbers).
I just tested the tool with some random inputs and it seems that everything is working well.
There's only a small bug. If you set the number of winners to 50, only 1 winner will be selected.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 25, 2023, 12:46:07 PM
here, the giveaway initiator just randomly adds participants? how does each participant know his winning number?
Are you thinking about the possibility that everyone chooses their own "random" number, like the previous way by an entry in the thread? I know that many people have their "lucky" numbers and always choose them if they are available.


About this, the conclusion of the discussion was that those giveaways are easily verified.

This giveaway manager is more focused in the contests like this, where people just add their addresses. Each participant number is in the order of applications.
https://bitcointalk.org/index.php?topic=5435424.0

However, I can add this feature. I will take a look.

There's only a small bug. If you set the number of winners to 50, only 1 winner will be selected.

Thanks, That is really a bug. Now fixed.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: CoinEraser on January 25, 2023, 01:52:37 PM
It's really great that you are doing the work and creating such a page. This will surely enable newbies to hold a fair and equitable free raffle to avoid misunderstandings like current Rbah´s free raffle. While it doesn't solve the payout issue, but it is a good step in the right direction.  :)

-snip- However, I can add this feature. I will take a look. -snip-
Examplens suggestion is really good. This definitely gives more opportunities to use the site in different ways.  :)


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: LoyceV on January 25, 2023, 03:41:00 PM
Bugs:
Now we have an integer (1 to 16777215) from the blockhash.
This should be:
0 to 16777215

For additonal winners, the past winners are removed from the list and one more digit is added from the blockhash. A maximum 50 was added to avoid bugs.
That's too much. The shortest block hash (https://loyce.club/blockdata/hash.txt.gz) until now is:
Code:
0000000000000000000000005d6f06154c8685146aa7bc3dc9843876c9cefd0f
That's only 40 characters. To be (very) safe, you should probably limit it to about 30 winners if you keep the method of adding a digit. My preferred alternative: add a nonce for each winner: take the sha256sum of "blockhash+1" and use those digits. That gives an unlimited number of potential winners.

I did a test, and checked the first 3 winners: your math checks out :)



Feature requests :D
Can you add a "unique URL" and "future block" feature? Example: If I enter 4 usernames, I'd like to be able to pick a block in the future. That should give me an URL that I can share here, so participants can follow the giveaway.
Even if the block isn't in the future, a unique URL would be really cool to share the results.
To select the Target Block: can you show the current latest block (773565) instead of "1" to start with?


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 25, 2023, 11:16:36 PM
Bugs:
Now we have an integer (1 to 16777215) from the blockhash.
This should be:
0 to 16777215

For additonal winners, the past winners are removed from the list and one more digit is added from the blockhash. A maximum 50 was added to avoid bugs.
That's too much. The shortest block hash (https://loyce.club/blockdata/hash.txt.gz) until now is:
Code:
0000000000000000000000005d6f06154c8685146aa7bc3dc9843876c9cefd0f
That's only 40 characters. To be (very) safe, you should probably limit it to about 30 winners if you keep the method of adding a digit. My preferred alternative: add a nonce for each winner: take the sha256sum of "blockhash+1" and use those digits. That gives an unlimited number of potential winners.

I did a test, and checked the first 3 winners: your math checks out :)

Thank you, very sharp. I made all your suggestions, already in production.

About this nonce alternative, I will take a look later.

Quote
Feature requests :D
Can you add a "unique URL" and "future block" feature? Example: If I enter 4 usernames, I'd like to be able to pick a block in the future. That should give me an URL that I can share here, so participants can follow the giveaway.
Even if the block isn't in the future, a unique URL would be really cool to share the results.

I hade to use some cryptography to accomplish this, but it was easier than I expected.

I saved all the saved data in a variable and encrypted it using AES.
Then I saved this encrypted data in a unique URL.

As it is not sensible data, I just added the private keys in the script file.
When you load it, data decrypted and will fill out automatically.

Easily shareable!

Take a look:
https://bitcoindata.science/giveaway-manager/?U2FsdGVkX19E08GaMJDY1QNCMHlxQ4BoXO/TAEUSso0BHeEzziPww8wbM4/X+GHSUiyN6SPx0ilvAwu+//A7plknuyGUx/JgM/n8+qEoXeQzEisCo5zzepIskxTJefviVPRZwtFw6sUujNaJmlryVkWJ5t6eicz+NAemMECso3+8BS5hJ9k1qiZi/OtbyZFV0KR3BXxGKcZab+zKKDitPw==

Quote
To select the Target Block: can you show the current latest block (773565) instead of "1" to start with?

Done!!!




Tell me if you see any more bugs !


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: hosseinimr93 on January 26, 2023, 12:02:07 AM
Tell me if you see any more bugs !
Now, it's possible that a participant wins more than once. It wasn't like this before the changes.

Edit:
I found another bug.
When I save the link, it doesn't save the target block. It's set to the current block again when I open the saved link.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 26, 2023, 01:09:35 AM
Tell me if you see any more bugs !
Now, it's possible that a participant wins more than once. It wasn't like this before the changes.

Edit:
I found another bug.
When I save the link, it doesn't save the target block. It's set to the current block again when I open the saved link.

Thank you.
I found the bugs. I guess all fixed now.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: LoyceV on January 26, 2023, 06:33:53 AM
I saved all the saved data in a variable and encrypted it using AES.
Then I saved this encrypted data in a unique URL.
Even better than what I imagined, I like it :)


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: PowerGlove on January 27, 2023, 04:33:49 AM
Nice work, bitmover! :)

Although it's not a big deal for this application, using the modulo operator like that will introduce a bias into the results (look up "modulo bias", if you're interested). One way to ensure a mathematically fair selection process is to generate a random number, and then (instead of applying a modulo operation to force it within range) throw it away and try again if it's not already in the range you need it to be in. A naive implementation of this "rejection sampling" would be very inefficient, so what you need to do is mask off some bits (based on the nearest enclosing power of two) and then apply the range check after that.

To make the above feasible, you need a good source of deterministically-generated random numbers. One way to get them would be to use Loyce's tip, and treat the blockhash as a "seed" which you append a running counter onto and then take the SHA-256 of that. Another way, which I prefer (only by a little) is to do effectively what Loyce suggested, but use SHA-256 in an HMAC construction (i.e. as a keyed hash function, with the seed as the "key", and a running counter as the "data").

I implemented the above as a Python (3.7+) script that you can play around with, if you like:

Code:
#!/usr/bin/env python3

import sys

import hmac

import math

def fail_if(condition, message):

    if condition:

        sys.exit('fatal error: ' + message)

def random_uint32(seed_bytes, index):

    fail_if(index < 0 or index > 4294967295, 'index is out of range.')

    return int.from_bytes(hmac.digest(seed_bytes, index.to_bytes(4, 'little'), 'sha256')[:4], 'little')

def pick_winners_unbiased(how_many, from_list, seed_bytes):

    fail_if(how_many < 0 or how_many > len(from_list), 'how_many is out of range.')

    remaining = from_list[:]

    winners = []

    counter = 0

    while len(winners) != how_many:

        mask = 2 ** math.ceil(math.log2(len(remaining))) - 1

        random = random_uint32(seed_bytes, counter) & mask

        counter += 1

        if random < len(remaining):

            winners.append(remaining.pop(random))

    return winners

def main(args):

    fail_if(len(args) != 3, 'expected 3 arguments (how_many_winners, names_file_path, seed_text).')

    how_many_winners, names_file_path, seed_text = int(args[0]), args[1], args[2]

    with open(names_file_path) as file:

        list_of_names = [name for name in file.read().splitlines() if name.strip()]

    seed_bytes = seed_text.encode('utf-8')

    print(f'Picking {how_many_winners} winner(s) from a list of {len(list_of_names)} participant(s), using seed "{seed_text}": {pick_winners_unbiased(how_many_winners, list_of_names, seed_bytes)}.')

if __name__ == '__main__':

    main(sys.argv[1:])

To test it, I used this list of 54 names:

Code:
TryNinja
BitMaxz
hilariousetc
HeRetiK
pooya87
HCP
OmegaStarScream
LeGaulois
mocacinno
DooMAD
jackg
mk4
buwaytress
LoyceV
AdolfinWolf
Hydrogen
stompix
Potato Chips
The Sceptical Chymist
Welsh
d5000
Steamtyme
ranochigo
DdmrDdmr
Rath
ETFbitcoin
Lucius
o_e_l_e_o
1miau
malevolent
mikeywith
Husna QA
suchmoon
Halab
nc50lc
mole0815
fillippone
Trofo
NotATether
Bthd
GazetaBitcoin
bitmover
NeuroticFish
dkbit98
Pmalek
BlackHatCoiner
DireWolfM14
SFR10
DaveF
Ratimov
webtricks
Rikafip
hosseinimr93
n0nce

It's mostly intended as reference code, so I didn't spend much time making it user-friendly. It expects 3 arguments: the number of winners to pick, a path to a file containing the names of participants (one per line), and a seed. So, if you place the script into a file named "pick_winners.py", and the above list into a file named "names.txt", and you wanted (for example) to pick 3 winners from that list (using the hash of the genesis block as the seed), then you would invoke it like this:

Code:
$ python3 pick_winners.py 3 names.txt 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

Which should produce the following:

Picking 3 winner(s) from a list of 54 participant(s), using seed "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f": ['n0nce', 'BlackHatCoiner', 'dkbit98'].

Obviously, JavaScript would be more useful to you, and if you don't mind leaving older browsers out in the cold, then the above algorithm can actually be implemented pretty easily (by using the built-in SubtleCrypto (https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto) interface; you just have to be careful to always serve the page over HTTPS, which is a requirement for this interface to be available on some browsers). Here are the important functions (forgive my clumsy JavaScript, I'm a bit out of practice):

Code:
async function seed_from_text(text) {

    // assumes: "text" is a non-empty string

    const encoder = new TextEncoder();

    return { hmac_key: await window.crypto.subtle.importKey("raw", encoder.encode(text), { name: "HMAC",  hash: "SHA-256" }, false, ["sign"]) };
}

async function random_uint32(seed, index) {

    // assumes: "seed" came from seed_from_text(...)

    // assumes: "index" is an integer >= 0 && <= 4294967295

    const data = new Uint8Array([index & 255, index >> 8 & 255, index >> 16 & 255, index >> 24 & 255]);

    const hash = await window.crypto.subtle.sign("HMAC", (await seed).hmac_key, data);

    return new DataView(hash).getUint32(0, true);
}

async function pick_winners_unbiased(how_many, from_array, seed) {

    // assumes: "how_many" is an integer >= 0 && <= from_array.length

    // assumes: "from_array" is an array of all participants

    // assumes: "seed" came from seed_from_text(...)

    const remaining = from_array.slice();

    const winners = [];

    let counter = 0;

    while(winners.length != how_many) {

        const mask = 2 ** Math.ceil(Math.log2(remaining.length)) - 1;

        const random = await random_uint32(seed, counter++) & mask;

        if(random < remaining.length) {

            winners.push(remaining.splice(random, 1)[0]);
        }
    }

    return winners;
}

The SubtleCrypto interface is asynchronous, so (unfortunately) these functions have to be, too. Here's a small example of me testing them, in the browser console:

>> seed = await seed_from_text("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f");
<< Object { hmac_key: CryptoKey }
>> names = ["TryNinja", "BitMaxz", "hilariousetc", "HeRetiK", "pooya87", "HCP", "OmegaStarScream", "LeGaulois", "mocacinno", "DooMAD", "jackg", "mk4", "buwaytress", "LoyceV", "AdolfinWolf", "Hydrogen", "stompix", "Potato Chips", "The Sceptical Chymist", "Welsh", "d5000", "Steamtyme", "ranochigo", "DdmrDdmr", "Rath", "ETFbitcoin", "Lucius", "o_e_l_e_o", "1miau", "malevolent", "mikeywith", "Husna QA", "suchmoon", "Halab", "nc50lc", "mole0815", "fillippone", "Trofo", "NotATether", "Bthd", "GazetaBitcoin", "bitmover", "NeuroticFish", "dkbit98", "Pmalek", "BlackHatCoiner", "DireWolfM14", "SFR10", "DaveF", "Ratimov", "webtricks", "Rikafip", "hosseinimr93", "n0nce"];
<< Array(54) [ "TryNinja", "BitMaxz", "hilariousetc", "HeRetiK", "pooya87", "HCP", "OmegaStarScream", "LeGaulois", "mocacinno", "DooMAD", ... ]
>> winners = await pick_winners_unbiased(3, names, seed);
<< Array(3) [ "n0nce", "BlackHatCoiner", "dkbit98" ]


I hope all of the above can help you to improve your nice project! :)

One cool thing about this approach is that it would remove the 30-winner limit you currently have. Also, having the algorithm available as a Python script might find a use case in giving people a second way to verify results.

P.S. Two small spelling mistakes I spotted: "For additonal winners" and "Provaly fair giveaway manager".


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 27, 2023, 04:11:18 PM
Nice work, bitmover! :)

Although it's not a big deal for this application, using the modulo operator like that will introduce a bias into the results (look up "modulo bias", if you're interested). One way to ensure a mathematically fair selection process is to generate a random number, and then (instead of applying a modulo operation to force it within range) throw it away and try again if it's not already in the range you need it to be in. A naive implementation of this "rejection sampling" would be very inefficient, so what you need to do is mask off some bits (based on the nearest enclosing power of two) and then apply the range check after that.


Hello PowerGlove.
Thank you a lot for pointing this out.

I believe you read this article, where the author make similar suggestions about this problem:

https://research.kudelskisecurity.com/2020/07/28/the-definitive-guide-to-modulo-bias-and-how-to-avoid-it/

The problem is that players who are in the first spots will more likely be winners than the ones in last (if we have more than 40 competitors), as pointed in the article.
https://cybermashup.files.wordpress.com/2020/06/modulo_distribution-2.png


As in this project I am not dealing with cryptographic schemes that could be attacked and things like that,  I believe I can make an easier way out for this bias.

I can just shuffle the list of competitors before applying the modulo operator.

As I need to always have the same result, I will not use the built in  shuffle function. I can use the blockhash decimal in a custom function to shuffle it.

Code:
competitors.sort((a, b) => 0.5 - blockhash_lastdigit/10);

This function will take move the item up or down depending if 0.5-blockhash_lastdigit is positive or negative.
It will shuffle the array.  

I will take a look if I can find a better way to shuffle it in a more random way.
(Good ideas here: https://stackoverflow.com/questions/16801687/javascript-random-ordering-with-seed)

I believe this will solve the problem in a much simpler way. What do you think?


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: LoyceMobile on January 27, 2023, 04:33:19 PM
Hello PokerPlayer.
Long day? ;)

I wouldn't shuffle the list of participants, it makes it harder to verify the result.
If you really mind the (give or take) 0.001% difference in chance of winning, just use more digits of the hash to make the difference even smaller. I'd say it's negligible already.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 27, 2023, 06:59:56 PM
I wouldn't shuffle the list of participants, it makes it harder to verify the result.

Yeah, this is certainly important. Easily verified by anyone who wants and always get the same result.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: Pmalek on January 27, 2023, 07:24:48 PM
I might sound like an unknowledgeable jackass, but what is the problem with random.org and why can't it be used for all kinds of random draws?

Is it the 'it's closed-source so we don't like it' problem or something else? Is there proof the randomness is unsatisfactory?
I fail to see the motive of random.org creators to built a non-random generator that shows a tendency to certain selections over others. Especially because they don't know what kind of selection and for what purpose the software is being used.

I understand that 3rd-parties can't verify that the selection was indeed random and drawn only once. Therefore, there is a need to trust whoever makes the draw to have done it fairly and not generate draws until the number/numbers he wants comes up. Is that the main reason or is there something else?


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 27, 2023, 08:03:04 PM
I might sound like an unknowledgeable jackass, but what is the problem with random.org and why can't it be used for all kinds of random draws?

I didn't know this website until now, but let me point a few things this tool can do (I don't know if random.org can do all of not  , but probably not all)

1 - code and results easily verified. No trust involved.
2 - it uses blockhash of a block to select multiple winners, which is already a culture of this forum for a long time.
3 - it allows anyone to share the results here in a simple URL.
4 - easy to use. Random.org is much more complicated as it is much bigger.
5 - you will always get the same winner no matter how many times you roll


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: Pmalek on January 28, 2023, 07:24:42 AM
1 - code and results easily verified. No trust involved.
...
3 - it allows anyone to share the results here in a simple URL.
...
5 - you will always get the same winner no matter how many times you roll
It essentially boils down to creating a system where each participant has a way to check that the draw was fair and not staged. The fairness is already guaranteed using the hash method of an upcoming block as you mentioned. But this improves and builds upon that idea with more advanced features.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: PowerGlove on January 30, 2023, 05:14:20 AM
Thank you a lot for pointing this out.
No problem, happy to help. :)

I believe this will solve the problem in a much simpler way. What do you think?
Hmm, that's getting a bit too hacky I would say (i.e. adding one imperfect approach onto another). It's worth pointing out that if you did have a good way to shuffle the participants (e.g. with a modern variant of the Fisher-Yates algorithm, and enough entropy to feed it with) then you wouldn't need anything else, because after a properly-executed shuffle you could simply read off the winners according to position (i.e. index 0 = winner 1, index 1 = winner 2, etc.)

Shuffling is actually a very natural way to think about this problem, and the code I provided earlier is capable of doing a "perfect" (unbiased) shuffle. If you call pick_winners_unbiased with a list of (let's say) 50 names and you ask it to return 50 winners, then what results is a shuffled version of the original list.

Sorry to be discouraging, but I don't think the approach you've settled on (i.e. using the available entropy directly, and coming up with an improvised scheme to generate a sequence of random numbers) is one that you should stick with for too long. Stretching out the entropy (with a cryptographic hash function, like Loyce and I have suggested) will let you explore better algorithms (and remove the limit on the number of winners, too).

Your algorithm suffering from "modulo bias" is not that much of a big deal (particularly for this application, and especially with the divisors being so much smaller than your random numbers), but it's a strong indication that you've probably gotten other details wrong, too. For example, the way you're prepending an extra hexadecimal digit for each winner beyond the first one, seems very sketchy to me. Aside from it having no mathematical basis for working correctly (at least, not one that I can see), that approach will overflow; after around 8 "steps" you'll start to lose precision (i.e. you'll be passing things into parseInt that are greater than Number.MAX_SAFE_INTEGER).

One way to test the quality of algorithms like these is to run simulations and compute histograms. So, I simulated 1,000,000 giveaways between 30 participants using your algorithm. In each giveaway "LoyceV" was included in the list of participants with 29 other names. Here's how many times LoyceV won, or came second, or third, etc:

https://imgvb.com/images/2023/01/30/b88d6c73c1cc36b80fa29c634f68be46.png

For reference, here's the same simulation data run through the algorithm I proposed earlier:

https://imgvb.com/images/2023/01/30/570befcb155b2315d483c14bd9398aa5.png

Now, one of these is clearly working correctly and the other needs a little help. :D

If you feel like I must have made some kind of mistake, then feel free to replicate my results:

  • Use the first 30 names from my previous post as the list of participants.
  • Keep an array H of 30 integers, initialized to 0.
  • Keep a running counter T set to the current trial # (from 0 to 999,999).

Then, do this 1,000,000 times:

  • Compute an imaginary "blockhash" by taking the SHA-256 of T (as a string; e.g. "0" == "5feceb66ffc86f38d952786c6d696c79c2dbc239dd4e91b46729d73a27fb57e9").
  • Pick 30 winners from the list of participants using your algorithm and the above blockhash.
  • Find the position of "LoyceV" in the list of winners and increment H accordingly.

My suggestion to you would be to take a more careful look at my previous post. But, if you're committed to sticking with your approach (which you seem to be), then I suggest changing this line:

Code:
let n_winner_decimal = parseInt(blockhash.slice(-(6 + step)), 16)

To this:

Code:
let n_winner_decimal = parseInt(blockhash.slice(-(6 + step), blockhash.length - step), 16)

That will improve things quite a bit (both by avoiding overflow, and also by producing a less correlated sequence). With the above fix applied, your histogram looks very similar to mine:

https://imgvb.com/images/2023/01/30/708c1b492a78459c9a7596cf355da886.png


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: LoyceV on January 30, 2023, 09:01:58 AM
For example, the way you're prepending an extra hexadecimal digit for each winner beyond the first one, seems very sketchy to me. Aside from it having no mathematical basis for working correctly (at least, not one that I can see), that approach will overflow; after around 8 "steps" you'll start to lose precision (i.e. you'll be passing things into parseInt that are greater than Number.MAX_SAFE_INTEGER).
An easy fix would be to remove a digit on the right when adding one on the left.

Quote
One way to test the quality of algorithms like these is to run simulations and compute histograms. So, I simulated 1,000,000 giveaways between 30 participants using your algorithm. In each giveaway "LoyceV" was included in the list of participants with 29 other names. Here's how many times LoyceV won, or came second, or third, etc:
Without checking the math, a question: did you remove one participant each time you calculated a next winner? The 0 wins at 15th (16th?) makes me think you didn't.

Quote
If you simulated 1 million giveaways and made a graph for places 1-30, why is the average around 33,333, and not 1 million?


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 30, 2023, 12:48:05 PM
Shuffling is actually a very natural way to think about this problem, and the code I provided earlier is capable of doing a "perfect" (unbiased) shuffle. If you call pick_winners_unbiased with a list of (let's say) 50 names and you ask it to return 50 winners, then what results is a shuffled version of the original list.

This kind of approach will not work for 2 reasons:

1 - not verifiable by third party and cannot be reproduced.
2 -  don't use blockhash, which is a "must" for this project, as this is a deep culture in forum giveaways.

For example, the way you're prepending an extra hexadecimal digit for each winner beyond the first one, seems very sketchy to me. Aside from it having no mathematical basis for working correctly (at least, not one that I can see), that approach will overflow; after around 8 "steps" you'll start to lose precision (i.e. you'll be passing things into parseInt that are greater than Number.MAX_SAFE_INTEGER).
An easy fix would be to remove a digit on the right when adding one on the left.

Thats a very good idea, I will implement it.

But also, I believe giveaways with more than one winner are quite unusual, and we won't see this is a lot here.

Anyway, I will implement LoyceV idea to remove a digit when adding another


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: PowerGlove on January 30, 2023, 01:55:49 PM
An easy fix would be to remove a digit on the right when adding one on the left.
Did you read my whole post? :P

Without checking the math, a question: did you remove one participant each time you calculated a next winner? The 0 wins at 15th (16th?) makes me think you didn't.
Yes. That's a pretty essential detail, so I made sure to get it right.

With bitmover's algorithm, where you appear in the list of participants (in your case, near the middle) affects the probabilities of where you'll appear in the list of winners (which is a sign that something is very wrong). For example, here's the same simulation, but done for o_e_l_e_o (who's near the end of the participants list):

https://imgvb.com/images/2023/01/30/dbaf68e8166f0c80db4444c197482b13.png

And here's how it looks (i.e. as it should) when repeated with the reference algorithm:

https://imgvb.com/images/2023/01/30/38cd5eac4a2d98fc2aad1385e2942b35.png

If you simulated 1 million giveaways and made a graph for places 1-30, why is the average around 33,333, and not 1 million?
Hmm, these graphs must be much more confusing than I think they are. 1,000,000 simulated outcomes distributed between 30 "bins", leaves each bin with ~33,333 (on average).



An easy fix would be to remove a digit on the right when adding one on the left.
Thats a very good idea, I will implement it.
I don't know how both of you have managed to miss the fact that I suggested that. :D


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: LoyceV on January 30, 2023, 05:08:16 PM
Did you read my whole post? :P
Lol, no, I missed clearly the last part.

Quote
With bitmover's algorithm, where you appear in the list of participants (in your case, near the middle) affects the probabilities of where you'll appear in the list of winners (which is a sign that something is very wrong).
Thanks for simulating this, otherwise it could have gone unnoticed for a very long time! O_e_l_e_o's graph doesn't look like what I expected at all.

Quote
I don't know how both of you have managed to miss the fact that I suggested that. :D
We're not alts :P


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on January 30, 2023, 09:28:20 PM
An easy fix would be to remove a digit on the right when adding one on the left.
Thats a very good idea, I will implement it.
I don't know how both of you have managed to miss the fact that I suggested that. :D

Just added your suggestions here (https://github.com/bitmover-studio/bitcoin-giveaway-manager/commit/5887bd2aab083ca79fcd18a1efb1eea93472c25f)




I also add a feature to tell when the future block will be mined, considering 10 minutes per block.

https://i.imgur.com/FD2V4Rk.png


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on February 01, 2023, 03:51:29 PM
I am happy to announce that this tool is already being used, and it was first used in this giveaway :
https://bitcointalk.org/index.php?topic=5437138.0;all

Results can be verified in this link
 shareable result (https://bitcoindata.science/giveaway-manager/?U2FsdGVkX1+IlMvYv77tuhsB/vQt4pcye/qxevNN6SJJTQnPx46mub06mApcePE5eOgtGfTrMnEh4HZ/ET/XTsJdoX2D7S1eEHF4XAR35Wk/WvmF13kYhB3SaIjWnMauhFT9IH1D+/zEoSQUlNmK4TlgzhdJbzyCpuuREjtFxYcKYLZ6RNVshw4ZjeMkANiYS5MWkE/XqRdr/j5z77c2xT7+Z7iISiQaLvxBWoOKCGVn68cGMp/qC3s7Q16+Bjrn2K2CtTGg8XUYiC5zC0sXKxypzRROWV9ldOoibubBD286t/ika+gPCVUMMJgapUuR9GqhMjBgZELn8722JH46fcjU77uYv0aWKO1X/RDg3CpEk9DV6cYtErhNNXcSOY2NOLBlnrSNzkMlyXLMu+ywzHpRTy9tU1x7AyoVRsq2O7Sr2Ewc1QxwtswJ3wKOYkmdwli3yjqJQ7HoFRU0GLVhVmzYzGnVMa2P5VY9ndYV4pWaNj0P+Qw2UXPR3FtlJQDs)


I am open for ideas for similar projects!


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: examplens on February 07, 2023, 01:03:52 PM
I am happy to announce that this tool is already being used, and it was first used in this giveaway :
https://bitcointalk.org/index.php?topic=5437138.0;all

Results can be verified in this link
 shareable result (https://bitcoindata.science/giveaway-manager/?U2FsdGVkX1+IlMvYv77tuhsB/vQt4pcye/qxevNN6SJJTQnPx46mub06mApcePE5eOgtGfTrMnEh4HZ/ET/XTsJdoX2D7S1eEHF4XAR35Wk/WvmF13kYhB3SaIjWnMauhFT9IH1D+/zEoSQUlNmK4TlgzhdJbzyCpuuREjtFxYcKYLZ6RNVshw4ZjeMkANiYS5MWkE/XqRdr/j5z77c2xT7+Z7iISiQaLvxBWoOKCGVn68cGMp/qC3s7Q16+Bjrn2K2CtTGg8XUYiC5zC0sXKxypzRROWV9ldOoibubBD286t/ika+gPCVUMMJgapUuR9GqhMjBgZELn8722JH46fcjU77uYv0aWKO1X/RDg3CpEk9DV6cYtErhNNXcSOY2NOLBlnrSNzkMlyXLMu+ywzHpRTy9tU1x7AyoVRsq2O7Sr2Ewc1QxwtswJ3wKOYkmdwli3yjqJQ7HoFRU0GLVhVmzYzGnVMa2P5VY9ndYV4pWaNj0P+Qw2UXPR3FtlJQDs)


I am open for ideas for similar projects!

you responded well to the giveaway transparency problem, the solution has already been used.

why didn't you integrate it with your main site?
on these giveaway pages, you could at least have a menu linked. There are some interesting tools there, why not inform the giveaway visitors about them?
Just my 2c.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: bitmover on February 07, 2023, 07:38:52 PM
you responded well to the giveaway transparency problem, the solution has already been used.

why didn't you integrate it with your main site?
on these giveaway pages, you could at least have a menu linked. There are some interesting tools there, why not inform the giveaway visitors about them?
Just my 2c.

Thanks your suggestion. you are right about them.

The point is that this website needs a new version.
Most of the stuff there is somehow broken already. API changed, many web standards changed, my skills improved and so on. There are many broken links as well.

I need to change so many things there. I also need to figure out a different model for it.

I might make something in react.

Edit: For example, I just took a look at the website and I saw that the most used tool (balance checker) is down because mempool.space changed their API structure. I will try to fix it soon.


Title: Re: [ANN] Bitcointalk Giveaway Manager
Post by: PowerGlove on February 08, 2023, 11:43:45 AM
Shuffling is actually a very natural way to think about this problem, and the code I provided earlier is capable of doing a "perfect" (unbiased) shuffle. If you call pick_winners_unbiased with a list of (let's say) 50 names and you ask it to return 50 winners, then what results is a shuffled version of the original list.

This kind of approach will not work for 2 reasons:

1 - not verifiable by third party and cannot be reproduced.
2 -  don't use blockhash, which is a "must" for this project, as this is a deep culture in forum giveaways.
Hmm, we must have our wires crossed somewhere because both of those points are wrong. Let me try again: I'm saying that if you were to call pick_winners_unbiased (from this post (https://bitcointalk.org/index.php?topic=5436655.msg61661741#msg61661741)) with the blockhash as the "seed" and with the same number of winners as there are contestants, then you would have performed a completely deterministic shuffle. By deterministic I mean that it would always come out the same way (which is your first point) and that the particular way it comes out would depend on the blockhash (which is your second point).

I think maybe I overcomplicated that post with too many ideas (modulo bias, rejection sampling, HMACs, using the SubtleCrypto interface, etc.) and that (as a result) you didn't properly understand the code. :)

I'm not suggesting that you switch to shuffling, I'm just pointing out that it would work correctly and that you already have everything you need if you decided to approach the problem from that angle.

I think your algorithm (since the overflow fix) is fine as is, and it has the important advantage of being simple to understand (and easy to explain). But, if you wanted to remove the 30-winner limit, and you wanted to keep things simple (no rejection sampling), then one way to do it (since you're already using CryptoJS) would be to change these two lines:

Code:
let decimal = parseInt(blockhash.slice(-6), 16)

Code:
let n_winner_decimal = parseInt(blockhash.slice(-(6 + step), blockhash.length - step), 16)

To these:

Code:
let decimal = parseInt(CryptoJS.HmacSHA256("0", blockhash).toString(CryptoJS.enc.Hex).slice(-8), 16)

Code:
let n_winner_decimal = parseInt(CryptoJS.HmacSHA256(String(step), blockhash).toString(CryptoJS.enc.Hex).slice(-8), 16)

That would adjust the range of your random numbers from 0-16777215 to 0-4294967295 (making modulo bias even less of a problem) and would let you pick as many winners as you like.

Here's a simulation/histogram (same parameters as before) to confirm that the above algorithm works as expected:

https://imgvb.com/images/2023/02/08/cdb6ba16c7ecccaf0c789d118a6b0470.png