How to build your own Multipool - the Open Source Way

[PereguineBerty]

Haven't tried it yet but if your info is good as it looks, you'll be a hero around here.

[1714181162]

1714181162

[1714181162]

1714181162

[1714181162]

1714181162

[1714181162]

1714181162

[paradigmflux]

Haven't tried it yet but if your info is good as it looks, you'll be a hero around here.

This is all live directly off of the BTCDPool.com site.
I will be uploading a snapshot of all of the config files from that pool later tonight.

[goodluckpool]

thank you for your sharing, waiting to read more.

[zikomoto]

Thank you for sharing , was already working on one and this is helping me a lot

[paradigmflux]

Here's a cronjob equivalent of the payment processor.  This way it is entirely independent of the node.js application - it still logs to the redis instance and dumps any failed txns through to a text file.  I will try and clean this up some.

coming up next will be the exchange interactions as well as the front end reporting.

Code:

#!/bin/bash
cp /Scripts/payouts /Scripts/old_payouts
rm /Scripts/payouts
counter=0
                redis-cli del Pool_Stats:CurrentRoundBTCD
                redis-cli del Pool_Stats:CurrentRoundBTC
now="`date +%s`"
thisShift=`redis-cli hget Pool_Stats This_Shift`
ShiftStart=`redis-cli hget Pool_Stats:$thisShift starttime`
BTCDPrice=`redis-cli hget Exchange_Rates btcdcoin`
TotalEarned=0
TotalEarnedBTCD=0
# loop through algos
while read line
do
        AlgoTotal=0
        AlgoTotalBTCD=0
        logkey2="Pool_Stats:"$thisShift":Algos"
        logkey2BTCD="Pool_Stats:"$thisShift":AlgosBTCD"
echo "LOGKEY2: $logkey2"

        # loop through each coin for that algo
        while read CoinName
        do
                coinTotal=0
                coinTotalBTCD=0
                thiskey=$CoinName":balances"
                logkey="Pool_Stats:"$thisShift":Coins"
                logkeyBTCD="Pool_Stats:"$thisShift":CoinsBTCD"
                #Determine price for Coin
                coin2btc=`redis-cli hget Exchange_Rates $CoinName`
#               echo "$CoinName - $coin2btc"
                workersPerCoin=`redis-cli hlen $thiskey`
                if [ $workersPerCoin = 0 ]
                then
                        echo "do nothing" > /dev/null
                else

                        while read WorkerName
                        do
                                thisBalance=`redis-cli hget $thiskey $WorkerName`
                                thisEarned=`echo "scale=4;$thisBalance * $coin2btc" | bc -l`
                                coinTotal=`echo "scale=4;$coinTotal + $thisEarned" | bc -l`
                                AlgoTotal=`echo "scale=4;$AlgoTotal + $thisEarned" | bc -l`
                                BTCDEarned=`echo "scale=4;$thisEarned / $BTCDPrice" | bc -l`
                                coinTotalBTCD=`echo "scale=4;$coinTotalBTCD + $BTCDEarned" | bc -l`
                                AlgoTotalBTCD=`echo "scale=4;$AlgoTotalBTCD + $BTCDEarned" | bc -l`

#                               echo "$WorkerName earned $BTCDEarned from $CoinName"
                                redis-cli hincrbyfloat Pool_Stats:CurrentRoundBTCD $WorkerName $BTCDEarned
                                redis-cli hincrbyfloat Pool_Stats:CurrentRoundBTC $WorkerName $thisEarned
                        done< <(redis-cli hkeys $CoinName:balances)
                        redis-cli hset $logkey $CoinName $coinTotal
                        redis-cli hset $logkeyBTCD $CoinName $coinTotalBTCD
                        echo "$CoinName: $coinTotal"

                fi
        done< <(redis-cli hkeys Coin_Names_$line)
          redis-cli hset $logkey2 $line $AlgoTotal
        redis-cli hset $logkey2BTCD $line $AlgoTotalBTCD
TotalEarned=`echo "scale=4;$TotalEarned + $AlgoTotal" | bc -l`
TotalEarnedBTCD=`echo "scale=4;$TotalEarnedBTCD + $AlgoTotalBTCD" | bc -l`

done< <(redis-cli hkeys Coin_Algos)
redis-cli hset Pool_Stats:$thisShift Earned_BTC $TotalEarned
redis-cli hset Pool_Stats:$thisShift Earned_BTCD $TotalEarnedBTCD

echo "Total Earned: $TotalEarned"

redis-cli hset Pool_Stats:$thisShift endtime $now
nextShift=$(($thisShift + 1))
redis-cli hincrby Pool_Stats This_Shift 1
echo "$thisShift" >>/Scripts/Shifts
redis-cli hset Pool_Stats:$nextShift starttime $now
echo "Printing Earnings report" >> /Scripts/ShiftChangeLog.txt
echo "Shift change switching from $thisShift to $nextShift at $now" >>/Scripts/ShiftChangeErrorCheckerReport
while read WorkerName
do

        PrevBalance=`redis-cli zscore Pool_Stats:Balances $WorkerName`
        if [[ $PrevBalance == "" ]]
        then
                PrevBalance=0
        fi
        thisBalance=`redis-cli hget Pool_Stats:CurrentRoundBTCD $WorkerName`
        TotalBalance=`echo "scale=4;$PrevBalance + $thisBalance" | bc -l` >/dev/null
        echo $WorkerName $TotalBalance
#       echo "$WorkerName $TotalBalance" >> /Scripts/PayoutReport
        echo "$WorkerName $TotalBalance - was $PrevBalance plus today's $thisBalance" >> /Scripts/ShiftChangeErrorCheckerReport

        redis-cli zadd Pool_Stats:Balances $TotalBalance $WorkerName
        redis-cli hset Worker_Stats:Earnings:$WorkerName $thisShift $thisBalance
        redis-cli hincrbyfloat Worker_Stats:TotalEarned $WorkerName $thisBalance
        #Log each earning in the redis instance in the HTML format.
        #But first, mask the worker address.
                str=$WorkerName
                n=7
                hiddenWorkerName="***************"${str:${#str} - $n}


         redis-cli hincrby Pool_Stats Earning_Log_Entries 1
EarningNumber=`redis-cli hget Pool_Stats Earning_Log_Entries`
redis-cli hset Pool_Stats:EarningsLog $EarningNumber "<tr><td><center>$thisShift</center></td><td><b><center>Shift Earning</center></b></td><td><b>$hiddenWorkerName</td><td><center>$thisBalance</center></td><td><center>$PrevBalance</center></td><td><center>$TotalBalance</center></td><td><center>$now</center></td></tr>"


done< <(redis-cli hkeys Pool_Stats:CurrentRoundBTCD)
echo "Done adding coins, clearing balances now." >> /Scripts/ShiftChangeLog.log


# Save the total BTC/BTCD earned for each shift into a historical key for auditing purposes.
redis-cli rename Pool_Stats:CurrentRoundBTCD Pool_Stats:$thisShift:ShiftBTCD
redis-cli rename Pool_Stats:CurrentRoundBTC Pool_Stats:$thisShift:ShiftBTC

#for every coin on the pool....
while read Coin_Names2
do
        #Save the old balances key for every coin into a historical key.
        redis-cli rename $Coin_Names2:balances Prev:$thisShift:$Coin_Names2:balances

        # This loop will move every block from the blocksConfirmed keys into the blocksPaid keys. This means only blocksConfirmed are unpaid.
        while read PaidLine
        do
                redis-cli sadd $Coin_Names2:"blocksPaid" $PaidLine
                redis-cli srem $Coin_Names2:"blocksConfirmed" $PaidLine
                echo "nothing" > /dev/null
        done< <(redis-cli smembers $Coin_Names2:"blocksConfirmed")


done< <(redis-cli hkeys Coin_Names)

echo "Done script at $now" >> /Scripts/ShiftChangeLog.log

#Calculate workers owed in excesss of 5 BTCDcoin and generate a report of them.
while read PayoutLine
do

        amount=`redis-cli zscore Pool_Stats:Balances $PayoutLine`
        roundedamount=`echo "scale=4;$amount - 1" | bc -l`
        echo "$PayoutLine $roundedamount"
#log a file named /Scripts/pallp0pp
        echo "$PayoutLine $amount" >> /Scripts/payouts
#send all of the payments using the coin daemon
        txn=`bitcoindarkd sendtoaddress $PayoutLine $amount`
        if [ -z "$txn" ]
        then
	#log failed payout to txt file.
                echo "payment failed! $PayoutLine" >>/Scripts/AlertLog
        else
                urlstring="http://btcd.explorer.ssdpool.com:9050/tx/$txn"

                 newtotal=`echo "scale=4;$amount - $roundedamount" | bc -l` >/dev/null
                redis-cli hincrby Pool_Stats Earning_Log_Entries 1
                redis-cli hset Worker_Stats:Payouts:$PayoutLine $thisShift $amount
                redis-cli hincrbyfloat Worker_Stats:TotalPaid $PayoutLine $amount
EarningNumber=`redis-cli hget Pool_Stats Earning_Log_Entries`
                str=$PayoutLine
                n=7
                hiddenPayoutLine="***************"${str:${#str} - $n}
#log in redis for displaying on website
redis-cli hset Pool_Stats:EarningsLog $EarningNumber "<tr><td>$thisShift</td><td><b>Payout</b></td><td>$hiddenPayoutLine</td><td>$amount</td><td>$amount</td><td><a href=\"$urlstring\">here</a></td><td>$now</td></tr>"
                redis-cli zadd Pool_Stats:Balances 0 $PayoutLine
        fi
done< <(redis-cli zrangebyscore Pool_Stats:Balances 5 inf)
#move old profitability stats
redis-cli rename Pool_Stats:CurrentShift:Profitability Pool_Stats:$thisShift:Profitability

[YarkoL]

Been looking for a guide like this for some time already..

[paradigmflux]

Anybody going through and setting one of these bad boys up?

[paradigmflux]

Hashrate pie charts - this would go onto your home.html for example.

Code:

<!--- Start Pie --->
 <script>

                                        var data_arr3=[];
                                        init.push(function () {
                                                var item = { "label": "Scrypt",
                                                        "value": {{=it.stats.algos['scrypt'].hashrate/1024/1024}}
                                                        }
                                                data_arr3.push(item);
                                                var item = { "label": "SHA",
                                                        "value": {{=it.stats.algos['sha256'].hashrate/1024/1024/1024/100}}
                                                        }
                                                data_arr3.push(item);
                                                var item = { "label": "X11",
                                                        "value": {{=it.stats.algos['x11'].hashrate/1024/1024/4}}
                                                        }
                                                data_arr3.push(item);
                                                var item = { "label": "Keccak",
                                                        "value": {{=it.stats.algos['keccak'].hashrate/1024/1024/500}}
                                                        }
                                                data_arr3.push(item);
                                                var item = { "label": "ScryptN",
                                                        "value": {{=it.stats.algos['scrypt-n'].hashrate/1024/1024}}
                                                        }
                                                data_arr3.push(item);
                                                Morris.Donut({
                                                        element: 'hero-donut',
                                                        data: data_arr3,
                                                        colors: PixelAdmin.settings.consts.COLORS,
                                                        resize: true,
                                                        labelColor: '#888',
                                                        formatter: function (y, data) {
                                                                if(data.label == "Keccak")
                                                                        return(Math.round(y * 500,2) + ' MH');
                                                                else if(data.label == "Scrypt")
                                                                        return(Math.round(y,2) + ' MH');
                                                                else if(data.label == "SHA")
                                                                        return(Math.round(y * 100,2) + ' GH');
                                                                else if(data.label == "X11")
                                                                        return(Math.round(y * 4,2) + ' MH');
                                                                else
                                                                        return(y + ' MH');

                                                        }

                                                });
                                        });
                                </script>
                                <!-- / Javascript -->


                        <div class="row">
                                <div class="col-md-5">
                                                <div class="graph-container">
                                                        <div id="hero-donut" class="graph"></div>
                                                </div>
                                </div>
<!--- END PIE -->

Worker distribution pie charts:

Code:

<!--- start worker pie chart --->
  <script>

                                        var data_arr4=[];
                                        init.push(function () {
                                                var item = { "label": "Scrypt",
                                                        "value": {{=it.stats.algos['scrypt'].workers}}
                                                        }
                                                data_arr4.push(item);
                                                var item = { "label": "SHA",
                                                        "value": {{=it.stats.algos['sha256'].workers}}
                                                        }
                                                data_arr4.push(item);
                                                var item = { "label": "X11",
                                                        "value": {{=it.stats.algos['x11'].workers}}
                                                        }
                                                data_arr4.push(item);
                                                var item = { "label": "Keccak",
                                                        "value": {{=it.stats.algos['keccak'].workers}}
                                                        }
                                                data_arr4.push(item);
                                                var item = { "label": "ScryptN",
                                                        "value": {{=it.stats.algos['scrypt-n'].workers}}
                                                        }
                                                data_arr4.push(item);
                                                Morris.Donut({
                                                        element: 'hero-donut2',
                                                        data: data_arr4,
                                                        colors: PixelAdmin.settings.consts.COLORS,
                                                        resize: true,
                                                        labelColor: '#888',
                                                        formatter: function (y, data) {
                                                                        return(y + ' Workers');
                                                        }

                                                });
                                        });
                                </script>
                                <!-- / Javascript -->

                                                <div class="graph-container">
                                                        <div id="hero-donut2" class="graph"></div>
                                                </div>
<!-- end worker pie chart --->

[OmarGsPools]

Anybody going through and setting one of these bad boys up?

I'll be attempting to set one up later tomorrow. I like the open-source route and would like to help in the future development.

[mnporter2001]

Nice share !

However I cant get it to work, this is my fault not yours i must say, I just dont understand it all........

So how much would it cost me for you/ anyone else reading this to do it for me ?

Thanks
Mark

[rcloss]

Do you have any suggestions for hosting providers that can run this kind a MultiPool install? If not, how much bandwidth does one of these installs typically use? I'm guessing I can't run this on my old DSL line

[goodluckpool]

great info, looking forward to that automatic coin exchange part.

[billotronic]

Do you have any suggestions for hosting providers that can run this kind a MultiPool install? If not, how much bandwidth does one of these installs typically use? I'm guessing I can't run this on my old DSL line

ooo good question for the OP. I too would be very interested to know what kind of hardware you need for this.

[DreamSpace]

nice guide thank you

[Biomech]

Do you have any suggestions for hosting providers that can run this kind a MultiPool install? If not, how much bandwidth does one of these installs typically use? I'm guessing I can't run this on my old DSL line

ooo good question for the OP. I too would be very interested to know what kind of hardware you need for this.

I'm not involved, just another spectator. But I know a bit about NOMP, which is what he's basing this off of. I ran it in a test rig with a single core running 2.8 GHz and 1 meg ram with no issues at all, and rented rigs to bombard it. On a basic cable connection with a lot of other shit running. Barely slowed it down. NOMP is not terribly resource intensive.

[paradigmflux]

You can easily run one of these off of a 4GB Digital Ocean VM (which costs $40 a month).
Just make sure that you use a free cloudflare.com account (to help reduce DDOS attacks)
Use the cloudflare.com accunt to sign up for a free Dome9 account (and use Dome9 to firewall all HTTP access to only accept connections from cloudflare)
and voila!


PS, I will be posting some more in this thread a bit later today with some more info. Thanks for all of the nice PMs so far.

[s1kx]

Awesome, great code quality and definitely an improvement in the scene. Proper redis usage etc will definitely help a lot of pools scale.

[paradigmflux]

Awesome, great code quality and definitely an improvement in the scene. Proper redis usage etc will definitely help a lot of pools scale.

It could be lots better. It could all be in node.js
but then again, I don't really feel that the whole pool operation should be in a single node.js application.
And using stuff like bash scripts - as long as they're well written (and run entirely from in memory) - they're pretty much just as efficient.  I'm just wrapping up extending the NOMP api quite a bit (to reveal stuff like user hashrates, profitabilities, payouts etc)
I will post the updated stats.js and website.js once I am done.

[goodluckpool]

updates?

[paradigmflux]

updates?

i will update this thread a bit later tonight
for a sample one of these pools check out BTCDPool.com
every 10 minutes it liquidate all the earning coins and continually applies buy pressure on bitcoindark at cryptsy and poloniex.
it's even mining every possible profitable SHA altcoin (including PPC, which up until today I never figured out how to get working with NOMP)

i will post a link to a zip of the entire pool tonight