How to run your own P2Pool in Ubuntu 14.04 LTS with merged mining

[raskul]

Haven't gone through the whole post yet but I think this is/was EXACTLY what I was looking for- an updated and easy way to setup a p2pool.

I'll start working on this now

good to see more and more p2pool nodes getting set up, best luck with yours waldohoover  

I'm happy to see people wanting to support the network as well and move towards p2pool, thanks

Setting up the node (in my sig) wasn't too bad with Morblias initial guidance but I def want to get a pool going now. I run /r/TeamRedditMining on reddit and would love to have our own pool going.  

aha! a US node, i'll direct US friends to it.. my node is also in my sig (Central Cavern - which is an Amsterdam node) - but be careful, the page is noisy  

Very nice and thank you!

No harm in making mining a bit of fun, and since murdof did hold my hand rather a lot in setting up that node, credit should go to him; he's taught me loads over the past couple of days... my only linux work previously was setting up nodes for coins - what I really want to do is get my head around DGM. Always mined on small DGM pools up 'til now and I'd love to be able to know the ins and outs of it - even if I don't essentially set up a DGM pool, it's all learning huh?

[1714018841]

1714018841

[1714018841]

1714018841

[1714018841]

1714018841

[KyrosKrane]

A few months ago, I was working on a script that would automate the installation and rollout of a VPS with a bitcoind full node and a p2pool install. I never got it finalized with all the bells and whistles that I wanted, nor really polished, but it was good enough that I was able to adapt it for my local and remote p2pool installs, with some obvious changes. I'm posting it here in the hope you (or others) find it useful. I'd encourage you to scavenge whatever bits you can incorporate into your already-excellent guide.

Code:

###############################################################################
# Prerequisites
###############################################################################
#
# Before you run this, do the following:
#   * Log in as root
#   * Be in your home directory
#   * The parent of home directories has to be specified below (e.g., if user home directories are like /home/user1, then HOME_ROOT must be set to /home).
#
#   * (Optional but recommended) Create a file with the name specified below in REGULAR_USER_PUBLIC_KEY_FILE that contains your SSH public key.  This should be in the copy/paste format you get from PuttyGen, not the saved public key file.

# Assumptions:
#   * Running on a relatively recent version of Ubuntu (12.04 or later) -- not tested with anything else, but it might work anyway!
#   * Python is already installed. If not, the script attempts to install Python, but right now I can't guarantee if you'll get Python 2.7 or 3.x.  P2Pool doesn't work with 3.x.
#   * This script was designed for use on a VM or VPS that's basically empty, and only has the root user created.



###############################################################################
# This section has to be run as root
###############################################################################

PrintStatus()
{
    echo
    echo "----- STATUS: $*"
    echo
}

#############
# Script settings - EdIT THESE before running.
#############

HOME_ROOT=/home

# These are general script settings.  Edit these before running the script.
REGULAR_USER_NAME=heybrother  # Randomly inspired by the song on the radio as I wrote this!
REGULAR_USER_PUBLIC_KEY_FILE=$REGULAR_USER_NAME.pub
# @TODO: Prompt the user for these two values.


#############
# System updates and patches
#############

# @TODO: Maybe create a swap file? For now, the script assumes the host system either has enough memory or has swap already set up.
# For a sample command, see:
# https://raw.github.com/XertroV/BitcoinAutoNode/master/bitcoinAutoNode.sh

PrintStatus "Updating the system"
# Update your packages and package list
apt-get update
apt-get -y dist-upgrade # This does the functions of upgrade as well.

PrintStatus "Installing system tools"
# @TODO: To get the add-apt-repository command, we may need to install one of the following, as per:
# https://stackoverflow.com/questions/13018626/add-apt-repository-not-found
# for Ubuntu <= 12.04
# sudo apt-get install python-software-properties
# for Ubuntu >= 12.10
# sudo apt-get install software-properties-common
#
# At some point I'll have to figure out how to distinguish versions and load only the needed one, but for now, just load both.
apt-get -y install python-software-properties software-properties-common


# Add the bitcoin repository - used for some bitcoind dependencies
# So it seems this is an official bitcoin-core repository, or at least, official enough that the devs endorse downloads from it in their readme file.
#So, I'll trust the binary download from here, rather than try to muddle through the insane mess required to build bitcoind cleanly from source on multiple arbitrary target systems.
PrintStatus "Adding bitcoin repository"
add-apt-repository -y ppa:bitcoin/bitcoin
apt-get update


# Install some useful utilities and the pre-reqs for P2Pool and bitcoin
PrintStatus "Installing utilities"
apt-get -y install nano htop ufw monit git p7zip-full screen


#############
# Python & P2Pool dependencies
#############

# Make sure Python is installed
PYTHON="$(which python)"
if [ -z "$PYTHON" ] ; then
    PrintStatus "Installing Python."
    apt-get -y install python
    # @TODO: review this; make sure we don't install Python3 by mistake.
    PYTHON="$(which python)"
fi

if [ -z "$PYTHON" ] ; then
    PrintStatus "WARNING! Failed to install Python. P2Pool may not run."
    # @TODO: Any special handling here for the error?
else
    PYTHON_VERSION="$("$PYTHON" -V 2>&1)"
    PYTHON_26=$(awk 'BEGIN { if ("'"$PYTHON_VERSION"'" ~ /Python 2.6.*/) print "YES";}')
    #PYTHON_27=$(awk 'BEGIN { if ("'"$PYTHON_VERSION"'" ~ /Python 2.7.*/) print "YES";}') # Just for testing
    PYTHON_3x=$(awk 'BEGIN { if ("'"$PYTHON_VERSION"'" ~ /Python 3.*/) print "YES";}')

    if [ -n "$PYTHON_26" ] ; then
        PrintStatus "Python 2.6.* requires an additional dependency, installing."
        apt-get -y install python-argparse
    elif [ -n "$PYTHON_3x" ] ; then
        PrintStatus "WARNING! Python 3.x detected. This may not work with P2Pool."
        # @TODO: Any special handling here for the error?
    elif [ -n "$PYTHON_27" ] ; then
        PrintStatus "Python 2.7 detected, all ok!"
    fi
fi

# Install the pre-reqs for P2Pool
PrintStatus "Installing P2Pool dependencies"
apt-get -y install python-zope.interface python-twisted python-twisted-web


#############
# bitcoind
#############

# For the UI, not strictly needed, but nice to have
PrintStatus "Installing Bitcoin dependencies"
apt-get -y install libqtgui4
# Side note - this installs a metric crapton of dependencies...

# Download and install bitcoind
PrintStatus "Installing bitcoind"
apt-get -y install bitcoind


#############
# User creation
#############

# @TODO: Check if user already exists and skip this step if needed

# Create your regular user
PrintStatus "Creating your regular user $REGULAR_USER_NAME"
useradd $REGULAR_USER_NAME

# Grant your user sudo permission
usermod -a -G sudo $REGULAR_USER_NAME

# Make the user's SSH directory and add the private key
if [ -s "$REGULAR_USER_PUBLIC_KEY_FILE" ] ; then
    PrintStatus "Setting up SSH keys for your user"

    # First ensure the .ssh directory exists.
    if [ ! -d $HOME_ROOT/$REGULAR_USER_NAME/.ssh ] ; then
        mkdir -p $HOME_ROOT/$REGULAR_USER_NAME/.ssh
        chown $REGULAR_USER_NAME $HOME_ROOT/$REGULAR_USER_NAME
        chown $REGULAR_USER_NAME $HOME_ROOT/$REGULAR_USER_NAME/.ssh
        chmod 700 $HOME_ROOT/$REGULAR_USER_NAME/.ssh
    fi

    # Add the key and assign ownership of the file.
    cat ~/$REGULAR_USER_PUBLIC_KEY_FILE >> $HOME_ROOT/$REGULAR_USER_NAME/.ssh/authorized_keys
    chown $REGULAR_USER_NAME $HOME_ROOT/$REGULAR_USER_NAME/.ssh/authorized_keys
    chmod 600 $HOME_ROOT/$REGULAR_USER_NAME/.ssh/authorized_keys
    rm ~/$REGULAR_USER_PUBLIC_KEY_FILE # don't need it any more.

else
    PrintStatus "SSH key NOT found for user $REGULAR_USER_NAME. Please enter a login password."
    passwd $REGULAR_USER_NAME
fi

# PrintStatus "Locking out root password."
# @TODO


#############
# Firewall setup
#############

# Next, let's set up a firewall
PrintStatus "Setting up the firewall"
ufw default deny    # Deny everything unless expressly permitted
ufw allow 22/tcp    # SSH
ufw allow 8333/tcp  # Bitcoind peer to peer
    # Note that we do NOT enable the Bitcoind RPC port!
ufw allow 9333/tcp  # P2pool peer to peer
ufw allow 9332/tcp  # P2Pool connections and Web interface
ufw --force enable  # Turn it on


#############
# Monit setup
#############

# set up monit, a monitoring tool.
PrintStatus "Configuring Monit"
# @TODO

# @TODO: Also check out Zabbix as per:
# https://bitcointalk.org/index.php?topic=153232.0
# http://www.zabbix.com/


#############
# Ideas for future growth:
#############

# Log rotation: http://manpages.ubuntu.com/manpages/jaunty/man8/logrotate.8.html
# Also https://bitcoin.stackexchange.com/questions/18857/how-to-limit-the-debug-log-file-while-running-bitcoind



###############################################################################
# This section has to be run as your regular user
###############################################################################

# @TODO: Maybe run this directly as the user?
# http://www.cyberciti.biz/open-source/command-line-hacks/linux-run-command-as-different-user/



PrintStatus()
{
    echo
    echo "----- STATUS: $*"
    echo
}



#############
# Bitcoind setup
#############

# Create the bitcoind configuration file
PrintStatus "Creating the bitcoind configuration file"
mkdir -p $HOME/.bitcoin
config="$HOME/.bitcoin/bitcoin.conf"
randUser=`< /dev/urandom tr -dc A-Za-z0-9 | head -c40`
randPass=`< /dev/urandom tr -dc A-Za-z0-9 | head -c40`

cat <<EOF > $config
server=1
daemon=1

rpcuser=$randUser
rpcpassword=$randPass
connections=40

blockmaxsize=1000000
mintxfee=0.00001
minrelaytxfee=0.00001
EOF


# Download the bitcoin bootstrap file. This saves a massive amount of time when starting bitcoind the first time - literally days.
# The right way to do this is via bittorrent - the seed file is at:

# Torrent info hash: 2d4e6c1f96c5d5fb260dff92aea4e600227f1aea
# Torrent magnet link:
# magnet:?xt=urn:btih:2d4e6c1f96c5d5fb260dff92aea4e600227f1aea&dn=bootstrap.dat&tr=udp://tracker.openbittorrent.com:80&tr=udp://tracker.publicbt.com:80&tr=udp://tracker.ccc.de:80&tr=udp://tracker.istole.it:80

# Details: https://bitcointalk.org/index.php?topic=145386.0  (Check here for more recent versions every few months.)

# However, most VPS providers don't allow torrents.  So alternately, we'll download using http. Not nearly as efficient, but still faster than syncing with the network.
PrintStatus "Downloading Bitcoin bootstrap.dat file.  This could take a while..."
cd $HOME/.bitcoin

#wget https://bitfetch.com/static/bootstrap.7z
# Note: as of this writing, the SSL certificate for bitfetch.com has expired.
# If you just try to wget the file, wget will fail, complaining about it.
# So, we have to force wget to ignore the expired certificate. Once the
# certificate on the site has been properly renewed, delete the line below
# and uncomment the line above.
wget --no-check-certificate https://bitfetch.com/static/bootstrap.7z


PrintStatus "Uncompressing Bitcoin bootstrap.dat file."
7z x bootstrap.7z

# Start bitcoind so that it can begin processing the bootstrap and syncing with the network
# actually don't do this; instead force a reboot at the end of the script.
# PrintStatus "Starting bitcoind"
# bitcoind -daemon


#############
# P2Pool setup
#############

# Download and install p2pool
cd
git clone https://github.com/forrestv/p2pool.git
# no actual install needed

# Start P2pool in the background.
# Note that if you enable this in this script, P2Pool will spit out a few hundred errors until bitcoind is caught up.
# So, disabling this here.
# nohup python $HOME/p2pool/run_p2pool.py &

#############
# cron setup
#############

# Add the entries for the cron jobs
PrintStatus "Setting up cron jobs"
crontab -l > $HOME/Old_cron.txt
echo "@reboot bitcoind -daemon" >> $HOME/Old_cron.txt
echo "@reboot python $HOME/p2pool/run_p2pool.py" >> $HOME/Old_cron.txt
crontab $HOME/Old_cron.txt
rm $HOME/Old_cron.txt


# @TODO: Auto update and restart P2Pool (may or may not be desirable)
# http://pastebin.com/HtD0uku4

EDIT 2014-06-20: I just noticed that the download of the blockchain file was disabled. That's because I was futzing around with the script the last time I used it, and I forgot to re-enable that line. It's corrected now.

[raskul]

breakthrough moment.

I used to get all confused and dizzy when I looked at code. scanning through that box, Kyros, it would seem I may very well have 'learned a new language'
I'll devote some time over the weekend to look into this in more depth.

[KyrosKrane]

This is also very help and really great, thanks for the contribution!

(Were you the guy who posted something similar about this on reddit?)

No, but I remember that post. It inspired me to create this script, and in fact, the section for setting up the bitcoin configuration file was adapted from that post.

[murdof]

Good stuff Kyros!

I don't have in the guide any information about firewalling - so I guess this could be in the extras.
I did add rpcallow=127.0.0.1 - but still people could find an exploit to that - so firewalling will be the best way to go.

What I understand is that people want to just get on a Linux box and just type 2 commands and set up everything.
I don't think that people prefer to build things (I don't personally but I have to).
So that's why I went with the binaries and a mini script to execute all the commands to have you up and running ASAP.
I have tried the script in 3 scenarios already and works great.
The only thing that I found is that when tar extracts some files are created with root as owner. Maybe I'm missing a tar option to use current user? Haven't checked.

All in all I invite you to try it out the 2 line script (you can just wget the script and check the contents) and let me know what you think.

[nreal]

All make commands could go like make -j12.... if you have lets say 8 cores, it makes it alot faster that way... time is coins


make -j12 -f makefile.unix

[murdof]

huntercoin is fixed and compiles ok

still though downloading the blockchain so haven't tried it (for some reason downloading is very slow)

so when it has finished and i confirm it works with merge mining I will add it to the guide.

[jonnybravo0311]

Hey murdof... just wanted to say good job on this - especially the "Lazy Man's" version.  Personally, I hacked my way through setting everything up (Disclaimer: I have everything running natively on OSX Mavericks).  The work you've done to compress the installation on Ubuntu 14.04LTS down to two commands is impressive.  Nicely done!

[KyrosKrane]

Something I just noticed:

Install Ubuntu 14.04 LTS Server from here: http://www.ubuntu.com/download/server
(note do not install Ubuntu 14.04 LTS - you need Server version for text only install)

I think in the second sentence, you meant to say, "do not install Ubuntu Desktop 14.04 LTS" -- but either way, this setup should work just fine even on a Desktop install of Ubuntu. You'd just need more disk space for the graphical stuff. In the past, I've run plenty of server-type software (including bitcoind and p2pool) on the Desktop versions of Ubuntu.

I'll spin up a VM with Ubuntu Desktop 14.04 installed and see how the script runs.

[murdof]

Thanks Kyros,

fixed.

I don't think there is a point to run a server with GUI.
Some people might want to run it on their main desktop on the other hand - so yes it would be good to know if you tried it and worked.
Normally it should work ok.

[Collider]

Murdof, how much of your RAM are you actually using on p2pool and Bitcoin client (aswell as the server OS)?

[murdof]

Murdof, how much of your RAM are you actually using on p2pool and Bitcoin client (aswell as the server OS)?

I'm running with 4GB and it utilizes about 2.5GB.
Huntercoin is something that takes up all CPU + Memory so I stopped it cause anyway I found it useless. Too slow and too many resources.

[wknight]

With memory being so cheap now I found 8 Gigs to be VERY comfortable running my p2pool node. no need to pop into swap file EVER!

[Collider]

While memory used to be more expensive, it is stil nice to know what the actual requirements are.

I would therefore think it is fair to say that you should get away with a 3 GB Ram VPS safely.

Most VPS providers allow you to exceed the allocated memory by atleast 10-20% anyway

[wknight]

While memory used to be more expensive, it is stil nice to know what the actual requirements are.

I would therefore think it is fair to say that you should get away with a 3 GB Ram VPS safely.

Most VPS providers allow you to exceed the allocated memory by atleast 10-20% anyway

Totally passed up the idea of a VPS concept.. good point.. though i like to keep my memory on a even number call me old fashioned. 4GB Ram would be ideal and yes I have run it on a vps at 4GB and no problems.. course it was for litecoin

The only reason I stay away from VPS is i want something local. Local is always the best results with p2pool shares

[raskul]

While memory used to be more expensive, it is stil nice to know what the actual requirements are.

I would therefore think it is fair to say that you should get away with a 3 GB Ram VPS safely.

Most VPS providers allow you to exceed the allocated memory by atleast 10-20% anyway

Totally passed up the idea of a VPS concept.. good point.. though i like to keep my memory on a even number call me old fashioned. 4GB Ram would be ideal and yes I have run it on a vps at 4GB and no problems.. course it was for litecoin

The only reason I stay away from VPS is i want something local. Local is always the best results with p2pool shares

definitely, 100% agree, unfortunately I don't have the ability to run a local node, so VPS is a good option for folks like me.
and with murdof's guide and how-to, those who can, can, and those who can't, can too!

[wknight]

definitely, 100% agree, unfortunately I don't have the ability to run a local node, so VPS is a good option for folks like me.
and with murdof's guide and how-to, those who can, can, and those who can't, can too!

Since i have no done it with bitcoin on a VPS.. how are the results? Are you seeing a lot more stales/DOA?

[raskul]

see for yourself; http://pool.centralcavern.uk:9332/

 

looks ok to me.. how is yours locally?

[wknight]

see for yourself; http://pool.centralcavern.uk:9332/

 

looks ok to me.. how is yours locally?

3.18TH/s (1.7% DOA)

so that is a bit of difference

[raskul]

see for yourself; http://pool.centralcavern.uk:9332/

 

looks ok to me.. how is yours locally?

3.18TH/s (1.7% DOA)

so that is a bit of difference

very good indeed.please always mine p2pool