A UNIX Training Server Guide

Introduction to Setup The training server

This guide explains how to build an online training server to be used together with the ISOC Introduction to Network Operations Course and the Network Services and Monitoring Course. The motivations for seting up such a server are explained here. Such a server can be setup in a learning institution or at a NOG/NREN.

For a simpler setup, there is a script that will take you at least up to the end of Part 1 of this guide. The code in that script is still beta so if you get stuck at any step, the error it produces should give you some clue as to where to proceed from in this guide.

Server Hardware Requirements for the Host Server

FreeBSD Host Server - the host operating system that will house the trainee virtualized jail servers. You can use FreeBSD 9.3 or higher. It is advised that you use FreeBSD 12.0 (or higher) since its well documented in the course. Please use the 64Bit version which is called "AMD64" version in order to take advantage of more RAM available on your system. Minimum specs are:

  • At least a dual core processor (more is better)
  • At least 4GB RAM (more is better)
  • At least 100GB Storage (more is better)

PLEASE NOTE: Each Virtual Server takes up about 3GB of storage so for 30 trainees, the setup will need at least 90GB of storage

it is advisable to have the training server as being totally dedicated for this role without it being used to provide other services. Though the firewall will adequately prevent the virtualized machines from running unwanted services, tools like "tcpdump" which form part of the labs can be used to expose un-encrypted packets originating from or being sent to the FreeBSD Host Server.
Each Jail gets an IPv4 address that is local ( range). It is also possible to provide each Virtual Machine with public IPv4 or IPv6 addresses by altering some parts of the setup below.
Here is what the FreeBSD Virtual Machines are permitted to do on the lab:

  • Permit DNS Queries originating from the Virtual Machines (UDP port 53)
  • Permit web traffic (Port 80 and Port 443) from the Internet destined to the Virtual Machines (to allow downloading of 3rd party software used in the labs)
  • Permit traffic between the Virtual machines

All other traffic is blocked unless specifically permitted using the FreeBSD Host Firewall open (See firewall section)

The FreeBSD Master Jail Server

The FreeBSD Master Jail Server, which runs within your host server, is a virtual server that will run within the host server that will host the following:

  • An FTP server - to serve FreeBSD ports locally as well as other files. This saves a lot of bandwidth since the 200MB of files needed by the trainee servers will only be downloaded once and then served from this machine
  • The FreeBSD ports directory to be used by all the trainees servers
  • Installation of a local DNS Cache
  • Testing of other softwares needed in the future

Download FreeBSD 12.0 AMD64

You can download the FreeBSD 12.0 64 bit image from HERE:

Setup the Training Server for the Online Lab

Part 1 - Create the FreeBSD Host Server

1. Install FreeBSD 12.0 AMD64

2. Update the Ports tree after installing

#portsnap fetch extract

3. Disable X11 stuff (graphical environment) and enable ezjail (we will install ezjail later):

#echo OPTIONS_UNSET=X11 >> /etc/make.conf
#echo ezjail_enable=YES >> /etc/rc.conf
#echo gateway_enable=YES >> /etc/rc.conf

3.1 Also enable these options in sysctl

#echo security.jail.sysvipc_allowed=1 >> /etc/sysctl.conf
#echo security.jail.allow_raw_sockets=1 >> /etc/sysctl.conf
#echo sysctl net.inet.ip.forwarding=1 >>   /etc/sysctl.conf

And run them so that they take effect immediately

#sysctl security.jail.sysvipc_allowed=1 
#sysctl security.jail.allow_raw_sockets=1
#sysctl net.inet.ip.forwarding=1

In some cases, you may want to increase the amount of SWAP memory that you have available using the harddisk space available. You can do so as follows to build a swap partition of 32GB

#dd if=/dev/zero of=/usr/swap0 bs=1m count=32000
#chmod 0600 /usr/swap0

Then add the following line to /etc/fstab

md99	none	swap	sw,file=/usr/swap0,late	0	0

Then activate the new swap partition:

#swapon -aL

4. Install Portupgrade, this will allow installation of ports to be done in a simpler fashion:

#cd /usr/ports/ports-mgmt/portupgrade
#make –DBATCH install clean

5. Install Sudo, Screen, EzJAIL and Bash:

#portinstall --batch sudo screen bash ezjail
#sysrc ezjail_enable=YES
#service ezjail start

Now we have screen installed, we will use it to ensure we can run the following commands without necessarily being connected to the SSH session.

screen -m

Incase you lose connectivity, you can rejoin the session with "screen -x"

Fetch the FreeBSD /usr/src code. It is needed by Ezjail:

#cd /
#fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE/src.txz

Then run:

#tar xvzf src.txz

6. Enable the FreeBSD cloned interface to be used by the virtual jails as their own loopback interfaces

#echo cloned_interfaces=lo1 >> /etc/rc.conf
#service netif cloneup
#ezjail-admin install -p

7. Make buildworld - this will take some time :-), please use screen. This can take between 20mins to an hour depending on CPU power of the host machine:

#rm -rf /usr/obj/*
#cd /usr/src
#make buildworld && ezjail-admin update -i -p

8. Edit the ezjail-admin command to enable creation of jails that will allow the jail to connect to the Internet

open this file with your favourite editor (ee is used for this example):

ee  /usr/local/bin/ezjail-admin

and look for this line

echo export jail_${ezjail_safename}_retention_policy=\"${ezjail_retention_policy}\"

and below it add:

echo export jail_${ezjail_safename}_devfs_ruleset=\"5\"
echo export jail_${ezjail_safename}_parameters=\"allow.raw_sockets=1 allow.sysvipc=1\"

save and exit the file

9. Create the master FreeBSD Jail that will contain all the pre-downloaded files and servers:

#ezjail-admin create masterserver "em0|x.x.x.x,lo1|"

NB: For IPv6 capability on the jails, you need to add the entire address like so:

#ezjail-admin create masterserver "em0|,em0|2001:43f8:220:219:0000:0000:0000:0100,lo1|"

Replace x.x.x.x with the IP address that will be assigned to the master Jail server. Replace em0 with the network interface name

10. Start the new jail

#ezjail-admin start masterserver

This marks the end of the first part!

Part 2 - Configure the Master Jail server

1. Type jls list
Use jexec to get to the new server

#jexec 1 csh

2. Please note that the number may change if you restart ezjail (check by typing "jls"). Also, you need to enable DNS on the Masterserver so that it can be able to resolve domains needed for the next part. You can use any DNS server that you wish, we will use Google's for now:

#echo "nameserver" > /etc/resolv.conf

and also since this is a headless server, disable installing of any dependencies with graphical support

#echo OPTIONS_UNSET=X11 >> /etc/make.conf

3. Build the master server

#pkg install perl5 python27 vim screen shellinabox portupgrade vsftpd-ext bind910
#portinstall --batch sudo bash zsh unbound textmaze

Shellinabox is useful if trainees are unable to SSH in, they can use a browser to access their virtual machine. Tested and it worked :-)

4. Enable a local DNS Server for use by all the trainee servers

cd /usr/local/etc/unbound/

then create an empty unbound.conf with the following in that file:

# ee unbound.conf

Then add the following (match for you jails' IP Block):

    interface: #listen on all interfaces
    access-control: allow #allow this network
    verbosity: 1

Add unbound to start at FreeBSD boot. The below is the same as adding unbound_enable=YES to /etc/rc.conf

#sysrc unbound_enable=YES

Save and exit. Then start unbound:

#service unbound start



Part 3 - Build a flavor to be used to create all the trainee servers

The following commands are to be run on the host server (ie not the master jail)

1. A flavour allows creation of several jails using a simple template. Here is what you need to do.

Go to /usr/jails/flavours

#cd /usr/jails/flavours
#cp -R example afnog

2. Now go to this folder: /usr/jails/flavours/afnog/etc/rc.d

Create a file called ezjail.flavour and ensure it is executable:

#touch ezjail.flavour && chmod ugo+x ezjail.flavour

Copy and paste the below text into that file exactly as it is:

. /etc/rc.subr
flavour_setup() {
  rm -f "/etc/rc.d/ezjail.flavour"

#This is where to add groups. Default group is "afnog"
pw groupadd -q -n afnog # -g 1004

#This is where to add users. Default username is "afnog" and password "admin"
echo -n '$1$p75bbfK.$Kz3dwkoVlgZrfLZdAXQt91' |\
pw useradd -n afnog -u 1005 -p 04-oct-2016 -s /bin/csh -m -d /home/afnog -G afnog,wheel -c 'Afnog User' -H 0

#This is where to specify what should be installed by default in the jails. At a bare minimum, you need python and bind910. The remaining lines should remain commented as they are part of the exercises in the course. However, if you want them to move faster in some sections of the course, you can uncomment the lines and they will be able to still run the commands to install the softwares on their servers but they will receive a message that the softwares have been already installed.

env ASSUME_ALWAYS_YES=YES pkg bootstrap
pkg install -y python27
#pkg install -y vim
#pkg install -y perl5
pkg install -y bind910
#pkg install -y bash
#pkg install -y sudo
run_rc_command "$1"

The above will tell ezjail to create a default group called "afnog" for all the jail servers with a default username of "afnog" and a password of "admin". To create your own password, you can replace what is in the file with your own password hash.

For example, if you want to generate a password hash for the password "xyz123abcd" you can type this on the UNIX shell:

#openssl passwd -1 "xyz123abcd"

You will get output from the shell. I got the following output (yours will be different!)

The output you get should then be added into the file under the user section like so:

echo -n '$1$02Ne1mPe$23qKEJM/18UPJiGL3kLgk.' |\
pw useradd -n afnog -u 1005 -p 04-oct-2016 -s /bin/csh -m -d /home/afnog -G afnog,wheel -c 'Afnog User' -H 0

3. Each jail needs to have python install by default to allow for use of ansible later. Here is how to do it:

#chown u+x  /usr/jails/flavours/afnog/etc/rc.d/ezjail.flavour
#mkdir /usr/jails/flavours/afnog/pkg
#cd /usr/jails/flavours/afnog/pkg
#cp /usr/jails/masterserver/var/cache/pkg/*python* .
#cp /usr/jails/masterserver/var/cache/pkg/*bind* .

3.1 Also, enable SSH to be available by default on all jails

echo sshd_enable=YES >> /usr/jails/flavours/afnog/etc/rc.conf

4. Enable the jail to connect to the network

Open this file /usr/local/bin/ezjail-admin
and look for this line

echo export jail_${ezjail_safename}_retention_policy=\"${ezjail_retention_policy}\"

and below it put:

echo export jail_${ezjail_safename}_parameters=\"allow.raw_sockets=1 allow.sysvipc=1\"

save and exit

Also create a file called /etc/devfs.rules which contains the following:

add include $devfsrules_jail
add path 'bpf*' unhide

5. Create the default files to be used in the jails. Go to the below folder:

#cd /usr/jails/flavours/afnog/etc

  • Add the resolv.conf that will be installed in all jails: Match the IP address to that of your masterserver.

#echo "nameserver" >> resolv.conf

  • Add a make.conf file that will be used for all the jails. Match the IP address to that of your masterserver.

#ee make.conf

Then paste this:

WRKDIRPREFIX=           /var/ports
DISTDIR=                /var/ports/distfiles
PACKAGES=               /var/ports/packages
INDEXDIR=               /var/ports

Save and exit

6. Create a test jail

#ezjail-admin create -f afnog mytestjail 'em0|,lo1|'

Then start the jail as follows:

#ezjail-admin start mytestjail

Login and check whether python and bind are installed by typing:

#which python27
#which named

Your trainees will need to access the FreeBSD Jails in order to carry out the hands on parts of the course. It is possible to assign each jail a public IP address and allow them to SSH in if you have enough IPv4 or IPv6 Public addresses. However, past runs of the course have shown that some trainees experience problems if they are in a network that does not allow SSH traffic out or has a firewall or Proxy server that restricts access.

If direct access via SSH is not possible, then you can provide web based access using "ShellInABox" (or equivalent). This guide will show you how to use "ShellInABox" to allow your trainees to access the remot]e lab.

It is assumed that you have already followed the guide on https://isoc-inforum.afrinic.net/wiki/tiki-index.php?page=Enabling+Web+Access+to+Jails" class="wiki wikinew text-danger tips"> how to setup the Jails.

How to setup Web based Shell access to the Virtual Servers

Install Apache on the Host Server Only

1. We will use Apache WebServer to act as a Proxy to the ShellInABox instances that will run on the individual jails:

pkg install apache24

Create a Self Signed Certificate for HTTPS

Then create a self signed certificate. If you have your own domain name, its better to use LetsEncrypt. For this part, we will use a Self Signed Certificate:

#cd /usr/local/etc/apache24
#mkdir ssl
#cd ssl
#openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout server.key -out server.crt

Alternatively, you can obtain a certificate from LetsEnrypt or similar
Answer the questions as guided. Once complete, 2 files will be created: "server.crt" and "server.key".

Edit Apache's SSL Configuration File

We will next edit Apache's SSL configuration file to provide the path to the two files.

#cd /usr/local/etc/apache24/extra
#mv httpd-ssl.conf httpd-ssl.conf.old
#ee httpd-ssl.conf

Now below is a sample you can edit to match your server.

Listen *:443
<VirtualHost *:443>
DocumentRoot "/usr/local/www/apache24/data/"
ServerName lab2.isoc.afrinic.net
SSLProxyEngine On
RequestHeader set Front-End-Https "On"
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

proxypass /mytestjail/              #Match with respective IP and Port
proxypassreverse /mytestjail/  #Match with respective IP and Port

SSLEngine on
SSLProtocol all -SSLv2
SSLCertificateFile "/usr/local/etc/apache24/ssl/server.crt"
SSLCertificateKeyFile "/usr/local/etc/apache24/ssl/server.key"
<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
<Directory "/usr/local/www/apache22/cgi-bin">
    SSLOptions +StdEnvVars
BrowserMatch "MSIE [2-5]" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog "/var/log/httpd-ssl_request.log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

Edit Apache's main configuration file

3. Now, you need to edit the main Apache configuration file located at "/usr/local/etc/apache24/httpd.conf" Open the file and find and uncomment the following the lines:

LoadModule proxy_html_module libexec/apache24/mod_proxy_html.so
LoadModule proxy_module libexec/apache24/mod_proxy.so
LoadModule proxy_connect_module libexec/apache24/mod_proxy_connect.so
LoadModule proxy_ftp_module libexec/apache24/mod_proxy_ftp.so
LoadModule proxy_http_module libexec/apache24/mod_proxy_http.so
LoadModule proxy_fcgi_module libexec/apache24/mod_proxy_fcgi.so
LoadModule proxy_scgi_module libexec/apache24/mod_proxy_scgi.so
LoadModule proxy_fdpass_module libexec/apache24/mod_proxy_fdpass.so

Include etc/apache24/extra/httpd-ssl.conf

4. Save and exit the file. Then start apache:

#service apache24 start

Steps to run on the Jails (Virtual Machines)

1. You need to install ShellInaBox if you have not done so on the jail:

#jexec mytestjail csh
#pkg install shellinabox

2. Then enable it in /etc/rc.conf of the Jail as follows.

#sysrc shellinaboxd_enable=YES
#sysrc shellinaboxd_port="5222"

3. Start ShellInaBox inside the jail

#service shellinaboxd start

Browse your server IP (or host name) via HTTPS and ensure you have a trailing slash at the end of the path you use. In this example, we would access the server as follows:


Automation of activities

Bulk Provisioning of Virtual Servers

To generate several jails from the command line, you can do the following:

  • First create a file called "users" with the indvidual server names that you want. This can be the email addresses of each of the trainees.
  • Then run the below script, adjust to fit your needs. The counter number is the final Octet in the IP address that will be assigned to the first user. Adjust the IP addresses to match what you prefer. An example for those with IPv6 networks is included (hashed out) below

for i in `cat /path/to/file/users`

ezjail-admin create -f afnog $i "em0|192.168.0.$counter,lo1|127.0.0.$counter"
#ezjail-admin create -f afnog $i  "em0|196.168.0.$counter,em0|2001:43f8:220:219:0000:0000:0000:$counter,lo1|127.0.1.$counter"
let counter=counter+1

2. To Start or Stop all the jails

#ezjail-admin stop  
#ezjail-admin start

To delete a stuck jid

#jail -r “number”

To find which Jail is chewing resources

ps -o pid,jid -awux

To quickly get a new set of jails upto speed with everything in place:

counter=677 #The first IP addressed assigned
for i in `cat users2`
jexec $counter echo nameserver > /etc/resolv.conf
pw -R /usr/jails/$i/ groupadd afnog
echo -n '$1$p75bbfK.$Kz3dwkoVlgZrfLZdAXQt91' | pw -R  /usr/jails/$i/ useradd -n afnog -u 1200 -p 04-oct-2016 -s /bin/csh -m  -d /home/afnog -G afnog,wheel -
pw -R /usr/jails/$i/ groupadd isoc
echo -n '$1$azWgq2ZL$7plNDfiyyc19GrqzVmGG21' | pw -R /usr/jails/$i/ useradd -n isoc -u 2005 -s /bin/csh -m -d /home/isoc -G isoc,wheel -c 'ISOC Moderator' -
chroot /usr/jails/$i/ env ASSUME_ALWAYS_YES=YES pkg bootstrap
chroot /usr/jails/$i/ pkg  install -y shellinabox bash bind911
chroot /usr/jails/$i/ service shellinaboxd start
jexec -U afnog $counter bash  -c "/usr/bin/alias fetch='fetch --no-verify-peer'"
jexec -U afnog $counter csh  -c "/usr/bin/alias fetch='fetch --no-verify-peer'"
jexec $counter csh -c "ln -s /usr/local/etc/namedb /etc/namedb"
printf "proxypass /$i/ https://192.168.1.$counter:5222/\nproxypassreverse /$i/ https://192.168.1.$counter:5222/\n" >> apacheconf
cp apacheconf /usr/local/etc/apache24/Includes/sdnog-`(date +"%F")`.conf
apachectl restart
let counter=counter +1