HOWTO : Highest secured Hiawatha Web Server 10.5 on Ubuntu Server 16.04 LTS

(A) Introduction

Hiawatha Web Server is designed with security in mind. It already built-in some security features to protect common attacks, such as SQLi, XSS, CSRF. Meanwhile, it can be configured to prevent scanning from vulnerability scanners too.

Hiawatha is a light weight and fast as well as secure web server in the market. Hiawatha is working well with PHP and MySQL. The following guide is showing how to configure Hiawatha in a very high secure way on Ubuntu Server LTS.

(B) Software Prerequisite

The current version as at the time of this writing :
(1) Ubuntu Server 16.04.2 LTS
(2) CMake 3.7.2
(3) Hiawatha 10.5

(C) Installation of PHP7.0 and MySQL

sudo apt-get install php7.0-cgi php7.0 php7.0-cli php7.0-mysql php7.0-curl php7.0-gd php7.0-intl php7.0-imap php7.0-mcrypt php7.0-pspell php7.0-recode php7.0-sqlite3 php7.0-tidy php7.0-xmlrpc php7.0-xsl apache2-utils php7.0-fpm php-memcache php-imagick php-cache mysql-server mysql-client

(D) Installation of Hiawatha

sudo apt-get install libc6-dev libssl-dev dpkg-dev debhelper fakeroot libxml2-dev libxslt1-dev

(a) Install CMake

wget https://cmake.org/files/v3.7/cmake-3.7.2.tar.gz
tar -xvzf cmake-3.7.2.tar.gz
cd cmake-3.7.2
./configure
make
sudo make install

(b) Install Hiawatha

wget http://www.hiawatha-webserver.org/files/hiawatha-10.5.tar.gz
tar -xzvf hiawatha-10.5.tar.gz

cd hiawatha-10.5/extra

./make_debian_package

cd ..

sudo dpkg -i hiawatha_10.5_amd64.deb

(E) Configuration of PHP7.0

sudo nano /etc/php/7.0/fpm/php.ini

Make changes as is.

allow_url_fopen = Off
session.cookie_httponly = 1
disable_functions = [EXIST_FUNCTION],system, show_source, symlink, exec, dl, shell_exec, passthru, phpinfo, escapeshellarg, escapeshellcmd,

* [EXIST_FUNCTION] is the functions that already at "disable_functions" of php.ini

(F) Let's Encrypt on Hiawatha

(a) Configuration of Hiawatha

sudo mkdir -p /etc/hiawatha/enable-sites
sudo mkdir -p /etc/hiawatha/disable-sites

Edit "cgi-wrapper.conf".

sudo nano /etc/hiawatha/cgi-wrapper.conf

Change the following as is.

CGIhandler = /usr/bin/perl
CGIhandler = /usr/sbin/php7.0-fpm
CGIhandler = /usr/bin/python
CGIhandler = /usr/bin/ruby
CGIhandler = /usr/bin/ssi-cgi

Wrap = jail_mysite ; /var/www/mysite ; www-data:www-data

Change the ownership of the log files.

cd /var/log/hiawatha
sudo chown www-data:www-data access.log
sudo chown www-data:www-data error.log
sudo chown www-data:www-data exploit.log
sudo chown www-data:www-data garbage.log
sudo chown root:root system.log

Change the ownership of the web application files.

cd /var/www/mysite
sudo chown -R root:root *

The following are examples of "hiawatha.conf" and "mysite.com".

/etc/hiawatha/hiawatha.conf example :


/etc/hiawatha/enable-sites/mysite.com example :


(b) Self Signed SSL Certificate Generation

To generate SSL certificate for the web root.

openssl genrsa -out default.pem 4096
openssl req -new -x509 -days 3650 -key default.pem -out server.crt
echo "" >> default.pem
cat server.crt >> default.pem
echo "" >> default.pem
rm -f server.crt
sudo mkdir -p /etc/hiawatha/tls
sudo cp default.pem /etc/hiawatha/tls
sudo chown www-data:www-data /etc/hiawatha/tls/default.pem
sudo chmod 400 /etc/hiawatha/tls/default.pem
sudo chmod 400 -R /etc/hiawatha/tls

(c) Let's Encrypt Generation and Configuration

To generate SSL certificate for the www.mysite.com.

wget https://www.hiawatha-webserver.org/files/letsencrypt.tar.gz
tar -xvzf letsencrypt.tar.gz
cd letsencrypt

nano letsencrypt.conf

Change the email "info@example.org" to your email as Let's Encrypt bot will inform you about the expire date of the certificate :
ACCOUNT_EMAIL_ADDRESS = samiux@gmail.com

Change the RSA Key size from "2048" to "4096" :
CERTIFICATE_RSA_KEY_SIZE = 4096

Uncomment "Production" and comment out "Testing" :
LE_CA_HOSTNAME = acme-v01.api.letsencrypt.org # Production
#LE_CA_HOSTNAME = acme-staging.api.letsencrypt.org # Testing

Make sure Port 80 is working and run the following command.

For the first time, you need to register to Let's Encrypt. Make sure keep the generated "account.key" in a safe place. "account.key" should be in the original place when doing SSL certificate renewal.

./letsencrypt register

To generate the SSL certificate.

sudo ./letsencrypt www.mysite.com

To revoke the SSL certificate (Optional).

sudo ./letsencrypt /etc/hiawatha/tls/www.mysite.com.pem

To renew SSL certificate (Optional).

sudo ./letsencrypt renew

To get the Let's Encrypt X3 certificate at https://letsencrypt.org/certificates/ and select :

Let’s Encrypt Authority X3 (IdenTrust cross-signed)
https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt

wget https://letsencrypt.org/certs/letsencryptauthorityx3.pem.txt
echo "" >> www.mysite.com.pem
cat lets-encrypt-x3-cross-signed.pem.txt >> www.mysite.com.pem

echo "" >> default.pem
cat lets-encrypt-x3-cross-signed.pem.txt >> default.pem

To generate sha256 base64 hash of the certificates. The first one is the "mysite.com" SSL certification and the second one is the SSL certification for web root directory.

openssl x509 -in /etc/hiawatha/tls/www.mysite.com.pem -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | base64

openssl x509 -in /etc/hiawatha/tls/default.pem -pubkey -noout | openssl rsa -pubin -outform der | openssl dgst -sha256 -binary | base64

Then change "CustomHeaderClient = Public-Key-Pins:" values at "/etc/hiawatha/enable-sites/mysite.com". The first "pin-sha256" is for "mysite.com" and the second "pin-sha256" is for web root. The others "pin-sha256" are get from Qualys SSL Labs Test site (see below).

With the help of Qualys SSL Labs, you can further configure the HPKP and test the grading of your site. The highest grade is A+.

The grade of the site are A+ on both Qualys SSL Labs and High-Tech Bridge SSL certificate testings. Meanwhile, it is also compliance with PCI DSS 3.1 Requirements that reported by High-Tech Bridge.

The Let's Encrypt SSL Certificate will be expired about 28 days and it requires to be renewed. Make sure you update "CustomHeaderClient = Public-key-Pins:" at /etc/hiawatha/enable-sites/mysite.com (as example). The first "pin-sha256" is required to be updated. After that, restart Hiawatha.

(G) Hardening of Ubuntu Server

(a) sysctl

sudo nano /etc/sysctl.d/60-hiawatha.conf



sudo sysctl /etc/sysctl.d/60-hiawatha.conf -p

(b) Apparmor

sudo apt-get install apparmor-profiles apparmor-utils
sudo nano /etc/apparmor.d/usr.sbin.hiawatha



sudo aa-enforce hiawatha

If you have change some settings, you should reload the profile.

sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha

If you want to disable this profile.

sudo ln -s /etc/apparmor.d/usr.sbin.hiawatha /etc/apparmor.d/disable/
sudo apparmor_parser -R < /etc/apparmor.d/usr.sbin.hiawatha

If you want to re-enable this profile after it has been disabled.

sudo rm /etc/apparmor.d/disable/usr.sbin.hiawatha
sudo apparmor_parser -r < /etc/apparmor.d/usr.sbin.hiawatha

Remarks :

If you encounter "500 Internal Server Error", you may consider to make the Apparmor to "Complain mode".

sudo aa-complain hiawatha

After several days browsing the website, you may consider to turn the Apparmor to "Enforce mode".

sudo aa-logprof

sudo aa-enforce hiawatha

It is because the captioned usr.sbin.hiawatha may not 100% work for you.

(c) Linux Malware Detect (Optional)

Linux Malware Detect Installation

* the captioned link may be out-dated and it is for your reference only

(d) MySQL

Create Normal User on MySQL

(e) fail2ban

sudo apt-get install fail2ban

Change the setting at /etc/fail2ban/jail.conf when necessary.

(H) Storage Performance Tuning

It is recommended to use SSD for the storage instead of hard drive for the excellent performance.

(a) SSD

Verify TRIM is supported :

sudo hdparm -I /dev/sda | grep TRIM

If the output is similar to the below which is supported :

* Data Set Management TRIM supported (limit 1 block)

If you install your Ubuntu in LVM, the TRIM is usually enabled by default. You can confirm it :

cat /etc/lvm/lvm.conf | grep issue_discards

If the output is similar to the below which is enabled :

issue_discards = 1

Then set the following to "deadline" if it is not done yet.

cat /sys/block/sda/queue/scheduler

noop [deadline] cfq

If not, set it :

sudo nano /etc/rc.local
Insert the following before "exit 0" :

echo 2048 > /sys/block/sda/queue/read_ahead_kb
echo 2048 > /sys/block/sda/queue/nr_requests
echo deadline > /sys/block/sda/queue/scheduler

* make sure your device is sda (or sdb ...)

To reload it or reboot your system :

sudo bash /etc/rc.local

After that, you need to edit the partition table (/etc/fstab) :

To make it looks like the following :

/dev/mapper/ubuntu--vg-root / ext4 noatime,nodiratime,norelatime,errors=remount-ro 0 1

sudo mount -a
sudo mount -o remount /

If you have an error after running the captioned commands, DO NOT reboot your system. You should correct the typo before doing so; otherwise, you cannot boot to your system again.

(b) Hard Drive

sudo nano /etc/rc.local

Insert the following before "exit 0" :

echo 2048 > /sys/block/sda/queue/read_ahead_kb
echo 2048 > /sys/block/sda/queue/nr_requests

* make sure your device is sda (or sdb ...)

To reload it or reboot your system :

sudo bash /etc/rc.local

After that, you need to edit the partition table (/etc/fstab) :

To make it looks like the following :

ext4 noatime,nodiratime,norelatime,errors=remount-ro 0 1

sudo mount -a
sudo mount -o remount /

If you have an error after running the captioned commands, DO NOT reboot your system. You should correct the typo before doing so; otherwise, you cannot boot to your system again.

(I) Optional

To further hardening Ubuntu Server, you may consider to set up firewall (UFW/iptables) and place the Ubuntu Server behind Unified Threats Management System (UTM) or Intrusion Prevention System (IPS).

Reference
Qualys SSL Labs
High-Tech Bridge
securityheaders.io
URL Rewrite for Hiawatha

That's all! See you.