Tuesday, September 24, 2013

HOWTO : High Performance IDS/IPS with SmoothSec 3.4

The following Intrusion Detection/Prevention System (IDS/IPS) setup is using AF_PACKET with SmoothSec 3.4. The following setup is for low traffic flow home and SOHO users (or you can say that it is a Proof-of-Concept). If your traffic is heavy, please consider to use a high-end hardware.

(A) Hardware

IDS/IPS -

Motherboard - Intel Desktop Board D510MO
CPU - Intel Atom D510 (Dual-core with HT)
RAM - 4GB (2 x 2GB)
Hard Drive - 320GB
Network Card 0 (eth0) - Onboard Gigabit
Network Card 1 (eth1) - TP-Link TG-3269 Gigabit PCI Network Adapter (with low profile)
Network Card 2 (eth2) - D-Link DUB-E100 USB 2.0 Fast Ethernet Adapter (up to 200MB)

* You can select (1) Level One USB-0401 USB Gigabit Ethernet Adapter or (2) PCi USB 3.0 Gigabit LAN Adapter UE-1000T-G3 for eth2. However, you need to compile and install the driver yourself.

Update : If you are using Backports latest kernel, the Level One USB-0401 and PCi UE-1000T-G3 are workable out of the box.

Router -

Motherboard - Intel Desktop Board D510MO
CPU - Intel Atom D510 (Dual-core with HT)
RAM - 4GB (2 x 2GB)
Hard Drive - 320GB
Network Card 0 (eth0) - Onboard Gigabit
Network Card 1 (eth1) - TP-Link TG-3269 Gigabit PCI Network Adapter (with low profile)

(B) Software

IDS/IPS -

Operating System - Debian 7.0 (Wheezy)
IDS/IPS Pre-configure system - SmoothSec 3.4 (64-bit)
IDS/IPS Engine - Snort (or Suricata)
Unified2 Spooler - Pigsty
Web Interface - Snorby
Rules Management - PulledPork

Router -

Operating System - Untangle 9.4.2 (64-bit)

* Basically, Untangle is a router and Unified Threat Management System (UTM).

(C) Hardware Setup

Internet ---- Router ---- SmoothSec ---- Switch ---- Personal Computers

Router -- eth0 connect to Internet; eth1 connect to SmoothSec

SmoothSec -- eth0 connect to Router; eth1 connect to Switch (uplink or port 1); eth2 connect to Switch (any port at 2 to 4)

* You can use any router to replace Untangle.

(D) Installation of SmoothSec

Download SmoothSec 3.4 at here or here.

Make sure the SmoothSec box can surf the internet; otherwise, installation will be failed. Or, you may re-arrange the cables when necessary.

Updated : Since the scripts for 3.4 has been updated to 3.4.1, you should follow the below link to upgrade the script to 3.4.1.

Upgrade to scripts version 3.4.1

The scripts 3.4.1 will install Backports newest kernel instead of Unstable kernel for Suricata IPS mode with AF_PACKET.

After installed SmoothSec, type the following :

smoothsec.first.setup

Snort -

Select "ips-standard" and following the instruction to install. Please also refer to here for the configuration of the config file. Make sure "AF_ENGINE" is set to "snort". The rules are "et" by default.

After install, reboot your box.

Suricata -

Select "ips-standard" and following the instruction to install. Please also refer to here for the configuration of the config file. Make sure "AF_ENGINE" is set to "suricata". The rules are "et" by default.

A new Linux kernel 3.10.2 will be installed at the end.

After install, reboot your box.

(E) Configuration of IDS/IPS

You may need to disable or/and drop some rules (sid).

Snort -

You may need to configure the /etc/snort/snort.conf :

nano /etc/snort/snort.conf

For example, the subnet is 192.168.1.0/24.

Replace "ipvar HOME_NET any" with "ipvar HOME_NET [192.168.1.0/24]"

Replace "ipvar EXTERNAL_NET any" with "ipvar EXTERNAL_NET ![192.168.1.0/24]"

Restart Snort :

/etc/init.d/snort restart

Disable rules :

nano /etc/pulledpork/snort/disablesid.conf

Drop rules :

nano /etc/pulledpork/snort/dropsid.conf

After doing that, reload the rules :

smoothsec.snort.rules.update

Suricata -

Disable rules :

nano /etc/pulledpork/suricata/disablesid.conf

Drop rules :

nano /etc/pulledpork/suricata/dropsid.conf

After doing that, reload the rules :

smoothsec.suricata.rules.update

* If the rule is too long to disable or drop, you may consider to edit "modifysid.conf". For example, to disable and drop the following rule :

To disable the rule :

2013437 "alert" "##alert";

To drop the rule :

2013437 "alert" "drop";

(F) Configuration of Snorby

For example, the sensor IP address is 192.168.1.180.

Point your browser to https://192.168.1.180. Enter the username and password of Snorby.

Setup the Snorby according to SmoothSec WiKi.

* If you want to sent your Snorby reports by Postfix, you need to install yourself and configure it then.

sudo apt-get install postfix

Select "Internet Site" when asked.

(G) IDS/IPS Tuning

There may be some false positive records. You need to tune the setting by adding the sid to the disablesid.conf or dropsid.conf when necessary. When use with dropsid.conf, yon may need to set firewall at Router to make the job done.

(H) Remarks

In the captioned setting, the SmoothSec acts as IDS and IPS in one box.

If you want to install IDS only, your SmoothSec only requires one Network Card which is connected to Switch.

In addition, SmoothSec 3.4 comes with Distributed IDS/IPS which allows you to deploy multi-sensors with one control panel (Snorby).

Furthermore, you can also reset your box to the fresh install environment :

smoothsec.reset

When using Snorby, you may need to browse for the rules sid. This link is for the rules lookup.

Make sure you create an empty file namely restart.txt under /var/www/snorby/tmp :

touch /var/www/snorby/tmp/restart.txt

That's all! See you.

Monday, September 09, 2013

HOWTO : 15 Minutes to Deploy an IDS with SmoothSec 3.4

You can deploy your IDS (Intrusion Detection System) to your LAN within 15 minutes with SmoothSec 3.4. You can install SmoothSec 3.4 in virtual machine, such as VMWare, VirtualBox or Parallels.

The virtual machine is required at least one CPU, 1GB RAM and 8G+ HDD to run the IDS. The NIC is a bridged adaptor.



That's all! See you.

Sunday, September 08, 2013

HOWTO : 30 minutes to deploy a distributed IDS with SmoothSec 3.4



Distributed IDS is one of the features of SmoothSec 3.4. It allows you to monitor more than one sensor with one web interface (Snorby). Furthermore, the distributed IDS can be deployed by virtual machine, such as VMWare, VirtualBox and Parallels.

The basic virtual machines (console and sensor) requirement is 1 CPU, 1GB RAM and 8GB+ virtual storage.

You can install up to 2 IDS engines, Snort and Suricata in one box or either one of them.

Hints for installation

When installing sensor, you will be asked for ssh passphrase, you just press "Enter" and leave it empty. The path of the ssh key pairs should be default (nothing to change).

That's all! See you.

Sunday, September 01, 2013

HOWTO : Intrusion Detection System (IDS) made easy with SmoothSec 3.2

Building an IDS (Intrusion Detection System) at home/SOHO is not a dream today. SmoothSec 3.2 is designed for deploying IDS without tear. You can use a very low-end hardware for this purpose. To deploy IDS, you are required at least 2 NICs (Network Interface) if you have a management switch with SPAN port. If you do not have this piece of expensive device, you can build one with 3 NICs.

Hardware

Motherboard - Intel Desktop Board D510MO
RAM - 4GB DDR2 (2 x 2GB)
Hard Drive - 320GB
Network Card 0 - Onboard Gigabit
Network Card 1 - TP-Link TG-3269 Gigabit PCI Network Adapter (with low profile)
Network Card 2 - D-Link DUB-E100 USB 2.0 Fast Ethernet Adapter (up to 200MB)

Software

Operating System - Debian 7 (Wheezy) IDS/IPS pre-configured system - SmoothSec 3.2 (64-bit).
IDS/IPS Engine - Suricata

Setup

Internet -- Router -- SmoothSec -- Switch -- Personal Computers

Network Card 0 and 1 will be bridged up while Network Card 2 will be a management interface.

Step 1 :

First of all, SmoothSec (Network Card 2) is connected to the Switch while Network Card 0 and 1 do not connect to the router. It is because you need to connect to the internet for the installation.

Step 2 :

Install SmoothSec as usual or follow the wiki. During the installation, you may be prompted some firmware are missing, you just ignore it as some firmware for Realtek 8169 are missing. It is harmless to do so. After installed, the box will be reboot.

Login as "root" with password "toor".

Step 2a :

To improve the SmoothSec :

apt-get --purge remove arpwatch
apt-get install arpalert openjdk-7-jre fail2ban
cd /etc/arpalert/
mv oui.txt oui.txt.old
wget http://standards.ieee.org/regauth/oui/oui.txt


Step 3 (Configuration of Suricata) :

Suricata

nano /etc/suricata/suricata.yaml

Locate "- fast:" and change "enabled: no" to "enabled: yes".

Locate "- drop:" and change "enabled: no" to "enabled: yes".

Time Zone for Snorby

If your time zone is not UTC, you should run the following command :

dpkg-reconfigure tzdata

Set the time zone for "UTC" at "None of the above"; otherwise, the Snorby will reported wrong timestamp.

Set your time zone at the Snorby web interface when it is available after Step 5.

Email feature of Snorby

apt-get install postfix

Configure the Postfix properly according to your network at "/etc/postfix/main.cf".

nano /var/www/snorby/config/initializers/mail_config.rb

Uncomment the lines just below "# Sendmail Example:".

Step 4 :

Connect Network Card 0 to router and Network Card 1 to Switch.

Normally, Network Card 0 will be eth0, Network Card 1 will be eth1 and Network Card 2 will be eth2.

nano /etc/network/interfaces

Comment all about eth2 entries.

Append the following :

auto eth2
iface eth2 inet static
   address 192.168.0.120
   netmask 255.255.255.0
   gateway 192.168.0.1


* where the "address" is your SmoothSec IP address and "gateway" is the IP address of your router.

nano /etc/init.d/bridge

Change "net1=eth1" to "net1=eth0"
Change "net2=eth2" to "net2=eth1"
Change "brctl addif $br eth1" to "brctl addif $br eth0"
Change "brctl addif $br eth2" to "brctl addif $br eth1"

update-rc.d bridge defaults

Step 5:

Run the script "smoothsec.first.setup" in the terminal.

Type "br0" when asking for monitor network interface.

Select "Snort" or "Suricata" as IDS Engine. I choose "Suricata".

Email address and password are for Snorby web interface login.

Then reboot.

Once boot up, go to one of the Personal Computers and browse "https://192.168.0.120". Then set the time zone and your report email address accordingly.

Step 6:

To update SmoothSec, you need to do the following commands (you can make a script to do so). The rules will be updated automatically in the early morning every day.

apt-get update
apt-get dist-upgrade
apt-get --purge autoclean
apt-get --purge autoremove
# update SmoothSec
cd /root/updates/
git pull origin master


Debug the mailing feature

Do not run the following commands unless you really need to.

cd /var/www/snorby

bundle exec rails c production
Snorby::Jobs::SensorCacheJob.new(true).perform

Known Issue

Pigsty will crash randomly. As a result, no capture in Snorby. The problem has been reported, see here. --> The workaround is to run a testing bash script on every 5 minutes to start Pigsty again.

nano /root/chkpigstylog

#!/bin/bash
# Check if "Error: " in pigsty.log or not. If yes, start Pigsty again.
STRING="Error: "
if grep -R "$STRING" /var/log/pigsty.log
then
   /root/runpigsty
fi


nano /root/runpigsty

#!/bin/bash
/usr/local/bin/pigsty -c /etc/pigsty/suricata.pigsty.config.js -i eth0 -n "Suricata" -d /var/log/suricata/ -m unified2.alert.* -D


crontab -e
*/5 * * * * /root/chkpigstylog


Update

Developers just fixed the problem. You just upgrade the Pigsty with the following command :

npm update pigsty-mysql -g
npm update pigsty -g


Reference

Snorby GitHub
Suricata
SmoothSec
Pigsty
SmoothSec WiKi - for installation

That's all! See you.

HOWTO : High Performance IDS/IPS with SmoothSec 3.2

Since the previous setup (HOWTO) of SmoothSec are not perfect, I am going to use AF_PACKET as packet acquisition engine. In this setting, you are required to have at least 3 network interfaces, one for the management purpose.

As AF_PACKET has high performance, even the very low-end hardware is benefited. The following setup is ideal for home/SOHO environment.

(A) Hardware

Motherboard - Intel Desktop Board D510MO
CPU - Intel Atom D510 (2-core with HT)
RAM - 4GB (2 x 2GB)
Hard Drive - 320GB
Network Card 0 (eth0) - Onboard Gigabit
Network Card 1 (eth1) - TP-Link TG-3269 Gigabit PCI Network Adapter (with low profile)
Network Card 2 (eth2) - D-Link DUB-E100 USB 2.0 Fast Ethernet Adapter (up to 200MB)

(B) Software

Operating System - Debian 7.0 (Wheezy)
IDS/IPS pre-configure system - SmoothSec 3.2 (64-bit)
IDS/IPS Engine - Suricata
Spooler - Pigsty
Web Interface - Snorby
Rules Management - PulledPork

(C) Setup

Internet -- Router -- SmoothSec -- Switch -- Personal Computers

The SmoothSec will monitor all the incoming and outgoing traffic between router and the switch.

Step a - Cable connection :

First of all, SmoothSec (Network Card 2) is connected to the Switch while Network Card 0 and 1 do not connect to the router at the moment. It is because you need to access the internet for the SmoothSec installation.

Step -1 - Installation of SmoothSec :

Install SmoothSec as usual or refer to SmoothSec Wiki. When you are prompted to install non-free network interface firmware, you just ignore it. It is because Debian missed some firmware for Realtek 8169. After the installation, reboot the box as advice. The username is "root" while the password is "toor".

Step 0 - Install the missing packages :

apt-get install ethtool postfix fail2ban openjdk-7-jre

If you want to use Postfix as mail server for the Snorby report, you should install it and configure it after the install. For the configuration of Postfix, you may ask Google if you do not know how to.

Make sure you select "Internet Site" when installing Postfix.

You may consider to install fail2ban to protect your ssh connection inside the network.

To improve the SmoothSec :

apt-get --purge remove arpwatch
apt-get install arpalert
cd /etc/arpalert/
mv oui.txt oui.txt.old
wget http://standards.ieee.org/regauth/oui/oui.txt


Step 1 - Get new Linux Kernel :

In order to install a high performance IDS/IPS, you need a newer kernel that the version should be 3.7 or greater.

apt-cache search linux-image

To look for Linux Kernel version that is greater than 3.7. If not, just add the following repos :

nano /etc/apt/sources.list

Append the following line (the address of the source may be different from yours, but it must be "unstable" :
deb http://ftp.us.debian.org/debian/ unstable main
deb-src http://ftp.us.debian.org/debian/ unstable main


Then look for Linux Kernel version that is greater than 3.7 :

apt-get update

I select the version 3.10 :

apt-get install linux-image-3.10-2-amd64 linux-headers-3.10-2-amd64

When you are asked to restart some services when install, just reply "yes".

You will be warned for some firmware missing, just ignore it. It is because Debian do not has some firmware for Realtek 8169. Anyway, it is harmless.

After the new kernel install, comment out what you added in "/etc/apt/sources.list". This step is VERY IMPORTANT as the newer version of Apache (2.4.x) and Perl will break the Snorby and PulledPork, the web interface of the SmoothSec and rules management tool.

Then reboot the SmoothSec and select the new kernel when it is available.

Step 2 - Configure Suricata :

nano /etc/suricata/suricata.yaml

Locate "#- delayed-detect: yes" and replace with "- delayed-detect: yes".

Locate "- fast:" and replace "enabled: no" with "enabled: yes".

Locate "- drop:" and replace "enabled: no" with "enabled: yes".

Locate "af-packet:" and replace "threads: 1" with "threads: 4". Or, the number of core of CPU you have.

Locate "#checksum-checks: kernel" and replace it with "checksum-checks: kernel".

Locate "#copy-mode: ips" and replace it with "copy-mode: ips".

Locate "#copy-iface: eth1" and replace it with "copy-iface: eth1".

Add "buffer-size: 64535" just below "copy-iface: eth1".

Locate "- interface: eth1" and replace "threads: 1" with "threads: 4". Or, the number of core of CPU you have.

Add the following lines just below "# disable-promisc: no" :

buffer-size: 64535
copy-mode: ips
copy-iface: eth0
use-mmap: yes
checksum-checks: kernel


Locate "rule-files:" and add "- local.rules" just below "- emerging.rules".

touch /etc/suricata/rules/local.rules

nano /etc/init.d/suricata

Locate "/usr/local/bin/suricata --user suricata -c /etc/suricata/suricata.yaml -i $INTERFACES -D" with "/usr/local/bin/suricata --user suricata -c /etc/suricata/suricata.yaml --af-packet -D"

There are 2 entries, you should replace them all.

Step 3 - Time Zone :

Make sure your SmoothSec is UTC no matter your time zone is. It is because, Snorby is only working on UTC. Otherwise, the timestamp of Snorby will be wrong.

To check time zone of SmoothSec :

date

If the time is not UTC, you need to change back to it :

dpkg-reconfigure tzdata

Set the time zone to "UTC" at "None of the above".

Step 4 - Configure email feature of Snorby :

If you installed Postfix, configure it properly according to your network at "/etc/postfix/main.cf".

nano /var/www/snorby/config/initializers/mail_config.rb

Then uncomment the lines just below "#Sendmail Example:". Or, refer to the SmoothSec Wiki for the installation.

Step 5 - Configure network interfaces :

Make it looks like the following. Make sure your eth2 has your IP "address" and "gateway" instead of "192.168.2.180" as it is an example only :

nano /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
   up ifconfig eth0 0.0.0.0 up
   down ifconfig eth0 down
   post-up ethtool -K eth0 gro off

auto eth1
iface eth1 inet manual
   up ifconfig eth1 0.0.0.0 up
   down ifconfig eth1 down
   post-up ethtool -K eth1 gro off

# The primary network interface
#allow-hotplug eth2
#iface eth2 inet dhcp
auto eth2
iface eth2 inet static
   address 192.168.2.180
   netmask 255.255.255.0
   gateway 192.168.2.1


* Please note that ethtool is used as the Realtek network interfaces will produce error when working as AF_PACKET method.

Error messages when doing debugging with "suricata -c /etc/suricata/suricata.yaml --af-packet" :

[ERRCODE: SC_ERR_SOCKET(200)] - Sending packet failed on socket 10: Message too long
[ERRCODE: SC_ERR_INVALID_ACTION(142)] - Unable to release packet data


Step 6 - Configure SmoothSec :

Run the following script to setup SmoothSec :

smoothsec.first.setup

Type "eth0" when asks for the monitor interface. Enter "192.168.2.0/24" when asked for the network. Please note that the address here is an example only. When asked for Intrusion Detection Engine, you type "2" for Suricata. The email address and password asked are for the login purpose of Snorby (web interface).

Step b - Cable connection :

Connect Network Card 0 to the router and Network Card 1 to the switch. Network Card 2 connect to the switch.

When done, reboot the SmoothSec.

Step 7 - Browse Snorby :

You need to wait for several minutes before you can connect to the internet as Suricata need some time to do with the rules.

Open your browser and enter the following url :

https://192.168.2.180

Accept the certificate and wait for about a minute the Snorby will show up.

Step 8 - IPS Setup :

Now your SmoothSec is running as IDS (Intrusion Detection System) and it will not block or drop any malicious traffic.

To configure the SmoothSec to running as IPS (Intrusion Prevention System), you need to :

nano /etc/pulledpork/suricata/dropsid.conf

Append the following :

pcre:MS(0[0-9]|1[0-9])-\d+,bugtraq:\d+,cve:20[0-9][0-9]-\d+

So, you will drop/block any malicious traffic that match the vulnerabilities in the vulnerability reports, such as cve and bugtraq as well as Microsoft's. Meanwhile, you can add your own rules in "/etc/suricata/rules/local.rules". Make sure to run "smoothsec.suricata.rules.update" after your add them.

You may want to disable some rules :

nano /etc/pulledpork/suricata/disablesid.conf

Append the following :

1:2210000-1:2210049

It will disable the rules serial 2210000 to 2210049. A total of 50 rules to be disabled.

After done, run the following script :

smoothsec.suricata.rules.update

* Please also note that you are required to wait for several minutes before you can connect to the internet as Suricata requires some time to do with the rules.

(D) Troubleshooting

(1) In case you find there is no GeoIP information on the events, you should check if the file "snorby-geoip.dat" in /var/www/snorby/config/ or not. If not, just download it, please follow the below commands. If the file does not exist, that mean you cannot connect to the internet when installing Snorby.

cd /tmp/
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
gzip -d GeoIP.dat.gz
mv GeoIP.dat snorby-geoip.dat
chown www-data:www-data snorby-geoip.dat
cp snorby-geoip.dat /var/www/snorby/config/


(2) In case you do not capture the events, you should make sure the correct interface name (such as eth0, eth1 and eth2) are connected properly. You should examine the MAC address of the Network Card to determine the correct interface name.

(E) Performance

The SmoothSec is installed on a low-end hardware (Intel Atom D510 CPU with Realtek Gigabit NICs). It is also behind a router, which is running Untangle (Intel Atom D510 CPU with Realtek Gigabit NICs). Untangle is an UTM (Unified Threat Management System) which can block some malicious traffic (but a few only). The switch is D-Link DGS-1008D (Home) Gigabit switch.

To test the performance, I am watching a youtube at 1080p on PC-1 (via wifi), watching a youtube at 720p on PC-2 (via wifi) and watching a youtube at HD on Android smartphone with wifi. The result is very smooth without any lagging for all the devices.

The CPU loading for the test is below 4.x and memory used is below 3GB.

AF_PACKET is ideal for IDS/IPS implementation when you have a very low-end hardware.

(F) Limitation

Since SmoothSec 3.2 is build on Debian 7.0 (Wheezy), the system will be broken when you upgrade to Sid (Unstable). The newer version of Apache (2.4.x) and Perl will refuse to run due to error. Therefore, when you installed the newer kernel (for AF_PACKET purpose), make sure you comment out the repos that you added in order to prevent the system upgrade to the Sid (Unstable) by accident.

Another limitation is that you are requested to have at least 3 NICs for IDS or IPS.

One more limitation is the Snorby cannot show the dropped traffic at the moment.

Known Issue

Pigsty will crash randomly. As a result, no capture in Snorby. The problem has been reported, see here. --> The workaround is to run a testing bash script on every 5 minutes to start Pigsty again.

nano /root/chkpigstylog

#!/bin/bash
# Check if "Error: " in pigsty.log or not. If yes, start Pigsty again.
STRING="Error: "
if grep -R "$STRING" /var/log/pigsty.log
then
   /root/runpigsty
fi


nano /root/runpigsty

#!/bin/bash
/usr/local/bin/pigsty -c /etc/pigsty/suricata.pigsty.config.js -i eth0 -n "Suricata" -d /var/log/suricata/ -m unified2.alert.* -D


crontab -e
*/5 * * * * /root/chkpigstylog


Update

Developers just fixed the problem. You just upgrade the Pigsty with the following command :

npm update pigsty-mysql -g
npm update pigsty -g


That's all! See you.