Break down of attack types

Most web developers know that they should sanitize their web input. However recent figures from the UK Security Breach Investigations Report 2010 indicate that 40 per cent of all website attacks are due to SQL injections.

SQL injection attacks allow perpetrators to leak data, usually by making a web application perform a query it wasn’t intended to do. However, what most fail to realize is under the right conditions SQL injection attacks can be much more potent than data exposure (which is a serious breach in itself). A well crafted attack has the potential to subvert your entire system where circumstances allow.

To begin, let’s discuss what the SQL injection attack is, and how it works.

A Basic Example

We shall take a PHP MySQL query and consider the problem with it.

mysql_query("SELECT id,username,password FROM user_table \
 WHERE username="'.$_GET['username']."');

So when a user executes a query genuinely, the variable will typically be replaced and the query such, I.E:

mysql_query("SELECT id,username,password FROM user_table WHERE username='matthew'");

The problem arises however when the data input contains characters which are meaningful in an SQL statement. Consider for example logging in with the username ma’tthew (note the intentional quotes in the middle of the username). When we do the variable expansion the query ends up appearing as:

mysql_query("SELECT id,username,password FROM user_table WHERE username='ma'tthew";

When you run this, the query is invalid SQL because the entire statement is syntactically incorrect. What has happened is the attacker has altered the behaviour of the SQL statement – actually gaining control of it. This allows the attacker to continue the statement altering to fetch data that is normally not permitted by the original statement.

This kind of attack is well known by web developers. Unfortunately for system administrators and web developers alike the problem doesn’t stop here. If the privileges that have been set by the system/database administrator are too lax it’s possible to reap data right off the disk and worse still, deploy arbitrary data onto the disk.

The Worst Case Scenario

mysql> desc data;
+-------+-------------+------+-----+---------+----------------+
| Field | Type        | Null | Key | Default | Extra          |
+-------+-------------+------+-----+---------+----------------+
| id    | int(11)     | NO   | PRI | NULL    | auto_increment |
| info  | varchar(32) | YES  |     | Nothing |                |
+-------+-------------+------+-----+---------+----------------+
2 rows in set (0.00 sec)

The webpage used is PHP written as follows:

<?php
mysql_connect('localhost','root','xxxxxx') or die(mysql_error());
mysql_select_db('mywebapp') or die(mysql_error());

echo "<table>\n";
echo "<tr><td>ID</td><td>Info</td></tr>\n";

if (isset($_GET['search'])) {
   $r = mysql_query("SELECT * from data where info like '".$_GET['search']."'") \
      or die(mysql_error());
   echo "SELECT * from data where info like '".$_GET['search']."'";
else {
   $r = mysql_query("SELECT * from data") or die(mysql_error());
}

while ($row = mysql_fetch_array($r, MYSQL_NUM)) {
   echo "<tr><td>$row[0]</td><td>$row[1]</td></tr>\n";
}

echo "</table>";
?>

<form name="test">
Search: <input type=text name=search value=""><br/>
<input type="submit"/>
</form>

However, the developer also has also run “chmod 777” on a directory called “images” which is used for another part of the website. This is a common work-around used to avoid permission problems when creating files, by allowing anyone to create files.

The SQL injection vulnerability occurs on line 9. Because the input is not sanitized, the user can perform a fake search and take control of the SQL. The attacker, having already tried standard SQL injection techniques has seen little data of interest remains on the databases. Rather than find the other databases, the attacker wants to spawn a shell. But, can this be done from within mysql?

The answer is, yes of course it can. This is because the db user has the FILE privilege set that means he can read files in and write files out. The attacker needs to know where the document root is for the website. It’s not outright retrievable from SQL but it is readable in the httpd.conf.

By utilizing the LOAD_FILE privilege and UNION selecting it out, the attacker can add it to the existing table to read the total contents of the file! It’s not a pretty read but thats not relevent. By exploiting the FILE privilege the attacker has obtained a means to get the sites document root.

Armed with this infomation, we can look at the design/layout/source code (again, with more LOAD_FILE tricks it’s possible to determine the most likely place that has a globally writable directory). For example an images/avatar folder for perhaps, joomla, if the site was written as such would be a great target. Because there is a tendency to make folders world-writable when they cannot be normally written to, the attacker can exploit this weakness to deploy a new file within the sites’ document root through mysql. Normally an attacker wants to deploy PHP code into the document root because it will execute. Since it contains lots of meta-characters the attacker typically translates the actual code he wants to use into hexadecimal output. Using the INTO OUTFILE syntax in MySQL he can dump the contents of said file right into his target directory. In this example I will be using simple PHP code that generates “hello world” when the page visits.

The image illustrates what’s happened here. The attacker has injected his own custom string and dumped it into an outfile that’s globally writable and present in the document root of an existing website. Now all that’s left is to visit the file you wrote. The big issue with this type of attack is that it will subvert any coding you might put in place, typically in uploads, to prevent php files being written into sensitive areas on disk.

In Conclusion

The potency of SQL injection and commonness of not sanitizing input is a real threat to system security over and above what’s contained inside of your database. A series of failures have to be reached to get to a point like the one demonstrated above. These failures may include: not enforcing least privileges on database users, not sanitizing all input that comes from a untrusted source, lax file permissions in directories and no defensive layers in sensitive directories.

The trouble is, it’s incredibly simple for a web developer to overlook the sanitization of input, especially with the tight deadlines and rapid application development process that is typical. Not only this but the general consensus to use vulnerable libraries to connect to mysql make such situations common and a concern. Most people are unaware that it’s possible to convert a data leakage vulnerability into a system compromize which can mean IT managers dont give SQL injection threats the priority they deserve in the development process.

Fixing the Situation

There are many ways to fix SQL injection:

• Sanitize your input!
• Use MySQLi or another modern SQL library that supports prepared (or pseudo prepared) SQL satements.
• Use the setfacl command to give apache only access to directories that are meant to be writable by it.
• Deploy .htaccess files into sensitive folders (like uploads) to whitelist what files should be accessible in the folder, so image folders should only allow access to jpg, png and gif for example.
• Dont give FILE privileges to DB users if they dont need it.
• Use SELinux as a last line of defense (its not possible for mysql to write to http content in SELinux).

Such exploits are the result of lax security measures and poor coding and can undermine the confidence of visitors to your site. There’s no need to be victim to the most common form of web attack.

Most of the more prominent features are laid out at the Redhat website but one of the things it neglects to mention is how much more access control it comes with.

Role Based Access Controls (RBAC) offer a system or security administrator a means to define a role of some sort. In our example below we’ll be using a web admin role.

Since Fedora 9, the SELinux maintainers for Redhat have pulled out all the stops to properly deploy a framework for SELinux that is more flexible than what you see with EL5. The problem with EL5′s SELinux policy is that although it works, it really does not scratch the surface of how powerful SELinux really is. RBAC simply is not implemented. This means that delegation of trust and enforcement of a corporate security policy is difficult.

Normal access controls are fraught with problems of trust. To make somebody a true webadmin in traditional Linux systems requires a lot of effort:

• The user must be able to read/write web content.
• The user must be able read/write configuration files.
• The user must be able to restart web services.
• The user must be able to alter php configuration files.
• The user must be able to read/write home directory content (if say apache uses mod_userdir).
• The user must be able to read/write the temporary files that the http service generates (php sessions and genuine temp files).
• The user must be able to change permissions of web content.

To manage this level of access on a traditional system would be nigh on impossible. You might be able to get a lot of it done through the use of file ACLs and sudo but it would be a nightmare to manage and make sure not to permit too much or too little access.

EL6 dips more than just its toe into the water of SELinux and with it comes a more flexible implementation of role based access control that is worthy of consideration.

Normally one needs to be able to define what the limits of the role are in order to implement it. But the SELinux policy in EL6 already comes with predefined roles, such as web admin which can be implemented without too much trouble.

Demonstrating RBAC

I am going to demonstrate how to do the above in a secure way which gives a system administrator the confidence to delegate trust.

For starters you’ll need either an Fedora 12 box or EL6 Beta. Once here we can prepare our system to do this in a few relatively simple steps.

Firstly, we’ll add the user onto the system as a web administrator.

[root@krbsrv ~]# useradd webadministrator
[root@krbsrv ~]# passwd webadministrator
Changing password for user webadministrator.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Next, we’ll create an SELinux User and assign our UID to use it.

[root@krbsrv ~]# semanage user -a -R "staff_r system_r webadm_r" -L s0 -r s0 webadm_u
[root@krbsrv ~]# semanage login -a -r s0 -L s0 -s webadm_u webadministrator

Line 1 creates the webadm_u SELinux user (this is distinctly different from a UNIX user account) which is mapped to roles it can be part of.

What we have done is assigned it to the staff, system and webadm roles. ‘Staff’ is a restricted account which can su and sudo which is what we’re going to need to permit, the system role is used here because its needed to run init scripts (to start/stop httpd), and finally our webadm role is the actual primary role of this user. It’s not possible to map the webadm role directly and only to this user because webadm_r doesnt actually have enough privileges to get it to login via SSH. So instead we use the loginable staff role and transition to the webadm role when we want to do work. The -l and -r  are sensitivities. This isnt used in SELinux but its mandatory to pass something to it.

Line 2 maps the actual UNIX user webadministrator to the SELinux user webadm_u, so when the user logs in this will be their identifiable user.

Now we have done this theres still a few more steps left yet.

We have listed 3 roles the SELinux user webadm_u can transition into. But, how do we know which one to transition into by default? Well – the answer to this is the folder: /etc/selinux/targeted/contexts/users. This folder contains a list of SELinux users you already have. If you open the file staff_u file you’ll see something like this:

system_r:local_login_t:s0	staff_r:staff_t:s0 sysadm_r:sysadm_t:s0
system_r:remote_login_t:s0	staff_r:staff_t:s0
system_r:sshd_t:s0		staff_r:staff_t:s0 sysadm_r:sysadm_t:s0
system_r:crond_t:s0		staff_r:staff_t:s0
system_r:xdm_t:s0		staff_r:staff_t:s0
staff_r:staff_su_t:s0		staff_r:staff_t:s0
staff_r:staff_sudo_t:s0		staff_r:staff_t:s0
system_r:initrc_su_t:s0		staff_r:staff_t:s0
staff_r:staff_t:s0		staff_r:staff_t:s0
sysadm_r:sysadm_su_t:s0		sysadm_r:sysadm_t:s0
sysadm_r:sysadm_sudo_t:s0	sysadm_r:sysadm_t:s0

This file is a two columned list of which role/types to map to users depending on how they enter the system. So for example the type “local_login_t” represents accessing from a console directly whereas the type “sshd_t” represents logging in via SSH. To the right of these entries is a left-to-right priority list of what contexts the staff_u user ends up getting when they login. Its not important to know all about how this works. All we really need to do is copy this file and name it webadm_u in the same directory.

[root@krbsrv ~]# cp /etc/selinux/targeted/contexts/users/staff_u \
/etc/selinux/targeted/contexts/user/webadm_u

OK so now we have initialized our webadm_u user for logging in. But theres one final task..

The UNIX user webadministrator cant do some of the things it needs to to properly function – such as restart the httpd service or change file ownerships/permissions when necessary. To do this webadm must become root. Becoming root means nothing to SELinux. It will enforce its policy all the same, so even as root webadministrator is restricted purely to the role that is needed. Thus we can safely do this without compromizing our system. We use sudo to do this which takes special tags we use to transition to our webadm role automatically so the user doesnt need to worry about the selinux particulars:

[root@krbsrv ~]# echo 'web_admin ALL=(ALL) TYPE=webadm_t ROLE=webadm_r ALL' >> /etc/sudoers

This means that when the webadministrator runs sudo it will automatically transition into the webadm_t type and webadm_r role.

Great, now we’ve fixed up our user lets test him out!

[root@krbsrv ~]# ssh webadministrator@192.168.122.73
webadministrator@192.168.122.73's password:
Last login: Wed Aug 11 22:55:45 2010 from 192.168.122.1

[webadministrator@krbsrv ~]$ id -Z
webadm_u:staff_r:staff_t:s0

[webadministrator@krbsrv ~]$ sudo -s
[root@krbsrv ~]# id -Z
webadm_u:webadm_r:webadm_t:s0

So, we login via SSH as per the norm. When we login we check our ID (getting SELinux context). You can see we have logged in with webadm_u as the user but staff_r as the role and staff_t as the type. We can’t do much to our web content in this role and we’re also not root. When we sudo what happens is sudo auto-transitions the user into the webadm_r role and webadm_t type – just what the doctor ordered.

This role runs a very restricted set of actions it can take. Lets see what we can do…

We should be able to change the apache configuration and restart the service:

[root@krbsrv ~]# echo "# Add a comment to this file" >> /etc/httpd/conf/httpd.conf
[root@krbsrv ~]# /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

However we can’t restart other services:

[root@krbsrv ~]# /etc/init.d/sshd restart
bash: /etc/init.d/sshd: Permission denied

We can read, create and modify files in the document root:

[root@krbsrv ~]# cd /var/www/html
[root@krbsrv html]# touch new_file.txt
[root@krbsrv html]# rm new_file.txt

However we can’t modified files outside this:

[root@krbsrv ~]# echo "Port 20000" >> /etc/ssh/sshd_config
bash: /etc/ssh/sshd_config: Permission denied
[root@krbsrv ~]# cat /etc/shadow
cat: /etc/shadow: Permission denied

Looks good!

So, here we are. As you can see, in the webadm role we can restart httpd (which webadministrator needs to do), write to our configuration files and alter our webcontent. However we can’t change anything outside of our remit or attempt to perform anything nefarious – all despite the fact we are root!

In Conclusion..

Practically speaking, the SELinux policy that comes with EL6 is meant to be a framework, not really a turn-key solution to just fit in with your current system. mAs such webadm as a role itself needs tweaking.

For starters, in the webadm role you can’t read your own home directory which is a little impractical. But also you can’t manage the php.ini or any session files created within php.ini. Therefore I’ve tweaked the policy and added the ability for webadm to be able to test websites from within the role, resolve DNS name entries and also allow SSL certificates to be written in the appropriate places. I decided not to permit webadministrator to be able to use FTP to download files directory in the webadm role. If he wants to do this however he can use the non-root login (using the staff role) to download to his home directory and then copy it accross in the webadm role afterwards. I have supplied the policy I wrote as an idea of how you would do this (download here: mywebadm).

It should be worth nothing that nearly every SELinux policy needs fine-tuning to suit your needs. One size definitely does not fit all. SELinux policy however gives you the specific tools you need to build a working, guaranteed access policy meaning you can delegate system administrator work without giving away root privileges and assign the specialists in their fields the power they need to do their work and no more.

I’m a bit of a fan of what SELinux is and does and I thought it was a shame that Redhat failed to mention the amount of effort and progress gone into the policy EL6 ships with. In the real world managing security threats outside and inside your network is a high priority. EL6 finally gives Linux the power to do this.

At least control groups gets a mention. But thats a story for another time…

First, a little background. Many of the consumer grade motherboards on the market use low-end NICs which under high network load can incur a substantial cost compared to enterprise grade NICs. This is because of shortcuts that have been used for getting the device onto the market.

With this in mind, lets take a closer look at what impact a bad NIC can have on Linux compared to one that has been properly optimized.

The Test Setup

Our test machine is a virtual machine running with QEMU + KVM. Configured on the VM are two network devices, an emulated rt8139 chipset device (eth1) and the newer, and hopefully more efficient virt-io paravirtualized network device (eth0).

ip address show
[...]
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 52:54:00:73:67:73 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.73/24 brd 192.168.122.255 scope global eth0
inet6 fe80::5054:ff:fe73:6773/64 scope link
valid_lft forever preferred_lft forever

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 52:54:00:73:67:74 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.177/24 brd 192.168.122.255 scope global eth1
inet6 fe80::5054:ff:fe73:6774/64 scope link
valid_lft forever preferred_lft forever
[...]

And just to be thorough, the kernel modules we’re running:

lsmod | egrep '^(8139|virtio_net)'
8139too                27638  0
8139cp                 19191  0
virtio_net             14013  0

The Benchmarking

To start we’ll take down the virt-io device and see what kind of performance we are able to obtain from the 8139 device when we give it some work to do:

ip link set dev eth0 down

To benchmark this properly requires the use of a system profiler and we have two options; Oprofile and Perf.

Perf is typically the one you should choose on newer kernels, and since test server is running Fedora 12 we’ll be using this as our profiler.

The way that profiling works is through special hardware performance counter registers on the CPUs which are utilized to obtain our statistics with very little overhead and thus causing lesser fudging of our benchmark.

The test file we’ll be downloading is a file of random data using wget on the hypervisor itself. The idea here is that we attempt to max our throughput by selecting a file on a neighbouring machine where as little network interference could effect our results. Perf will record the data which we can analyze:

perf record -af wget http://192.168.122.1/stuff/bigfile.img
--2010-08-09 14:08:22--  http://192.168.122.1/stuff/bigfile.img
Connecting to 192.168.122.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 419419444 (400M) [application/octet-stream]
Saving to: "bigfile.img.1"

100%[===================================================>] 419,419,444 21.0M/s   in 14s

2010-08-09 14:08:37 (27.9 MB/s) - "bigfile.img.1"

[ perf record: Woken up 15 times to write data ]
[ perf record: Captured and wrote 2.413 MB perf.data (~105447 samples) ]

What we have done here is got the profiler to monitor the entire system at the same time running the ‘wget‘ command. This has given us reference samples. The percentages that the report creates are relative to the total load the system produced, thus to get the overall load on the system at the same time we have ran mpstat to collate overall system load. These results are listed below:

02:17:35 PM  CPU      %usr   %nice    %sys...
02:17:36 PM  all      0.00    0.00    0.00...
02:17:37 PM  all      2.00    0.00    8.00...
02:17:38 PM  all      0.00    0.00   17.44...
02:17:39 PM  all      1.01    0.00   18.18...
02:17:40 PM  all      0.00    0.00   12.12...
02:17:41 PM  all      1.01    0.00    9.09...
02:17:42 PM  all      0.00    0.00    4.08...
02:17:43 PM  all      1.00    0.00   11.00...
02:17:44 PM  all      0.00    0.00   16.83...
02:17:45 PM  all      0.00    0.00    2.02...
02:17:46 PM  all      0.94    0.00    5.66...
02:17:47 PM  all      0.00    0.00    5.56...
02:17:48 PM  all      0.00    0.00    2.11...
02:17:49 PM  all      0.98    0.00   16.67...
02:17:50 PM  all      0.00    0.00    9.20...
02:17:51 PM  all      0.99    0.00    7.92...
02:17:52 PM  all      0.00    0.00    2.08...

You can see here during the run system cpu load ramped up to about 13% whilst the download took place.

The results for our perf and our 8139 module grepped out are thus listed. They give us more insight as to what is going on:

perf report | grep 8139
7.15%          swapper  [kernel] [k] cp_start_xmit        [8139cp]
4.72%          swapper  [kernel] [k] cp_interrupt [8139cp]
3.88%             wget  [kernel] [k] cp_rx_poll   [8139cp]
3.23%             wget  [kernel] [k] cp_start_xmit        [8139cp]
1.73%             wget  [kernel] [k] cp_interrupt [8139cp]
0.92%          swapper  [kernel] [k] cp_rx_poll   [8139cp]
0.17%      flush-253:0  [kernel] [k] cp_start_xmit        [8139cp]
0.12%      flush-253:0  [kernel] [k] cp_interrupt [8139cp]
0.03%             sshd  [kernel] [k] cp_start_xmit        [8139cp]
0.03%          swapper  [kernel] [k] dma_unmap_single_attrs.clone.2       [8139cp]
0.02%      flush-253:0  [kernel] [k] cp_rx_poll   [8139cp]
0.02%          swapper  [kernel] [k] dma_map_single_attrs.clone.1 [8139cp]
0.02%             sshd  [kernel] [k] cp_interrupt [8139cp]
0.01%             wget  [kernel] [k] dma_unmap_single_attrs.clone.2       [8139cp]
0.01%            ata/0  [kernel] [k] cp_start_xmit        [8139cp]
0.01%            ata/0  [kernel] [k] cp_interrupt [8139cp]
0.01%             sshd  [kernel] [k] cp_rx_poll   [8139cp]
0.01%             wget  [kernel] [k] dma_map_single_attrs.clone.1 [8139cp]
0.01%          kswapd0  [kernel] [k] cp_interrupt [8139cp]
0.01%  hald-addon-stor  [kernel] [k] cp_interrupt [8139cp]
0.01%          swapper  [kernel] [k] netif_wake_queue     [8139cp]
0.01%  hald-addon-stor  [kernel] [k] cp_start_xmit        [8139cp]
0.01%        scsi_eh_0  [kernel] [k] cp_start_xmit        [8139cp]
0.01%                X  [kernel] [k] cp_start_xmit        [8139cp]

Of the total load on the system, the 8139 driver used about 20% of the entire load. If we take our 15% system usage from before and take 20% from it this indicates that about 3% of the cpu was used handling the network traffic.

Lets take a look at virt-io. We’ll enable it and run the same test.

perf record -af wget http://192.168.122.1/stuff/bigfile.img
--2010-08-09 14:25:35--  http://192.168.122.1/stuff/bigfile.img
Connecting to 192.168.122.1:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 419419444 (400M) [application/octet-stream]
Saving to: "bigfile.img.7"

100%[================================================>] 419,419,444 59.1M/s   in 6.0s

2010-08-09 14:25:41 (66.6 MB/s) - "bigfile.img.7"

Interesting, this ran actually took half the time.

02:25:34 PM  CPU    %usr   %nice    %sys[...]
02:25:35 PM  all    4.00    0.00   23.00
02:25:36 PM  all    2.67    0.00   52.00
02:25:37 PM  all    1.23    0.00   33.33
02:25:38 PM  all    1.00    0.00   17.00
02:25:39 PM  all    0.00    0.00    2.00
02:25:40 PM  all    3.03    0.00   36.36
02:25:41 PM  all    0.00    0.00    6.93
02:25:42 PM  all    0.00    0.00    1.01

So, system load was much higher this run using virt-io. Lets check our perf results:

0.18%          swapper  [kernel] [k] virtnet_poll      [virtio_net]
0.08%             wget  [kernel] [k] virtnet_poll      [virtio_net]
0.04%          swapper  [kernel] [k] try_fill_recv     [virtio_net]
0.03%      flush-253:0  [kernel] [k] virtnet_poll      [virtio_net]
0.01%          swapper  [kernel] [k] start_xmit        [virtio_net]
0.01%          kswapd0  [kernel] [k] virtnet_poll      [virtio_net]
0.01%             wget  [kernel] [k] xmit_skb  [virtio_net]
0.01%             wget  [kernel] [k] start_xmit        [virtio_net]
0.01%             wget  [kernel] [k] try_fill_recv     [virtio_net]
0.00%          swapper  [kernel] [k] xmit_skb  [virtio_net]
0.00%          swapper  [kernel] [k] free_old_xmit_skbs        [virtio_net]
0.00%          kswapd0  [kernel] [k] free_old_xmit_skbs        [virtio_net]
0.00%      flush-253:0  [kernel] [k] free_old_xmit_skbs        [virtio_net]
0.00%      flush-253:0  [kernel] [k] start_xmit        [virtio_net]
0.00%      flush-253:0  [kernel] [k] try_fill_recv     [virtio_net]

Well, this is much better. virt-io uses about 1% of the average 25% system usage for the task, thats 0.25% of the total CPU, about 12 times more efficient!

So, what does this show us?

Well, this test would be a no-contest race anyway because on a VM like this 8139 is not paravirtualized whereas virt-io is. Virt-IO was bound to win.

But what this does demonstrate is the difference in driver implementations can broadly affect the CPU. On consumer systems especially cheap NICs reduce performance over the long term by perhaps 3-4% of the CPU. This might not seem like a lot now, but when we delve into the realms of 10G ethernet, this will start to show on more modern CPUs. Having multiple CPUs handling incoming traffic will spread this load out leaving your system free to handle other tasks – or at least not block so much which could lead to increased throughput.

Conclusion

This change, ultimately, will make Linux CPUs perform better in very high speed networks. With the enterprise trend beginning to move to high speed SANS using ISCSI, and perhaps further in the future Fibre Channel over Ethernet, it becomes important for system adminstrators to know where their overheads are. 10G NICs in these environments will really benefit from multi-core CPUs which by the time 10G becomes the norm, most people should be using.

As a system administrator myself, I have a keen interest in resource accounting. Measuring efficiency is important in our business and improving it without having to do much effort on my own behalf I will always welcome.

Anyone worth their salt in the world of technology has used Apache, and every single internet user will have been visited sites powered by it.

The press release reinforces all of the achievements of the project during the last decade and a half. But one of the introductory paragraphs sums it up nicely:

A triumph for the all-volunteer Foundation, the Apache HTTP Server reliably delivers petabytes of data across the world’s most demanding uses, including real-time news sources, Fortune 100 enterprise portals, cloud computing clusters, financial services platforms, mission-critical military intelligence applications, aerospace communications networks, and more. The server software can be downloaded, modified and installed by anyone free of charge.

Well done to the Apache Foundation, and all the developers who have contributed to the HTTP Server project over the years, and made it the most popular webserver on the planet!

Usual warning: this is a testing beta release, so don’t rely on it just yet!

Firstly, one intersting point that people don’t realize – the Ubuntu version numbers are simply a reflection of the date they were released. Therefore this version, 10.4, will be released April 2010. This is key part of Ubuntu’s organization – that the release cycle is planned in advance, so everyone know what to expect.

The significance of this particular realase is that this is a LTS version, Ubuntu’s “Long Term Support” version, which are only released every two years. These special versions are supported for an extended period, three years on the Desktop version and five years on the Server version.

The result of this is that as a server admin you can install the LTS version, and be assured that security and bug fixes will continue to be supplied for the next five years. This means you can avoid having to upgrade your operating system every couple of months just to stay secure, like you do for other distributions.

The actually release notes are pretty long and detailed, however I’ve extract some of the keys changes over the last few months:

Ubuntu 10.04 LTS Desktop and Netbook Editions continue the trend of ever-faster boot speeds, with improved startup times and a streamlined, smoother boot experience.

Ubuntu 10.04 LTS brings many improvements over Ubuntu 8.04 LTS to keep your servers safe and secure for the next five years, including AppArmor profiles for many key services, kernel hardening, and an easy-to-configure firewall.

And for those who like version numbers (as all Linux geeks should!), here’s some key details to oggle over:

On the Desktop: GNOME 2.30, KDE SC 4.4, XFCE 4.6.1, OpenOffice.org 3.2.0, X.Org server 1.7.5

On the Server: Apache 2.2, PostgreSQL 8.4, PHP 5.3.1, LTSP 5.2

Under the hood: GCC 4.4.3, eglibc 2.11, Linux 2.6.32.9, Python 2.6.5

Of course all the information is publically avaliable, and worth a read over here on the beta release site.

I want to give you two examples taken from live severs to demonstrate the usefulness for monitoring servers, and in particular graphing their stats to show problems and illustrate long term trends which may need addressing in future.

short term graphing

If you’re processing hundreds of thousands of emails a day it’s hard, if not impossible to spot trends in your activity. If one day you send 12,000 messages instead of 8,000 how can you easily notice, and more importantly if it’s extraordinary?

Firstly doesn’t that look pretty? OK, maybe in quite a geeky way, but it shows you some important things which lets you make some presumptions.

• Most legitimate emails are sent and received during working hours Monday to Friday.
• At the weekend a lot less legitimate emails are sent and received.
• The background level of rejecting illegitimate email doesn’t adhere to a weekly cycle.

On the whole the mail service seem pretty health, and shows a steady weekly pattern. It’s worth pointing out this server isn’t under huge load so the numbers aren’t massive. However it demonstrates the point well.

longer term graphing

Now for a graphs which shows how various serious extraordinary activities can be easily identified in a longer time period. Take a plot from another server over the last year, this time of its load average.

Again, a pretty looking graph, with three key events:

• A complete gap in the graph in November.
• A unique spike in load in March.
• A drop in average load from the start of August.

Let’s address these points in order. The gap in graphing could represent the server going down (a power outage, hardware failure etc). Now in reality it is actually due to the graphing system itself being upgraded, but for this article let’s call it an outage to demonstrate what it would like look if it really had happened. We can see after the outage the machine returned to around normal (for that period) load.

The second point, the massive spike in load was due to a DDOS attack against one of the hosted websites. It didn’t bring the server down (due to well configured apache, and quick action by the administrators) but it made the server work a lot harder than for the rest of the entire year. The results of this attack made us look at the general load levels of the server, and with a little more tweaking after the attack you can see the load average was leveled out to a more even average.

Four months later, and after trying to reduce the average load and memory usage further we decided to update the RAM in the server. The use of other graphs (not shown here) indicated that swap usage was increasing, as a physical memory upgrade was on the books. The results of this upgrade (which took so little time that you can’t see it on the graph) has dropped the average load to a fraction of the amount.

in summary

Graphing your stats provides a long term record of health and performance, and gives an interesting interactive method of keeping track of your servers. I certainly wouldn’t pore over pages of numbers to check the server daily, but instead I can at a glance see things are normal. For those wanting to try it themselves, I would recommend the powerful (bit a little complex) Cacti graphing suite, which is based on SNMP and rrdtool. There are simpler systems such as Munin too, but all run on LAMP systems well.

The new version of Ubuntu brings the usual bug fixes and package updates, but also lots of new software. This release includes more support for virtualization technology, both natively on Ubuntu servers themselves and via third parties services like Amazon. It now offers an entire enterprise grade platform for running virtual servers, all for free.

All the information about Ubuntu releases are publicly available long in advance. There are no surprises when the final version is ready for download. No excuse for developers to claim unexpected changes have broken the website – new features and bug fixes are addressed and added to the public development version of Ubuntu daily. You can download and install any development branch, which can give you a feel of an upcoming version in advance of it’s official stable release. Users can give their feedback and criticism to the developers and then submit their own fixes to problems, just like all well maintained Open Source projects.

It’s worth pointing out that Karmic is a standard 18 month support release, running through to April 2011, after which users will be expected to have moved onto a newer version in order to be kept secure and stable. But don’t worry! Ubuntu’s next Long Term Support (LTS) edition is due in April 2010, and will keep servers secure through to 2015.

Ubuntu Timeline

For the uninitiated (catch up Windows!), SSH gives you an encrypted connection to your server wherever on the internet it is. OpenSSH has evolved greatly from what a lot of people perceive to be a secure version of telnet, but the modern truth is far from it. The feature list of OpenSSH is very impressive, and it not only allows seemless secure command line, it can handle dynamic SOCKS proxying for impromptu VPNs, public-key logins for password-less access, along with port fowarding and file transfers. Using the highly flexible X display system it can even forward graphical displays to remote machines as if they were on our own PC. These are all techniques which someone who wants the most from their system should learn to use.

OpenSSH has evolved from an extra layer of security into a whole suite of networking tools – all of which just happen to be fully encrypted and secure to use across public internet connections at the same time! It has also been ported to a whole raft of platforms (Windows, Solaris, HP, etc) and so taken a place right at the heart of the internet.

All this is a testament to the power of Open Source software, and demonstrates how a transparent and public security policy when developing software leads to very great things.

Traditionally, change is brought about by ideas, contributions, team work and communities. This is no less so in IT.

Just a few years ago software and applications were seen as magical entities and few people understood how they worked. This lack of understanding inevitably led to problems like users being locked into applications that vendors no longer promoted, even while they continued to collect support fees.

But now, as our lives plunge into the digital, people are more understanding of how applications work and in turn more able and willing to contribute to the science of IT. Today online organisations and businesses have a vision of how they want things to work, they want more flexibility online and are willing to fund the building of bespoke software.

UKFast puts 25 per cent of all resources into its R&D department to develop software and applications. Most of our systems are written in-house by the UKFast R&D community, to the exact specifications that our business needs.

Community is how Linux has developed over the years and with Google opening its speeding the web applications to the scrutiny of programmers we see a level of outsider contribution here too.

Facebook and other social networking sites have allowed third parties to create add-on applications which contribute to the level of customer enjoyment. But what about those companies that charge us for licence fees and bar us from personal modifications – well even they’re on the turn, it seems.

Microsoft has over the past year, donated code to PHP, offered support to the Apache Foundation, and it made its first code submission to the Linux kernel just last month.

There have also been suggestions of more schools and workplaces adopting open source in their organisations to cut license costs. It’s long been part of the battle that the world’s kids are introduced to Microsoft at a young and pliable age – so many don’t even know there is an alternative.

So maybe the future is different. Will we see IT lessons shift from creating a PowerPoint presentation to building the programme itself? I certainly hope so!

OK, obviously I lie, Wine is actually a program which lets you run Windows software in Linux. For many this idea seems a bit crazy – in most respects Windows & Linux are so different just running a program built for one OS, out of the box on the other is pretty much inconceivable.

So before I go further, let me just give you the proper definition of Wine, as taken from the development team’s website: Wine HQ.

Wine is a translation layer (a program loader) capable of running Windows applications on Linux and other POSIX compatible operating systems. Windows programs running in Wine act as native programs would, running without the performance or memory usage penalties of an emulator, with a similar look and feel to other applications on your desktop.

There’s one little line in that definiton that gives rise to Wine’s name, “without the performance or memory usage penalties of an emulator“. Wine stands for (in the classic Linux Recursive Acronym way): Wine Is Not an Emulator. Now I’m not going to go into the details of how it works, but you can consider modern versions of wine as almost stripped-down versions of Windows.

And there are actual real life examples of this – Windows application which run better on Linux than on their native OS. Yes, programs never designed to run on Linux, actually performing better on the same hardware.

Word running on Linux? Yes!

Now being a true Linux advocate, I’d always prefer running software developed by the open source community. However I also understand sometimes you have no other choice – which of course is what drives the development of Wine. The biggest driving force of this development – Games. There was big fanfare when tests showed Quake 3 out performing the native Windows version.

However over here at UKFast the call of Wine in day-to-day use is not great, indeed generally in the world of high-spec Linux servers, running Windows software is not needed. But the simple reason I wanted to discuss it was to help people try, and eventually full-migrate to, Linux, and as my previous post said – Submerge Yourself in Linux.

A Modern Linux Desktop: not so scary any more!

As the title of my first post touched on, Linux has a steep learning curve, and I don’t think many people would disagree.

The power, freedom and control of Linux is certainly what draws and maintains the vast majority of its users. The first steps into Linux could be from the demands of an over-worked office mail server, right down to getting frustrated one too many times with having to reboot your Windows desktop computer.

In the last five years the development of the Linux desktop has been amazing, and has done a massive amount to recruit new users, most of whom want to escape the frustration of day to day Windows use. However in the world of servers, firewalls, routers and racks, the image of a flashing white cursor on a black background is not escapable regardless of how much the desktop distributions have developed. As the endless blinking continues, awaiting your beckoning command, for many uninitiated users, this gives a sense of horror – “What on Earth do I do?”

Actually – take that back – my previous statement isn’t entirely correct. You can bypass the blinking light, and administer a server, hosted in a purpose build datacentre in a remote location without punching commands into a console. Control panels like Plesk, which we provide to many clients, allow access to the internal workings of a Linux server, without getting your hands dirty. But I’ve always questioned how much this actually teaches about the system your running, regardless of its operating system. I’ve learnt so much from my exploration of Linux that I encourage everybody to explore and learn for them selves what’s going on.

Inversely to the horror experienced by many first-timers, the command line is something I, and my colleagues on the Linux team adore. Its immediate uninhibited access to control the machine is something you become so used to, having it withdrawn from you can make you feel utterly powerless (as I do when presented with a graphical desktop and told to “fix the web server!”).

So how do you go about submerging yourself in Linux without scarying yourself witless. The Ubuntu Linux distribution is an excellent example of how to approach Linux with this intent. It provides a feature rich and easy to install desktop environment, which is straight forward enough for a long term Windows user to pick up in a minute.

But the code and software behind the scenes is exactly the same as runs the server edition of Ubuntu. Actually, the desktop edition is just an extension of the server edition, the same edition we supply in the rack-mounted, quad-core, ultra-quick, super-dooper servers we host for some of our largest clients.

And I would encourage everyone to try it. Get it installed, play around – and then one day, click that little icon of white text on black background. You may have no idea what to do when you see that blinking cursor, however the entire evolution of Linux has been based on communities (something else I plan on discussing in a later post), and now they are stronger than ever. So just pop over to the Ubuntu Forums and say hi, you’ll probably see one of us chipping in. Hopefully one day you’ll end up with a blinking cursor on a high-spec UKFast Linux server all of your own!

For me it started around ten years ago with an inspirational school teacher, a CD burner and a bit of free time. Looking back at this makes me think about how exactly I was able to turn this serendipitous introduction into a profession.

For myself, the astounding power and freedom of the Linux operating system makes it a natural choice. However this comes around from a confidence I have developed in its use and abilities (including fixing it when it goes wrong!). But I don’t think many people will disagree with the statement that Linux has a steep learning curve – people just can’t get off the ground.

Just the other day I overheard a brilliant example which embodies probably the very central issue new users have: “The Black Screen With White Text”.

Black screen with white text

But there are many ways to overcome this – including the use of the Plesk control panel – which we will talk about in one of my next posts.

For me, one of the main reasons to keep learning is being able to put back into the community I’ve taken from. Just like the teacher who inspired me in the first place a decade ago. The pleasure of being able to give back knowledge after you’ve taken so much is excellent. This strong driving force of the open source community is something I certainly plan on talking about more in my up coming posts.

However, the queston I’ll tackle next time is – How to submerge yourself in linux.