Security audits

[Introduction]

Please note that this article is about practical side of security audit.
You can easily find many teoretical materials on this subject (for example at wikipedia http://en.wikipedia.org/wiki/Information_technology_security_audit).

One of our customers asked us to make a security audit of his server. He had suspicions that his system has been compromised (this indirectly has been confirmed by the increased load, which was not so high before). He had a CentOS 5.4 server running Apache, MySQL, vsftpd server and so on. In a nutshell, it was a typical LAMP configuration.

[Start:: definining security perimeter]

Before proceeding to the audit, we have discussed with customer detailed list of components over which we have to focus. This stage helped us to determine security perimeter. Final list consisted of the following items:

* System kernel and modules
* System services
* Logging
* Backups
* Firewall rules
* General security policy

[Analysing system:: detailed picture of the perimeter]

Given the high probability of breaking the system, we immediately started to collect information and security checks. It is nearly impossible to create strict scenario of such work. It always include some kind of improvisation. However we came to few steps that help us. These steps convenient to use in the form of questions.

— Who is working on a server right now?

To answer this question, we have used system utilities such as ‘w’ and ‘last’.

• w - Displays list of logged users and shows what commands they run at the moment.”

• last -i -  Will show who, when and from which ip addresses logined to server.

Logs were analyzed as well. We have checked /var/log/audit/audit.log and /var/log/secure. After that we checked for suspicious entries /etc/passwd, /etc/shadow, /etc/group, /etc/sudoers

— What processes are running?

— Which ones should run on this host?

Right, these are two questions. And this step is most important. It needs the maximum attention and highest  concentration. You should rembember that there is possibility of changing the system binaries. So we used few ways of checking the running processes.

• ps auxf

• pstree

• cat /proc/*/stat | awk ‘{print $1,$2}’

There is common trick used by hackers together with binaries replacement. They put to their applications immutable flag, which is not seen by usual ls. But there is an effective way of finding such binaries:

• lsattr `echo $PATH | tr ‘:’ ‘ ‘` | grep i–

In most cases the process list shows us that there was intrusion and this case was the one of them. We have found two crond processess running: /usr/sbin/cron and /usr/sbin/crond. Path to default cron daemon binary on rh-based systems is /usr/sbin/crond. Second one was listening on port 2283 and telnet to this port opened a root shell on the server.
To find out what is doing a process on server usually are used strace and lsof utilities. If you didn’t get enough information from these utilities – then can be checked /proc/PID filesystem:

• lsof -p PID - Displays files, sockets and ports opened by a process.

• strace -p PID -F – Can help to trace what a process doing (call and signals) and its forks.

• mc -sd – /proc/PID Browsins proc using a file manager.

Here you can find a little description of procfs pseudo filesystem.

/proc/PID/cmdline -  Command which started the process
/proc/PID/cwd - Symlink to working directory.
/proc/PID/environ - Enviroment variables which affect the process.
/proc/PID/exe - Symlink to executable file
/proc/PID/fd - Directory with symlinks to opened file descriptors
/proc/PID/root - Symlink to filesystem root of this process
/proc/PID/status - Contains basuc information on process status
/proc/PID/task – Directory with hardlinks to child processes

— What processes opened ports of connections?

Besides all of above we are checking the network activity for all processes. This can be easily done using netstat, lsof and nmap utilities:

• netstat -tapn – Displays network activity on server

• lsof -i -n – Will display same result as netstat

• nmap -sT -sU -p 1-65535 localhost – Scanning for open ports

— What automated checks say?

At the last we check the system using some automated utilities: rkhunter, chkrootkit and clamdscan. They can be installed using default package manager (yum install application-name)

• rkhunter -с – Checks the system using series of automated checks. Before start is better to start rkhunder –update.

• chkrootkit – Checks system in the same way as rkhunder does, but has more false-positive alerts. Analyzing chkrootkit -x | less is more easy.

• clamscan - Checks file for injections, trojans, shell-scripts. Time of checking depends on harddrive size.

Check for / may take long time, so it is better to check separate folders / mount points (/home, /tmp, etc)

In our case rkhunder output was not so usefull, but clamscan found few interesting files”

/home/vhosts/domain.tpl/httpdocs/tmp/cc.php: PHP.Hide FOUND
/home/vhosts/domain.tpl/httpdocs/tmp/mass.pl: Perl.Defacer FOUND

Files were created few hour ago, that increased the possibility to find intruision signs in logs. After checking the apache logs for domain we could find entries about accessing these scripts from web. These scripts were accessed from the same ip, that simplified for us next steps of analyzing the atatckers actiosn. First attacker requests were looking like this:

xxx.xxx.xxx.xxx [DATE:11:19:24 /index.php?option=com_ckforms&controller=ck
data&view=ckformsdata&layout=detail&task=detail&fid=-2+union+select+1,2,3,
concat(0x3a,username,0x3a,email,0x3a,activation),5,6,7,8,9,10,11,12,13,14,
15,16,17,18,19+from+jos_users--

In this request we noticed two known things. First of all, option=com_ckforms, gives us idea that site has Joomla installed on it. Second, the last part of request is a SQL-injection. As we found later, attacke used ckforms-index-sql-injection (56988) vulnerability to gain administrator rights in cms.

FTP logs contained same ip. Attacker used ftp to upload few perl scripts and a kernel exploit. Firstg things that we did - we checked the config file for Joomla to check if mysql and ftp password for this domain are the same. We found that passwords were the same, so most probably attackers used same technology to gain the passwords.

Thus we could understand how attackers tactic and causes of successful attack:

• Old Joomla version

• Same password for ftp and mysql

• Weak firewall policy, that allowed to open ports for incoming connections

• Old version of kernel prone to attacks (CVE-2009-3547)

Elimination of foresaid causes is only a part of work we are doing during audit. This case is not so tipic as during analyzis we found that system is already compromised. In most cases we have to find the ways of break-ins. There is some irony in the fact that attacker did a part of work for us.

[Risk analysis]

Another part of the work that we perform in such cases is
risk analysis. The process of analysis is divided into three phases:
1) Analysis of processes related to perimeter of security
2) Analysis in a perimeter
3) Analysis of the human factor (random error, the dismissed employee)

In this article we will not go into details of the analysis, we describe only
basic concepts. The concept is to find opportunities to compromise the perimeter. Each of such possibilities, depending on the situation, is assigned a level. In our practice, we use three levels “High”, “medium” and “low”. Each level includes certain points. Correspondingly more points perimeter gains – worse it is protected.

[Finding the appropriate solution]

We must understand that absolute security is not a subject to discuss in real life. Security is a process that assumes a cyclical and even better permanent search for solutions and methods of their use. At the same time we should not forget about comfort and usability. In most cases enhancing security makes us to decrease comfort or usability. For anyone is not so pleasant to take off shoes at the airport, but the possibility to be in the same plane with a terrorrist is more unpleasant. It is very important to search suitable compromises. For example using ftp is not secure, because data and moreover passwords are transmitted in an unencrypted form. A good solution might be to use ssh authentication with rsa keys (+ passphrase). However, the probability of successful man-in-the-middle attack to ftp is not so high. It is worth to carefully consider the risks before changing the working processes that are already used. Ftp problem can be solved in several ways. Though its replacement by sftp is reliable but not always is acceptable. In this case we are allowed to access ftp from certain ips only, which solves the problem and thus reduces the discomfort to the minimum.

[Implementation and Monitoring]

Of course setting up the firewall with the default policy to DENY is only a part of measures aimed at improve the security. We preffer to use nagios and nrpe. We even built a distributed cluster based on it, to reduce to minimum false-positive alerts. Distributed monitoring will be discussed here but later, in other article. Of course we performed update for all software on the server and set up a common security policy (using sudo, configured passwords lifetime, etc.). We have developed procedures for updating and checking the server, which help client to be informed of how things are going on on his server. Customer continues to use our services and we are pleased in our partnership. We hope this article helped you to discover and learn something new from the world of security.
Thank you for your attention.