Get a quoteGet a quote
Live ChatLive chat
Client ZoneClient Zone
ContactsContacts
Hide
Live Support
Sales department
Technical Support
Security audits

[Introduction]

Please note that this article is about practical side of security audit.
You can easily find many teoretical materials on this subject (for example at wikipedia http://en.wikipedia.org/wiki/Information_technology_security_audit).

One of our customers asked us to make a security audit of his server. He had suspicions that his system has been compromised (this indirectly has been confirmed by the increased load, which was not so high before). He had a CentOS 5.4 server running Apache, MySQL, vsftpd server and so on. In a nutshell, it was a typical LAMP configuration.

[Start:: definining security perimeter]

Before proceeding to the audit, we have discussed with customer a detailed list of components which we had to focus on. This stage helped us to determine the security perimeter. Final list consisted of the following items:

* System kernel and modules
* System services
* Logging
* Backups
* Firewall rules
* General security policy

[Analysing system:: detailed picture of the perimeter]

 

Given the high probability of breaking the system, we immediately started collecting information and security checks. It is nearly impossible to create a strict scenario of such work. It always includes some kind of improvisation. However, we managed to outline a few steps that usually help us. These steps are convenient to use in the form of questions.

— Who is working on a server right now?

To answer this question, we have used system utilities such as ‘w’ and ‘last’.

  • w - Displays list of logged users and shows what commands they run at the moment.”
  • last -i -  Will show who, when and from which ip addresses logined to server.

Logs were analyzed as well. We have checked /var/log/audit/audit.log and /var/log/secure. After that we checked for suspicious entries /etc/passwd, /etc/shadow, /etc/group, /etc/sudoers

— What processes are running?

— Which ones should run on this host?

Right, these are the two questions. And this step is the most important. It needs the maximum attention and the highest  concentration. You should rembember that there is a possibility of changing the system binaries. So we used a few ways of checking the running processes.

  • ps auxf
  • pstree
  • cat /proc/*/stat | awk ‘{print $1,$2}’

There is a common trick, used by hackers together with binaries replacement. They put to their applications immutable flag, which is not seen by usual ls. But there is an effective way of finding such binaries:

  • lsattr `echo $PATH | tr ‘:’ ‘ ‘` | grep i–

In most cases the process list shows us that there was intrusion and this case was the one of them. We have found two crond processess running: /usr/sbin/cron and /usr/sbin/crond. Path to default cron daemon binary on rh-based systems is /usr/sbin/crond. Second one was listening on port 2283 and telnet to this port opened a root shell on the server.
To find out what is doing a process on server, strace and lsof utilities are usually used. If you didn’t get enough information from these utilities – then can be checked /proc/PID filesystem:

  • lsof -p PID - Displays files, sockets and ports opened by a process.
  • strace -p PID -F – Can help to trace what a process doing (call and signals) and its forks.
  • mc -sd – /proc/PID Browsins proc using a file manager.

Here, you can find a little description of procfs pseudo filesystem.

/proc/PID/cmdline -  Command, which started the process
/proc/PID/cwd - Symlink to working directory.
/proc/PID/environ - Enviroment variables which affect the process.
/proc/PID/exe - Symlink to executable file
/proc/PID/fd - Directory with symlinks to opened file descriptors
/proc/PID/root - Symlink to filesystem root of this process
/proc/PID/status - Contains basuc information on process status
/proc/PID/task – Directory with hardlinks to child processes

— What processes opened ports of connections?

Besides all of the above we are checking the network activity for all processes. This can be easily done using netstat, lsof and nmap utilities:

  • netstat -tapn – Displays network activity on server
  • lsof -i -n – Will display same result as netstat
  • nmap -sT -sU -p 1-65535 localhost – Scanning for open ports

— What automated checks say?

Finally, we've checked the system for usage of some automated utilities: rkhunter, chkrootkit and clamdscan. They can be installed using default package manager (yum install application-name)

  • rkhunter -с – Checks the system, using series of automated checks. Before start it is better to start rkhunder –update.
  • chkrootkit – Checks system in the same way as rkhunder does, but has more false-positive alerts. Analyzing chkrootkit -x | less is more easy.
  • clamscan - Checks file for injections, trojans, shell-scripts. Time of checking depends on harddrive size.

Check for / may take long time, so it is better to check separate folders / mount points (/home, /tmp, etc)

In our case rkhunder output was not so usefull, but clamscan found few interesting files”

/home/vhosts/domain.tpl/httpdocs/tmp/cc.php: PHP.Hide FOUND
/home/vhosts/domain.tpl/httpdocs/tmp/mass.pl: Perl.Defacer FOUND

Files were created few hours ago, that increased the possibility to find the intruision signs in logs. After checking the apache logs for domain we could find entries about accessing these scripts from web. These scripts were accessed from the same ip, that simplified the next steps of analyzing the atatcker's actions for us. First attacker's requests were looking like this:

xxx.xxx.xxx.xxx [DATE:11:19:24 /index.php?option=com_ckforms&controller=ck
data&view=ckformsdata&layout=detail&task=detail&fid=-2+union+select+1,2,3,
concat(0x3a,username,0x3a,email,0x3a,activation),5,6,7,8,9,10,11,12,13,14,
15,16,17,18,19+from+jos_users--

In this request we noticed two known things. First of all, option=com_ckforms, gives us idea that site has Joomla installed on it. Second, the last part of request, there is a SQL-injection. As we found out later, attacker used ckforms-index-sql-injection (56988) vulnerability to gain administrator's rights in cms.

FTP logs contained the same ip. Attacker used ftp to upload few perl scripts and a kernel exploit. First things that we did - we checked the config file for Joomla to check if mysql and ftp password for this domain are the same. We found that passwords were the same, so attackers, most likely used the same technology to gain the passwords.

Thus, we could understand attacker's tactic and causes of successful attack:

  • Old Joomla version
  • Same password for ftp and mysql
  • Weak firewall policy, that allowed to open ports for incoming connections
  • Old version of kernel prone to attacks (CVE-2009-3547)

Elimination of foresaid causes is only a part of work we are doing during audit. This case is not so typical as during analyzis we found out that system is already compromised. In most cases we have to find the ways of break-ins. There is some irony in the fact that attacker did a part of work for us.

[Risk analysis]

Another part of the work that we perform in such cases is risk analysis. The process of analysis is divided into three phases:
1) Analysis of processes, related to perimeter of security
2) Analysis in a perimeter
3) Analysis of the human factor (random error, the dismissed employee)

In this article we will not go into details of the analysis, we will only describe
basic concepts. The concept is to find opportunities to compromise the perimeter. Each of such possibilities, depending on the situation, is assigned a level. In our practice, we use three levels “High”, “Medium” and “Low”. Each level includes certain points. Correspondingly, the more points the perimeter gains – the worse it is protected.

[Finding the appropriate solution]

We must understand that absolute security is not a subject to discuss in real life. Security is a process that assumes a cyclical and even better permanent search for solutions and methods of their use. At the same time, we should not forget about comfort and usability. In most cases enhancing security makes us to decrease comfort or usability. For anyone is not so pleasant to take off shoes at the airport, but the possibility to be in the same plane with a terrorrist is more unpleasant. It is very important to search suitable compromises. For example, using ftp is not secure, because data and moreover passwords are transmitted in an unencrypted form. A good solution might be to use ssh authentication with rsa keys (+ passphrase). However, the probability of successful man-in-the-middle attack to ftp is not so high. It is worth to carefully consider the risks before changing the working processes that are already used. Ftp problem can be solved in several ways. Though its replacement by sftp is reliable but not always is acceptable. In this case we are allowed to access ftp from certain ips only, which solves the problem and thus, reduces the discomfort to the minimum.

[Implementation and Monitoring]

Of course, setting up the firewall with the default policy to DENY is only a part of measures aimed at the improvement of the security. We preffer to use nagios and nrpe. We've even built a distributed cluster based on it, to reduce to minimum false-positive alerts. Distributed monitoring will be discussed here but later, in other article. Of course, we performed update for all software on the server and set up a common security policy (using sudo, configured passwords lifetime, etc.). We have developed procedures for updating and checking the server, which helps client to be informed of how things are going on on his server. Customer continues to use our services and we are pleased in our partnership. We hope this article helped you to discover and learn something new from the world of security.
Thank you for your attention.

Tags: 
Request a Quote
Request a quote
Fill out a small form and let us contact you shortly
Client Zone
Access Client Zone
Ticket system, knowledge base and other services for our clients
Contact Us
Contact us
Send us a request or ask any questions via this contact form