Today I’ve received an alert from the monitoring system, the mails count from server’s queue was too high.
Depending on the numbers of the clients hosted on the server more than 500 of mails lasting more than half hour in the queue is meaning that someone has sent a newsletter or spam.
Let’s ssh there and study the problem. Firstly we should look at the server’s queue:
[root@ulise ~]# /var/qmail/bin/qmail-qstat messages in queue: 758 messages in queue but not yet preprocessed: 0
We do have 758 mails in the queue. Let’s examine the queue with qmail-qread. Seeing a bunch of strange email addresses in the recipient list usually it’s meaning spam.
[root@ulise ~]# /var/qmail/bin/qmail-qread [...]
You can examine the email content of the emails in the queue using Plesk interface or just less command. Firstly we should find message’s id using qmail-qread, then find the file holding the email in /var/qmail/queue with find command.
[root@ulise ~]# /var/qmail/bin/qmail-qread [...] 18 Jul 2008 02:01:11 GMT #22094026 1552 <> remote user@yahoo.com [...] [root@ulise ~]# find /var/qmail/queue/ -name 22094026 /var/qmail/queue/mess/19/22094026 /var/qmail/queue/remote/19/22094026 /var/qmail/queue/info/19/22094026 [root@ulise ~]# less /var/qmail/queue/mess/19/22094026 Received: (qmail 10728 invoked from network); 22 Jul 2008 19:40:46 +0300 Received: from unknown (HELO User) (86.107.221.138) by domain.com with SMTP; 22 Jul 2008 19:40:46 +0300 Reply-To: <support@PayPal.Inc.com> From: "PayPal"<support@PayPal.Inc.com> Subject: Dispute Transaction Date: Tue, 22 Jul 2008 19:40:52 +0300 MIME-Version: 1.0 Content-Type: text/html; charset="Windows-1251" Content-Transfer-Encoding: 7bit X-Priority: 1 X-MSMail-Priority: High X-Mailer: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 [...]
Oops, we do have some spam in the queue that’s received from the network (IP: 86.107.221.138). We should remove spam from the queue or the server IP address will finish listed in the RBLs, qmail-remove is the right tool for this job.
Check the number of the spams with the spam pattern (”PayPal.Inc.com” in this case):
[root@ulise ~]# qmail-remove -p 'PayPal.Inc.com'
Now, remove spams (notice the ‘-r’ switch), they all will end up in the /var/qmail/queue/yanked directory. Don’t forget to stop qmail daemon before (/etc/init.d/qmail stop) :
[root@ulise ~]# qmail-remove -r -p 'PayPal.Inc.com'
In a few minutes we do have more emails with the same patterns from the same ip address. That’s great, we do have opportunity to examine smtp traffic from the spammer’s ip address. Run tcpdump and wait a few minutes.
[root@ulise ~]# tcpdump -i eth0 -n src 86.107.221.138 \or dst 86.107.221.138 -w smtp.tcpdump -s 2048
Examining log file with less or wireshark we found that spammer is sending spam using LOGIN authentication:
220 ulise.domain.com ESMTP ehlo User 250-ulise.domain.com 250-AUTH=LOGIN CRAM-MD5 PLAIN 250-AUTH LOGIN CRAM-MD5 PLAIN 250-STARTTLS 250-PIPELINING 250 8BITMIME AUTH LOGIN 334 VXNlcm5hbWU6 dGVzdA== 334 UGFzc3dvcmQ6 MTIzNDU= 235 go ahead
Interesting, let’s decode the user/pass to see which account is used:
[root@ulise ~]# perl -MMIME::Base64 -e 'print decode_base64("dGVzdA==")' test [root@ulise ~]# perl -MMIME::Base64 -e 'print decode_base64("MTIzNDU=")' 12345
So, someone created a test account with a weak password and someone else guessed it and is sending spam through the server.
Let’s find the domain owning of the mailbox:
[root@ulise ~]# mysql -uadmin -p`cat /etc/psa/.psa.shadow` psa [...] mysql> SELECT m.mail_name, d.name, a.password FROM mail AS m LEFT JOIN (domains AS d, accounts AS a) ON (m.dom_id = d.id AND m.account_id = a.id) WHERE m.mail_name='test' AND a.password='12345'; +-----------+------------+----------+ | mail_name | name | password | +-----------+------------+----------+ | test | example.com | 12345 | +-----------+------------+----------+ 1 row in set (0.01 sec)
Next step is to delete test mailbox and send a warning to client.
To improve your server’s security you’ll need to enable:
Server -> Mail -> Check the passwords for mailboxes in the dictionary
Creating a mailbox “test” with password “12345″ is a stupid thing and spammers just love to exploit it.
11 comments ↓
Glad to see one of our open source tools getting some use here. FYI we just launched MagicSpam, a complete anti-spam solution designed for full integration with PLESK. It’s at magicspam.com if you’re curious. Cheers.
Ryan, LinuxMagic
When i do the TCP dump the file contains a lot of binary data. less does not filter it, and wireshark is not compatible with it.
also i tried to decode on two separate linuxes with the perl libraries installed and both do not give a result even to thee examples you gave.
You don’t need the binary data, just look at the smtp conversation to find what’s happening there.
Regarding second problem, maybe you are missing MIME::Base64 module, install it with:
perl -MCPAN -e ‘install MIME::Base64′
[...] Debug Plesk Spam Problems? [...]
I think is better to analize logs instead sniffing network traffic, what if the spammer has stopped sending mail?
we (at work) parse our logs looking for smtp_auth’s strings, so we got thouse used a lot of times in a short period, later we use this query:
$ mysql -uadmin -pPASSWORD psa -e “select mail.mail_name, accounts.password from mail,accounts where mail.account_id=accounts.id and mail.dom_id=(select id from domains where name like ‘DOMAIN_OF_SMTP_AUTH’);”
and 99% of the cases $DOMAIN_OF_SMTP_AUTH has weak passwords and are used for spamming.
currently i’m working in identify smtp_auth’s brute force attack, do you have some knowledge about it?
great blog! =)
kind regards.
Reading logs and sniffing traffic are both useful
, when the spam is sent by exploiting insecure php scripts (the most common case) you’ll not see too much information in the log files.
Regarding brute force attacks, password strength verification should be enabled in Plesk as users tends to use weak passwords (Server->Mail->Preferences->Check the passwords for mailboxes in the dictionary).
Another recommendation is “Only use of full POP3/IMAP mail accounts names is allowed” as it will slow down brute force attacks because they should guess the domain too.
I tried and I see the binary info too.
The problem is that it’s the user and password part:
250-AUTH=LOGIN CRAM-MD5 PLAIN
250-AUTH LOGIN CRAM-MD5 PLAIN
250-STARTTLS
250-PIPELINING
250 8BITMIME
^^VJ”^K^@>^@^@^@>^@^@^@^@^@^L^G^B^@^V^W^@E^@^@$@^@@^FJPE@Nv^@^Y^Ne\CWIP^X^V^C7^@^@250 ok
^^VJ+^K^@V^@^@^@V^@^@^@^@^V^WC^@^Y^C^@E^@^@H?@^@m^FkvE@N^L^@^Y(g
P^X^\s^B^@^@RCPT TO:
^^VJ]h^K^@<^@^@^@<^@^@^@^@^V^WC^@^Y^C^@E^@^@(?@^@m^FvE@N^N^\^@^Y]7ȴ?P^PF^@^@^@^@^@^@^@^@^^VJ{^K^@>
^@^@^@>^@^@^@^@^@^L^G^B^@^V^W^@E^@^@0Y@^@@^F^ZE@Nv^@^Y^Lg
(P^X^V͒^@^@250 ok
If you found the mail accounts with problems then go ahead, change the passwords and see what happens. If you had weak passwords then the problem should be fixed.
I didn’t find any accounts because I can’t see the binary information.
How can I see the file in ASCII or save it in ASCII? I tried -A but it didn’t work.
Please, some guy is really giving me a hard time spamming with my server.
You can use Wireshark to inspect your tcpdump file.
This blog helped me a lot. Thank you all.
My way:
1. use the open source ‘iptraf’ program to get spammer’s ip. This only works if the spamming is still going on.
2. use ‘tcpdump’ to get a log file. This file can have binary code (like Daniel Valdes said).
3. use ‘wireshark’ to open the log file made by ‘tcpdump’.
4. you can now study the logfile and see all packages and their details.
5. this will give you a hint about what’s going on…
(in my case it was an insecure php script).
6. delete the buggar (the php script, in my case) and use ‘qmail-remove’ or, even better, ‘qmhandle’ (which is a clever script that doesn’t need install) to delete the qmail queue.
7. good luck
8. and always install all trusted software secury upgrades
Leave a Comment