Sorry for the 1 hour downtime last night from 12:30am to about 1:30am (Singapore time). The site is up and functional now. I jumped when I got a downtime message from my servers, strange, the server can be reached and pinged, so not a hardware or network problem, but my website cannot be reached. A short search later led me to finding Apache stopped and unable to restart automatically. After viewing my logs, it seems that my server was issued a SIGTERM to terminate Apache, possibility hacked with a Trojan:
A snippet of the logs:
--08:07:28-- http://[some ip]/barbut
Connecting to [some ip]:80... barbut: no process killed
brb: no process killed
barbutLinux: no process killed
--08:07:28-- http://[some ip]/barbut
Connecting to [some ip]... barbut: no process killed
brb: no process killed
barbutLinux: no process killed
--08:07:28-- http://[some ip]/barbut
Connecting to [some ip]... connected.
HTTP request sent, awaiting response... connected.
HTTP request sent, awaiting response... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35336 (35K) [text/plain]
Saving to: `barbut'
0K .200 OK
Length: 35336 (35K) [text/plain]
Saving to: `barbut.1'
0K ................ ........... .......... .......... .... 100% 54.5K=0.6s
08:08:33 (54.5 KB/s) - `barbut' saved [35336/35336]
chmod: cannot access `barbut': No such file or directory
sh: ./barbut: No such file or directory
This is like 1 of the 10 trys this hacker from Germany tried to spam my servers with. These bastards seem to trick the server to killing my processes and downloading a Trojan called Kaiten. But based on what I see in the logs, they are unable to run it. Think setting all downloaded files unable to execute in the temp folder was a good choice.
So it’s very much a DDoS attack. It managed to get a service to run under the “apache” username which blocked the ports required by Apache itself. I managed to find the pesky process blocking port 80, killed it, restarted Apache successfully and changed all my passwords. I am puzzled if the file was not run, then how the file managed to kill Apache in the first place.
So if you try to run Apache, you get an error with this message:
httpd start 2>&1for a quick fix to the solution, what you can do is to find the pesky service running by entering the following command as root
ps -aux
Carefully look through all the processes for suspicious ones. Then kill the process using the process id listed by it’s side XXXX
kill -9 XXXX
Then restart Apache
/sbin/service httpd restart
The next thing is to do a complete scan of your server for any residual files lurking. Who knows what could be installed with the Trojan, I will be damned how they got in the first place. Now I have to consider whether to reload all my backups and do a fresh install of the OS on the infected servers.
Mmmm, I guess say the wise saying goes “No server is unhackable”, but I think with every such incident, you just get stronger every time you combat them.
Update: Updated post and removed the hacker’s IP to prevent it from getting indexed here.
Cursed those bloody hackers, two of our client VPS was hacked as well and it was sending DDOS and adding themself to the botnet.
yea, seems that it all set to go at one time.
Hey btw how you check whether your IP is listed on the bot net?
Hmm I am not too sure, that is what the datacenter told us lol