OK, I've been hacked, now what?

[sedwards () sedwards com]

Yes it's true, one of my client's web page was hacked. The attack
occurred on March 27.

Here's the text of the page he left:

        "This Page was hacked by Homicide :P cause .. I was bored hehe
        well anyways ph34r the Blue Candy Bar and this ones for u Lina (:

There were no graphics on the page.

Here's what I've done so far:

1) Disconnected the host from the net.

2) Removed the disk drives from the host and mounted them "read-only, no
suid" on another host.

3) Changed all root and user passwords on all hosts.

4) Changed all router and switch access and enable passwords.

5) Examined all other hosts for signs of compromise -- none detected.

6) Examined the directories to determine which files were modified during
the attack.

7) Examined the directories to determine which files were accessed during
the attack.

8) Since he deleted the HTTP access and error log files (and linked them
to /dev/null to help cover his tracks), I've reconstructed  the logs
using custom n-th generation tools -- "dd if=/dev/foo | grep 27/Mar"
This was surprisingly successful -- it looks like I'm only missing about
40 minutes of logs from 02:30 to 03:10! Unfortunately, it's the most
important 40 minutes :)

9) Examined all hosts for "well known" lame cgi's.

The attack came from a PPP server run by BBN in Lexington, KY. I've
contacted their NOC and they have indicated a willingness to help.

The host was a Solaris 2.5.1 SPARC running several web sites on Apache
1.3.1. It (Solaris and Apache) probably were not up to current patch
level. Note that one of the CGI's abused below is SGI's infamous "handler"
-- my admins have been admonished not to blindly copy stuff from one host
to another.

Here's the chronology as best as I can reconstruct it. Note that the
hacker deleted the log files and linked them to /dev/null. The log
entries presented below came from scavenging disk blocks from the raw
disk devices. Thus, the 2 scans documented below could be to 2 different
sites on the same host. Also, when a time is show for file activity,
that is the time of last activity -- previous accesses are not recorded.

Time            Action
--------        ------
scan #1
02:24:57        unsuccessfully tries to use phf cgi to execute "ls -lF"
02:24:58        unsuccessfully tries to use faxsurvey cgi to execute "ls -lF"
02:25:01        successfully uses handler cgi to execute "ls -lF /etc"
02:25:06        successfully tries to use webdist.cgi to execute "ls -lF /etc"
02:25:07        unsuccessfully tries to use php.cgi to retrieve "/etc/passwd"
02:25:08        unsuccessfully tries to use view-source to retrieve "/etc/passwd"
02:25:09        unsuccessfully tries to use htmlscript to retrieve "/etc/passwd"
02:25:10        unsuccessfully tries to use campas to execute "ls -lF /etc"
02:25:11        unsuccessfully tries to use info2www to execute "ls -lF /etc"
02:25:12        unsuccessfully tries to use aglimpse to execute "ls -lF /etc"
02:25:12        unsuccessfully tries to use pfdisplay.cgi to execute "ls -lF /etc"
02:25:13        unsuccessfully tries to GET /_vti_pvt/service.pwd

scan #2
02:28:40        unsuccessfully tries to use phf cgi to execute "ls -lF"
02:28:42        unsuccessfully tries to use faxsurvey cgi to execute "ls -lF"
02:28:45        successfully uses handler cgi to execute "ls -lF /etc"
02:28:48        successfully tries to use webdist.cgi to execute "ls -lF /etc"
02:28:51        unsuccessfully tries to use php.cgi to retrieve "/etc/passwd"
02:28:52        unsuccessfully tries to use view-source to retrieve "/etc/passwd"
02:28:54        unsuccessfully tries to use htmlscript to retrieve "/etc/passwd"
02:28:55        unsuccessfully tries to use campas to execute "ls -lF /etc"
02:28:56        unsuccessfully tries to use info2www to execute "ls -lF /etc"
02:28:57        unsuccessfully tries to use aglimpse to execute "ls -lF /etc"
02:28:58        unsuccessfully tries to use pfdisplay.cgi to execute "ls -lF /etc"
02:28:59        unsuccessfully tries to GET /_vti_pvt/service.pwd

02:29:32        successfully uses handler cgi to execute "uname -a"
02:30:52        successfully uses handler cgi to execute "uname -a"

03:10:xx        /d2/www/site1/* is accessed
03:15:xx        /etc/dfs/sharetab is accessed
03:15:xx        /usr/lib/fs/nfs/nfsfind is accessed
03:30:xx        /mnt/d2/www/cgi-bin/hand is accessed
03:32:xx        created file "/d2/www/cgi-bin/.s" which is a list of all
                SUID executables. The file is created nobody:nobody
                indicating that it probably was created by tricking a
                cgi.
03:32:xx        /* is accessed
03:39:xx        created a small SUID/GUID executable file
                "/mnt/usr/bin/sh2" which is owned by root:root and
                contains the string "/bin/sh"
03:42:xx        /etc/passwd and /etc/oshadow are modified
                At some point he edits /etc/shadow and prefixes all of
                the encrypted passwords with "1". He also gives bin a
                password. 
03:52:xx        /d2/www/site1/index.html is modified
03:54:xx        /d2/www/cgi-bin is modified
03:54:31        gets index.html
03:57:53        gets index.html
03:58:03        gets index.html
04:01:xx        /logs is linked to /dev/null
04:03:xx        /.bash_history is linked to /dev/null
04:04:xx        /root/etc/shadow is modified
04:05:xx        /var/adm/messages is cleared.
04:09:xx        /d2/www/site2/index.html is modified
04:09:xx        /d2/www/site3/index.html is modified
04:22:xx        /.sh_history is linked to /dev/null
04:25:xx        /etc/mnttab is edited
04:26:xx        /d2/www/logs is linked to /dev/null
04:34:xx        /d2/logs is linked to /dev/null
04:35:xx        /d2/errors is accessed
04:35:xx        /d2/logs is accessed
04:35:xx        all files are deleted from /d2/errors
04:36:xx        /.bash_history is accessed
04:36:xx        /usr/openwin/bin/xterm is accessed

09:55:10        gets index.html

While the "agent" field in the HTTP access logs says he is running MSIE
4.01 on W98, I suspect an automated tool named "vito" -- the probes are
too close together for him to be entering this text in the "location"
box or selecting "bookmarks."

Here's where I'm asking for help:

1) What should I do now?

2) What should I have done differently?

3) What should I do to reduce the probability of this happening again?

4) What should I do to make detection of a hack easier?

I have some ideas on these questions but I don't want to "steer" the
discussion.

I still don't have the "smoking gun" that says exactly how he got root
access. Opinions and conclusions from the above chronology are welcomed.

Thanks in advance,
------------------------------------------------------------------------
Steve Edwards      sedwards () sedwards com      Voice: +1-760-723-2727 PST
Newline            Pager: +1-760-740-1220           Fax: +1-760-731-3000