
Unless you have no logging enabled, the first time you run Logcheck you will
almost certainly get an GIGANTIC e-mail message with thousands of log
entries. While this may seem like a bad thing, it's not.

What happened is that logcheck when through your entire accumulated logs at
once, and told you about everything it found.  If you look at the logs by
day or hour, there are probably not *that* many entries.  It's just the
entire batch at once that kills you.

Also, you may have 20 or 30 thousand "license violation," bad blocks on a
device, synchronization failures, etc.  These are things you want to know
about -- you just don't want to see all 30 thousand entries.

There are a few ways to deal with this. The first and most simply way is to
just ignore it. Delete the message, and look forward to your NEXT message,
which will be a reasonable size. If it's not a reasonable size, then you
have something going on in your environment that I did not tune the keyword
files for. You'll have to tune then yourself. See Craig's "INSTALL" and
"README.keywords" files for information about what each file is, and how to
tune them a bit.

The second way to deal with it is to save the message out and try to use
Excel or Vernon D. Buerg's "list.com" to look at it. The problem here is
that you'll need to filter out the 30 thousand "license violation" messages
to see the few that are important. This is assuming you have fixed the
license violation, of course!  You can do search for and replace with
nothing the messages you don't want to see anymore, but there is a better
way.

This requires more UNIX tools and the use of the command line, but it's the
best way I've found to deal with huge "startup" log problems.  First you
export the message to text (and hope REALLY, REALLY hard that your mailer
does not insert spurious line breaks).  Then you start "grepping" the file.
You already have egrep.exe, as logcheck.sh depends on it.  But it's probably
not in your path.

Anyway, use "egrep --help | more" to see the usage information.  The two
most useful options here are -c and -v.  -c simply counts the occurrences --
so you can really say, "There are 30 thousand license violation messages."
-v inVerts or reverses grep's usual behavior.  So you can filter out message
you don't want to see by something like:
	grep -v "something I don't want to see" MyGiganticEMail.txt > MySmallerEMail.txt

This is, in part, how logcheck.sh works.  You can get even more tricky and
filter out a bunch of entries you don't want to see using the -f argument.
This is exactly what I did when scanning log files to do the initial tune on
the keyword files.  I'd start out with a 13 Gig log file (all Event Logs
exported to text and merged into one file), then I'd end up with 12 Gig of
text after logcheck.sh finished.  It turned out that a small number of
things would be in the logs thousands and thousands of times.

So, it seems like the easy thing would be to add those things to one of the
"ignore" files and call it a day, right?  Except that this list of things is
comprised of events that you WANT to know about!  All the things we don't
care about are ALREADY in the ignore files.

Here's the list. There are some things you could argue you don't care about.
Fine, add them to the ignore file. But what about "Logon Failure?"

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The partial synchronization replication of the.*database from the  primary domain controller  failed with the following error
The partial synchronization replication of the.*database from the primary domain controller failed with the following error
Logon Failure:
The device, .*, has a bad block
server announcement from the computer.*that believes that it is the master browser for the domain
The server could not expand a table because the table reached the maximum size
Norton AntiVirus.*Scan could not open file C:\\
Norton AntiVirus.*Could not scan.*files inside
Norton AntiVirus.*Scan Complete: Viruses:0   Infected:0   Scanned:
Norton AntiVirus.*Scan started on all drives and all extensions
Norton AntiVirus.*Virus definitions are current
TimeServ.*Time set \(offset
Success
LicenseService
Security,627,Failure,Account Management
Security,577,Failure,Privilege Use
The session setup from the computer
NIC Agents,515,Warning,Drive Array
The server received an incorrectly formatted request from
Otman4,.,Information,None
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When I created the above list and called it jpfilter.txt, then ran the
following "post processing" command, my 12 Gig file went down to 100 K.

	egrep -v -f jpfilter.txt message.pre > message.txt

The problem, as I mentioned, is that I really DO want to know about all the
bad stuff that's been happening on my network that I didn't know about.  I
can't add the above list to the ignore file because it contains too many
things I want to know about -- I just want to know a few at a time.


So, what else can we do. Well, go out and get the rest of the UNIX tools I
mention in README-Win-Tools.txt, specifically
http://www.weihenstephan.de/~syring/win32/UnxUtils.html.  then you can do
things like the following, where "message.txt" is a text file containing a
(large) logcheck e-mail.


Count the number of "Logon Failure:" events (170 in this example):

	egrep -c "Logon Failure:" message.txt
	170
	egrep "Logon Failure:" message.txt | wc -l
	170


Using the DOS sort command, and the UNIX egrep, cut, uniq and wc commands, find
out how many *different* people had logon failures (37).  Leave off the " | wc" to
list them:


	egrep "Logon Failure:" message.txt | cut -d , -f 10- | sort | uniq | wc -l
	37
	egrep "Logon Failure:" message.txt | cut -d , -f 10- | sort | uniq
	...
	Logon Failure:^` 	Reason:		Account locked out^` 	User Name:	xxxxx^` 	Domain:	xxx^` 	Logon Type:	3^` 	Logon Process:	NtLmSsp ^` 	Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0^` 	Workstation Name:	xxxx 
	Logon Failure:^` 	Reason:		Unknown user name or bad password^` 	User Name:	xxxxx^` 	Domain:		xxx^` 	Logon Type:	3^` 	Logon Process:	KSecDD^` 	Authentication Package:	MICROSOFT_AUTHENTICATION_PACKAGE_V1_0^` 	Workstation Name:	\\xxxxxxxx 
	Logon Failure:^`        Reason: The user has not be granted the requested^`        logon type at this machine^`    User Name:      rxxxxx^`         Domain:        xxx^`      Logon Type:     2^`     Logon Process:  IIS     ^`Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0^`         Workstation Name:       xxxxxxxx
	...


Create a new file, without all the "Logon Failure:" events:

	egrep -v "Logon Failure:" message.txt > message.1


Use the "--help" switch to get help for the UNIX commands.  Note that the
syntax for searching (grepping) can be a little odd, since some characters
have special meanings.  If you get odd results, try escaping symbols with
the backslash character ( \ ).  Backslashes have to be doubled to work as
you expect.  The period symbol means "any character" and the asterisk means
zero or more, so the regular expression " .* " means zero or more of any
character.  There are a zillion references to "Regular Expressions" on the
web -- find one you like.

* Cut is used to cut out some fields in a line.
* Wc is "word count" (for those on the Continent -- no, not water closet),
and -l shows the number of lines.
* Uniq shows only lines that are (wait for it) not duplicates.  The file does
need to be sorted.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Last Updated: Tue Sep  4 20:33:44 2001
-- JP Vossen <jp@jpsdomain.org>

