Re: I don't uderstand ISA Logs

From: Joris Dobbelsteen (none.of_at_your.business)
Date: 03/31/04 

Date: Wed, 31 Mar 2004 19:58:15 +0200

This really has not much todo with the ISA server, but rather the way the
HTTP protocol works. When you really need to know this information, you need
to catch clicks in the browser of the user.

Getting this information from a log file would be possible, but it is very
hard, and no log can ever help you much with doing this. At least I'm not
aware of any proxy that can do this, but I'm not interested in such
information, so this doesn't apply to me. In this situatuation you can
probably use any proxy you like, but they won't give you more information,
unless you build a filter for them, which is very tricky, always.

Generally user requests are to pages like:
*.htm
*.html
*.asp
*.aspx
*.php
*.jsp
*.sht
*.shtm
and such, these have usually the text/html mime type.
In case of Exchange, this would probably be *.asp or *.aspx all of the time.

Many files can be filtered out:
*.jpg *.gif are pictures commonly used in a document for pictures, but are
not usually clicks
*.css are style sheets, probably never clicked on.
*.js are java scrips, probably part of a page
....

When the mime type is text/*, it is probably a click (exceptions above).
image/* is usually never a click.
Types like application/* point usually to downloads or activex stuff in a
web page.

You can use timings, as where closely related timing indicate a single page,
commonly within a second or something (depending on the connection).

When you are programming an ISAPI filter, you should try to catch the
"referer" HTTP header, however it might not always be included in the
request. This gives you some information about how the user is navigating.

Analyzing how a user is navigating from somewhere in the middle (proxy) is
very hard to do, and it is a very tricky bussiness. You cannot guarentee to
be correct always! At the end-point is a little less tricky, because you got
more information, but still you cannot guarentee anything.

In the case you are investigating, it would probably be more useful to see
wich actual sites are visited and the nature of these sites. As an manager
of a company I would not be interested in the clicks the user has made,
rather the nature of the web page they are visiting. See the bigger picture
instead of the tiny details.

When I monitor my packet filters I'm interested in seeing:
* which port has many packets dropped
* which IP has many packets dropped (easly identifies attacks)
For the top ports and IPs I would like to know more. I don't care about that
single hit on port xxx or that few dropped packets from IP a.b.c.d...
I would really go back to the customer and see if you cannot develop a much
simpler solution that satisfies their needs.

Besides, I would even consider working for a company that wants to monitor
my e-mail and web traffic. It points out how much trust the company puts in
me. But this part of the consideration is up to your customer (or customer's
employees). However this is severaly besides the point.

Hope this helps a little.

- Joris

"Michele Massari" <mmassari@omniway.sm> wrote in message
news:1623001c41710$46f20c00$a501280a@phx.gbl...
> Sorry.. first I don't reply you about the situation that
> force me to know only the user requestes.
>
> My customer want monitoring the employees activities.
> In this company there are Exchange 2000 and ISA 2000.
> My software must process and combine exchange and ISA
> logs to make some kinds of reports to know all webpage
> visited and all the e-mail sent and received.
>
> I've thought that the right way was to install a Proxy
> Server and analyze the logs... but ISA server have me
> disappointed for the lack of a field in the log files
> that can allow to identify a single user request / click.
>
> You think that there are other solutions at this problem
> instead use a Web Proxy? However i need a server-level
> software because I don't want to install a software on
> each client...
>
> Thank a lot
>
> Michele Massari
> >-----Original Message-----
> >It is not possible, clicks and pictures inside a web
> page are translated
> >into HTTP requests, which look all the same to the ISA
> server. The only
> >thing you can do is guess what site is visited.
> >
> >Usually text/* files are clicked on (some exceptions
> though).
> >Can use time to filter this out. When timings are very
> close, this MIGHT
> >indicate a single page.
> >
> >Complexity increases when:
> >user used multiple browser windows (visits are
> interleaved somewhat)
> >pages require resources from other web sites (such as
> counters and ads,
> >which are the most common).
> >
> >You can do much with SQL to filter something out, but
> you probably need much
> >more complex logic do so.
> >
> >Before starting this job I would rather investigate
> whether you really need
> >to know what links the user clicked. I cannot imagine a
> situation where you
> >want to know this...
> >
> >- Joris
> >
> >"MIchele Massari" <mmassari@omniway.sm> wrote in message
> >news:112cb01c41672$d199b1d0$a001280a@phx.gbl...
> >> Hi,
> >> Overall I'm sorry for my bad english...
> >>
> >> I'm developing an application that analyze ISA server
> >> logs.
> >>
> >> I don't understand how I can group the records in SQL
> to
> >> diplay only actual user 'clicks', not just 'hits' in
> the
> >> log files.
> >> I try to better explain myself:
> >> 1. With the browser i open the url www.isaserver.org,
> >> then I click on Books section and finally i close the
> >> browser.
> >> 2. ISA write a several records... (images, js, css,
> html
> >> in the frames...)
> >> 3. With a SQL Query I want obtain only 2 records as:
> >> User1 2/2/04 15.35 www.isaserver.org 150Kb
> >> User1 2/2/04 15.39 www.isaserver.org/pages/books.asp
> 190Kb
> >>
> >> I hope you have understand my problem...
> >>
> >> Please Help me...
> >>
> >> Michele from Italy
> >
> >
> >.
> >