Re: what is the best way to read text file

From: Alvin Bruney [MVP] (vapor)
Date: 02/14/05 

Date: Mon, 14 Feb 2005 13:47:30 -0500

logfiles follow a different formats depending on the various supported
standards.

If you are looking to parse the log file for errors or just security
concerns, you should instead resort to Microsoft's log parser utility which
ships free as an IIS resource download instead of re-inventing the wheel to
implement your own functionality. It contains the ability to parse logfiles
by firing legitimate sql queries against the raw text file.

It's quite a remarkable piece of machinery actually. The log parser utility
is a command line tool with no GUI interface so you have to learn all the
nuts and bolts. But i've built a web GUI front end that sits on top of
logparser available on www.logparser.com. Serverstats, which is open source
by the way, eliminates the need for the user to learn sql queries and the
nuts and bolts of the logparser utility which can be a bit intimidating. You
can point and click your way thru any w3c extended log file format. 

-- 
Regards,
Alvin Bruney [Microsoft MVP ASP.NET]
[Shameless Author plug]
The Microsoft Office Web Components Black Book with .NET
Now Available @ http://www.lulu.com/owc
----------------------------------------------------------
"Paul Clement" <UseAdddressAtEndofMessage@swspectrum.com> wrote in message 
news:mgk1119p5tq0nfvtl8gsscgpdeppsnt581@4ax.com...
> On Mon, 14 Feb 2005 04:53:31 -0600, "Guoqi Zheng"<no@sorry.com> wrote:
>
> ¤ Dear sir/lady,
> ¤
> ¤ I need to process our web server log file every day. Normally, this file 
> is about 50MB - 100s MB big. How can I read the file in a effective way.
> ¤
> ¤ I can open a streamreader to read it and use readline() function to read 
> line by line and process them. However, this sounds very slow. I can also 
> use readtoend to read everything into a string, but then I am afraid that 
> it will be too big for memory.
> ¤
> ¤ How should I read the file?
>
> Don't think I've ever seen the contents of this file. Does it have any 
> type of structure? Does it
> use field delimiters?
>
>
> Paul ~~~ pclement@ameritech.net
> Microsoft MVP (Visual Basic)

Relevant Pages

  • Re: what is the best way to read text file
    ... logfiles follow a different formats depending on the various supported ... If you are looking to parse the log file for errors or just security ... It contains the ability to parse logfiles ...
    (microsoft.public.dotnet.framework)
  • [ANN] Apache::LogRegex 1.1 minor bugfix release
    ... Apache::LogRegex - Parse a line from an Apache logfile into a hash ... Designed as a simple class to parse Apache log files. ... The field names of the hash are derived from the log file format. ...
    (comp.lang.perl.modules)
  • Re: Logrotate is a pain
    ... >>important feature is that the program, which writes the logfiles does ... > to have missed is that if you rename an open log file, ... > writing to it continues writing to it regardless of the name. ... You can simply put a simple line in it to send SIGHUP to your daemon and the ...
    (comp.os.linux.misc)
  • Re: Parsing a odd log file format
    ... like to understand how to deal with the part of my log file that is ... of my goal is to understand the language not just parse logs :-) ... once, so you should be able to solve it again, of learning the grammar ... Parse all the lines as csv, and the ones where you don't have the ...
    (comp.lang.lisp)
  • ANN: Apache::LogRegex 1.2
    ... Designed as a simple class to parse Apache log files. ... lines from the log file line by line returning a hash of each line. ...
    (comp.lang.perl.modules)