Re: Re: A problem with rapid growth of the transaction log

From: Matthew Byrd [MSFT] (matbyrd_at_online.microsoft.com)
Date: 03/25/04 

Date: Thu, 25 Mar 2004 10:31:04 -0500

Hi Goran,

My understanding of your tool is that it is hitting the calendar of the user
and not the users Free/Busy information? If that is the case then you are
not touching the public folder store in any significant way as long as your
are not making modifications to the calendar itself.

Hope this Helps, 

-- 
Matthew Byrd
Microsoft PSS
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
This posting is provided "AS IS" with no warranties, and confers no rights.
"Goran Kimovski" <goran.kimovski@no.spam.please.atseavus.com> wrote in
message news:uyqOWjZEEHA.3716@TK2MSFTNGP10.phx.gbl...
> Me again :-)
>
> I just found this article: 815769 XADM: Many Transaction Logs Are Created
> When Clients Search for Free/Busy Calendaring Information
> (http://support.microsoft.com/default.aspx?scid=kb;en-us;815769) where it
> basically says that the views are not cached if searching for free/busy
> information and the locales between the client and the server differ ...
>
> Could this be the cause of our headaches?
>
> /Kima
>
> "Goran Kimovski" <goran.kimovski@no.spam.please.atseavus.com> wrote in
> message news:uIWzwKYEEHA.1092@TK2MSFTNGP12.phx.gbl...
> > Hello Matthew,
> >
> > the situation with the transaction logs is becoming more and more
alarming
> > ... now we have a new customer with the same problem, where it's been
> > verified that the problem is not caused by a virus scanning or so by
> hiding
> > the M drive and even stopping the anti-virus software entirely ...
> >
> > We also got some new information that may bring some new light in
> > understanding the whole problem ... the Exchange administrator reports
> that
> > the Public Folder store is growing rapidly too (even though it's not
used
> by
> > the users) and crashes or hangs the Exchange Server after a while ... Is
> CDO
> > using this store for the temporary search table for the view based on
the
> > filter?
> >
> > What is also different at this site is that we're talking about a single
> > Exchange Server with only 40 mailboxes ...
> >
> > Is there maybe a problem in the way we create the filter on the Messages
> > collection in CDO, which makes the CDO to e.g. re-create the search
table
> or
> > at least re-populate it again every time we run the synchronization
cycle
> in
> > our service?
> >
> > Any kind of help is highly appreciated at this point ...
> >
> > Thanks in advance ...
> >
> > /Kima
> >
> > "Goran Kimovski" <goran.kimovski@no.spam.please.atseavus.com> wrote in
> > message news:eGOooJfDEHA.3692@tk2msftngp13.phx.gbl...
> > > Hello Matthew,
> > >
> > > thanks for the e-mail ... it may bring an entirely new light on our
case
> > ...
> > > unfortunately, testing at the customer site is real hard to convey ...
> the
> > > customer is rather angry and the hosting company not exactly willing
to
> > > cooperate ... I'll have to see if something can be done to persuade
them
> > to
> > > verify your finding ...
> > >
> > > Meanwhile, we'll try out your suggestion in our own lab ... I would,
> > though,
> > > like to ask you few more things regarding the problem ...
> > >
> > > You say that you create search tables within the database, using the
> > filter
> > > we set in CDO and if the filter doesn't change (as we're using a date
> > > filter, it doesn't change during the day), than the same search table
is
> > > reused upon the subsequent accesses to the same view. In that case, I
> > would
> > > expect that the log files will be created only at the start-up of our
> > > service or in the first synchronization cycle after midnight (the
> service
> > is
> > > configured to synchronize the calendar every 10 minutes ... or rather,
> to
> > > read the appointments matching the 7 day filter and than sleep for 10
> > > minutes till the next cycle).
> > >
> > > Unfortunately, at this particular customer site, the transaction logs
> are
> > > created constantly while the service synchronizes the appointments (1
or
> 2
> > > files per minute) and than they stop piling up during the idle time of
> the
> > > service ... also, if you simply stop the service, than the creation of
> > > transaction logs stops too and immediately after you start it, the
logs
> > > start coming up immediately ... actually, this is the reason why the
> > hosting
> > > company is unwilling to cooperate as they claim they have their
> > > configuration according to the MS recommendations for the virus
scanning
> > and
> > > backup and they believe our service misbehaves for some reason ...
> > >
> > > The code synchronizing the calendar in our service is rather simple
...
> > I've
> > > taken out the important part (without any error checking and other
> non-CDO
> > > code) to show you how it looks like:
> > >
> > > startDate = Date
> > > endDate = DateAdd("d", 7, startDate)
> > >
> > > Set mapiSession = New MAPI.Session
> > > mapiSession.Logon , , False, True, 0, True, server & vbLf & user
> > >
> > > Set calendarFolder =
> > mapiSession.GetDefaultFolder(CdoDefaultFolderCalendar)
> > > Set appointmentList = calendarFolder.Messages
> > >
> > > Set calendarFilter = appointmentList.Filter
> > > Set appointmentStart = calendarFilter.Fields.Add(CdoPR_START_DATE,
> > endDate)
> > > Set appointmentEnd = calendarFilter.Fields.Add(CdoPR_END_DATE,
> startDate)
> > >
> > > For Each appointment In appointmentList
> > >     ' read the appointment attributes like:
> > >     ' StartTime, EndTime, BusyStatus, AllDayEvent, Sensitivity,
> Location,
> > > Subject, Text
> > > Next appointment
> > >
> > > The code above is run for all the users found in the selected address
> list
> > > (at the particular customer in question, that is the
GlobalAddressList).
> > As
> > > you can see, even if search tables are created in the Exchange
database
> > when
> > > using the filter for the first time, all subsequent cycles will simply
> > reuse
> > > the search table ...
> > >
> > > Do you think something can cause the search tables to be always
> recreated?
> > > If yes, that could explain the constant increase of the transaction
logs
> > > throughout the day ...
> > >
> > > The only things that come to my mind as being different than the rest
of
> > the
> > > customers which are successfully using our service without this
problem
> > are:
> > > - The server we use in the MAPI Logon is always the same, but the
> > mailboxes
> > > are actually stored on 11 different servers (the GlobalAddressList is
> > > replicated among all 12 servers)
> > > - Our service runs on a separate machine and the CDO installed on that
> > > machine was coming from an Exchange 5.5 CD (installed by installing
the
> > > Exchange 5.5 Administrator and also patched with Exchange 5.5. SP4,
> > post-SP4
> > > CDO patch and Exchange 5.5 CDO Patch 2657.55) ... I don't know why 5.5
> was
> > > used, I guess the reseller installing the software had 5.5 CD close at
> > hand
> > > and didn't bother to get a 2000 CD :-) ... however, I don't believe
that
> > > using CDO 5.5.xxxx or CDO 6.0.xxx would make any difference (guess you
> can
> > > tell me if I'm wrong or not)
> > >
> > > The situation is getting rather critical as the customer feels that
> we're
> > > responsible for the problems they're experiencing with their Exchange
> > > Servers, so any help is highly appreciated ...
> > >
> > > /Kima
> > >
> > > "Matthew Byrd [MSFT]" <matbyrd@online.microsoft.com> wrote in message
> > > news:Oeg18QcDEHA.2804@tk2msftngp13.phx.gbl...
> > > > Goran Kimovski,
> > > >
> > > > Here is what I think you have happening with CDO and transaction log
> > file
> > > > generation.  In Exchange anytime we access content with a specific
> view
> > we
> > > > have to create a search table within the database so that this
> > information
> > > > can be presented to the client.  This is done so that subsequent
view
> of
> > > the
> > > > folder with that given view are just simple accesses of the search
> > table.
> > > >
> > > > Depending on the view used these search tables can either be
updated,
> > > > unchanging, or recreated when accessed on a new day.  Given your
> > > description
> > > > of what is happening I am guessing that when the CDO app hits the
> store
> > > and
> > > > requests the 7 day filter that we are creating a search table in the
> > store
> > > > to support this view.
> > > >
> > > > The act of creating this search table and or updating it will cause
> > > > transaction log files to be generated as these are modifications to
> the
> > > > database.
> > > >
> > > > If you want to test this out I recommend the following test
sequence.
> > > >
> > > > 1. Wait until late at night when a minimum number of users are
logged
> > into
> > > > the system.
> > > > 2. Make sure nothing is touching the M: drive (hitting the M drive
is
> a
> > > big
> > > > culprit for creating translogs) Using the following article will
> ensure
> > > that
> > > > nothing is inadvertently hitting the M: Drive.
> > > >
> > > > 305145 HOW TO: Remove the IFS Mapping for Drive M in Exchange 2000
> > Server
> > > > http://support.microsoft.com/?id=305145
> > > >
> > > > 3. Stop the SMTP virtual server thru the Exchange System manager.
This
> > > will
> > > > stop all mailflow.
> > > > 4. Note the current number of transaction log files
> > > > 5. Run the CDO application
> > > > 6. After the application stops running determine the new number of
> > > > transaction log files.
> > > > 7. Start the SMTP virtual server thru the Exchange System manager.
> > > >
> > > > This test sequence will minimize the number of things that will be
> > > touching
> > > > store so that hopefully the only thing is now the CDO application.
> > > >
> > > > Hope this helps,
> > > >
> > > > --
> > > > Matthew Byrd
> > > > Microsoft PSS
> > > >
> > > > When responding to posts, please "Reply to Group" via your
newsreader
> so
> > > > that others may learn and benefit from your issue.
> > > >
> > > > This posting is provided "AS IS" with no warranties, and confers no
> > > rights.
> > > >
> > > > "Goran Kimovski" <goran.kimovski@no.spam.please.atseavus.com> wrote
in
> > > > message news:u%23fAmGSDEHA.2308@tk2msftngp13.phx.gbl...
> > > > > <<<Forward as a new post in hope to draw higher attention from
more
> > > > users>>>
> > > > >
> > > > >
> > > > > Forced to continue this thread as we still have problems ...
> > > > >
> > > > > I've sent the KB articles to the hosting company where the 12
> Exchange
> > > > > Servers are hosted and they removed all of the folders listed in
the
> > MS
> > > > > recommendation (some were indeed scanned by the anti-virus ...
btw,
> > > > they're
> > > > > using GroupShield ... don't know what version). They also checked
> the
> > > > backup
> > > > > routines (they're using Computer Associates software for that) and
> say
> > > > they
> > > > > were set according to the recommendations already. However, they
> > report
> > > no
> > > > > changes in the behavior of the transaction logs, i.e. they still
> > rapidly
> > > > > grow when our service reads the appointments from the user's
> > mailboxes.
> > > It
> > > > > is also noted that the store.exe process, which is usually using
> very
> > > low
> > > > > CPU resources, now runs at 15-20% on average and has peaks as high
> as
> > > > > 90-100% ...
> > > > >
> > > > > Unfortunately I have too many communication layers to the hosting
> > > company,
> > > > > plus they haven't been much trustworthy up to now (and they're
big,
> so
> > > > they
> > > > > can easily convince the customer that they run everything
perfectly
> > and
> > > it
> > > > > is our software that causes all the trouble), so I can't tell if
the
> > > > > anti-virus and backup are really excluding the M drive and
Exchange
> > > > folders
> > > > > on all Exchange servers.
> > > > >
> > > > > I would like to ask you, under the assumption that everything is
> > really
> > > > set
> > > > > properly, if you can think of any other reason why we still have
the
> > > same
> > > > > problem ...
> > > > > We're using CDO to:
> > > > > - logon to the mailboxes,
> > > > > - open the calendar folder,
> > > > > - set a filter to read only 7 days and
> > > > > - retrieve the appointment entries
> > > > > ... without any write/set operation whatsoever ...
> > > > >
> > > > > As I said in my first post in this thread, the installation is set
> so
> > > that
> > > > > mailboxes are distributed over 11 Exchange servers and we access
> them
> > by
> > > > > performing a logon to the 12th, central Exchange Server and then
> using
> > > CDO
> > > > > to retrieve the calendar.
> > > > >
> > > > > Another thing worth noting may be that our service is run on a
> > separate
> > > > > machine, i.e. not on the central Exchange Server and the CDO on
that
> > > > machine
> > > > > is installed from an Exchange 5.5 CD, i.e. CDO.DLL is of version
5.5
> > > (with
> > > > > post-SP4 update). We plan to install a new CDO from Exchange 2000
CD
> > > > > tomorrow just to rule out if this could be of any relation to the
> > > original
> > > > > problem or not, but don't expect any change ...
> > > > >
> > > > > Any input from anyone who can say something to help us find a
> solution
> > > to
> > > > > our problem is highly appreciated ...
> > > > >
> > > > > /Kima
> > > > >
> > > > > "Lanwench [MVP - Exchange]"
> > > > > <lanwench@heybuddy.donotsendme.unsolicitedmail.atyahoo.com> wrote
in
> > > > message
> > > > > news:uq0Gd2PDEHA.1452@TK2MSFTNGP09.phx.gbl...
> > > > > > You're most welcome! Yes, these articles are great.
> > > > > >
> > > > > >
> > > > > > Goran Kimovski wrote:
> > > > > > > Thanks Lanwench, you've been of great help ...
> > > > > > >
> > > > > > > I also found several articles in the MS KB related to virus
> > scanning
> > > > > > > and Exchange ... I guess some of the readers of the newsgroup
> > could
> > > be
> > > > > > > interested to read them (the company hosting the Exchange
Server
> > > > > > > apparently didn't :-) ), so I'll place them in this post:
> > > > > > >
> > > > > > > 245822 XGEN: Recommendations for Troubleshooting an Exchange
> > Server
> > > > > > > Computer with Antivirus Installed
> > > > > > >
(http://support.microsoft.com/default.aspx?scid=kb;en-us;245822)
> > > > > > >
> > > > > > > 299046 XADM: Calendar Items Disappear from User's Folders
> > > > > > >
(http://support.microsoft.com/default.aspx?scid=kb;EN-US;299046)
> > > > > > >
> > > > > > > 298551 XADM: Large Number of Transaction Logs Created
> > > > > > >
(http://support.microsoft.com/default.aspx?scid=kb;EN-US;298551)
> > > > > > >
> > > > > > > 298924 XADM: Do Not Back Up or Scan Exchange 2000 Drive M
> > > > > > >
(http://support.microsoft.com/default.aspx?scid=kb;EN-US;298924)
> > > > > > >
> > > > > > > 328841 XADM: Exchange and Antivirus Software
> > > > > > > (http://support.microsoft.com/?id=328841)
> > > > > > >
> > > > > > > /Kima
> > > > > > >
> > > > > > > "Lanwench [MVP - Exchange]"
> > > > > > > <lanwench@heybuddy.donotsendme.unsolicitedmail.atyahoo.com>
> wrote
> > in
> > > > > > > message news:ehkDCw3CEHA.3748@TK2MSFTNGP11.phx.gbl...
> > > > > > >> Goran Kimovski wrote:
> > > > > > >>> Hello Lanwench,
> > > > > > >>>
> > > > > > >>> unfortunately I can't answer your question as the servers
are
> > > hosted
> > > > > > >>> by a third party service provider and the information about
> the
> > > > > > >>> mailbox replication is something I've got through them ...
> > > > > > >>>
> > > > > > >>> However, it seems that the problem was actually related to
an
> > > > > > >>> anti-virus program scanning the M: partition ... we could
> easily
> > > > > > >>> reproduce the same behaviour on a simple installation of a
> > single
> > > > > > >>> Exchange 2000 Server with dozen of mailboxes ... by running
> > Norton
> > > > > > >>> Corporate Anti-virus on the M: partition and starting our
> > service,
> > > > > > >>> the transaction log files started to pile up ... this seems
to
> > > slow
> > > > > > >>> down the service too, actually ... don't know if the problem
> > makes
> > > > > > >>> the read operation take more time or something ... we didn't
> > > > > > >>> investigate much further since it seems we now have what to
> look
> > > for
> > > > > > >>> at the customer site ...
> > > > > > >>>
> > > > > > >>> One thing regarding this ... do you recon it is safe to
> exclude
> > > the
> > > > > > >>> M: partition and/or other folders from virus scaning? I'm
not
> > sure
> > > > > > >>> it'll be acceptable for the service provider hosting the
> > Exchange
> > > > > > >>> Servers to do that, though, so if you have any other
> > suggestions,
> > > > > > >>> please let me know them ...
> > > > > > >>
> > > > > > >> You must *always* exclude the M drive and all Exchange
folders
> > from
> > > > > > >> backup and file-level virus scanning. It can cause major
> > problems.
> > > > > > >>
> > > > > > >>
> > > > > > >>>
> > > > > > >>> /Kima
> > > > > > >>>
> > > > > > >>> "Lanwench [MVP - Exchange]"
> > > > > > >>> <lanwench@heybuddy.donotsendme.unsolicitedmail.atyahoo.com>
> > wrote
> > > in
> > > > > > >>> message news:%23CL7Qw2CEHA.1544@TK2MSFTNGP09.phx.gbl...
> > > > > > >>>> Goran Kimovski wrote:
> > > > > > >>>>> Hello Mark,
> > > > > > >>>>>
> > > > > > >>>>> the service can't be more simple ... it simply is
scheduled
> to
> > > go
> > > > > > >>>>> through all accounts in the global address list, open the
> > > > > > >>>>> mailboxes for each account and read the appointments
> scheduled
> > > > > > >>>>> for today and the coming 7 days ... it does not write
> anything
> > > in
> > > > > > >>>>> the mailboxes nor anywhere else ... Since it is purely
> > > CDO-based,
> > > > > > >>>>> the only possible way that something is written is by the
> CDO
> > > > > > >>>>> itself, but I highly doubt that ...
> > > > > > >>>>>
> > > > > > >>>>> What made me suspect it could be so that some temporal
> > > replication
> > > > > > >>>>> is performed is that we have another customer with 6
> Exchange
> > > > > > >>>>> Servers and total of 600 mailboxes on them, where the
> > mailboxes
> > > > > > >>>>> are synchronised to the central server (don't aks me why
> :-) )
> > > > > > >>>>> and we're reading the appointments from the central server
> > using
> > > > > > >>>>> the same service, but we do not have any problems with the
> > > > > > >>>>> transaction log there ... As far as we can see, the only
> > > > > > >>>>> difference between the two customers is the fact that the
> > > > > > >>>>> mailboxes are replicated to the central server at one
> > customer,
> > > > > > >>>>> while not at the other ...
> > > > > > >>>>
> > > > > > >>>> How are mailboxes being replicated? Nothing native to
> > Exchange/AD
> > > > > > >>>> does this.
> > > > > > >>>>
> > > > > > >>>>>
> > > > > > >>>>> /Kima
> > > > > > >>>>>
> > > > > > >>>>> "Mark Arnold [MVP]" <mark@mvps.org> wrote in message
> > > > > > >>>>> news:b1fc01c40781$6fd25f10$a401280a@phx.gbl...
> > > > > > >>>>>> Nothing is making Exchange replicate mailbox information
> > > > > > >>>>>> anywhere. The creation of the log files is possibly due
> > > > > > >>>>>> to your application extracting data from and writing to
> > > > > > >>>>>> the mailboxes some form of data or flag etc. Would that
> > > > > > >>>>>> make sense based on your knowledge of your application?
> > > > > > >>>>>> If your app just read some data then it wouldn't create
> > > > > > >>>>>> so many logs.
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>> -----Original Message-----
> > > > > > >>>>>>> Hello folks,
> > > > > > >>>>>>>
> > > > > > >>>>>>> My company produces a service that connects to an
> > > > > > >>>>>> Exchange Server using CDO
> > > > > > >>>>>>> to collect users from a selected address list and
> > > > > > >>>>>> appointments from a 7 day
> > > > > > >>>>>>> period for the user's mailboxes. The service has been
> > > > > > >>>>>> working fine on more
> > > > > > >>>>>>> than 30 customer sites, both connecting to Exchange 5.5
> > > > > > >>>>>> and Exchange 2000
> > > > > > >>>>>>> without any problems. However, now we have a customer
> > > > > > >>>>>> that runs 12 Exchange
> > > > > > >>>>>>> Servers in a network, where one central server is
> > > > > > >>>>>> replicating the address
> > > > > > >>>>>>> lists from all local servers, but the user's mailboxes
> > > > > > >>>>>> are not replicated to
> > > > > > >>>>>>> it. It's been reported to us that when our service runs
> > > > > > >>>>>> the synchronisation
> > > > > > >>>>>>> cycle, i.e. reads the appointments from the mailboxes
> > > > > > >>>>>> (around 250 in total),
> > > > > > >>>>>>> the transaction log increases rapidly.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I know that by performing regular online backups the log
> > > > > > >>>>>> files from the
> > > > > > >>>>>>> MDBDATA folder get purged and all that stuff, but the
> > > > > > >>>>>> question is not how to
> > > > > > >>>>>>> workaround the problem, but rather why reading
> > > > > > >>>>>> appointments causes the
> > > > > > >>>>>>> transaction log to increase?
> > > > > > >>>>>>>
> > > > > > >>>>>>> We're suspecting that the fact that the mailboxes are
> > > > > > >>>>>> found on the local
> > > > > > >>>>>>> servers, but we're accessing them through the central
> > > > > > >>>>>> server is making
> > > > > > >>>>>>> Exchange to do some temporal replication of the mailbox
> > > > > > >>>>>> our service is
> > > > > > >>>>>>> opening, which is seen as a transaction and placed in
> > > > > > >>>>>> the log.
> > > > > > >>>>>>>
> > > > > > >>>>>>> I would like to hear if anyone can confirm our suspicion
> > > > > > >>>>>> and have any
> > > > > > >>>>>>> suggestions?
> > > > > > >>>>>>>
> > > > > > >>>>>>> Btw, we have a customer with 6 Exchange Servers and 600
> > > > > > >>>>>> maiboxes where we
> > > > > > >>>>>>> again connect to a central server, but they replicate
> > > > > > >>>>>> the mailboxes to the
> > > > > > >>>>>>> central server too and we do not see this problem
> > > > > > >>>>>> there ...
> > > > > > >>>>>>>
> > > > > > >>>>>>> /Kima
> > > > > > >>>>>>>
> > > > > > >>>>>>> Goran Kimovski
> > > > > > >>>>>>> Chief Technical Officer
> > > > > > >>>>>>> Seavus Group
> > > > > > >>>>>>>
> > > > > > >>>>>>> www.seavus.com
> > > > > > >>>>>>>
> > > > > > >>>>>>> "Turning your visions into technical solutions and
> > > > > > >>>>>> products."
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> .
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>