Re: Re: A problem with rapid growth of the transaction log

From: Goran Kimovski (goran.kimovski_at_no.spam.please.atseavus.com)
Date: 03/24/04 

Date: Wed, 24 Mar 2004 10:31:32 +0100

Hello Matthew,

the situation with the transaction logs is becoming more and more alarming
... now we have a new customer with the same problem, where it's been
verified that the problem is not caused by a virus scanning or so by hiding
the M drive and even stopping the anti-virus software entirely ...

We also got some new information that may bring some new light in
understanding the whole problem ... the Exchange administrator reports that
the Public Folder store is growing rapidly too (even though it's not used by
the users) and crashes or hangs the Exchange Server after a while ... Is CDO
using this store for the temporary search table for the view based on the
filter?

What is also different at this site is that we're talking about a single
Exchange Server with only 40 mailboxes ...

Is there maybe a problem in the way we create the filter on the Messages
collection in CDO, which makes the CDO to e.g. re-create the search table or
at least re-populate it again every time we run the synchronization cycle in
our service?

Any kind of help is highly appreciated at this point ...

Thanks in advance ...

/Kima

"Goran Kimovski" <goran.kimovski@no.spam.please.atseavus.com> wrote in
message news:eGOooJfDEHA.3692@tk2msftngp13.phx.gbl...
> Hello Matthew,
>
> thanks for the e-mail ... it may bring an entirely new light on our case
...
> unfortunately, testing at the customer site is real hard to convey ... the
> customer is rather angry and the hosting company not exactly willing to
> cooperate ... I'll have to see if something can be done to persuade them
to
> verify your finding ...
>
> Meanwhile, we'll try out your suggestion in our own lab ... I would,
though,
> like to ask you few more things regarding the problem ...
>
> You say that you create search tables within the database, using the
filter
> we set in CDO and if the filter doesn't change (as we're using a date
> filter, it doesn't change during the day), than the same search table is
> reused upon the subsequent accesses to the same view. In that case, I
would
> expect that the log files will be created only at the start-up of our
> service or in the first synchronization cycle after midnight (the service
is
> configured to synchronize the calendar every 10 minutes ... or rather, to
> read the appointments matching the 7 day filter and than sleep for 10
> minutes till the next cycle).
>
> Unfortunately, at this particular customer site, the transaction logs are
> created constantly while the service synchronizes the appointments (1 or 2
> files per minute) and than they stop piling up during the idle time of the
> service ... also, if you simply stop the service, than the creation of
> transaction logs stops too and immediately after you start it, the logs
> start coming up immediately ... actually, this is the reason why the
hosting
> company is unwilling to cooperate as they claim they have their
> configuration according to the MS recommendations for the virus scanning
and
> backup and they believe our service misbehaves for some reason ...
>
> The code synchronizing the calendar in our service is rather simple ...
I've
> taken out the important part (without any error checking and other non-CDO
> code) to show you how it looks like:
>
> startDate = Date
> endDate = DateAdd("d", 7, startDate)
>
> Set mapiSession = New MAPI.Session
> mapiSession.Logon , , False, True, 0, True, server & vbLf & user
>
> Set calendarFolder =
mapiSession.GetDefaultFolder(CdoDefaultFolderCalendar)
> Set appointmentList = calendarFolder.Messages
>
> Set calendarFilter = appointmentList.Filter
> Set appointmentStart = calendarFilter.Fields.Add(CdoPR_START_DATE,
endDate)
> Set appointmentEnd = calendarFilter.Fields.Add(CdoPR_END_DATE, startDate)
>
> For Each appointment In appointmentList
> ' read the appointment attributes like:
> ' StartTime, EndTime, BusyStatus, AllDayEvent, Sensitivity, Location,
> Subject, Text
> Next appointment
>
> The code above is run for all the users found in the selected address list
> (at the particular customer in question, that is the GlobalAddressList).
As
> you can see, even if search tables are created in the Exchange database
when
> using the filter for the first time, all subsequent cycles will simply
reuse
> the search table ...
>
> Do you think something can cause the search tables to be always recreated?
> If yes, that could explain the constant increase of the transaction logs
> throughout the day ...
>
> The only things that come to my mind as being different than the rest of
the
> customers which are successfully using our service without this problem
are:
> - The server we use in the MAPI Logon is always the same, but the
mailboxes
> are actually stored on 11 different servers (the GlobalAddressList is
> replicated among all 12 servers)
> - Our service runs on a separate machine and the CDO installed on that
> machine was coming from an Exchange 5.5 CD (installed by installing the
> Exchange 5.5 Administrator and also patched with Exchange 5.5. SP4,
post-SP4
> CDO patch and Exchange 5.5 CDO Patch 2657.55) ... I don't know why 5.5 was
> used, I guess the reseller installing the software had 5.5 CD close at
hand
> and didn't bother to get a 2000 CD :-) ... however, I don't believe that
> using CDO 5.5.xxxx or CDO 6.0.xxx would make any difference (guess you can
> tell me if I'm wrong or not)
>
> The situation is getting rather critical as the customer feels that we're
> responsible for the problems they're experiencing with their Exchange
> Servers, so any help is highly appreciated ...
>
> /Kima
>
> "Matthew Byrd [MSFT]" <matbyrd@online.microsoft.com> wrote in message
> news:Oeg18QcDEHA.2804@tk2msftngp13.phx.gbl...
> > Goran Kimovski,
> >
> > Here is what I think you have happening with CDO and transaction log
file
> > generation. In Exchange anytime we access content with a specific view
we
> > have to create a search table within the database so that this
information
> > can be presented to the client. This is done so that subsequent view of
> the
> > folder with that given view are just simple accesses of the search
table.
> >
> > Depending on the view used these search tables can either be updated,
> > unchanging, or recreated when accessed on a new day. Given your
> description
> > of what is happening I am guessing that when the CDO app hits the store
> and
> > requests the 7 day filter that we are creating a search table in the
store
> > to support this view.
> >
> > The act of creating this search table and or updating it will cause
> > transaction log files to be generated as these are modifications to the
> > database.
> >
> > If you want to test this out I recommend the following test sequence.
> >
> > 1. Wait until late at night when a minimum number of users are logged
into
> > the system.
> > 2. Make sure nothing is touching the M: drive (hitting the M drive is a
> big
> > culprit for creating translogs) Using the following article will ensure
> that
> > nothing is inadvertently hitting the M: Drive.
> >
> > 305145 HOW TO: Remove the IFS Mapping for Drive M in Exchange 2000
Server
> > http://support.microsoft.com/?id=305145
> >
> > 3. Stop the SMTP virtual server thru the Exchange System manager. This
> will
> > stop all mailflow.
> > 4. Note the current number of transaction log files
> > 5. Run the CDO application
> > 6. After the application stops running determine the new number of
> > transaction log files.
> > 7. Start the SMTP virtual server thru the Exchange System manager.
> >
> > This test sequence will minimize the number of things that will be
> touching
> > store so that hopefully the only thing is now the CDO application.
> >
> > Hope this helps,
> >
> > --
> > Matthew Byrd
> > Microsoft PSS
> >
> > When responding to posts, please "Reply to Group" via your newsreader so
> > that others may learn and benefit from your issue.
> >
> > This posting is provided "AS IS" with no warranties, and confers no
> rights.
> >
> > "Goran Kimovski" <goran.kimovski@no.spam.please.atseavus.com> wrote in
> > message news:u%23fAmGSDEHA.2308@tk2msftngp13.phx.gbl...
> > > <<<Forward as a new post in hope to draw higher attention from more
> > users>>>
> > >
> > >
> > > Forced to continue this thread as we still have problems ...
> > >
> > > I've sent the KB articles to the hosting company where the 12 Exchange
> > > Servers are hosted and they removed all of the folders listed in the
MS
> > > recommendation (some were indeed scanned by the anti-virus ... btw,
> > they're
> > > using GroupShield ... don't know what version). They also checked the
> > backup
> > > routines (they're using Computer Associates software for that) and say
> > they
> > > were set according to the recommendations already. However, they
report
> no
> > > changes in the behavior of the transaction logs, i.e. they still
rapidly
> > > grow when our service reads the appointments from the user's
mailboxes.
> It
> > > is also noted that the store.exe process, which is usually using very
> low
> > > CPU resources, now runs at 15-20% on average and has peaks as high as
> > > 90-100% ...
> > >
> > > Unfortunately I have too many communication layers to the hosting
> company,
> > > plus they haven't been much trustworthy up to now (and they're big, so
> > they
> > > can easily convince the customer that they run everything perfectly
and
> it
> > > is our software that causes all the trouble), so I can't tell if the
> > > anti-virus and backup are really excluding the M drive and Exchange
> > folders
> > > on all Exchange servers.
> > >
> > > I would like to ask you, under the assumption that everything is
really
> > set
> > > properly, if you can think of any other reason why we still have the
> same
> > > problem ...
> > > We're using CDO to:
> > > - logon to the mailboxes,
> > > - open the calendar folder,
> > > - set a filter to read only 7 days and
> > > - retrieve the appointment entries
> > > ... without any write/set operation whatsoever ...
> > >
> > > As I said in my first post in this thread, the installation is set so
> that
> > > mailboxes are distributed over 11 Exchange servers and we access them
by
> > > performing a logon to the 12th, central Exchange Server and then using
> CDO
> > > to retrieve the calendar.
> > >
> > > Another thing worth noting may be that our service is run on a
separate
> > > machine, i.e. not on the central Exchange Server and the CDO on that
> > machine
> > > is installed from an Exchange 5.5 CD, i.e. CDO.DLL is of version 5.5
> (with
> > > post-SP4 update). We plan to install a new CDO from Exchange 2000 CD
> > > tomorrow just to rule out if this could be of any relation to the
> original
> > > problem or not, but don't expect any change ...
> > >
> > > Any input from anyone who can say something to help us find a solution
> to
> > > our problem is highly appreciated ...
> > >
> > > /Kima
> > >
> > > "Lanwench [MVP - Exchange]"
> > > <lanwench@heybuddy.donotsendme.unsolicitedmail.atyahoo.com> wrote in
> > message
> > > news:uq0Gd2PDEHA.1452@TK2MSFTNGP09.phx.gbl...
> > > > You're most welcome! Yes, these articles are great.
> > > >
> > > >
> > > > Goran Kimovski wrote:
> > > > > Thanks Lanwench, you've been of great help ...
> > > > >
> > > > > I also found several articles in the MS KB related to virus
scanning
> > > > > and Exchange ... I guess some of the readers of the newsgroup
could
> be
> > > > > interested to read them (the company hosting the Exchange Server
> > > > > apparently didn't :-) ), so I'll place them in this post:
> > > > >
> > > > > 245822 XGEN: Recommendations for Troubleshooting an Exchange
Server
> > > > > Computer with Antivirus Installed
> > > > > (http://support.microsoft.com/default.aspx?scid=kb;en-us;245822)
> > > > >
> > > > > 299046 XADM: Calendar Items Disappear from User's Folders
> > > > > (http://support.microsoft.com/default.aspx?scid=kb;EN-US;299046)
> > > > >
> > > > > 298551 XADM: Large Number of Transaction Logs Created
> > > > > (http://support.microsoft.com/default.aspx?scid=kb;EN-US;298551)
> > > > >
> > > > > 298924 XADM: Do Not Back Up or Scan Exchange 2000 Drive M
> > > > > (http://support.microsoft.com/default.aspx?scid=kb;EN-US;298924)
> > > > >
> > > > > 328841 XADM: Exchange and Antivirus Software
> > > > > (http://support.microsoft.com/?id=328841)
> > > > >
> > > > > /Kima
> > > > >
> > > > > "Lanwench [MVP - Exchange]"
> > > > > <lanwench@heybuddy.donotsendme.unsolicitedmail.atyahoo.com> wrote
in
> > > > > message news:ehkDCw3CEHA.3748@TK2MSFTNGP11.phx.gbl...
> > > > >> Goran Kimovski wrote:
> > > > >>> Hello Lanwench,
> > > > >>>
> > > > >>> unfortunately I can't answer your question as the servers are
> hosted
> > > > >>> by a third party service provider and the information about the
> > > > >>> mailbox replication is something I've got through them ...
> > > > >>>
> > > > >>> However, it seems that the problem was actually related to an
> > > > >>> anti-virus program scanning the M: partition ... we could easily
> > > > >>> reproduce the same behaviour on a simple installation of a
single
> > > > >>> Exchange 2000 Server with dozen of mailboxes ... by running
Norton
> > > > >>> Corporate Anti-virus on the M: partition and starting our
service,
> > > > >>> the transaction log files started to pile up ... this seems to
> slow
> > > > >>> down the service too, actually ... don't know if the problem
makes
> > > > >>> the read operation take more time or something ... we didn't
> > > > >>> investigate much further since it seems we now have what to look
> for
> > > > >>> at the customer site ...
> > > > >>>
> > > > >>> One thing regarding this ... do you recon it is safe to exclude
> the
> > > > >>> M: partition and/or other folders from virus scaning? I'm not
sure
> > > > >>> it'll be acceptable for the service provider hosting the
Exchange
> > > > >>> Servers to do that, though, so if you have any other
suggestions,
> > > > >>> please let me know them ...
> > > > >>
> > > > >> You must *always* exclude the M drive and all Exchange folders
from
> > > > >> backup and file-level virus scanning. It can cause major
problems.
> > > > >>
> > > > >>
> > > > >>>
> > > > >>> /Kima
> > > > >>>
> > > > >>> "Lanwench [MVP - Exchange]"
> > > > >>> <lanwench@heybuddy.donotsendme.unsolicitedmail.atyahoo.com>
wrote
> in
> > > > >>> message news:%23CL7Qw2CEHA.1544@TK2MSFTNGP09.phx.gbl...
> > > > >>>> Goran Kimovski wrote:
> > > > >>>>> Hello Mark,
> > > > >>>>>
> > > > >>>>> the service can't be more simple ... it simply is scheduled to
> go
> > > > >>>>> through all accounts in the global address list, open the
> > > > >>>>> mailboxes for each account and read the appointments scheduled
> > > > >>>>> for today and the coming 7 days ... it does not write anything
> in
> > > > >>>>> the mailboxes nor anywhere else ... Since it is purely
> CDO-based,
> > > > >>>>> the only possible way that something is written is by the CDO
> > > > >>>>> itself, but I highly doubt that ...
> > > > >>>>>
> > > > >>>>> What made me suspect it could be so that some temporal
> replication
> > > > >>>>> is performed is that we have another customer with 6 Exchange
> > > > >>>>> Servers and total of 600 mailboxes on them, where the
mailboxes
> > > > >>>>> are synchronised to the central server (don't aks me why :-) )
> > > > >>>>> and we're reading the appointments from the central server
using
> > > > >>>>> the same service, but we do not have any problems with the
> > > > >>>>> transaction log there ... As far as we can see, the only
> > > > >>>>> difference between the two customers is the fact that the
> > > > >>>>> mailboxes are replicated to the central server at one
customer,
> > > > >>>>> while not at the other ...
> > > > >>>>
> > > > >>>> How are mailboxes being replicated? Nothing native to
Exchange/AD
> > > > >>>> does this.
> > > > >>>>
> > > > >>>>>
> > > > >>>>> /Kima
> > > > >>>>>
> > > > >>>>> "Mark Arnold [MVP]" <mark@mvps.org> wrote in message
> > > > >>>>> news:b1fc01c40781$6fd25f10$a401280a@phx.gbl...
> > > > >>>>>> Nothing is making Exchange replicate mailbox information
> > > > >>>>>> anywhere. The creation of the log files is possibly due
> > > > >>>>>> to your application extracting data from and writing to
> > > > >>>>>> the mailboxes some form of data or flag etc. Would that
> > > > >>>>>> make sense based on your knowledge of your application?
> > > > >>>>>> If your app just read some data then it wouldn't create
> > > > >>>>>> so many logs.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>> -----Original Message-----
> > > > >>>>>>> Hello folks,
> > > > >>>>>>>
> > > > >>>>>>> My company produces a service that connects to an
> > > > >>>>>> Exchange Server using CDO
> > > > >>>>>>> to collect users from a selected address list and
> > > > >>>>>> appointments from a 7 day
> > > > >>>>>>> period for the user's mailboxes. The service has been
> > > > >>>>>> working fine on more
> > > > >>>>>>> than 30 customer sites, both connecting to Exchange 5.5
> > > > >>>>>> and Exchange 2000
> > > > >>>>>>> without any problems. However, now we have a customer
> > > > >>>>>> that runs 12 Exchange
> > > > >>>>>>> Servers in a network, where one central server is
> > > > >>>>>> replicating the address
> > > > >>>>>>> lists from all local servers, but the user's mailboxes
> > > > >>>>>> are not replicated to
> > > > >>>>>>> it. It's been reported to us that when our service runs
> > > > >>>>>> the synchronisation
> > > > >>>>>>> cycle, i.e. reads the appointments from the mailboxes
> > > > >>>>>> (around 250 in total),
> > > > >>>>>>> the transaction log increases rapidly.
> > > > >>>>>>>
> > > > >>>>>>> I know that by performing regular online backups the log
> > > > >>>>>> files from the
> > > > >>>>>>> MDBDATA folder get purged and all that stuff, but the
> > > > >>>>>> question is not how to
> > > > >>>>>>> workaround the problem, but rather why reading
> > > > >>>>>> appointments causes the
> > > > >>>>>>> transaction log to increase?
> > > > >>>>>>>
> > > > >>>>>>> We're suspecting that the fact that the mailboxes are
> > > > >>>>>> found on the local
> > > > >>>>>>> servers, but we're accessing them through the central
> > > > >>>>>> server is making
> > > > >>>>>>> Exchange to do some temporal replication of the mailbox
> > > > >>>>>> our service is
> > > > >>>>>>> opening, which is seen as a transaction and placed in
> > > > >>>>>> the log.
> > > > >>>>>>>
> > > > >>>>>>> I would like to hear if anyone can confirm our suspicion
> > > > >>>>>> and have any
> > > > >>>>>>> suggestions?
> > > > >>>>>>>
> > > > >>>>>>> Btw, we have a customer with 6 Exchange Servers and 600
> > > > >>>>>> maiboxes where we
> > > > >>>>>>> again connect to a central server, but they replicate
> > > > >>>>>> the mailboxes to the
> > > > >>>>>>> central server too and we do not see this problem
> > > > >>>>>> there ...
> > > > >>>>>>>
> > > > >>>>>>> /Kima
> > > > >>>>>>>
> > > > >>>>>>> Goran Kimovski
> > > > >>>>>>> Chief Technical Officer
> > > > >>>>>>> Seavus Group
> > > > >>>>>>>
> > > > >>>>>>> www.seavus.com
> > > > >>>>>>>
> > > > >>>>>>> "Turning your visions into technical solutions and
> > > > >>>>>> products."
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> .
> > > >
> > > >
> > >
> > >
> > >
> >
> >
>
>

Loading