Re: Why the following code is faster executed when placed in the form?

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Dmitriy Antonov (antonovdima_at_netzero.net_NOT_FOR_SPAM)
Date: 04/22/04 

Date: Thu, 22 Apr 2004 01:50:55 -0400

First of all, Jack, you don't pay attention to my replies. I told you that
everything that is not changed inside the loop must not be calculated there.
You think doing "Ch" & Chan is not a big deal? Big mistake. Even doing 2+2
should not be done inside the loop. You should calculate it before the loop,
assign to variable and use variable (equal to 4 in this case) in the body of
the loop. Operation "Ch" & Chan is dozens (maybe hundreds) times slower than
2+2, and you do it repeatedly inside the inner loop, while the value of Chan
is changed inside the outer loop. This is just one example. Again, it
doesn't save you 50% of the time, but it is the simplest step toward
optimization.
Another simple thing: you use function Chr(). It should work really quickly,
but it is still a function. Some of the characters you use are predefined as
VB constants. E.g. Chr(13)=vbCr; Chr(10)=vbLf. There is combined constant
vbCrLf. Others can be predefined in the beginning of the procedure as local
variables. This is another small step.

For example, concatenation Chan & Chr(135) & NetID & Chr(135) /don't forget
to substitute Chr(135) with variable/ could be "prebuilt" before inner
loop, as well.

Why it is important? VB-way of working with strings is really convenient.
But it has one big BUT - it is exceptionally slow. You think, that, when you
concatenate something, VB happily append new string to existing one. Nope.
Let's see how VB does aforementioned operation:

Chan & Chr(135) & NetID & Chr(135)

It allocates memory for Chr(135) /in your case you can add call to Chr()/.
Then it allocates new piece of memory and copies Chan and this character
both strings into new one. Then it destroys memory taken for Chr(135),
because it is not a variable. Then it allocates memory for new memory and
puts result of previous operation and content of NetID. Then it allocates
memory for another Chr(135) . Finally result of previous operation is copied
along with this new character into final piece of memory and last 2 bytes
required for Chr(135) are destroyed. It is possible, that I miss some other
steps. Maybe it doesn't destroy those temporary memory chunck immediately
(below you can find couple words about it). But the main idea should be
clear (I hope).

Now imaging if you build extremely big string (like you do), how it works.
The bigger string the more time is required to do it. One interesting
example. Let say we want to concatenate three variables - sBig (100'000
characters) and two small variables sSm1 and sSm2, both are just one
character in length. How would you do it. If you tell me the following is
correct answer:

sNew=sBig & sSm1 & sSm2

then you are wrong.

sNew=sBig & (sSm1 & sSm2)

In the first case, VB allocates memory for 100'001 characters, then copy
first two variables into it, then allocates memory for 100'002 characters
and then copy previous result and the last variable into it.

In the second case, it allocates place for two characters combine what's
inside parentheses and then do the last expensive concatenation. Purists
will tell me that even the second case is not the best, because there are
techniques to further optimize this (see below).

I hope you understand the picture. This is something discussed to death and
you will find enormous amount of information about it on the Internet. Being
the most disorganized person in the world, I can't give you links. You
should find them yourself.

I hope you now understand why building the entire content into one huge
string doesn't work as you expect. And this is exactly, why at the end of
the process it is much slower than at the beginning. The last statement
definitely true in that case of building large string. If you experience the
same thing, when dumping strings line by line, I don't have explanation for
that. May be VB do not clean memory immediately, because it is busy and as a
result all those allocated strings wait its time to be destroyed. In this
case you may go out of your physical memory and Windows starts swapping
everything using slow swap file (which is on HD). I think this is the most
likely, but can't tell you for sure. You can indirectly "see" it, if your
harddrive becomes making those specific sounds of intensive IO operations.
Having that you use HD for building your file, load on your disk is
additionally increased.

One of the best ways to REALLY improve speed is to return to big string
again. But now you should prebuild this big string before adding information
to it. If you know the biggest possible length of the final string, then use
it this way

s=Space(10000000)

Then instead of concatenation you use Mid statement. Do not confuse it with
Mid function. Mid statement allows you to replace characters within existing
string. Which is much-much-much faster. Its syntax

Mid(stringvar, start[, length]) = string

Check out VB help for more info.

At the end just use s=RTrim$(s)

If you don't know upfront (at least approximately) the length of the final
string, then there are techniques to do it dynamically. You can find them on
the Internet. Main idea - you define some big string. And check its length.
If before the next replacement you see that the length is not enough, then
you define another bigger string. It takes time but still much better than
what you have now.

Further optimizing, would require using of CopyMemory API function and
undocumented tricky VB functions VarPtr and StrPtr. If you going to go so
deep - just use Google.

Just one more (really smallest) thing. They say (I've never tested), Title
= "" better to be replaced with Title=vbNullString, because the latter is
faster. In this case I am sure the difference is insignificant (if any).

I didn't make thorough studying of your code. May be there are other ways
for improvement.

Dmitriy.

"Jack" <replyto@newsgroup> wrote in message
news:%23Wfh7eAKEHA.268@TK2MSFTNGP11.phx.gbl...
> Thank you very much, Dmitriy.
> I am sorry I am bugging you so much. I appreciate each of your reply.
> If you feel I am overboard please just let me know and I will open the new
> thread.
> There is one more thing I would like to understand the behaviour.
> It is the saving content of the ListView to the file.
>
> 1. First of all the procedure takes about 5 minutes when saving to the
file
> one ListView row at the time.
> The size of the saved file is about 9 MB.
> Logically, it should be much faster storing all data in the string and
then
> saving it to file. But that is not the case.
> Doing the latter way saving data takes forever!!!! Why it is so?
>
> 2. When saving data I am displaying progress bar, which Max value is set
to
> ListItems.Count
> The behaviour of it is very strange. As closer to the end the bar is
> progressing slower and slower.
> It takes about 1 minute to reach 50%, 2 minutes to reach 75% and 5 minutes
> to reach 100%.
> The data is about evenly spread over the whole ListView grid, and besides
> that I am saving each subitem, empty or not.
> How to explain that?
>
> The saving routine is shown below:
> ============================
>
> Public Sub SaveGuide(frm As Form)
> Dim z As Byte 'sets lv 24hrs or 44hrs
> Dim hdrs2 As ColumnHeaders
> Dim itms(1 To 11) As ListItems
> With frm
> Set hdrs2 = .lvEPG(2).ColumnHeaders
> Set itms(1) = .lvEPG(1).ListItems
> Set itms(2) = .lvEPG(2).ListItems
> Set itms(7) = .lvEPG(7).ListItems
> Set itms(8) = .lvEPG(8).ListItems
> Set itms(10) = .lvEPG(10).ListItems
> Set itms(11) = .lvEPG(11).ListItems
> End With
> Open App.Path & "\Data\GuideA" For Output As FileNum 'create new file
> frmSignal.probar(6).max = itms(1).Count
> lName = "" ' just recycle global variable
> For i = 1 To itms(1).Count
> Title = ""
> Desc = ""
> Lang = ""
> frmSignal.probar(6).Value = i - 1
> z = 1
> '---------CHAN--------
> Chan = itms(1)(i)
> '---------NetID--------
> NetID = itms(1)("Ch" & Chan).Tag
> For j = 2 To 49
> '---------TITLE--------
> Title = Title & itms(z + 1)("Ch" & Chan).SubItems(j) &
> Chr(134)
> '---------DESC--------
> lpString = itms(z)(itms(z + 1)("Ch" &
Chan).Key).SubItems(j)
> 'DESC
> For k = 1 To Len(lpString) 'replacing CR with Chr(140)
> If Mid(lpString, k, 1) = Chr(13) Then
> lpString = Left(lpString, k - 1) & Chr(140) &
> Mid(lpString, k + 1)
> End If
> If Mid(lpString, k, 1) = Chr(10) Then
> lpString = Left(lpString, k - 1) & Chr(140) &
> Mid(lpString, k + 1)
> End If
> Next
> Desc = Desc & lpString & Chr(134)
> '---------LANG--------
> If j < 49 Then
> If InStr(1, itms(10)(itms(z)("Ch" &
> Chan).Key).SubItems(j - 1), Chr(10) = 0) Then
> Lang = Lang & itms(10)(itms(z)("Ch" &
> Chan).Key).SubItems(j - 1) & Chr(134)
> End If
> Else
> If InStr(1, itms(11)(itms(z)("Ch" &
> Chan).Key).SubItems(j - 1), Chr(10) = 0) Then
> Lang = Lang & itms(11)(itms(z)("Ch" &
> Chan).Key).SubItems(j - 1) & Chr(134)
> End If
> End If
> If j = 49 Then
> If z = 7 Then Exit For
> If o44Hrs = 1 Then j = 1: z = 7 'switching to 44hrs
> form
> End If
> Next
> Print #FileNum, Chan & Chr(135) & NetID & Chr(135) & Title &
> Chr(135) & Desc & Chr(135) & Lang & Chr(135)
> ' lName = lName & Chan & Chr(135) & NetID & Chr(135) & Title &
> Chr(135) & Desc & Chr(135) & Lang & Chr(135)
> Next
> ' Print #FileNum, lName
> Close FileNum
> End Sub
> =======================================================================
>
> Please notice there are 2 commented lines close to the end of sub.
> I tried to use those lines first to save the data to the string lName and
> later (after For-Next loop completes) saving lName to the file, expecting
it
> to be a little faster.
> As I said above there is a problem with it. The vbasic returns to idle
(not
> run) state but CPU stays at 100% and TaskManager reports that my exe is
> still running!!!! I have to use TaskManager to kill executable.
> Of course, when I uncomment these 2 lines I comment the line ( Print
> #FileNum, Chan & Chr(135)...) just above lName = ... line.
> Jack

Relevant Pages

  • Re: Fast string operations
    ... worried about string performance. ... > is why people use unsafe code now and then to use pointer arithmetic to loop ... > I'm aware that looping has to occur one way or the other, but with Trim, ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Fast string operations
    ... > is why people use unsafe code now and then to use pointer arithmetic to ... > loop over arrays without all the unnecessary bounds checking. ... don't return the trimmed string". ... The customer perceives this as a memory leak. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Get the path and namefile in run time
    ... function Get_Path_Only return String; ... -- This returns the first N characters of the program name. ... end loop; ...
    (comp.lang.ada)
  • Re: Help a beginner - simple lowercase to uppercase and so on function
    ... And then one to loop across the string calling that function ... copying at to a new array. ... are any characters other than lowercase letters */ ...
    (comp.lang.c)
  • Re: String Literals
    ... read-only section of memory. ... to retrieve specific characters, but limits your ability to change ... I can imagine some buggy compilers messing it ... the contexts in which a string ...
    (comp.lang.c.moderated)