Re: Random file access -con't

From: Ginny Caughey [MVP] (ginny.caughey.online_at_wasteworks.com)
Date: 08/10/04 

Date: Tue, 10 Aug 2004 19:51:02 -0400

I use a text file format similar to CSV. I separate records by CRLF and
fields by the pipe symbol. I read in the whole file when the program loads
and parse the parts into collections of business objects primarily using
StreamReader.ReadLine, String.Split, and various members of the Convert
class. (This is indeed the slowest part, but it's many times faster than
parsing XML and it only happens once when the app loads.) Whenever I need to
add a record to a file, I add it to the memory collection and I also append
a record to the text file itself. (This is fast.) If I need to delete or
modify a record, I use a special code in a new record I append that
indicates that the record is deleted or modified. This approach works well
for reasonably small amounts of data - a few thousand records - but it
requires a desktop app that can play back the log files and update
enterprise data correctly as well as the intelligence in the device app to
correctly interpret the files. But for large amounts of data requiring a
storage card, I would certainly use SqlCe instead rather than trying to keep
all the data in memory all the time.

I think for the volume of data you're expecting that you have a number of
options. I recommend trying several and see what you like, but I don't
recommend XML unless the data volume is very small. 

-- 
Ginny Caughey
.Net Compact Framework MVP
"Wapiti" <testme@spamcatcher.org> wrote in message
news:%234MYH$yfEHA.4092@TK2MSFTNGP10.phx.gbl...
> So, what method do you use then, for accessing your text files?  In the
> Compact Framework?  And how many records are you accessing - do you
perform
> record(column) modifications with the collection data?  If so, how?
>
> Regular text files seem like they would consume a lot of time.
>
>
> "Ginny Caughey [MVP]" <ginny.caughey.online@wasteworks.com> wrote in
message
> news:Osme7nyfEHA.3928@TK2MSFTNGP11.phx.gbl...
> > The main problem with XML with large amounts of data is that it's
> generally
> > slow to parse it on mobile devices as well as taking up more space than,
> for
> > example, as CSV file. I use text files instead of XML in most cases
where
> I
> > can.
> >
> > -- 
> > Ginny Caughey
> > .Net Compact Framework MVP
> >
> >
> >
> > "Wapiti" <testme@spamcatcher.org> wrote in message
> > news:O37m9SyfEHA.1764@TK2MSFTNGP10.phx.gbl...
> > > So, with that link in mind, what would anyone think regarding the use
of
> > xml
> > > to store, and otherwise manage, data for a data collection
application?
> > >
> > > The data doesn't have to be synchronized with any host database, but
> > > extracted to a ascii flat text file.
> > >
> > > Seems though, that xml will be slow as an ascii stream - isn't it just
a
> > big
> > > text file anyway?
> > >
> > > I've never used xml - maybe now is the time to try??
> > >
> > > Input appreciated.
> > >
> > > Mike
> > >
> > >
> > > "Noble" <noble@agforest.com> wrote in message
> > > news:eIh9jPxfEHA.720@TK2MSFTNGP11.phx.gbl...
> > > > "SetFilePointer" is a cool function but it does not provide TRUE
> random
> > > > access like the days of old.
> > > >
> > > > Take a look at this thread:
> > > > http://www.xtremevbtalk.com/archive/index.php/t-179063.html
> > > >
> > > > Thanks,
> > > > Noble
> > > >
> > > > "Paul G. Tobey [eMVP]" <ptobey no spam AT no instrument no spam DOT
> com>
> > > > wrote in message news:uFKCfjwfEHA.2592@tk2msftngp13.phx.gbl...
> > > > > You can call anything in any DLL anywhere on the device, but you
> have
> > to
> > > > > write your own declarations.  That's what P/Invoke is: Platform
> > Invoke.
> > > > You
> > > > > declare the functions which are found in DLLs as externs and then
> call
> > > > them,
> > > > > passing suitable parameters.  If you feel like something is
missing,
> > > then
> > > > > you haven't been involved in .NET CF development long enough, yet.
> > > > >
> > > > > Download the OpenNETCF SDF and look through the source code.  A
huge
> > > > > percentage of what is done there is working around the lack of
> managed
> > > > code
> > > > > wrappers for Win32 API functions.  In fact, CreateFile, WriteFile,
> > > > ReadFile
> > > > > and, maybe, SetFilePointer, are in there.
> > > > >
> > > > > Paul T.
> > > > >
> > > > > "Wapiti" <testme@spamcatcher.org> wrote in message
> > > > > news:O4SN3cwfEHA.3416@TK2MSFTNGP09.phx.gbl...
> > > > > >
> > > > > > "Ginny Caughey [MVP]" <ginny.caughey.online@wasteworks.com>
wrote
> in
> > > > > message
> > > > > > news:%23ScYlOvfEHA.2012@TK2MSFTNGP10.phx.gbl...
> > > > > > > Certainly you *can* use direct file I/O to accomplish what you
> > want.
> > > > > > Paul's
> > > > > > > suggestion of CreateFile/ReadFile/WriteFile would certainly
work
> > as
> > > > > would
> > > > > > > the .NetCF StreamReader and StreamWriter classes which wrap
> them,
> > > > > although
> > > > > > > neither of these gives you random access - you'd need to code
> that
> > > for
> > > > > > > yourself using these functions or classes as primitives.
> > > > > >
> > > > > > I guess I'll have to do some digging to figure out what you guys
> are
> > > > > talking
> > > > > > about here.  Even tho Paul 'already explained it' - explain to
me
> > why,
> > > > if
> > > > > CF
> > > > > > doesn't support random access, would the basic win32 api
functions
> > for
> > > > > file
> > > > > > access in CF be supported?
> > > > > >
> > > > > > Something doesn't sound right.
> > > > > >
> > > > > >
> > > > > > > 1-2 thousand records, depending on the size of each record,
> isn't
> > a
> > > > > large
> > > > > > > amount of data for modern devices. And where the amount of
data
> > *is*
> > > > > > large,
> > > > > > > the usual solution is to store the data on a storage card.
> > > > > >
> > > > > > Sounds reasonable.  A storage card might be the best way of
going
> > > > anyway.
> > > > > >
> > > > > >
> > > > > > > I don't know off hand the exact footprint of SqlCe, but that
> > product
> > > > was
> > > > > > > designed for exactly your type of scenario (and also for
amounts
> > of
> > > > data
> > > > > > > that are much larger.) What is the absolute minimal hardware
> that
> > > you
> > > > > need
> > > > > > > to target?
> > > > > >
> > > > > > I'm testing on an Axim right now.   'Absolute minimal' is
> difficult
> > to
> > > > > > define. I guess I'd have to resort to the answer, it depends.
But
> > > that
> > > > > > would be too vague so, I could set a minimum to my users.
They're
> > > going
> > > > > to
> > > > > > be collecting data - that data could span 1 day to a weeks worth
> of
> > > > data.
> > > > > > The records aren't large by any means, say, 10 fields per record
> > each.
> > > > > its
> > > > > > a warehouse application - Issue, Receipt, Transfer and Inventory
> > > > > > adjustments.
> > > > > >
> > > > > >
> > > > > > > There are also 3rd party database solutions available.
> > > > > >
> > > > > > I've been burned before on these, but if anyone has
> recommendations,
> > > I'd
> > > > > be
> > > > > > open to hearing them.  Speed in development, sufficient speed in
> > data
> > > > > access
> > > > > > and a company that will be around for the next year are the
> > important
> > > > > > factors pertaining to a 3rd party solution, at this point.
> > > > > >
> > > > > > >
> > > > > > > -- 
> > > > > > > Ginny Caughey
> > > > > > > .Net Compact Framework MVP
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > "Wapiti" <testme@spamcatcher.org> wrote in message
> > > > > > > news:OQS2cEvfEHA.2020@TK2MSFTNGP10.phx.gbl...
> > > > > > > > There could be 1 - 2 thousand records downloaded to the
device
> > > > > > (warehouse
> > > > > > > > parts).  But the reason I haven't been considering sqlce is
> > > because
> > > > > this
> > > > > > > > application needs to have the capability of running on
> different
> > > > > models
> > > > > > > and
> > > > > > > > classes of ppc's; some limited in memory.  I understand that
> > sqlce
> > > > has
> > > > > a
> > > > > > > > huge footprint - if I start requiring that my customers run
on
> > > high
> > > > > > > > memory/processor devices they're not going to be happy.
> > > > > > > >
> > > > > > > > What is the footprint of sqlce?   What mem/process
> is -actually-
> > > > > > required?
> > > > > > > > We've all learned that we can't trust the specs for the
> > products.
> > > > > Real
> > > > > > > > life, what is needed?
> > > > > > > >
> > > > > > > > This app only needs to import a flat text file, and export a
> > flat
> > > > text
> > > > > > > file,
> > > > > > > > to the host system.  There is no synchronization with
another
> > > > > > > database -this
> > > > > > > > hhc app is going to be interfacing with a legacy system of
> which
> > I
> > > > > have
> > > > > > no
> > > > > > > > control over.
> > > > > > > >
> > > > > > > > Why is it that everything has to be so bleeding edge?
Random
> > > files
> > > > > work
> > > > > > > > well for handheld applications.  I've been using them for
> years.
> > > > Now,
> > > > > > it
> > > > > > >
> > > > > > > > seems, we need to keep buying more and more powerful devices
> > just
> > > to
> > > > > > > perform
> > > > > > > > simple tasks, such as offsite data collection.  How
> frustrating.
> > > > > > > >
> > > > > > > > Please tell me I'm wrong - and that, as an example, I could
> > > install
> > > > > and
> > > > > > > run
> > > > > > > > sqlce, my cf vb.net app and data onto a default Axim device
> (as
> > an
> > > > > > > example)
> > > > > > > > without a hitch.  If there is a hitch, my customers will
> > complain
> > > > and
> > > > > > then
> > > > > > > > go away - and rightly so.  I really do hope I'm wrong, but
my
> > > > reading
> > > > > > > > suggests otherwise.
> > > > > > > >
> > > > > > > >
> > > > > > > > "Ginny Caughey [MVP]" <ginny.caughey.online@wasteworks.com>
> > wrote
> > > in
> > > > > > > message
> > > > > > > > news:u3xeJ%23ufEHA.1644@tk2msftngp13.phx.gbl...
> > > > > > > > > Mike,
> > > > > > > > >
> > > > > > > > > Is there a particular reason you aren't considering
> SqlServer
> > > CE?
> > > > > How
> > > > > > > many
> > > > > > > > > records do you expect the system to handle?
> > > > > > > > >
> > > > > > > > > -- 
> > > > > > > > > Ginny Caughey
> > > > > > > > > .Net Compact Framework MVP
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > "Wapiti" <testme@spamcatcher.org> wrote in message
> > > > > > > > > news:uePttiufEHA.3416@TK2MSFTNGP09.phx.gbl...
> > > > > > > > > > Could someone point me to some examples of random file
> > access
> > > > > using
> > > > > > > the
> > > > > > > > > cf -
> > > > > > > > > > vb.net?
> > > > > > > > > >
> > > > > > > > > > I've found plenty of examples for straight text access
on
> > msdn
> > > > and
> > > > > > > > > > elsewhere - but I'd like to use random access - using a
> > table
> > > > > > > structure
> > > > > > > > > and
> > > > > > > > > > be able to build a system to quickly access records
> > (relative
> > > to
> > > > > > text
> > > > > > > > > files
> > > > > > > > > > anyway).  I'm currently looking at streamwriter and
> > > filestream -
> > > > > not
> > > > > > > > sure
> > > > > > > > > > which is my best bet, if any.
> > > > > > > > > >
> > > > > > > > > > I thought I saw an example a long while back while first
> > > > studying
> > > > > > the
> > > > > > > > use
> > > > > > > > > of
> > > > > > > > > > .net in converting our msvc1.5 based handheld
> application -
> > > but
> > > > > > can't
> > > > > > > > find
> > > > > > > > > > it/them now.  Looking too hard?  Or do they not exist?
> > > > > > > > > >
> > > > > > > > > > Are there any good books on building cf vb.net apps for
> the
> > > > > purposes
> > > > > > > of
> > > > > > > > > data
> > > > > > > > > > collection?
> > > > > > > > > >
> > > > > > > > > > side question, it seems to me that handheld devices
> primary
> > > > > purpose
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > > for data collection of some sort, being that they are
> > mostly,
> > > > now
> > > > > > > > anyway,
> > > > > > > > > > not 'connected'.  I'm surprised that there isn't more
> > details
> > > on
> > > > > the
> > > > > > > > > > differing data storage methods.  I could be very wrong
> here
> > > and
> > > > > its
> > > > > > > all
> > > > > > > > > > over, but I'm just not finding it.
> > > > > > > > > >
> > > > > > > > > > Mike
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Relevant Pages

  • Re: What is the logic of storing XML in a Database?
    ... can't do with CSV. ... In what a header does constitute a schema. ... So a schema is the standard structure by which the user sending a file ... in much the same way that XML standards are. ...
    (comp.databases.theory)
  • Re: ultra newbie question (dont laugh)
    ... storing employee information in an XML file. ... but not as fun as writing the logic. ... On reason to use OOP is to insulate your main code from things like storage ... from CSV file. ...
    (comp.lang.python)
  • Re: What is the logic of storing XML in a Database?
    ... will trap most major errors. ... In fact one does, XML. ... CSV is useful for unloading an SQL table into a text ...
    (comp.databases.theory)
  • Re: Why do you use XML?
    ... different flavours of CSV. ... Sometimes you wrap all strings in quotes, ... we output to in this "standard" CSV format. ... The only benefit to XML is that someone already did most of the ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Why do you use XML?
    ... Sometimes you wrap all strings in quotes, sometimes you only wrap strings in quotes if they contain a comma. ... Our CSV output routines have a bajillion configuration options to accommodate all of the different systems we output to in this "standard" CSV format. ... The only benefit to XML is that someone already did most of the ...
    (microsoft.public.dotnet.languages.csharp)
Loading