Re: Regarding Opening a 1GB File for Processing.

From: Thomas Ganss (TGanss_RemoveForRealAdress_at_T-Online.de)
Date: 06/24/04 

Date: Thu, 24 Jun 2004 08:28:53 +0200

>its a text file each record is of 100 chars, and the
>following is the structure of the file. i need to sort the
>file based in the first field, second field is the record
>number, ids are not unique, record number (2nd filed) is
>unique.

0000000601 0000000001 Cherry Venus
sdjsdiuhndxjsnddxs
0000000054 0000000002 Orange Jupiter
xuhdexeudfhxe

You only need to read the first number, since the Record Number
should be the same as your counter - a bit of speed gained.

After sorting, you can read each block as an offset of the file -
kind of very simple ISAM.

Another option is to read in the file into a *local* filehandler /
datastorage mechanism: foxpro or dbase should do it under
5 minutes. A local SQL server would be another (possibly
slightly slower [because of heightened security]) option.

But since the file *you* get it is probably machine generated,
the quickest way would be to sort ***before***
outputting the file <g>.

HTH

thomas

Relevant Pages

  • Re: Sorting Alpha then Numeric
    ... I had to convert the chars, I used Ascand if numeric add 123, I also ... First field enter: ... You will need three blank fields and enter 3 sort order equations ...
    (microsoft.public.access.reports)
  • Re: Regarding Opening a 1GB File for Processing.
    ... its a text file each record is of 100 chars, ... file based in the first field, ... first two fields and sort the list, ... >to do during the sorting. ...
    (microsoft.public.dotnet.framework.performance)
  • Re: sort on single item only
    ... 1234 ABC 56 one-large-text-field ... they can also look like this since the first field is ... actually five chars, the 2nd field is actually four chars. ... What's your sort field? ...
    (comp.unix.shell)
  • Re: sort on single item only
    ... Man sort, option -t ... 1234 ABC 56 one-large-text-field ... they can also look like this since the first field is ... actually five chars, the 2nd field is actually four chars. ...
    (comp.unix.shell)
  • Re: The best elegant solution to override 65k rows limit in a sheet
    ... >select is about 100 times more powerful than spreadsheets. ... 5 point moving averages from a table ... in which the first field is observation number and the second field is ... What sort of gyrations does something this simple require in a SELECT ...
    (microsoft.public.excel)
Loading