Re: Regarding Opening a 1GB File for Processing.

From: Ashish M Bhonkiya (bhonkiya_at_hotmail.com.nospam)
Date: 06/23/04 

Date: Wed, 23 Jun 2004 11:57:35 -0700

Thanks for your reply.

its a text file each record is of 100 chars, and the
following is the structure of the file. i need to sort the
file based in the first field, second field is the record
number, ids are not unique, record number (2nd filed) is
unique.

0000000601 0000000001 Cherry Venus
sdjsdiuhndxjsnddxs
0000000054 0000000002 Orange Jupiter
xuhdexeudfhxe

as per ur suggestion i have also tried extracting the
first two fields and sort the list, but after the
producing the sorted list, when i start searching the
original list its taking much time as i have to read the
file again.

some more clues appreciated.

Thanks
Ashish M Bhonkiya

>-----Original Message-----
>"Ashish M Bhonkiya" <bhonkiya@hotmail.com.nospam> wrote in
>news:OEM9XvRWEHA.3664@TK2MSFTNGP12.phx.gbl:
>
>> Hi Group,
>>
>> I have a huge one GB file to do some processing
(sorting) using .Net
>> Framework (C-Sharp language). What is the best efficent
way to do
>> this.
><snip>
>
>I would say the approach differs depending on the file
and its contents.
>
>For instance, if the file contains records of some kind
that contain a
>value you want to sort on, and the size of this value is
small compared to
>the size of the full record, then perhaps construct a new
list containing
>that value + the position in the original file that the
full record can be
>found, then sort this list. Afterwards simply loop
through the list and
>grabbing the right record and outputting to a new file in
the correct
>order.
>
>Other than that I would do as you suggest, split up the
file and use
>quicksort on it.
>
>However, I would still try to minimize the amount of
writing you would need
>to do during the sorting.
>
>Is the file a text file or a structured file or ... ?
>
>--
>Lasse Vågsæther Karlsen
>lasse@vkarlsen.no
>PGP KeyID: 0x0270466B
>.
>

Relevant Pages

  • Re: Sorting a table by concatenating several fields in the same table
    ... I would not rely on either the sort order of the query or the OrderBy in the ... > to further sort any identical entries in the first field. ... > allow the user to select up to three fields for sorting. ...
    (microsoft.public.access.queries)
  • Re: Regarding Opening a 1GB File for Processing.
    ... your sorting there, and then BCP out the data? ... file based in the first field, ... first two fields and sort the list, ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Regarding Opening a 1GB File for Processing.
    ... > What about using DTS or BULK INSERT to insert this data into SQL Server, ... > your sorting there, and then BCP out the data? ... > file based in the first field, ... > first two fields and sort the list, ...
    (microsoft.public.dotnet.framework.performance)
  • Re: The best elegant solution to override 65k rows limit in a sheet
    ... >select is about 100 times more powerful than spreadsheets. ... 5 point moving averages from a table ... in which the first field is observation number and the second field is ... What sort of gyrations does something this simple require in a SELECT ...
    (microsoft.public.excel)
  • Re: Regarding Opening a 1GB File for Processing.
    ... >its a text file each record is of 100 chars, ... i need to sort the ... >file based in the first field, ... A local SQL server would be another (possibly ...
    (microsoft.public.dotnet.framework.performance)