Re: Scanning hard drive
- From: "Bela Istok" <bela_i@xxxxxxxxxxx>
- Date: Mon, 22 Oct 2007 17:30:45 -0400
Last run with code:
Using delegated function
Start time:10/22/2007 5:13:19 PM
End time:10/22/2007 5:16:58 PM
Total time:00:03:39.6027394
Size1: 769462539; Size2: 2074779247; Size3:52272046924; File Number:169593; Directory Number:21854;
Code:
Attached.
Regard,
Bela Istok
"Bela Istok" <bela_i@xxxxxxxxxxx> wrote in message news:FAFA5C8E-ECBD-47A8-A5D4-7E9441B1CF62@xxxxxxxxxxxxxxxx
I redo the code (your code), using Delegates and get a first run of 4 minutes, and second and later run under a minute. (This is fine if you have multi processor or multi core machine, with only 1 core you don't win much). I will try to post the code later when have time to clean it a bit./*
Regards,
Bela Istok
"Bela Istok" <bela_i@xxxxxxxxxxx> wrote in message news:1B4D202E-1CCD-4CE8-AE71-B43F0B28318C@xxxxxxxxxxxxxxxxThe first I see in you code is the use of Doubles, you don't need dobles you only need long types for the size value.
You code run in 4 min. Going to see if can perform better.
Regards,
Bela Istok
"Rotsey" <malcolm_smith@xxxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message news:eiuJvfEFIHA.4752@xxxxxxxxxxxxxxxxxxxxxxxOK. Here is the class the scans the drive.
Can you see any issues here?
Bela, would you be able to post your code that scans the drive.
I am scanning 56GB in about 10 minutes.
public struct PieFileInfo
{
public int RowID;
public PieInfoType Type;
public string Name;
public string FullName;
public double Filesize;
public DateTime LastAccess;
public DateTime Created;
public string Extension;
public DateTime LastWrite;
public int ParentFolderRowID;
public string Drive;
public double FileAge1;
public double FileAge2;
public double FileAge3;
public bool IsRoot;
}
public class ScanDrive
{
private DriveData mList = new DriveData();
public DriveData DataList
{
get { return mList; }
set { mList = value; }
}
private string mDrive;
private int mNewFolderID = 0;
public ScanDrive(string drivefolder)
{
mDrive = drivefolder;
mList.Drive = drivefolder;
}
public bool ScanFolder(string folder, int folderID, ref double psize1, ref double psize2, ref double psize3)
{
DirectoryInfo objDir = new DirectoryInfo(folder);
int newFolderID;
double size1 = 0;
double size2 = 0;
double size3 = 0;
System.DateTime fileDate;
try
{
newFolderID = WriteRow(objDir, folderID);
if (newFolderID == 0)
{
return false;
}
foreach (DirectoryInfo dir in objDir.GetDirectories())
{
if (!ScanFolder(dir.FullName, newFolderID, ref psize1,ref psize2,ref psize3))
{
return true;
}
size1 += psize1;
size2 += psize2;
size3 += psize3;
}
foreach (FileInfo fle in objDir.GetFiles())
{
WriteRow(fle, newFolderID);
fileDate = GetAgeDate(fle);
if (fileDate >= DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))
size1 += fle.Length;
else if (fileDate >= DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))
size2 += fle.Length;
else
size3 += fle.Length;
}
UpdateFolderRow(newFolderID, size1, size2, size3);
psize1 = size1;
psize2 = size2;
psize3 = size3;
return true;
}
catch (Exception)
{
return true;
}
}
private DateTime GetAgeDate(FileInfo f)
{
if (f.CreationTime >= DateTime.Today.AddDays(Settings.Instance.SettingsData.Age1Days * -1))
return f.CreationTime;
else if (f.LastWriteTime >= DateTime.Today.AddDays(Settings.Instance.SettingsData.Age2Days * -1))
return f.LastWriteTime;
else
return f.LastAccessTime;
}
private int WriteRow(DirectoryInfo dir, int folderID)
{
PieFileInfo info = new PieFileInfo();
if (mNewFolderID == 0)
info.IsRoot = true;
mNewFolderID++;
info.RowID = mNewFolderID;
info.Type = PieInfoType.Folder;
info.FullName = dir.FullName;
info.Name = dir.Name;
info.LastAccess = dir.LastAccessTime;
info.LastWrite = dir.LastWriteTime;
info.Created = dir.CreationTime;
info.ParentFolderRowID = folderID;
info.Drive = mDrive;
mList.DataList.Add(info);
return mNewFolderID;
}
private int WriteRow(FileInfo file, int folderID)
{
PieFileInfo info = new PieFileInfo();
info.Type = PieInfoType.File;
info.Name = file.Name;
info.FullName = file.FullName;
info.Filesize = file.Length;
info.Extension = file.Extension;
info.LastAccess = file.LastAccessTime;
info.LastWrite = file.LastWriteTime;
info.Created = file.CreationTime;
info.ParentFolderRowID = folderID;
info.Drive = mDrive;
if (file.LastAccessTime >= DateTime.Today.AddDays(-30))
info.FileAge1 = file.Length;
else if (file.LastAccessTime >= DateTime.Today.AddDays(-60))
info.FileAge2 = file.Length;
else
info.FileAge3 = file.Length;
mList.DataList.Add(info);
return folderID;
}
private void UpdateFolderRow(int folderID, double size1, double size2, double size3)
{
int index = mList.DataList.FindIndex(delegate(PieFileInfo fi)
{
if (fi.Type == PieInfoType.Folder && fi.RowID == folderID)
return true;
else
return false;
});
PieFileInfo info = mList.DataList[index];
info.FileAge1 += size1;
info.FileAge2 += size2;
info.FileAge3 += size3;
info.Filesize += size1 + size2 + size3;
mList.DataList[index] = info;
}
}
"Bela Istok" <bela_i@xxxxxxxxxxx> wrote in message news:ED0DEFAE-1EB6-4DEF-96F8-CE11CAB9B316@xxxxxxxxxxxxxxxxWhat do you scan? I do a directory and File scan in my 100GB (64GB of data) hard drive and took 24 seconds. (Scanned every directory and every file in the drive for info).
PD: for reference 170,187 files and 21,899 folders.
Regards,
Bela Istok
"Rotsey" <malcolm_smith@xxxxxxxxxxxxxxxxxxxxxxxxxx> wrote in message news:errJ9I6EIHA.1208@xxxxxxxxxxxxxxxxxxxxxxxHi,
I am writing an app that scans hard drives and logs info
about every fine on the drive.
The first iteration of my code used a class and a generic list
to store the data and rhis took 13min on my 60 GB drive.
I wanted it to be quicker.
So the second time I changed the class to a struct and still
with the generic list got the time down to 4 min.
I am wondering if I can improve on this even more??
Anyone know if it would be possible to improve this
using C# or would it need to be wrritten in assembler or the like?
rotsey
Copyright © 2007 Bela Istok (bela_i@xxxxxxxxxxx)
The File is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY KIND, either express or implied.
*/
using System;
using System.Threading;
using System.IO;
using System.Collections.Generic;
using System.Text;
namespace ConsoleApplication1
{
class Program
{
/// <summary>
/// The Start time.
/// </summary>
static DateTime start;
public static bool Runing = true;
static void Main(string[] args)
{
Console.WriteLine("Using delegated function");
start = DateTime.Now; //Set the Start Time
Console.WriteLine("Start time:{0}", start);
RootDirectories();
while (Runing) //Run until we get the stop signal.
{
Thread.Sleep(100);
}
Console.WriteLine("Press any key to exit");
Console.ReadLine();
}
/// <summary>
/// This lis contains the Directory and File information, there will be 1 element for each directory in the machine.
/// NOTE: The documentation states that the key parameter uses a hash table and because each DirectoryInfo we add is
/// a new instance we have a diferent HashCode for each directory, no matter what the directory name is.
/// </summary>
public static Dictionary<DirectoryInfo, List<FileInfo>> infos = new Dictionary<DirectoryInfo, List<FileInfo>>(22000/*Use a high number or equal to the number of dirs*/);
/// <summary>
/// The number of root directories (used to check when we finish).
/// </summary>
public static int times;
/// <summary>
/// Get the Root Directories.
/// </summary>
public static void RootDirectories()
{
DirectoryInfo[] dirs = new DirectoryInfo(@"G:\").GetDirectories(); //All Directories for drive G
int length = times = dirs.Length; // The number of directories.
for (int i = 0; i < length; i++)
{
// We create a delegate for each directory we found
ParameterizedThreadStart pts = new ParameterizedThreadStart(GetDirectoriesThread);
//We use the BeginInvoke, the documentation states that this will use the ThreadPoll for the machine.
pts.BeginInvoke(dirs[i], new AsyncCallback(Result), null);
}
}
/// <summary>
/// The Number of times Result got called.
/// </summary>
public static int endTimes;
/// <summary>
/// The Delegate call this method each time finished one root directory.
/// </summary>
/// <param name="r"></param>
public static void Result(IAsyncResult r)
{
// We increment this each time we get called.
endTimes++;
if (endTimes == times) //If the number of times is == the number of root dirs we finished.
{
DateTime end = DateTime.Now;
Console.WriteLine("End time:{0}", end);
TimeSpan time = end - start;
Console.WriteLine("Total time:{0}", time);
Console.WriteLine("Size1: {0}; Size2: {1}; Size3:{2}; File Number:{3}; Directory Number:{4};", size1, size2, size3, filesNumber, infos.Count);
// Call this to finish the execution.
Runing = false;
}
}
/// <summary>
/// The method that get called when we use BeginInvoke.
/// </summary>
/// <param name="obj"></param>
public static void GetDirectoriesThread(object obj)
{
GetDirectories(obj as DirectoryInfo); //Here we call each directory of that directory.
}
public static void GetDirectories(DirectoryInfo dir)
{
DirectoryInfo[] directories;
try
{
directories = dir.GetDirectories(); // Get the directories.
}
catch (Exception) //If we get an error.
{
return;
}
int length = directories.Length;
for (int i = 0; i < length; i++)
{
DirectoryInfo dp = directories[i];
GetDirectories(dp); //Call ourselves recursive to see if we have other dirs.
try
{
FileInfo[] files = dp.GetFiles(); //Now get the files in this directory.
CalculateFiles(files);
infos.Add(dp, new List<FileInfo>(files)); //Add this directory and files to the collection.
//NOTE: here we don't care to be Thread safe because we only add elements to the list, if we changed any
// element we need to check we are Thread-safe.
}
catch (Exception)
{
continue; //Continue the execution ...
}
}
}
/// <summary>
/// Public static because this way is Thread-Safe.
/// </summary>
public static long size1 = 0;
/// <summary>
/// Public static because this way is Thread-Safe.
/// </summary>
public static long size2 = 0;
/// <summary>
/// Public static because this way is Thread-Safe.
/// </summary>
public static long size3 = 0;
/// <summary>
/// Public static because this way is Thread-Safe.
/// </summary>
public static int filesNumber = 0;
/// <summary>
/// We only need to read this one time.
/// </summary>
static readonly DateTime old1 = DateTime.Now.AddDays(-10);
/// <summary>
/// We only need to read this one time.
/// </summary>
static readonly DateTime old2 = DateTime.Now.AddDays(-20);
/// <summary>
/// Calculates the file sizes by age.
/// </summary>
/// <param name="info">The files to calculate.</param>
public static void CalculateFiles(FileInfo[] info)
{
// We do this here because we can put more rules, like files with more than 2 dates.
int count = info.Length;
for (int j = 0; j < count; j++)
{
try //We use a try because some error scaning the C disk.
{
FileInfo fi = info[j];
DateTime age = GetAgeDate(fi);
if (age >= old1)
size1 += fi.Length;
else if (age >= old2)
size2 += fi.Length;
else
size3 += fi.Length;
filesNumber++;
}
catch (Exception)
{
continue;
}
}
}
private static DateTime GetAgeDate(FileInfo f)
{
if (f.CreationTime >=
DateTime.Today.AddDays(10 * -1))
return f.CreationTime;
else if (f.LastWriteTime >=
DateTime.Today.AddDays(20 * -1))
return f.LastWriteTime;
else
return f.LastAccessTime;
}
}
}
- Follow-Ups:
- Re: Scanning hard drive
- From: Rotsey
- Re: Scanning hard drive
- References:
- Scanning hard drive
- From: Rotsey
- Re: Scanning hard drive
- From: Rotsey
- Re: Scanning hard drive
- From: Bela Istok
- Re: Scanning hard drive
- From: Bela Istok
- Scanning hard drive
- Prev by Date: Re: MSMQ body format issue.
- Next by Date: getting errors while compiling the program written in .NET framework 1.1 using Visual c# 2005 Express Edition
- Previous by thread: Re: Scanning hard drive
- Next by thread: Re: Scanning hard drive
- Index(es):
Relevant Pages
|
