Re: speeding up a file conversion from text file to XML format

Tech-Archive recommends: Speed Up your PC by fixing your registry

On Mar 2, 2:26 pm, Peter Newman
<PeterNew...@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote:

See some thoughts inline:

to the SQL very quickly. The problem I have come up with is generating the
XML File, when files are received in with inexcess of 4000 data records (

Sample File

"111111"
"TEST"
"1"
"100"
"839"
"04/03/2008"
"001"
"HEADFOOD...@xxxxxxxxx"
"CompanyCont...@xxxxxxxxxxxxx"
"111111 SERVICE Serial: 839 THIS IS A TEST"
"22222233333333 COMPANY NAME BANK REF 000000000010099" ***
"TRAILER RECORD"
"QQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"
"QQQQQQQQQ"

What do the lines with QQQQQQQ stand for? Are they read?



BUILD XML FILE CODE

Imports System.Xml
Imports System.IO

Module XMLBuilder
Private XML_DOC As New XmlDocument
Public CURRENTFILE As String = ""
Dim ROOT As XmlElement
Public sr As StreamReader
Private iCount As Integer = 0
Private InputLine As String = ""
Public LICENCENUM As String = ""
Public PIN As String = ""
Public SERIAL As String = ""
Public ORIGACC As String = ""
Private PASSTRAILER As Boolean = False
Public LEDGERKEY As String = ""
Public lupBacsID As String = ""
Public lupOrigSC As String = ""
Public lupOrigACNO As String = ""
Public lupOrigACNA As String = ""
Public lupCountry As String = ""

Public Sub OpenFile(ByVal FileName As String)
Dim sr As New StreamReader(FileName)
End Sub

Public Function BuildXMLFile(ByVal FullFileName As String) As Boolean
'Create the Doc
Dim Root As XmlElement
' open the file and get info
CURRENTFILE = FullFileName
Dim sr As New StreamReader(CURRENTFILE)
For iCount = 1 To 10
Select Case iCount
Case 1
LICENCENUM = StripQuotes(sr.ReadLine)
Case 2
PIN = StripQuotes(sr.ReadLine)
Case 3
TRANSCOUNT = StripQuotes(sr.ReadLine)
Case 4
FILEVALUE = StripQuotes(sr.ReadLine)
Case 5
SERIAL = StripQuotes(sr.ReadLine)
Case 6
MOVEDATE = StripQuotes(sr.ReadLine)
Case 7
ORIGACC = StripQuotes(sr.ReadLine)
End Select
Next

What is the purpose of this for loop and select case? They don't seem
to serve a purpose. Just remove them:

Dim sr As New StreamReader(CURRENTFILE)
LICENCENUM = StripQuotes(sr.ReadLine)
PIN = StripQuotes(sr.ReadLine)
TRANSCOUNT = StripQuotes(sr.ReadLine)
FILEVALUE = StripQuotes(sr.ReadLine)
SERIAL = StripQuotes(sr.ReadLine)
MOVEDATE = StripQuotes(sr.ReadLine)
ORIGACC = StripQuotes(sr.ReadLine)
sr.Close()


Friend Sub CreateTransactionsElement(ByRef m_doc As Xml.XmlDocument,
ByVal parent As Xml.XmlNode)
Dim Newlayer As XmlElement = m_doc.CreateElement("Transactions") '
Creating an element
'Add a grand child element
' Write Tranmsactions to file
Dim sr As New StreamReader(CURRENTFILE)
' Move to line 11
For iCount = 1 To 10
sr.ReadLine()
Next

Here you are re-opening the file and reading the first 10 lines
again. In effect you are reading the first 10 lines of each file
twice. Instead, up in the code earlier where you read the first 10
lines and store their values, go ahead and read all the transactions
into an List(Of String) or something so you don't waste the time
duplicating the file I/O.

And finally, you might approach the problem a little differently.
Create two classes. One for the SqlImporter object and another for
the Transaction object. The SqlImporter will have a collection
property to hold the transactions. Then load the file and create the
objects and then use the Xml serializer to create the XML. You just
work with the object and let the serializer create the XML for you.
You can decorate your classes properties with the appropriate
attributes to control how the xml file will be created. It will make
your code much more readable and maintainable.


Here's some simple classes that shows how to serialize and deserialize
to and from XML. You just create instances of the class, populate
it's properties and then call the Serialize method to create the
xml.

Imports System.Xml.Serialization
Imports System.IO

Public Sub Main()

Dim s As New SqlImporter()
s.LicenseNumber = "123456"
s.HeaderFile = "header file"

Dim t As New Transaction()
'set t properties here

s.Transactions.Add(t)

'Finally, serialize to xml:
SqlImporter.Save("filename.xml", s)

End Sub

<Serializable()> _
Public Class SqlImporter

Public Sub New()
_transactions = New List(Of Transaction)()
End Sub

Private _licenseNumber As String
<XmlAttribute("name")> _
Public Property LicenseNumber() As String
Get
Return _licenseNumber
End Get
Set(ByVal value As String)
_licenseNumber = value
End Set
End Property

Private _headerFile As String
<XmlAttribute("headerFile")> _
Public Property HeaderFile() As String
Get
Return _headerFile
End Get
Set(ByVal value As String)
_headerFile = value
End Set
End Property

Private _transactions As List(Of Transaction)
<XmlArray("Transactions"), XmlArrayItem("Transaction",
GetType(Transaction))> _
Public Property Transactions() As List(Of Transaction)
Get
Return _transactions
End Get
Set(ByVal value As List(Of Transaction))
_transactions = value
End Set
End Property

Public Shared Sub Save(ByVal filename As String, ByVal obj As
SqlImporter)
Using wrt As New StreamWriter(filename)
Dim xSer As New XmlSerializer(GetType(SqlImporter))
xSer.Serialize(wrt, obj)
End Using
End Sub

Public Shared Function Load(ByVal filename As String) As SqlImporter
Dim importer As SqlImporter

Using rdr As New StreamReader(filename)
Dim xSer As New XmlSerializer(GetType(SqlImporter))
importer = DirectCast(xSer.Deserialize(rdr), SqlImporter)
End Using

Return importer
End Function

End Class

<Serializable()> _
Public Class Transaction
'Properties for Transaction here
End Class



Hope this helps,

Chis

.

Relevant Pages

  • RE: ASP.Net app cannot read XML with Anonymous Authentication disabled
    ... ASP.Net app cannot read XML with Anonymous Authentication ... | Dim ProjReader As XmlReader = Nothing ... | Dim ProjURL As String = Session ... | 'Get XML Data for Projector and load in Projector Dropdown ...
    (microsoft.public.dotnet.framework.aspnet.security)
  • RE: using myXslDoc.Transform
    ... and xml file ... Dim strXML As String = SqlHelper.ExecuteScalar(cn, ... ' get xsl file from database and load xsl file ...
    (microsoft.public.dotnet.xml)
  • RE: ASP.Net app cannot read XML with Anonymous Authentication disabled
    ... ASP.Net app cannot read XML with Anonymous Authentication ... | Dim ProjReader As XmlReader = Nothing ... | Dim ProjURL As String = Session ... | 'Get XML Data for Projector and load in Projector Dropdown ...
    (microsoft.public.dotnet.framework.aspnet.security)
  • Net.WebRequest/WebResponse ?s
    ... We have some legacy code which makes use of the MS XML Core Services and needed to be upgraded/updated. ... Public Function GetResponsePWP(ByVal ReqURL As String, ByVal ReqTimeout As Integer, ByVal ReqType As String, ByVal PostData As String, ByVal UserName As String, ByVal Password As String) As String ... Dim DataStream As Stream ... nmsStatusDescr = HTTPResponse.StatusDescription ...
    (microsoft.public.dotnet.languages.vb)
  • RE: using myXslDoc.Transform
    ... 'get xml file from the database and load xml file ... Dim strXML As String = SqlHelper.ExecuteScalar(cn, ... the transform can be applied as usual. ...
    (microsoft.public.dotnet.xml)