Re: How to break up a Flat File into individual files

Tech Tip: Click here to run a free scan for Windows Errors and optimize PC performance

From: Mike Jansen (mjansen_nntp_at_mail.com)
Date: 10/28/04

Next message: Lars W. Andersen: "Re: Updategram inserts when DB lookup fails"

Previous message: John: "Valid Schema will not pass"
In reply to: RichTheHSTGuy: "How to break up a Flat File into individual files"
Next in thread: Stephen W. Thomas: "Re: How to break up a Flat File into individual files"
Reply: Stephen W. Thomas: "Re: How to break up a Flat File into individual files"
Messages sorted by: [ date ] [ thread ]

Date: Thu, 28 Oct 2004 16:05:16 -0400

I know how to do this on an XML receive pipeline. I don't know if you can
do the same for a flatfile receive pipeline. But you could do this:

1. Create a Send Port/FILE adapter/XMLSendPipeline that has on its filter
the receiveport that reads in the flat file (so the whole flatfile.XML gets
written to another file)
2. Create a receive port/FILE adapter/custom pipeline. In the custom
pipeline, drop an XMLDisassembler and specify an Envelope Schema and a
Document schema. The envelope schema should be defined as the envelope part
of your big flatfile.XML file. The document schema should be the individual
documents that you want to process. These should be all in one place in
your envelop.

Example:

Your envelope schema would define
<bigfile>
    <somehierarchy>
        <any>
    </somehierarchy>
</bigfile>
With a document body xpath of "/bigfile/somehierarchy"

Your document schema would define:
<onedoc>
<etcetc/>
</onedoc>

The result of this will be:

flatfile -> flatfile_recvport -> flatfile.XML -> file send port -> xml
receive port (big file gets split up) -> individual docs

That is, if you subscribe to the receive port that had the custom XML
pipeline, you will be receiving the indivual documents not the big document.

We're currently doing something similar very successfully, but we have flat
file -> recv port -> SQL send port -> SQL recv port w/ envelope schema ->
individual docs. The incoming file is multiple denormalized records.
Sending to SQL and back allows us to normalize and then have individual
messages.

Hope that helps. I know some of the description was scant, but it should be
a start. Let me know if you need more details (or if someone has a better
solution!). The ideal solution is a custom receive pipeline component that
would do the flatfile conversion and envelope/document shredding all in one.
I've experimented with this but haven't had time to work through it.

Mike Jansen
Sr. Software Developer
Prime ProData, Inc.
North Canton, Ohio USA
(mjansen)(at)(primepro-com)

"RichTheHSTGuy" <RichTheHSTGuy@discussions.microsoft.com> wrote in message
news:B779D811-4B13-4698-87A8-7CCE73B8FBD4@microsoft.com...
> I have built a schema to take a flat file and convert to xml but what I
get
> is 1 huge xml instead of individual xml messages. I've enclosed the
schema,
> what settings do I need to change and/or add to make that happen, breaking
it
> into individual xml instead of 1 huge 1.
>
> <?xml version="1.0" encoding="utf-16" ?>
> - <xs:schema xmlns="http://Centricity_FF.Centricity_FF_schema"
> xmlns:b="http://schemas.microsoft.com/BizTalk/2003"
> targetNamespace="http://Centricity_FF.Centricity_FF_schema"
> xmlns:xs="http://www.w3.org/2001/XMLSchema">
> - <xs:annotation>
> - <xs:appinfo>
> <b:schemaInfo count_positions_by_byte="false" standard="Flat File"
> root_reference="Patient" />
> <schemaEditorExtension:schemaInfo namespaceAlias="b"
> extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension"
> standardName="Flat File"
>
xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/Schem
aEditorExtensions" />
> </xs:appinfo>
> </xs:annotation>
> - <xs:element name="Patient">
> - <xs:annotation>
> - <xs:appinfo>
> <b:recordInfo structure="delimited"
> preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="false"
> sequence_number="1" child_delimiter_type="hex" child_order="postfix"
> child_delimiter="0x0D 0x0A" />
> </xs:appinfo>
> </xs:annotation>
> - <xs:complexType>
> - <xs:sequence minOccurs="1" maxOccurs="unbounded">
> - <xs:annotation>
> - <xs:appinfo>
> <b:groupInfo sequence_number="0" />
> </xs:appinfo>
> </xs:annotation>
> - <xs:element name="Patient_Info" nillable="true">
> - <xs:annotation>
> - <xs:appinfo>
> <b:recordInfo sequence_number="1" structure="delimited"
> preserve_delimiter_for_empty_data="true"
suppress_trailing_delimiters="false"
> child_delimiter_type="char" child_order="prefix" child_delimiter=","
> tag_name="01" />
> </xs:appinfo>
> </xs:annotation>
> - <xs:complexType>
> - <xs:sequence>
> - <xs:annotation>
> - <xs:appinfo>
> <b:groupInfo sequence_number="0" />
> </xs:appinfo>
> </xs:annotation>
> - <xs:element name="Medical_Rec_Number" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="1" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Last_Name" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="2" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="First_Name" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="3" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Street_Address1" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="4" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Street_Address2" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo justification="left" sequence_number="5" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="City" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="6" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="State" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="7" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Zip_Code" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="8" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Home_Phone_Number" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="9" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Work_Phone_number" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="10" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="SSN" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="11" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Sex" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="12" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Date_of_Birth" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="13" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_Last_Name" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="14" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_First_Name" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="15" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_Address1" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="16" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_Address2" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="17" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_City" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="18" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_State" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="19" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_Zip" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="20" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_Home_Phone" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="21" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_Work_Phone" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="22" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_SSN" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="23" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="GT_Relationship" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="24" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Attending_Doctor_Last_Name" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="25" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Attending_Doctor_First" nillable="true"
type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="26" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Admitting_Doctor" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="27" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Patient_Employer" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="28" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_Name" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="29" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_Address" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="30" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_City" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="31" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_State" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="32" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_Zip" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="33" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_Insured_Last_Name" nillable="true"
> type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="34" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_Insured_First_Name" nillable="true"
> type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="35" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_ID_Name" nillable="true" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="36" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_Group_Nbr" nillable="true"
type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="37" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_Group_Name" nillable="true"
type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="38" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_IN_Carrier_Address" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="39" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> - <xs:element name="Primary_INCarrier_City" type="xs:string">
> - <xs:annotation>
> - <xs:appinfo>
> <b:fieldInfo sequence_number="40" justification="left" />
> </xs:appinfo>
> </xs:annotation>
> </xs:element>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
> </xs:schema>

Next message: Lars W. Andersen: "Re: Updategram inserts when DB lookup fails"

Previous message: John: "Valid Schema will not pass"
In reply to: RichTheHSTGuy: "How to break up a Flat File into individual files"
Next in thread: Stephen W. Thomas: "Re: How to break up a Flat File into individual files"
Reply: Stephen W. Thomas: "Re: How to break up a Flat File into individual files"
Messages sorted by: [ date ] [ thread ]

Relevant Pages

RE: HTTP Get
... would post the data into BizTalk as XML? ... | I can now confirm that using a custom pipeline and the flat file ... | issue I have, though, is that because the string that is arriving in is ... I have set the delimiter in the schema to be &, ...
(microsoft.public.biztalk.general)
Re: MSMQT and Orchestration
... use the XMLReceive pipeline with MSMQT receiver. ... The MSMQT subservice failed to process incoming message, ... Source: "XML disassembler" Receive ... more than one schema is deployed for the same message ...
(microsoft.public.biztalk.general)
Re: How to break up a Flat File into individual files
... I too have tried to get this to work in one single Receive Pipeline with no ... > Document schema. ... The envelope schema should be defined as the envelope part ... The document schema should be the individual ...
(microsoft.public.biztalk.general)
Re: File Adapter - schema matching?
... Schema deployed and if the same data is passed to the File Adapter then the ... pipeline component that recogizes the data using the probe interface. ... > the pipeline will automatically perform a validation of the received XML ...
(microsoft.public.biztalk.general)
Re: How to break up a Flat File into individual files
... And specify the document type in the document property of the Flat File ... disassembler of your custom pipeline. ... >> Document schema. ... The envelope schema should be defined as the envelope ...
(microsoft.public.biztalk.general)