JMonks.org
Home
Latest News
Getting Started
    Framework
    IO Services
Download
Development
Architecture
Javadoc
    Framework
    IO Services
Bugs/Requests

SourceForge.net

Getting Started - IO Services

Overview

IO Services allows developers to write the batch jobs which involves reading and writing of large files of different formats very quickly and efficiently. While writing the batch jobs which involves working with files of different formats like fixed width flat, delimited flat, XML and OFX, developers can concentrate more on the business logic than the core tasks like parsing and generating of the files. IO API parses the required file and return the records in the file as java objects and accepts the java objects represents the records to generate the file.

IO API requires some information from the developers to understand the file structure that it is going to prase and generate. It accepts this information in the format of XML and this is called the file spec. This file spec explains the details like file format and information to identify the records and fields in the file.


Top

System Requirements

IO Services can be used in any applications running on any JRE starting from 1.4.2.


Top

Download

IO Services can be downloaded here.

Download the Batch_IO_1.0.zip from the available downloadable files. This distribution contains the IO services jar (batch_io_1.0.jar), all the jars required in the runtime and source code to add to debug path if needed.


Top

Installation

Following are the Setps to install the IO Services.

  • Extract the downloaded Batch_IO_1.0.zip file into any directory.
  • Keep the batch_io_1.0.jar file available in the extracted directory in the classpath.
  • Keep all the jars available in the lib directory of extracted directory in the classpath.

Top

Working with Fixed Width flat files

Flat files are the most common type of files being used by the enterprises to exchange the data to and forth. IO Services provides the implementation to parse and generate the fixed width flat files. Lets take a task of reading and writing the following fixed width flat file to understand the implementation better.

Save the following content as sample-fixed-width-file.dat file. This file can be downloaded
here.

120060526
512345678910000012.0034343434347878787878
512345678920000013.0034343434347878787878
512345678930000014.0034343434347878787878
60000000.00

Lets take a moment to understand the structure of the file. This file basically has the 3 kind of records and each record starts with particular string and the length of each record type is fixed. In each record all the fields are starting at one position and ending at one position. Following sections will explain how to write file spec for this kind of structure and how to read and write the file using that file spec.

Understanding Fixed Width flat file spec

As mentioned in the Overview section, file spec will be defined in XML format and tells the structure of the file, information to identify the records and fields in those records.

All the fixed width flat file specs will have the following XML format.

<file-spec file-type="fixed-width-flat">
    <record-spec record-type="DETAIL" starts-with="5" record-length="42">
        <field-spec field-name="field-name1" start-pos="12" end-pos="23"/>
        <-- More field specs will follow -->
    </record-spec>
    <-- More records specs will follow -->
</file-spec>

In this file spec, file-spec element provides the information about the type of the file. For all the fixed width flat files the file-type attribute value should be fixed-width-flat. record-spec element provides the information to identify the record in the file. Fixed width flat file record-spec requires the attributes "record-type", "starts-with" and "record-length". "record-type" attribute value can be anything that you want to use in your java code, "starts-with" attribute value tells the parser that each record of that type should starts with that values, and "record-length" attribute values tells the length of the record of that type. field-spec element requires two attributes, "start-pos" which tells where the field starts and "end-pos" which tells where the field ends.

By using this information, we can define the file spec for our sample-fixed-width-file.dat file, which looks like the below. Save the following contents into a sample-fixed-width-file-spec.xml file. This can be downloaded
here.

<file-spec file-type="fixed-width-flat">
    <record-spec record-type="HEADER" starts-with="1" record-length="9">
        <field-spec field-name="timestamp" start-pos="2" end-pos="9"/>
    </record-spec>
    <record-spec record-type="DETAIL" starts-with="5" record-length="42">
        <field-spec field-name="field1" start-pos="2" end-pos="11"/>
        <field-spec field-name="field2" start-pos="12" end-pos="21"/>
        <field-spec field-name="field3" start-pos="22" end-pos="31"/>
        <field-spec field-name="field4" start-pos="32" end-pos="41"/>
    </record-spec>
    <record-spec record-type="TRAILER" starts-with="6" record-length="9">
        <field-spec field-name="recordCount" start-pos="2" end-pos="9"/>
    </record-spec>
</file-spec>

Reading fixed width flat files

Following set of Stpes explains how to read the sample-fixed-width-file.dat file using file spec sample-fixed-width-file-spec.xml defined in earlier step.

  1. Get the FileReader by passing the input streams of sample-fixed-width-file.dat file and sample-fixed-width-file-spec.xml file. Adjust path references according to your directory structure.

    FileReader fileReader=FileReader.getFileReader(
         new FileInputStream("C:\\sample-fixed-width-file.dat"),
         new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));

  2. Read the records from the file reader in iterative manner untile you receive the null.

    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
    }

  3. Read the fields from records being read from the file reader. Fields can be accessed using name given in the file spec.

    if(record.getRecordType().equals(RecordType.DETAIL))
    {
    System.out.println(record.readField("field1"));
    System.out.println(record.readField("field2"));
    System.out.println(record.readField("field3"));
    System.out.println(record.readField("field4"));
    }
    else if(record.getRecordType().equals(RecordType.HEADER))
    {
    System.out.println(record.readField("timestamp"));
    }
    else if(record.getRecordType().equals(RecordType.TRAILER))
    {
    System.out.println(record.readField("recordCount"));
    }
    else
    System.out.println(record.getRecordType().toString());

  4. Close the file reader.

    fileReader.close();

For better clarity, all the code is given below.

    FileReader fileReader=FileReader.getFileReader(
            new FileInputStream("C:\\sample_fixed.dat"),
            new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
        if(record.getRecordType().equals(RecordType.DETAIL))
        {
            System.out.println(record.readField("field1"));
            System.out.println(record.readField("field2"));
            System.out.println(record.readField("field3"));
            System.out.println(record.readField("field4"));
        }
        else if(record.getRecordType().equals(RecordType.HEADER))
        {
             System.out.println(record.readField("timestamp"));
        }
        else if(record.getRecordType().equals(RecordType.TRAILER))
        {
             System.out.println(record.readField("recordCount"));
        }
        else
            System.out.println(record.getRecordType().toString());
        }
    fileReader.close();

Writing fixed width flat files

Following set of Stpes explains how to generate the sample-fixed-width-file.dat file using file spec sample-fixed-width-file-spec.xml defined earlier.

  1. Get the FileWriter by passing the output stream of sample-fixed-width-file.dat file and input stream of sample-fixed-width-file-spec.xml file. Adjust path references according to your directory structure.

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample-fixed-width-file.dat"),
         new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));

  2. Create the writer record from the file writer by telling the type of record you want to create. Once you get the recrod, write all the field values by using the field name given in the file spec and submit the records to the file reader to write them into the file.

    WriterRecord headerRecord=fileWriter.createWriterRecord(RecordType.HEADER);
    headerRecord.writeField("timestamp", "20060812");
    fileWriter.writeRecord(headerRecord);

    WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeField("field1", "1234567891");
    detailRecord.writeField("field2", "0000012.00");
    detailRecord.writeField("field3", "3434343434");
    detailRecord.writeField("field4", "7878787878");
    fileWriter.writeRecord(detailRecord);

    WriterRecord trailerRecord=fileWriter.createWriterRecord(RecordType.TRAILER);
    trailerRecord.writeField("recordCount", "0000000.00");
    fileWriter.writeRecord(trailerRecord);

  3. Close the file writer.

    fileWriter.close();

For better clarity, all the code is given below.

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample-fixed-width-file.dat"),
         new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    WriterRecord headerRecord=fileWriter.createWriterRecord(RecordType.HEADER);
    headerRecord.writeField("timestamp", "20060812");
    fileWriter.writeRecord(headerRecord);

    WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeField("field1", "1234567891");
    detailRecord.writeField("field2", "0000012.00");
    detailRecord.writeField("field3", "3434343434");
    detailRecord.writeField("field4", "7878787878");
    fileWriter.writeRecord(detailRecord);

    WriterRecord trailerRecord=fileWriter.createWriterRecord(RecordType.TRAILER);
    trailerRecord.writeField("recordCount", "0000000.00");
    fileWriter.writeRecord(trailerRecord);
    fileReader.close();


Top

Working with XML files

Now-a-days XML files have become the standard for the information exchange between enterprises. IO Services provides the implementation to parse and read the records from the XML file and generate the XML files from record objects. Lets start reading and writing the following XML file to understand the implementation.

Save the following content as sample-xml-file.xml file. This file can be downloaded
here.

<?xml version='1.0' encoding='ISO-8859-1'?>
<sample-root>
    <sample-header>
        <file-type>Employee Records</file-type>
    </sample-header>
    <sample-detail>
         <first-name>Suresh</first-name>
         <last-name>Pragada</last-name>
         <dept-info>
             <dept-name>IT</dept-name>
             <dept-location>LOC1</dept-location>
         </dept-info>
    </sample-detail>
    <sample-trailer>
         <transaction-count>1</transaction-count>
    </sample-trailer>
</sample-root>

Understanding XML file spec

As mentioned in the Overview section, file spec will be defined in XML format and tells the structure of the file, information to identify the records.

All the XML file specs will have the following XML format.

<file-spec file-type="xml" root-element="sample-root">
    <record-spec record-type="DETAIL" record-xpath="/sample-root/sample-header"/>
    <-- More records specs will follow -->
</file-spec>

For all the XML files the file-type attribute value should be xml and it this requires an additional attribute "root-element" which tells the root element of the XML document. This will be used in generating the file and doing basic validation while parsing. record-spec element provides the information to identify the record in the file. In XML file implementation, records will be identified using XPATH. So, the record-spec requires the attribute "record-xpath" to identify the record along with the "record-type" attribute which will be used to identify the type of record. XML file spec doesnt require any field-spec, since it returns all the data underneath that XPATH element as an object.

By using this information, we can define the file spec for our sample-xml-file.xml file, which looks like the below. Save the following contents into a sample-xml-file-spec.xml file. This can be downloaded
here.

<file-spec file-type="xml" root-element="sample-root">
    <record-spec record-type="HEADER" record-xpath="/sample-root/sample-header"/>
    <record-spec record-type="DETAIL" record-xpath="/sample-root/sample-detail"/>
    <record-spec record-type="TRAILER" record-xpath="/sample-root/sample-trailer"/>
</file-spec>

Reading XML files

Following set of Stpes explains how to read the sample-xml-file.xml file using file spec sample-xml-file-spec.xml defined in earlier step.

  1. Get the FileReader by passing the input streams of sample-xml-file.dat file and sample-xml-file-spec.xml file. Adjust path references according to your directory structure.

    FileReader fileReader=FileReader.getFileReader(
         new FileInputStream("C:\\sample-xml-file.xml"),
         new FileInputStream("C:\\sample-xml-file-spec.xml"));

  2. Read the records from the file reader in iterative manner untile you receive the null.

    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
    }

  3. Read the fields from records being read from the file reader. Fields can be accessed using name given in the file spec.

    if(record.getRecordType().equals(RecordType.DETAIL))
    {
    XMLReaderRecord xmlDetailRecord=(XMLReaderRecord)record;
    System.out.println(xmlDetailRecord.readSimpleElement("first-name"));
    System.out.println(record.readSimpleElement("last-name"));

    XMLReaderRecord deptInfoComplexRecord=(XMLReaderRecord)xmlDetailRecord.readComplexElement("dept-info");
    System.out.println(deptInfoComplexRecord.readSimpleElement("dept-name"));
    System.out.println(deptInfoComplexRecord.readSimpleElement("dept-location"));
    }
    else if(record.getRecordType().equals(RecordType.HEADER))
    {
    System.out.println(record.readField("file-type"));
    }
    else if(record.getRecordType().equals(RecordType.TRAILER))
    {
    System.out.println(record.readField("transaction-count"));
    }
    else
    System.out.println(record.getRecordType().toString());

    XMLReaderRecord few more additional methods to read the nested elements and repeat elements from the reader records.
  4. Close the file reader.

    fileReader.close();

For better clarity, all the code is given below.

    FileReader fileReader=FileReader.getFileReader(
            new FileInputStream("C:\\sample_fixed.dat"),
            new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
        if(record.getRecordType().equals(RecordType.DETAIL))
        {
            XMLReaderRecord xmlDetailRecord=(XMLReaderRecord)record;
            System.out.println(xmlDetailRecord.readSimpleElement("first-name"));
            System.out.println(record.readSimpleElement("last-name"));

            XMLReaderRecord deptInfoComplexRecord=(XMLReaderRecord)xmlDetailRecord.readComplexElement("dept-info");
            System.out.println(deptInfoComplexRecord.readSimpleElement("dept-name"));
            System.out.println(deptInfoComplexRecord.readSimpleElement("dept-location"));
        }
        else if(record.getRecordType().equals(RecordType.HEADER))
        {
             System.out.println(record.readField("file-type"));
        }
        else if(record.getRecordType().equals(RecordType.TRAILER))
        {
             System.out.println(record.readField("transaction-count"));
        }
        else
            System.out.println(record.getRecordType().toString());
        }
    fileReader.close();

Writing XML files

Following set of Stpes explains how to generate the sample-xml-file.xml file using file spec sample-xml-file-spec.xml defined earlier.

  1. Get the FileWriter by passing the output stream of sample-xml-file.xml file and input stream of sample-xml-file-spec.xml file. Adjust path references according to your directory structure.

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample-xml-file.xml"),
         new FileInputStream("C:\\sample-xml-file-spec.xml"));

  2. Create the writer record from the file writer by telling the type of record you want to create. Once you get the recrod, write all the field values by using the field name given in the file spec and submit the records to the file reader to write them into the file. XMLWriterRecord provides few more methods to write the nested(complex) elements and repeat elements into the record.

    XMLWriterRecord headerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.HEADER);
    headerRecord.writeSimpleElement("file-type", "Employee Records & Details");
    fileWriter.writeRecord(headerRecord);

    XMLWriterRecord detailRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeSimpleElement("first-name", "Suresh");
    detailRecord.writeField("last-name", "Pragada");

    XMLWriterRecord deptComplexRecord=(XMLWriterRecord)detailRecord.createComplexElement("dept-info");
    deptComplexRecord.writeSimpleElement("dept-name", "IT");
    deptComplexRecord.writeSimpleElement("dept-location", "LOC1");
    fileWriter.writeRecord(detailRecord);

    XMLWriterRecord trailerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.TRAILER);
    trailerRecord.writeSimpleElement("transaction-count", "1");
    fileWriter.writeRecord(trailerRecord);

  3. Close the file writer.

    fileWriter.close();

For better clarity, all the code is given below.

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample-xml-file.xml"),
         new FileInputStream("C:\\sample-xml-file-spec.xml"));

    XMLWriterRecord headerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.HEADER);
    headerRecord.writeSimpleElement("file-type", "Employee Records & Details");
    fileWriter.writeRecord(headerRecord);

    XMLWriterRecord detailRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeSimpleElement("first-name", "Suresh");
    detailRecord.writeField("last-name", "Pragada");

    XMLWriterRecord deptComplexRecord=(XMLWriterRecord)detailRecord.createComplexElement("dept-info");
    deptComplexRecord.writeSimpleElement("dept-name", "IT");
    deptComplexRecord.writeSimpleElement("dept-location", "LOC1");
    fileWriter.writeRecord(detailRecord);

    XMLWriterRecord trailerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.TRAILER);
    trailerRecord.writeSimpleElement("transaction-count", "1");
    fileWriter.writeRecord(trailerRecord);

    fileReader.close();


Top

Working with Delimited flat files

Delimited flat files consists of the records in which fields are seperated by a special character called as a delimiter and every record consists of same number of fields in each record. Lets take a task of reading and writing the following delimited flat file to understand the implementation IO Services for delimited flat files.

Save the following content as sample-delimited-file.dat file. This file can be downloaded
here.

1234567891|0000012.00|3434343434|7878787878
1234567892|0000013.00|3434343434|7878787878
1234567893|0000014.00|3434343434|7878787878
1234567894|0000015.00|3434343434|7878787878

Understanding Delimited flat file spec

As mentioned in the Overview section, file spec will be defined in XML format and tells the structure of the file, information to identify the records and fields in those records.

All the delimited flat file specs will have the following XML format.

<file-spec file-type="delimited-flat">
    <record-spec record-type="DETAIL" delimiter="|" field-count="4">
        <field-spec field-name="field-name1" index="1"/>
        <-- More field specs will follow -->
    </record-spec>
</file-spec>

In this file spec, file-spec element provides the information about the type of the file. For all the delimited flat files the file-type attribute value should be "delimited-flat". record-spec element provides the information to identify the record in the file. Delimited flat file record-spec requires the attributes "record-type", "delimiter" and "field-count". "record-type" attribute value can be anything that you want to use in your java code, "delimiter" attribute value tells the parser that all the fields in that record will be delimited by that value and "field-count" attribute values tells the number of fields available in the record. field-spec element requires two attributes, "index" which tells where the field can be found in that record. Typically, all the delimited file specs will contain only one record spec.

By using this information, we can define the file spec for our sample-delimiter-file.dat file, which looks like the below. Save the following contents into a sample-delimited-file-spec.xml file. This can be downloaded
here.

<file-spec file-type="delimited-flat">
    <record-spec record-type="DETAIL" delimiter="|" field-count="4">
        <field-spec field-name="field1" index="1"/>
        <field-spec field-name="field2" index="2"/>
        <field-spec field-name="field3" index="3"/>
        <field-spec field-name="field4" index="4"/>
    </record-spec>
</file-spec>

Reading delimited flat files

Following set of Stpes explains how to read the sample-delimited-file.dat file using file spec sample-delimited-file-spec.xml defined in earlier step.

  1. Get the FileReader by passing the input streams of sample-delimited-file.dat file and sample-delimited-file-spec.xml file. Adjust path references according to your directory structure.

    FileReader fileReader=FileReader.getFileReader(
         new FileInputStream("C:\\sample-delimited-file.dat"),
         new FileInputStream("C:\\sample-delimited-file-spec.xml"));

  2. Read the records from the file reader in iterative manner untile you receive the null.

    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
    }

  3. Read the fields from records being read from the file reader. Fields can be accessed using name given in the file spec.

    if(record.getRecordType().equals(RecordType.DETAIL))
    {
    System.out.println(record.readField("field1"));
    System.out.println(record.readField("field2"));
    System.out.println(record.readField("field3"));
    System.out.println(record.readField("field4"));
    }
    else
    System.out.println(record.getRecordType().toString());

  4. Close the file reader.

    fileReader.close();

For better clarity, all the code is given below.

    FileReader fileReader=FileReader.getFileReader(
            new FileInputStream("C:\\sample-delimited-file.dat"),
            new FileInputStream("C:\\sample-delimited-file-spec.xml"));
    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
        if(record.getRecordType().equals(RecordType.DETAIL))
        {
            System.out.println(record.readField("field1"));
            System.out.println(record.readField("field2"));
            System.out.println(record.readField("field3"));
            System.out.println(record.readField("field4"));
        }
        else
            System.out.println(record.getRecordType().toString());
        }
    fileReader.close();

Writing delimited flat files

Following set of Stpes explains how to generate the sample-delimited-file.dat file using file spec sample-delimited-file-spec.xml defined earlier.

  1. Get the FileWriter by passing the output stream of sample-delimited-file.dat file and input stream of sample-delimited-file-spec.xml file. Adjust path references according to your directory structure.

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample-delimited-file.dat"),
         new FileInputStream("C:\\sample-delimited-file-spec.xml"));

  2. Create the writer record from the file writer by telling the type of record you want to create. Once you get the recrod, write all the field values by using the field name given in the file spec and submit the records to the file reader to write them into the file.

    WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeField("field1", "1234567891");
    detailRecord.writeField("field2", "0000012.00");
    detailRecord.writeField("field3", "3434343434");
    detailRecord.writeField("field4", "7878787878");
    fileWriter.writeRecord(detailRecord);

  3. Close the file writer.

    fileWriter.close();

For better clarity, all the code is given below.

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample_fixed.dat"),
         new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeField("field1", "1234567891");
    detailRecord.writeField("field2", "0000012.00");
    detailRecord.writeField("field3", "3434343434");
    detailRecord.writeField("field4", "7878787878");
    fileWriter.writeRecord(detailRecord);

    fileReader.close();


Top

More on XMLReaderRecord & XMLWriterRecord

XMLFileReader returns xml records as XMLReaderRecord objects. This provides additional methods to read the nested(complex) elements and repeat elements (elements that repeat). In the same way, XMLFileWriter returns XMLWriterRecord object which provides additonal methods to create the nested elements and repeat elements.

Please see the
org.jmonks.batch.io.xml package document for additonal details and complete example.


Top

Feedback

I am trying my level best to keep this tutorial understandable and it is upto date with the releases. If you see any mistakes or inconsistency in examples, please send me a message (if you are a member of sourceforge) or an email.


Top