Getting Started - IO Services

Overview

IO Services allows developers to write the batch jobs which involves reading and writing of large files of different formats very quickly and efficiently. While writing the batch jobs which involves working with files of different formats like fixed width flat, delimited flat, XML and OFX, developers can concentrate more on the business logic than the core tasks like parsing and generating of the files. IO API parses the required file and return the records in the file as java objects and accepts the java objects represents the records to generate the file.

IO API requires some information from the developers to understand the file structure that it is going to prase and generate. It accepts this information in the format of XML and this is called the file spec. This file spec explains the details like file format and information to identify the records and fields in the file.

System Requirements

IO Services can be used in any applications running on any JRE starting from 1.4.2.

Download

Installation

Following are the Setps to install the IO Services.

• Extract the downloaded Batch_IO_1.0.zip file into any directory.
• Keep the batch_io_1.0.jar file available in the extracted directory in the classpath.
• Keep all the jars available in the lib directory of extracted directory in the classpath.

Working with Fixed Width flat files

Understanding Fixed Width flat file spec

As mentioned in the Overview section, file spec will be defined in XML format and tells the structure of the file, information to identify the records and fields in those records.

All the fixed width flat file specs will have the following XML format.

<file-spec file-type="fixed-width-flat">
    <record-spec record-type="DETAIL" starts-with="5" record-length="42">
        <field-spec field-name="field-name1" start-pos="12" end-pos="23"/>
        <-- More field specs will follow -->
    </record-spec>
    <-- More records specs will follow -->
</file-spec>

Reading fixed width flat files

Following set of Stpes explains how to read the sample-fixed-width-file.dat file using file spec sample-fixed-width-file-spec.xml defined in earlier step.

1. Get the FileReader by passing the input streams of sample-fixed-width-file.dat file and sample-fixed-width-file-spec.xml file. Adjust path references according to your directory structure.

   FileReader fileReader=FileReader.getFileReader(
        new FileInputStream("C:\\sample-fixed-width-file.dat"),
        new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
   

2. Read the records from the file reader in iterative manner untile you receive the null.

   ReaderRecord record=null;
   while((record=fileReader.getNextRecord())!=null)
   {
   }
   

3. Read the fields from records being read from the file reader. Fields can be accessed using name given in the file spec.

   if(record.getRecordType().equals(RecordType.DETAIL))
   {
   System.out.println(record.readField("field1"));
   System.out.println(record.readField("field2"));
   System.out.println(record.readField("field3"));
   System.out.println(record.readField("field4"));
   }
   else if(record.getRecordType().equals(RecordType.HEADER))
   {
   System.out.println(record.readField("timestamp"));
   }
   else if(record.getRecordType().equals(RecordType.TRAILER))
   {
   System.out.println(record.readField("recordCount"));
   }
   else
   System.out.println(record.getRecordType().toString());
   

4. Close the file reader.

   fileReader.close();
   

    FileReader fileReader=FileReader.getFileReader(
            new FileInputStream("C:\\sample_fixed.dat"),
            new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
        if(record.getRecordType().equals(RecordType.DETAIL))
        {
            System.out.println(record.readField("field1"));
            System.out.println(record.readField("field2"));
            System.out.println(record.readField("field3"));
            System.out.println(record.readField("field4"));
        }
        else if(record.getRecordType().equals(RecordType.HEADER))
        {
             System.out.println(record.readField("timestamp"));
        }
        else if(record.getRecordType().equals(RecordType.TRAILER))
        {
             System.out.println(record.readField("recordCount"));
        }
        else
            System.out.println(record.getRecordType().toString());
        }
    fileReader.close();

Writing fixed width flat files

Following set of Stpes explains how to generate the sample-fixed-width-file.dat file using file spec sample-fixed-width-file-spec.xml defined earlier.

1. Get the FileWriter by passing the output stream of sample-fixed-width-file.dat file and input stream of sample-fixed-width-file-spec.xml file. Adjust path references according to your directory structure.

   FileWriter fileWriter=FileWriter.getFileWriter(
        new FileOutputStream("C:\\sample-fixed-width-file.dat"),
        new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
   

2. Create the writer record from the file writer by telling the type of record you want to create. Once you get the recrod, write all the field values by using the field name given in the file spec and submit the records to the file reader to write them into the file.

   WriterRecord headerRecord=fileWriter.createWriterRecord(RecordType.HEADER);
   headerRecord.writeField("timestamp", "20060812");
   fileWriter.writeRecord(headerRecord);
   
   WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
   detailRecord.writeField("field1", "1234567891");
   detailRecord.writeField("field2", "0000012.00");
   detailRecord.writeField("field3", "3434343434");
   detailRecord.writeField("field4", "7878787878");
   fileWriter.writeRecord(detailRecord);
   
   WriterRecord trailerRecord=fileWriter.createWriterRecord(RecordType.TRAILER);
   trailerRecord.writeField("recordCount", "0000000.00");
   fileWriter.writeRecord(trailerRecord);
   
   

3. Close the file writer.

   fileWriter.close();
   

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample-fixed-width-file.dat"),
         new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    WriterRecord headerRecord=fileWriter.createWriterRecord(RecordType.HEADER);
    headerRecord.writeField("timestamp", "20060812");
    fileWriter.writeRecord(headerRecord);

    WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeField("field1", "1234567891");
    detailRecord.writeField("field2", "0000012.00");
    detailRecord.writeField("field3", "3434343434");
    detailRecord.writeField("field4", "7878787878");
    fileWriter.writeRecord(detailRecord);

    WriterRecord trailerRecord=fileWriter.createWriterRecord(RecordType.TRAILER);
    trailerRecord.writeField("recordCount", "0000000.00");
    fileWriter.writeRecord(trailerRecord);
    fileReader.close();

Working with XML files

Understanding XML file spec

As mentioned in the Overview section, file spec will be defined in XML format and tells the structure of the file, information to identify the records.

All the XML file specs will have the following XML format.

<file-spec file-type="xml" root-element="sample-root">
    <record-spec record-type="DETAIL" record-xpath="/sample-root/sample-header"/>
    <-- More records specs will follow -->
</file-spec>

Reading XML files

Following set of Stpes explains how to read the sample-xml-file.xml file using file spec sample-xml-file-spec.xml defined in earlier step.

1. Get the FileReader by passing the input streams of sample-xml-file.dat file and sample-xml-file-spec.xml file. Adjust path references according to your directory structure.

   FileReader fileReader=FileReader.getFileReader(
        new FileInputStream("C:\\sample-xml-file.xml"),
        new FileInputStream("C:\\sample-xml-file-spec.xml"));
   

2. Read the records from the file reader in iterative manner untile you receive the null.

   ReaderRecord record=null;
   while((record=fileReader.getNextRecord())!=null)
   {
   }
   

3. Read the fields from records being read from the file reader. Fields can be accessed using name given in the file spec.

   if(record.getRecordType().equals(RecordType.DETAIL))
   {
   XMLReaderRecord xmlDetailRecord=(XMLReaderRecord)record;
   System.out.println(xmlDetailRecord.readSimpleElement("first-name"));
   System.out.println(record.readSimpleElement("last-name"));
   
   XMLReaderRecord deptInfoComplexRecord=(XMLReaderRecord)xmlDetailRecord.readComplexElement("dept-info");
   System.out.println(deptInfoComplexRecord.readSimpleElement("dept-name"));
   System.out.println(deptInfoComplexRecord.readSimpleElement("dept-location"));
   }
   else if(record.getRecordType().equals(RecordType.HEADER))
   {
   System.out.println(record.readField("file-type"));
   }
   else if(record.getRecordType().equals(RecordType.TRAILER))
   {
   System.out.println(record.readField("transaction-count"));
   }
   else
   System.out.println(record.getRecordType().toString());
   

   XMLReaderRecord few more additional methods to read the nested elements and repeat elements from the reader records.
4. Close the file reader.

   fileReader.close();
   

    FileReader fileReader=FileReader.getFileReader(
            new FileInputStream("C:\\sample_fixed.dat"),
            new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
        if(record.getRecordType().equals(RecordType.DETAIL))
        {
            XMLReaderRecord xmlDetailRecord=(XMLReaderRecord)record;
            System.out.println(xmlDetailRecord.readSimpleElement("first-name"));
            System.out.println(record.readSimpleElement("last-name"));

            XMLReaderRecord deptInfoComplexRecord=(XMLReaderRecord)xmlDetailRecord.readComplexElement("dept-info");
            System.out.println(deptInfoComplexRecord.readSimpleElement("dept-name"));
            System.out.println(deptInfoComplexRecord.readSimpleElement("dept-location"));
        }
        else if(record.getRecordType().equals(RecordType.HEADER))
        {
             System.out.println(record.readField("file-type"));
        }
        else if(record.getRecordType().equals(RecordType.TRAILER))
        {
             System.out.println(record.readField("transaction-count"));
        }
        else
            System.out.println(record.getRecordType().toString());
        }
    fileReader.close();

Writing XML files

Following set of Stpes explains how to generate the sample-xml-file.xml file using file spec sample-xml-file-spec.xml defined earlier.

1. Get the FileWriter by passing the output stream of sample-xml-file.xml file and input stream of sample-xml-file-spec.xml file. Adjust path references according to your directory structure.

   FileWriter fileWriter=FileWriter.getFileWriter(
        new FileOutputStream("C:\\sample-xml-file.xml"),
        new FileInputStream("C:\\sample-xml-file-spec.xml"));
   

2. Create the writer record from the file writer by telling the type of record you want to create. Once you get the recrod, write all the field values by using the field name given in the file spec and submit the records to the file reader to write them into the file. XMLWriterRecord provides few more methods to write the nested(complex) elements and repeat elements into the record.

   XMLWriterRecord headerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.HEADER);
   headerRecord.writeSimpleElement("file-type", "Employee Records & Details");
   fileWriter.writeRecord(headerRecord);
   
   XMLWriterRecord detailRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.DETAIL);
   detailRecord.writeSimpleElement("first-name", "Suresh");
   detailRecord.writeField("last-name", "Pragada");
   
   XMLWriterRecord deptComplexRecord=(XMLWriterRecord)detailRecord.createComplexElement("dept-info");
   deptComplexRecord.writeSimpleElement("dept-name", "IT");
   deptComplexRecord.writeSimpleElement("dept-location", "LOC1");
   fileWriter.writeRecord(detailRecord);
   
   XMLWriterRecord trailerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.TRAILER);
   trailerRecord.writeSimpleElement("transaction-count", "1");
   fileWriter.writeRecord(trailerRecord);
   

3. Close the file writer.

   fileWriter.close();
   

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample-xml-file.xml"),
         new FileInputStream("C:\\sample-xml-file-spec.xml"));

    XMLWriterRecord headerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.HEADER);
    headerRecord.writeSimpleElement("file-type", "Employee Records & Details");
    fileWriter.writeRecord(headerRecord);

    XMLWriterRecord detailRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeSimpleElement("first-name", "Suresh");
    detailRecord.writeField("last-name", "Pragada");

    XMLWriterRecord deptComplexRecord=(XMLWriterRecord)detailRecord.createComplexElement("dept-info");
    deptComplexRecord.writeSimpleElement("dept-name", "IT");
    deptComplexRecord.writeSimpleElement("dept-location", "LOC1");
    fileWriter.writeRecord(detailRecord);

    XMLWriterRecord trailerRecord=(XMLWriterRecord)fileWriter.createWriterRecord(RecordType.TRAILER);
    trailerRecord.writeSimpleElement("transaction-count", "1");
    fileWriter.writeRecord(trailerRecord);

    fileReader.close();

Working with Delimited flat files

Understanding Delimited flat file spec

As mentioned in the Overview section, file spec will be defined in XML format and tells the structure of the file, information to identify the records and fields in those records.

All the delimited flat file specs will have the following XML format.

<file-spec file-type="delimited-flat">
    <record-spec record-type="DETAIL" delimiter="|" field-count="4">
        <field-spec field-name="field-name1" index="1"/>
        <-- More field specs will follow -->
    </record-spec>
</file-spec>

Reading delimited flat files

Following set of Stpes explains how to read the sample-delimited-file.dat file using file spec sample-delimited-file-spec.xml defined in earlier step.

1. Get the FileReader by passing the input streams of sample-delimited-file.dat file and sample-delimited-file-spec.xml file. Adjust path references according to your directory structure.

   FileReader fileReader=FileReader.getFileReader(
        new FileInputStream("C:\\sample-delimited-file.dat"),
        new FileInputStream("C:\\sample-delimited-file-spec.xml"));
   

2. Read the records from the file reader in iterative manner untile you receive the null.

   ReaderRecord record=null;
   while((record=fileReader.getNextRecord())!=null)
   {
   }
   

3. Read the fields from records being read from the file reader. Fields can be accessed using name given in the file spec.

   if(record.getRecordType().equals(RecordType.DETAIL))
   {
   System.out.println(record.readField("field1"));
   System.out.println(record.readField("field2"));
   System.out.println(record.readField("field3"));
   System.out.println(record.readField("field4"));
   }
   else
   System.out.println(record.getRecordType().toString());
   

4. Close the file reader.

   fileReader.close();
   

    FileReader fileReader=FileReader.getFileReader(
            new FileInputStream("C:\\sample-delimited-file.dat"),
            new FileInputStream("C:\\sample-delimited-file-spec.xml"));
    ReaderRecord record=null;
    while((record=fileReader.getNextRecord())!=null)
    {
        if(record.getRecordType().equals(RecordType.DETAIL))
        {
            System.out.println(record.readField("field1"));
            System.out.println(record.readField("field2"));
            System.out.println(record.readField("field3"));
            System.out.println(record.readField("field4"));
        }
        else
            System.out.println(record.getRecordType().toString());
        }
    fileReader.close();

Writing delimited flat files

Following set of Stpes explains how to generate the sample-delimited-file.dat file using file spec sample-delimited-file-spec.xml defined earlier.

1. Get the FileWriter by passing the output stream of sample-delimited-file.dat file and input stream of sample-delimited-file-spec.xml file. Adjust path references according to your directory structure.

   FileWriter fileWriter=FileWriter.getFileWriter(
        new FileOutputStream("C:\\sample-delimited-file.dat"),
        new FileInputStream("C:\\sample-delimited-file-spec.xml"));
   

2. Create the writer record from the file writer by telling the type of record you want to create. Once you get the recrod, write all the field values by using the field name given in the file spec and submit the records to the file reader to write them into the file.

   WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
   detailRecord.writeField("field1", "1234567891");
   detailRecord.writeField("field2", "0000012.00");
   detailRecord.writeField("field3", "3434343434");
   detailRecord.writeField("field4", "7878787878");
   fileWriter.writeRecord(detailRecord);
   
   

3. Close the file writer.

   fileWriter.close();
   

    FileWriter fileWriter=FileWriter.getFileWriter(
         new FileOutputStream("C:\\sample_fixed.dat"),
         new FileInputStream("C:\\sample-fixed-width-file-spec.xml"));
    WriterRecord detailRecord=fileWriter.createWriterRecord(RecordType.DETAIL);
    detailRecord.writeField("field1", "1234567891");
    detailRecord.writeField("field2", "0000012.00");
    detailRecord.writeField("field3", "3434343434");
    detailRecord.writeField("field4", "7878787878");
    fileWriter.writeRecord(detailRecord);

    fileReader.close();

More on XMLReaderRecord & XMLWriterRecord

Feedback