Configure CAOS SOAP API batching in IDMC

CAOS SOAP API batching

Informatica Data Management Cloud (IDMC) sends API requests one record at a time with mappings unless you intentionally configure it to batch. While the CAOS is built for this kind of volume (and much more), this leads to slow and inefficient runtimes for jobs with large amounts of data on the IDMC side. Reducing the number of requests to the CAOS will decrease the processing burden on our Secure Agent servers and free up bandwidth for all users.

Prerequisites

• Access to CAOS data. This can be requested through this form.
• A list of values you can use to call the CAOS API — for instance: netIDs, PVIs, emplIDs, program codes, etc.

The mapping

This process differs from the Configure batching for Person API requests in IDMC in that we nest each of our values in XML within the body of the request instead of the URL (or query parameters). Below is a visual example of what we're constructing with our Data Integration, part of IDMC, mapping:

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:caos="http://services.wisc.edu/caos-messaging" xmlns:caos1="http://services.wisc.edu/caos">
   <soapenv:Header/>
   <soapenv:Body>
      <caos:GetStudentsRequest>
         <caos:personQuery>
            <caos:pvi>examplepvi1</caos:pvi>
         </caos:personQuery>
         <caos:personQuery>
            <caos:pvi>examplepvi2</caos:pvi>
         </caos:personQuery>
         <caos:personQuery>
            <caos:pvi>examplepvi3</caos:pvi>
         </caos:personQuery>
         <caos:studentDataOptions>
            <caos:option>ACADEMIC_OBJECTIVE</caos:option>
         </caos:studentDataOptions>
      </caos:GetStudentsRequest>
   </soapenv:Body>
</soapenv:Envelope>

Note that the studentDataOptions value ACADEMIC_OBJECTIVE is optional and not required for batching requests to CAOS. It is included here because it is common — though not required — for multiple elements to be mapped in the Web Services Transformation.

1. The source is a CSV file containing the data element needed for sending CAOS requests. Depending on the operation you're calling (GetStudents, GetPresentTerm, GetSubject, etc.), you will need a different element. For this tutorial, we're sending GetStudents requests using PVIs, which are unique person identifiers.

2. The first Expression Transformation creates a new column that numbers each record. We also add a third column hard-coded to ACADEMIC_OBJECTIVE for each record. This is optional and ensures we receive enrollment data for each PVI we request. Depending on your use case, this could be any number of options.

   Below are the settings for the three fields: - Field One: - Field Type: Variable Field - Name: v_Counter (the v_ prefix helps identify variable fields as you work through the Expression Transformation) - Type: bigint - Precision: 19 (default) - Expression: v_Counter+1 - Field Two: - Field Type: Output Field - Name: o_RowNum - Type: bigint - Precision: 19 (default) - Expression: v_Counter - Field Three (optional): - Field Type: Output Field - Name: o_AcademicObjective - Type: string - Precision: 100 - Expression: 'ACADEMIC_OBJECTIVE'

   

3. This Expression Transformation adds a batch number column. For the CAOS SOAP API, the upstream limit is upwards of 1,000 records per request; however, IDMC enforces response message size limits. Payload size varies by integration depending on how much data you're requesting — if you include several studentDataOption entries, each record's response is significantly larger. We recommend between 10 and 50 records per batch, but use your judgment. If batches are too large, you may see a size limit error. In this tutorial, the batch size is set to 10 records, so the expression is Ceil(o_RowNum/10).

   • Field output:
     • Field Type: Output Field
     • Name: o_Batch_Number
     • Type: bigint
     • Precision: 19
     • Expression: Ceil(o_RowNum/10)

   Below is a preview of the data at this stage:

   

4. Aggregator Transformation: Set Group By to the batch number only. No other settings are required in this transformation.

5. As shown in the full mapping image at the top of this page, both the BatchNumbers Expression Transformation and the Aggregator Transformation have links to the Web Services Transformation. This requires renaming a group of incoming fields. Update the incoming fields from the Aggregator Transformation to use a prefix of Parent_.

   • In the Web Service section of Properties, select the Business Service that contains your Web Services Consumer connection for CAOS.
   • In the Request Mapping section of Properties, hover over each incoming o_Batch_Number field and mark it as a primary key.
   • Hover over o_RowNumber from the Expression fork of the mapping and mark it as a foreign key.
   • Map the data element (in our case, PVI) and any studentDataOptions from the Expression fork to the request body.
   • In the Response Mapping section of Properties, select the data elements you want to retrieve. This will display two sets of output fields on the right — one for faults and one for data — and each requires its own target file.

   

   

   

6. In the full mapping image, there is an Expression Transformation between the Web Services Transformation and the data target. This Expression Transformation is intentionally blank and is used only to preview what we're receiving from CAOS without writing data to the target. If you have followed the steps to this point and have selected valid targets for both the Fault and OutputData branches, a preview in this Expression Transformation should yield 10 times the number of records shown at the Aggregator step.

References

• CAOS User Guide
• IDMC knowledge base article on batching in the body of a request
• IDMC knowledge base article on creating a Business Service