Configure batching for Person API requests in IDMC

Person API batching

Informatica Data Management Cloud (IDMC) sends API requests one record at a time with mappings unless you intentionally configure batching. While the Person API is built for this kind of volume (and much more), sending individual requests leads to slow and inefficient runtimes for jobs with large amounts of data on the IDMC side. Reducing the number of requests to the Person API will decrease the processing burden on your Secure Agent servers, and therefore free up bandwidth for all users in the long run.

Prerequisites

• Access to the Person API or Mock Person API. Here is where you can get started.
• A swagger file with the operation and endpoint you are using defined. For example, GET to /people.
  • If you need a place to host your swagger file, we have a repository here and you can send us a request for access by emailing us at integration-platform@doit.wisc.edu.
  • [Link for document 159980 is unavailable at this time] on creating a swagger file.
• A list of values you can use to call the Person API. For instance: SupOrgs, netIDs, PVIs, first names, etc. The API Team has documented possible filters here.

The mapping

We are passing the values of each batch as one string delimited by commas. Here is an example of the URL sent via IDMC:

https://api.wisc.edu/people?filter[identifiers.name]=pvi&filter[identifiers.value]=UWxxxxxxx,UWxxxxxxx,UWxxxxxxx&include=identifiers

The string of three fake PVIs is what the mapping above constructs, and it represents one batched request. While include=identifiers isn't necessary for sending PVI requests to the Person API, we include that segment as an example of a common filter, since we will be retrieving data from what is returned at the end of the mapping. Depending on your use case, you could leave that part out, or adjust it to jobs or jobs.costAllocations, etc. See the Person API Filter Query Parameter documentation for more information.

Note that if your source .csv has additional columns containing data you want to deliver to your target, you may need a Joiner transformation to re-associate that data with the identifiers after the Web Service transformation step.

1. Create a .csv file and put it in Box, an S3 bucket, or your team's folder on the Secure Agent. For this tutorial, the file only needs the list of identifiers or values you want to send to the API. Use this file as the source in your mapping.

2. The records need to be numbered. This Expression transformation uses a Variable field and then an Output field to produce a numbered column for your .csv. You can skip this step if your source .csv already has a numbering column (as is sometimes the case with a database source).

   Below are the settings for the two fields: - Field One: - Field Type: Variable Field - Name: v_Counter (we use an initial and underscore prefix to make variable fields easy to identify while working through an Expression transformation) - Type: bigint - Precision: 19 (default) - Expression: v_Counter+1 - Field Two: - Field Type: Output Field - Name: o_RowNum - Type: bigint - Precision: 19 (default) - Expression: v_Counter

   

3. This Expression transformation adds another column: the batch number. The Person API cannot exceed 10 values in one request, so the expression is Ceil(o_RowNum/10).

   • Field Output:
     • Field Type: Output Field
     • Name: o_Batch_Number
     • Type: bigint
     • Precision: 19
     • Expression: Ceil(o_RowNum/10)

   Below is a preview of what the mapping produces at this stage:

   

4. This Expression transformation concatenates all the PVIs into their respective batch number. It creates another output field, and the value cascades down for each record within the batch.
   • Field One:
     • Field Type: Variable Field
     • Name: v_BatchNumber
     • Type: bigint
     • Precision: 19
     • Expression: o_Batch_Number
   • Field Two:
     • Field Type: Variable Field
     • Name: v_PVIs
     • Type: String
     • Precision: 255
     • Expression: pvi (or your original .csv value column)
   • Field Three:
     • Field Type: Variable Field
     • Name: v_ConcatPVI
     • Type: String
     • Precision: 255
     • Expression: iif(o_Batch_Number = v_Prev_BatchNumb, v_PVIs||','||v_ConcatPVI,v_PVIs)
   • Field Four:
     • Field Type: Variable Field
     • Name: v_Prev_BatchNumb
     • Type: bigint
     • Precision: 19
     • Expression: v_BatchNumber
   • Field Five:
     • Field Type: Output Field
     • Name: o_BatchedPVIs
     • Type: String
     • Precision: 255
     • Expression: v_ConcatPVI

   

   Here is a visual of what this Expression transformation does:

   

5. The Aggregator transformation reduces the record set to only the last record of each batch — the record that contains all of the PVIs for that batch. As noted above, if your source .csv contains no other unique data per record, this is not an issue. If it does contain unique per-record data, that data will be lost during aggregation. You can work around this by either duplicating the source file and adding a Joiner transformation downstream, or by creating a second mapping that joins the output file from this mapping back to the source file.

   • The only setting in the Aggregator transformation is under Group By > o_Batch_Number. Your mapping will now have one-tenth of the records it started with.

6. This Expression transformation, Add_Filter_Values, further defines the URL for the API request. The query parameters are defined in the swagger file; the values of those query parameters are defined in this step. Included below is a visual of a Postman request alongside the expressions in the transformation. These values are strings and should be defined with single quotes around them.

   

7. Web Service transformation. In the Web Service section of this transformation's properties, select a Business Service that references the REST V2 connector for your swagger file. The Business Service should be configured for the endpoint you are calling. See [Link for document 159980 is unavailable at this time] if you do not have one set up already.

   In the Request Mapping, point the output fields you created to their respective options. Map Batched PVIs to the filter value field — not the original PVI column from your source.

   In the Response Mapping property, select the data elements you want to retrieve from the API response.

   

8. Two targets are required for this mapping: one for fault responses from the API, and one for the data elements you are retrieving. In the first image of the full mapping above, there is an Expression transformation between the Web Service transformation and the data target. That Expression is intentionally blank and is used purely to preview what is being received from the Person API without writing data to the target. If you have followed the steps to this point and have selected valid targets for both the Fault and Data branches, a preview in that Expression transformation should yield 10 times the number of records that the Aggregator transformation produced.