Person API Webhooks

Setting Up A New Webhook

If you are an authorized user of the Person API, you can set up to five webhooks by following the instructions below:

1. Set up your own webhook receiver per the specifications below.
2. Create a new webhook using the webhook endpoint .

Webhook Receiver Requirements

Your webhook receiver must meet the following requirements:

• Exposes a URL that accepts HTTP POST requests from any IP address.
• URL must use the HTTPS protocol for secure communication.
• Validates that requests are legitimate using the secret token as specified below.
• Accepts an application/vnd.api+json request body as specified below.
• Returns a 200 OK HTTP response code after successfully handling the webhook event.
• Returns a response within 10 seconds after receiving the webhook event.

Validate Payloads

When you create a new webhook, you will receive a secret token to validate received payloads with, which your application should store somewhere securely. This token is then sent with each webhook event in the X-Person-Api-Token HTTP header. Your webhook endpoint will have to check the token on each received request to ensure they are legitimate and reject requests with invalid or missing tokens. If the request is not legitimate, we recommend that you respond with HTTP 401 Unauthorized .

If you have multiple webhooks, you will have to make sure that you are checking the received token against the correct stored token, as they are unique per-webhook.

Token verification is required and consists of the following:

• Each webhook event has a 0.1% (1 in 1000) chance of being tested for token verification.
• When webhook events are selected for verification, one of three things will occur (each with an equal chance):
  • The X-Person-Api-Token header will be set to a randomly-generated token, whose format is not guaranteed to match that of currently issued tokens.
  • The X-Person-Api-Token header will be left blank (an empty string).
  • The X-Person-Api-Token header will be deleted.
• Upon being sent, the response code will be checked:
  • If the event was not rejected (e.g., we received an HTTP 200 response), this will be logged and flagged, and we will email you. The event will not be re-sent, as your application has indicated that it has already acted on it.
  • If the event was rejected (e.g., we received an HTTP 401 response), we will log that your application acted properly, and will immediately re-send the event with the correct token.

Please note that you should verify the token before processing the event, as we will re-send them upon the correct rejection of invalid events. Failing to verify before processing (i.e., emitting a 401 but still processing the event) could trigger undefined behavior as your application attempts to repeat a previous action (e.g., attempting to delete an object twice).

Webhook Events

Webhook events are sent in the request body and follow the JSON:API specification with the following data:

• Each request contains a single events type resource object in data .
• It contains an eventType attribute that will be one of created, updated, deleted, merged .
  • The merged event occurs when a person had multiple records with different PVIs merged into a single record. In this case, the person in the event is the single person that exists after the merge has completed. In their list of identifiers, you can find the PVIs of their old records by looking for identifiers with the name pvi and current set to false .
  • If a person record is split into multiple people, this will be represented as two or more created events and one deleted event. Historical identifiers are not preserved for the resulting people. Although a person splitting is rare, special care should be taken in downstream systems to ensure the correct data is being attributed to the correct person. All Person API users will be contacted through email whenever splits occur with information on how to handle the split in downstream systems.
• It contains a changedRelationships attribute that lists the relationships affected during the event.
  • This attribute allows you to identify which relationships require synchronization across systems when processing events.
  • Example: "changedRelationships": ["addresses", "identifiers", "self"]
• The event resource object has a relationship person to the associated person and identifiers to all the identifiers for that person. These relationships contain links to look up the related resource in the Person API.
• The included resources in the request contain all the identifiers associated with the person.

Example webhook event for an application registered to UW-Madison:

{
  "data": {
    "type": "events",
    "id": "3a8c6ff6-35d0-40de-8f01-f77a216d721e",
    "attributes": {
      "eventType": "updated",
      "changedRelationships": ["addresses", "identifiers", "self"]
    },
    "relationships": {
      "person": {
        "data": {
          "id": "80259",
          "type": "people"
        },
        "links": {
          "related": "https://api.wisc.edu/people/80259"
        }
      },
      "identifiers": {
        "data": [
          {
            "id": "9949757",
            "type": "identifiers"
          },
          {
            "id": "9949758",
            "type": "identifiers"
          },
          {
            "id": "9949759",
            "type": "identifiers"
          },
          {
            "id": "9949760",
            "type": "identifiers"
          },
          {
            "id": "11470381",
            "type": "identifiers"
          },
          {
            "id": "12183851",
            "type": "identifiers"
          },
          {
            "id": "12670596",
            "type": "identifiers"
          },
          {
            "id": "12670599",
            "type": "identifiers"
          }
        ],
        "links": {
          "related": "https://api.wisc.edu/people/80259/identifiers"
        }
      },
      "pastIdentifiers": {
        "data": [
          {
            "id": "8239757",
            "type": "identifiers"
          },
          {
            "id": "16759219",
            "type": "identifiers"
          }
        ]
      }
    }
  },
  "included": [
    {
      "attributes": {
        "active": true,
        "current": false,
        "name": "emplId",
        "source": "HRS",
        "value": "00712529",
        "institution": "UW-Madison"
      },
      "id": "9949757",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": true,
        "current": false,
        "name": "pvi",
        "source": "IAM",
        "value": "UW539D609",
        "institution": "Shared"
      },
      "id": "9949758",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": true,
        "current": false,
        "name": "pvi",
        "source": "IAM",
        "value": "UW706Q479",
        "institution": "Shared"
      },
      "id": "9949759",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": true,
        "current": true,
        "name": "pvi",
        "source": "IAM",
        "value": "UW736J839",
        "institution": "Shared"
      },
      "id": "9949760",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": false,
        "current": false,
        "name": "netId",
        "source": "IAM",
        "value": "BUCKY",
        "institution": "UW-Madison"
      },
      "id": "11470381",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": false,
        "current": true,
        "name": "netId",
        "source": "IAM",
        "value": "BADGER",
        "institution": "UW-Madison"
      },
      "id": "12183851",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": true,
        "current": true,
        "name": "emplId",
        "source": "SIS",
        "value": "0000600917",
        "institution": "UW-Madison"
      },
      "id": "12670596",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": true,
        "current": true,
        "name": "eppn",
        "source": "IAM",
        "value": "BADGER@WISC.EDU",
        "institution": "UW-Madison"
      },
      "id": "12670599",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": true,
        "current": true,
        "name": "emplId",
        "source": "SIS",
        "value": "0000577398",
        "institution": "UW-Madison"
      },
      "id": "8239757",
      "type": "identifiers"
    },
    {
      "attributes": {
        "active": true,
        "current": true,
        "name": "eppn",
        "source": "IAM",
        "value": "BUCKY@WISC.EDU",
        "institution": "UW-Madison"
      },
      "id": "16759219",
      "type": "identifiers"
    }
  ]
}

Specifying Relationships

By default, webhook events will only include a person's existing identifiers, which is to say identifiers present on the most recent version of a person. For instance, if someone's NetID changes from BUCKY to BADGER , by default, we will only send the most recent version ( BADGER ).

If your application needs no identifiers or if it needs past identifiers, you will need to specify the relationships attribute when creating or updating your webhook subscription. This attribute is an array that can have 0-2 values:

• identifiers - The existing identifiers of a person.
  • As mentioned above, this refers to identifiers present on the most recent version of the person.
  • If someone gets deleted, their final set of identifiers prior to deletion would count as their existing identifiers because the most recent version of that person was the one prior to their deletion.
• pastIdentifiers - The past identifiers of a person.
  • This is not to be conflated with the identifiers.current or identifiers.active attributes; "past" just means an identifier which is no longer present on the person.
  • If an identifier changes, is deleted, or becomes invisible due to your application's permission (e.g. an identifier gets marked as inactive and your application lacks permission to view inactive data), the old version of the identifier will be a pastIdentifier .
  • When an identifier becomes a pastIdentifier , it should be treated as an identifier which is going away. It will not be included in future events, as that past identifier (or past version of an identifier) is not archived in the Person API; it is ephemeral and exists only within that event.

If you specify both relationships, you will get both the existing and past identifiers of a person in the included section of each event. If you specify neither (i.e. [] ), you will not receive any identifiers in the included section of each event.

Note that both data.relationships.identifiers and data.relationships.pastIdentifiers will be populated regardless of what relationships your webhook is configured to receive; the relationships attribute on a webhook simply controls what will end up in the included section of that webhook's forwarded events.

If you specify both relationships, to separate out which identifiers are past identifiers and which are not, you will need to match the IDs of the identifiers in the included section with those listed in the data.relationships.identifiers and data.relationships.pastIdentifiers sections.

Test Your Webhook Receiver with Postman

Postman can be used to simulate webhook events being sent to your webhook receiver. Download this Postman collection and import into Postman to help test your webhook receiver.

Filtering Webhooks

After creating a webhook, changes to data for any person will trigger an event. Since the Person API contains a broad population of people, this default behavior can generate more events than necessary. To help with this, two categories of filters can be added to a webhook to reduce which events are considered when triggering an event.

Attribute Based Filtering

Attribute-based filtering is used to define "who" should be sent to your webhook receiver. Up to 5 of these filters can be created per webhook.

To create a filter, send a POST request to the /filters endpoint with the structure below, replacing the example values as needed. We recommend using the link to the /filters endpoint returned when getting or creating a webhook, located in the relationships section of a webhook, rather than building the link in your client.

POST https://api.wisc.edu/people/webhooks/1234/filters
Content-Type: application/vnd.api+json

{
  "data": {
    "type": "webhookFilters",
    "attributes": {
      "attribute": "degrees.degree",
      "operator": "STARTS_WITH",
      "values": [
        "BA"
       ]
    }
  }
}

In this example, events will only be sent to the webhook for people that have at least one degrees relationship with a degree that starts with "BA". The attribute can be modified to any attribute on a person or their related resources, using the same structure as query parameter filters relative to the /people endpoint. For example, to filter on an attribute on the person itself, the attribute would be firstName for first name.

If values has multiple items, the filter will use an OR condition. Multiple filters on a given webhook are evaluated using an AND condition.

Supported operator values are [ STARTS_WITH , CONTAINS , ENDS_WITH , EQUALS , IN ]. IN is only used for searching on values in a list attribute.

See the filter parameter documentation for more information on the filtering: Person API Filter Query Paramater

People That Enter or Leave a Filter

Filtering is straightforward for updates to a person that already exists in the population defined by the filter, but what if someone enters or leaves a filter population?

• A person who enters a filter will generate an event with an eventType of created , updated , or merged .
• A person who leaves a filter will generate an event with an eventType of updated , merged , or deleted .

Given that an event will be sent to a webhook for a person that has left a population, it is important to check the person referenced in each event to confirm whether they are still within the desired population.

For merged events, at least one of the following must be true for it to generate an event to a webhook:

• At least one of the persons being merged is within the desired filter.
• The person that resulted from the merge is within the desired filter.

Relationship Based Filtering

Relationship-based filtering is used to define "what" events should be sent to your webhook receiver.

To create a relationship filter, send a POST request to the /relationshipFilters endpoint with the structure below, replacing the example values as needed. We recommend using the link to the /relationshipFilters endpoint returned when getting or creating a webhook, located in the relationships section of a webhook, rather than building the link in your client.

POST https://api.wisc.edu/people/webhooks/1234/relationshipFilters
Content-Type: application/vnd.api+json

{
  "data": {
    "type": "webhookRelationshipFilters",
    "attributes": {
      "relationship": "identifiers"
    }
  }
}

In this example, events will only be sent to the webhook for people that have had at least one of their identifiers change.

The relationship can be set to any valid relationship on a person. Alternatively, it can be set to self , a special value that refers to the base person (that is, the self relationship monitors for changes to firstName , lastName , officePhoneNumber , etc. rather than changes to the names or phoneNumbers resources).

Multiple relationship filters on a given webhook are evaluated using an OR operator.

Retries and Expired Webhooks

After you receive a webhook event, return an HTTP status code. To acknowledge the event, return one of the following status codes: 102, 200, 201, 202, 204 . To send a negative acknowledgment for the event, return any other status code. If you send a negative acknowledgment or the acknowledgment deadline of 10 seconds expires, we will resend the event.

We will attempt to resend events with an exponential backoff for up to 7 days. If we do not receive a positive acknowledgment within that timeframe, the events will be deleted. If we do not receive any positive acknowledgments for 31 days, we will expire the webhook.

Replays

After a webhook has been created, we start keeping track of all the events for up to 7 days. You can request to replay events that occurred during that time by following the instructions below:

Send a POST request to /people/webhooks/{webhookId}/replay with the time to start replaying events from in the replayFromTime attribute. This should be an ISO 8601 date and time with the following format: YYYY-MM-DDTHH:MM:SSZ. The time must be in UTC.

This date and time must be within the last 7 days. If the date and time are not within the last 7 days, we will return a 400 Bad Request error.

The replay feature might be imprecise due to: * Possible clock skew among servers. * The fact that the service works with the arrival time of the event rather than when an event occurred in the source system.

{
  "data": {
    "type": "replay",
    "attributes": {
      "replayFromTime": "2020-01-01T00:00:00Z"
    }
  }
}

Event Ordering

Although we try to process incoming changes as they are published and send out webhook events in-order in a timely manner, these events can end up out-of-order, whether because they happened to arrive at our webhook processor out-of-order, the receiving application was offline and has a backlog of events to process when it comes back online, or because distributed queues are imperfect and occasionally attempt delivery more than once.

If not handled properly by receivers, this can cause problems, especially when upstream changes cause individuals to temporarily leave the populations (Person API Populations) that make up the Person API, resulting in people being deleted and re-created several hours (or even several minutes) apart due to circumstances beyond our control.

As such, webhook receivers should be prepared for events that may seem odd, like a 'create' or 'update' event for a deleted person that results in a 404. Even in the absence of application downtime creating a backlog of unacknowledged events. It is important to be prepared for these edge cases, both for the stability of webhook receiving applications and because returning an error will result in the event going back in the queue and being retried later on.

Occurrence of Duplicate and Out-of-Order Events 

For a properly configured application with an auto-scaling or buffering mechanism (to handle periods of high traffic) under ideal network conditions, up to 8% of events received may be duplicates and 0.5% may be out-of-order.

Full Statistics and Methodology

To safely handle duplicate and out-of-order events, we recommend:

1. When creating people, if they already exist in your application, update them instead.
2. When deleting people, if they do not exist in your application, no action is required.
3. When updating people, check if they are present in the Person API:
4. If you receive a 404 from the Person API, treat it as a delete event.
5. If you receive a 200 response, treat it as a 'create' or update as appropriate.

Uniqueness constraints in databases can also be helpful in dealing with duplicate events (e.g., if an insert fails due to the event ID not being unique, return 202 Accepted and don't act on the event).

Handling Periods of High Webhook Traffic

Webhooks usually allow for real-time gradual changes about people rather than large bulk changes. However, there are times when large upstream changes will result in a high amount of webhook traffic in a short period of time. Webhook receivers should be prepared for a sudden increase in traffic at any time. No advanced notice will be sent. Here are some recommendations for your webhook receiver to handle these times of high traffic:

• Use filters : Make sure you're not consuming the entire Person API population by using attribute and relationship filters to narrow down which events will get sent to your receiver.
• Throttle incoming requests : Reverse proxies like Apache HTTPD and Nginx allow for rate limiting incoming requests so that requests are explicitly denied if there are too many of them. This allows your webhook receiver to return a non-200 response code (such as 429: Too Many Requests ) and the Person API will retry the webhook event until it's successfully consumed (see documentation on retries).
• Use serverless tools : Serverless cloud services such as AWS Lambda or Google Cloud Functions can help with variable amounts in traffic by automatically scaling as needed. This allows a webhook receiver to scale dynamically instead of being at max capacity at all times.
• Test and monitor your webhook receiver : Monitor webhook traffic to make sure your receiver is properly handling high traffic events. Perform load testing to see how many requests your receiver can handle before breaking or exceeding the 10 second timeout set by the Person API.

More information about periods of high traffic can be found in the maintenance and operations documentation: Person API Operations

Webhook Metrics

When you query the GET /people/webhooks or GET /people/webhooks/{id} endpoints, you will see a series of metrics for each webhook subscription which can give some insight as to the subscription's overall health.

While these metrics can be skewed during periods of high change activity, it can be useful for diagnosing issues and identifying if a lack of webhook events is due to a problem on your end or if the change volume has just been lower than usual.

These metrics are:

• ackedInPastWeek - The total number of message acknowledgements within the last 7 days.
  • The phrasing "number of message acknowledgements" matters because this metric is affected by duplicates and Replays (which mark all messages within the specified time range as unacknowledged so they can be redelivered), both of which can create multiple acknowledgements for the same message.
• deadlineExceededInPastWeek - The total number of messages which were not acknowledged before timing out within the last 7 days.
  • This is primarily a measure of latency, as the timeout is 10 seconds.
  • If your application has extremely high latency, whether due to being under heavy load or having a poor network connection, you will see it in this metric.
  • Additionally, if your webhook endpoint is blocked by a firewall that silently drops incoming connections, you may also see it in this metric.
• oldestUnackedMessageAge - The age (in seconds) of the oldest unacknowledged message currently in your subscription.
  • As mentioned in Retried and Expired Webhooks, we will attempt to deliver messages for up to 7 days; after this period, undelivered messages expire and are dropped from your subscription automatically.
  • As such, this only includes unacknowledged messages which are still available for delivery.
• 4xxResponsesInPastWeek - The total number of HTTP 4xx responses to delivery attempts within the last 7 days.
  • This can be useful for identifying errors such as an incorrect URL being specified causing 404s or a bad reverse proxy configuration.
• 5xxResponsesInPastWeek - The total number of HTTP 5xx responses to delivery attempts within the last 7 days.
  • This can be useful for identifying errors such as the target application crashing or a bad reverse proxy configuration.
  • It can also be used to identify offline applications, as we treat unreachable systems as 5xx internally.

It should be noted that these metrics are not real-time; it may take 5-10 minutes for events and their responses to be counted.