IDMC Source Control User Guide

Overview

Source control helps teams manage changes to IDMC assets such as project folders, mappings, tasks, and related assets. It allows users to check out assets before editing them, check in completed work, and keep a version history in GitLab.

Source control is useful because it helps prevent users from overwriting each other’s work, makes changes easier to track, and provides a safer way to collaborate on shared IDMC assets.

Some common use cases include:

• Sharing the code
• Simplifying disaster recovery
• Archiving code
• Re-usable code base
• Simplifies moving assets across environments
• And more ...

Some rules and limitations of the IDMC Source Control

When it comes to Source Control service, We are following Informatica's recommendations.

• Currently our test and prod behave differently:
  • Test: users are able to check-in and check-out.
  • Prod: users can only check-out at this moment.
• Only people who have owner level access to both the repository and IDMC folder are able to perform Source Control functionalities (Pull/Commit/etc.).
• We are currently using GitLab for repository management.

Limitations

• IDMC Source Control service does not handle Connections. They have to already exist before you pull from an existing repository.
• IDMC locks the assets as soon as they are checked-out by a user (the functionality is more like SVN than Git in this sense).
• GitLab PATs can be created with an expiry date up to a year. When expired, the token in IDMC needs to be replaced by a new valid one.
• Each user has only access to their own repository.
• This service will only know about assets in IDMC. It will not manage any files that exists in Secure Agent (those still have to be manually maintained).

Before you start

Before using source control in IDMC, make sure the following items are complete:

• You have access to the IDMC folder you are checking in.
• You have access to your team’s GitLab project.
• Your team’s GitLab project has already been created by an administrator. If you do not have GitLab access or need a repository, contact us before continuing at integration-platform@doit.wisc.edu.
• You know which IDMC project folder should be connected to your team’s GitLab project.

For any questions or concerns or issues, please contact us at integration-platform@doit.wisc.edu.

Create a GitLab personal access token

Each user must create a GitLab personal access token (PAT). This token allows IDMC to connect to GitLab using your GitLab account.

• Sign in to GitLab at https://git.doit.wisc.edu/.
• Open your user profile by clicking your avatar at the top right-hand side of the page.
• Select Edit profile:

• Select Access Tokens.
• Create a new personal access token:

• Copy the token and save it in a secure location. Important: GitLab only displays the token once. Copy it before leaving the page.

Take note of your GitLab username

• Click your avatar at the top right-hand side of the page.
• Note the username which is identified through a trailing "@" symbol:

• The username is the text without the trailing "@" symbol + "@wisc.edu" at the end.

Add your token in IDMC

After creating your GitLab personal access token, you need to add it to your IDMC profile settings.

• Sign in to IDMC.
• Open any IDMC service (for example "Data Integration") so that your profile menu becomes visible.
• Select your profile icon at the top right hand corner of the page.
• Select Settings:

• Scroll to the Personal Access Token section.
• Enter your GitLab username.
• Enter the GitLab personal access token you created.
• Save your changes.

Set source control properties

Source control properties connect an IDMC project folder to the correct GitLab project. 

Note: you need to have special permission added to your profile in order to use this feature successfully. If you don't see the configuration steps in the following bullet points, please reach out to us at: integration-platform@doit.wisc.edu.

• In IDMC, open Data Integration.
• Go to Explore.
• Find the project folder that should be connected to source control.
• Do not open the folder.
• Open the actions menu for the folder from the right-hand side three dots button:

• Select Source Control Properties.
• Enter the GitLab repository URL and the branch (you need to create the branch in GitLab if not checking into main).
• Save the source control properties.

• Leave the branch as is.
• And hit the "Save" button.

Complete the first check-in/commit

After source control properties are set, the project folder must be checked in for the first time. This adds the folder and its contents to the repository.

• In IDMC, go to Data Integration.
• Go to Explore.
• Find the project folder.
• Open the actions menu for the folder.
• Select Check-in:

• Review the staged check-in:

• Add a commit message and complete the check-in:

• You will see a message on success or failure:

• And you can check the Source Control Logs for more details. Navigate to Monitor -> Source Control Logs -> click on the check-in you would like to see the details of:

Start working on a Source Control managed folder

After the first check-in your folder is set to be connected to the source control. In order to start working on an existing asset, you need to check out the asset you would like to work on. Assets can be either checked out at the asset level or at the folder level. Be warned that checking out an asset or folder will lock that asset or folder for anybody else who would like to work on it:

When you are ready to check-in existing assets, you need to do so at the level they were checked out. For example, if the entire folder was checked out, you will need to check-in the entire folder.

If the asset doesn't exist yet and you are just creating it, then you don't need to check-out anything. Just remember to check-in everything you created when ready.

Complete the first pull

After source control properties are set on a new folder, you can pull an existing code. Please note that the connections are not managed through the source control. You have to manually create them or use existing ones. Alternatively, you could do an export then import for the first round before pulling the code.

Pull will only update your code. After pull, you need to also checkout if you are working on an existing asset.

• In IDMC, go to Data Integration.
• Go to Explore.
• Find the project folder or create a new folder (the name has to match the folder name in the repository).
• Open the actions menu for the folder.
• Ensure the folder is pointing to the correct repository and branch.
• Select Pull and pick the hash you are interested in pulling (you need to click on the row in order to select the specific Git hash):

• Review the staged pull.
• Complete the check-in.

After the first check-in is complete, a source control indicator appears beside the folder:

The all green indicator shows for those assets that are in checked-in state.

The indicator with a human figure shows that asset has been checked out. When hovering over this indicator, it will show the user who has it checked out.

Important notes and tips

• A source-controlled project folder or asset can only be checked out by one user at a time.
• If an asset is already checked out, hover over or review the source control indicator to see who has it checked out.
• Check in your changes when your work is complete so other users can access the asset.
• Do not delete assets unless they are no longer needed. If an asset is deleted in IDMC, the deletion can also be reflected in GitLab after check-in.
• If you are unsure whether an asset may be needed again, move it to an archive folder instead of deleting it.
• Check out only the assets you need to work on.
• Check in your changes as soon as your work is complete.
• Use clear check-in comments when comments are available.
• Coordinate with your team before making large changes.
• Avoid keeping assets checked out for long periods of time.
• Use an archive folder instead of deleting assets that may be needed later.