AWS S3 and Box as flat file connector alternatives

Both AWS S3 and Box are excellent options for when you do not have easy access to the Secure Agent filesystem.

Issues with flat file connector

The IDMC flat file connector requires that you have filesystem access on the server that hosts the IDMC Secure Agent. The support overhead of managing user accounts on the hosting server will present challenges as the number of users who require use of the connector increases. This can potentially introduce delays in project completion and in provisioning and deprovisioning tasks.

IDMC Flat file connection documentation

AWS S3 as an alternative

An alternative to the flat file connector is to use the AWS S3 connector as the source or target for your files. AWS S3 is Amazon's object storage service. You can store files in S3 that you would otherwise store on the hosting server. IDMC provides a connector for AWS S3 that is provisioned in our IDMC environments.

To use the AWS S3 connector, you will need the following:

• An AWS account for your team. UW Cloud Services (Navigate to the AWS Onboarding Survey to get started.)
• Knowledge of AWS S3 buckets. AWS S3 Getting Started
• Knowledge of AWS IAM users. AWS IAM Users
• Knowledge of AWS access keys. AWS Access Keys

After you have acquired an AWS account, you can create an IAM user with read/write permissions to S3 buckets in your account. If you are using AWS S3 only as a source, read permission may be sufficient. An AWS access key and secret key can be generated for this IAM user and then used in the IDMC AWS S3 connection properties.

IDMC AWS S3 connection documentation

AWS S3 for testing and debugging integrations

AWS S3 provides a useful mechanism for testing pieces of a large ETL project. When an ETL project relies on multiple sources and transformations, you can design and test individual transformation steps by writing output to an S3 bucket for verification. Once verification is successful, you can design and test the next transformation in the same fashion. This method also works for testing changes to an existing ETL process: duplicate the affected parts and verify the output in an S3 bucket.

WiscAlerts as an example

The [Link for document 159996 is unavailable at this time] describes a successful use of the AWS S3 connector as a flat file connector alternative.

The Integration Platform team implemented the integration using AWS S3 instead of relying on filesystem access for flat files. In this integration, the team created an AWS S3 connection as the target location for the files required by the integration. The team then created mapping tasks that gather data from Oracle sources using an Oracle connector and write it to AWS S3 as a CSV file.

The WiscAlerts example linked above describes the integration in more detail and outlines the benefits and limitations of using AWS S3.

Using Box as an alternative

Box is a service for storing files securely. See this guide on how to get started with Box. IDMC provides a Box connector that can be used as an alternative to flat files. This documentation hosts a [Link for document 159994 is unavailable at this time].

When to use AWS S3 vs. Box

Both AWS S3 and Box are strong alternatives to filesystem-based flat file operations. The following considerations can help you choose between them.

• Box is suitable for sensitive information; however, it is not suitable for restricted data (see Box terms of use).
• Box has a much smaller learning curve than AWS S3 and can be set up quickly. AWS S3, and understanding how it fits into the broader AWS ecosystem, requires more ramp-up time.
• If your team is already familiar with AWS S3 or actively using it, AWS S3 is likely the better fit.
• If you are using other AWS services, AWS S3 will integrate more naturally with your existing infrastructure.
• The Box connector in IDMC offers less granular control over file names and formatting than the S3 connector. If you have requirements beyond a CSV file with a static file name, use AWS S3.

Training and experimentation

For training and experimentation, flat files are an easy and effective way to learn Informatica Data Management Cloud (IDMC). When possible, use your local machine during training. If you are using a hosted Secure Agent for training purposes, we recommend using the IDMC Box connector.

Recommendation for using the flat file connector

If there are limitations that make AWS S3 or Box impractical as flat file alternatives, we recommend hosting a Secure Agent on a machine that you manage.