ECMS - Using IBM Workload Scheduler File Transfer Job definition XML

This document explains how to Use the IBM Workload Scheduler File Transfer Job definition XML to create/modify a File Transfer job type.

IBM Workload Scheduler File Transfer Job definition XML

Below is an IBM Workload Scheduler File Transfer Job definition in its XML representation. This is how a File Transfer job is stored in IWS (much like EventRules are also XML).

This XML can be used in TWS/WebAdmin TextEdit mode to create and modify File Transfer jobs. The benefit of doing so is that the XML representation includes the KeyPath options for use of the SSH protocol even though the TWS/WebAdmin UI does not expose those fields in its File Transfer job UI.

CAVEAT The following is based on testing against a Windows 2016 VM running the BitVise SSH server. This BitVise SSH server provides a configuration option for a local bash.exe shell. If connecting to a different type of SSH-capable server, the file paths part of the information below may be different. A quick Google search shows that it is possible to install a Powershell version of OpenSSH on WindowsOS, but https://www.openssh.com/portable.html does not reveal a WindowsOS version available. This may be why applications like BitVise exist (?).

Things of note

PubKey Auth requires SSH
The File Transfer Job Type SSH protocol expects an SSH daemon on the remote system, which provides the connection on port 22
The SSH protocol expects that SSH daemon to be running in an environment that provides a shell, e.g., sh, bash, tcsh (though this is nowhere specified in the documentation, it’s perhaps not surprising that the developers expected that selecting SSH as the protocol is because the job will be connecting to an SSH server running on a system with a shell)
The SSH protocol does not run the SFTP command to make the connection to the remote system (like the Perl sendfiles.pl script, which calls the sftp command as installed on RIGID/WINGRA, which are AIX hosts)
The SSH Protocol requires EXEC privileges on the remote host; it literally executes the sh shell after successful connection negotiation, and then runs the sftp command in that exec’d sh shell
File paths in <jsdlfiletransfer:localfile></jsdlfiletransfer:localfile> and <jsdlfiletransfer:remotefile></jsdlfiletransfer:remotefile> must be full paths from the root of the file system; e.g., where use of sendfiles.pl accepts a partial path like /MILWAUKEE/imagenow/Admissions/MIL_WAK_WSH_DUR-BR/ the File Transfer job requires a full path like /c/Users/xferuser/MILWAUKEE/imagenow/ (NOTE: this example path shows how MSYS2’s bash shell accesses the WindowsOS directory structure C:\Users\xferuser\MILWAUKEE\imagenow\)
File paths must be in Unix format (no Windows OS volume F: or UNC path conventions like <host>\f\path\to\file; see item above)
For transfers of type <jsdlfiletransfer:uploadInfo>:
- <jsdlfiletransfer:localfile> denotes the source file(s)
- <jsdlfiletransfer:remotefile> denotes the target folder
For transfers of type <jsdlfiletransfer:downloadInfo>:
- <jsdlfiletransfer:remotefile> denotes the source file(s)
- <jsdlfiletransfer:localfile> denotes the target folder
One can, when using this XML formatted File Transfer configuration, input clear text passwords in <jsdl:password></jsdl:password>; on saving the job, IWS will encrypt the password with {aes} and from that point on, one sees something like {aes}xyNBNGlsKha+-actual-encrypted-password-hash-goes-here-E7HAz93N2e in the password configuration
The <jsdlfiletransfer:certificateGroup></jsdlfiletransfer:certificateGroup> section is only needed if using PubKey Authentication
The <jsdlfiletransfer:keyStoreFilePath></jsdlfiletransfer:keyStoreFilePath> item should contain the full path from root to the private key file that matches the public key file stored/configured on the remote system (e.g., if the user maestro public id_rsa.pub file is stored on the remote system for trusted connection negotiation, then the full path to the private key on RIGID or WINGRA goes here: /tws/doit/.ssh/id_rsa)
One can only know whether to configure the id_rsa or id_dsa private keys by confirming with the remote system which public key is used on it
<jsdlfiletransfer:remoteCredentials> password is not required if using PubKey Auth
NO NEED to change the <jsdlfiletransfer:localCredentials> sections; it should be true that the maestro credentials are what is correct for all jobs on RIGID and WINGRA

XML data file, with dummy data in [] brackets

This example XML uses <jsdlfiletransfer:uploadInfo>. For file downloads that XML data node is <jsdlfiletransfer:downloadInfo>.

[dynamic_workstation]#[job_name]
 TASK
<?xml version="1.0" encoding="UTF-8"?>
<jsdl:jobDefinition xmlns:jsdl="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdl" xmlns:jsdlfiletransfer="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdlfiletransfer" name="filetransfer">
  <jsdl:application name="filetransfer">
    <jsdlfiletransfer:filetransfer>
      <jsdlfiletransfer:uploadInfo>
        <jsdlfiletransfer:server>[remote.host.dns.name]</jsdlfiletransfer:server>
        <jsdlfiletransfer:localfile>[/full/path/to/local/source/files]</jsdlfiletransfer:localfile>
        <jsdlfiletransfer:remotefile>[/full/path/to/remote/target/folder]</jsdlfiletransfer:remotefile>
        <jsdlfiletransfer:localCredentials>
          <jsdl:userName>maestro</jsdl:userName>
          <jsdl:password>{aes}[xyNBNGlsKha+-actual-encrypted-password-hash-goes-here-E7HAz93N2e]</jsdl:password>
        </jsdlfiletransfer:localCredentials>
        <jsdlfiletransfer:remoteCredentials>
          <jsdl:userName>[remote_user_uid]</jsdl:userName>
          <jsdl:password/>[remote_user_pwd]</jsdl:password>
        </jsdlfiletransfer:remoteCredentials>
        <jsdlfiletransfer:certificateGroup>
          <jsdlfiletransfer:keyStoreFilePath>/tws/doit/.ssh/id_rsa</jsdlfiletransfer:keyStoreFilePath>
          <jsdlfiletransfer:password/>
        </jsdlfiletransfer:certificateGroup>
        <jsdlfiletransfer:protocol>SSH</jsdlfiletransfer:protocol>
        <jsdlfiletransfer:transferMode>binary</jsdlfiletransfer:transferMode>
      </jsdlfiletransfer:uploadInfo>
    </jsdlfiletransfer:filetransfer>
  </jsdl:application>
</jsdl:jobDefinition>
 DESCRIPTION "FT Move files to host PALLICK for conversion"
 RECOVERY STOP

File Transfer Job Type Log Entry

Follows is an IWS log entry for a successful run of a PubKey Auth SSH File Transfer Job type. Note that the TASK line, which in WINDOWS/UNIX job types shows the command/script executed, here shows runtime variables provided by IWS (which are not part of the File Transfer job type configuration) between <jsdl:variables> and </jsdl:variables>, with the complete File Transfer job XML configuration between <jsdl:application name="filetransfer"> and </jsdl:application>.

===============================================================
= JOB       : RIGID_1#JOBS[(0600 03/31/20),(JOBS)].SIADMIL_SEND_DUR-BR_TO_HOST_PALLICK_FT_7
= TASK      : <?xml version="1.0" encoding="UTF-8"?>
<jsdl:jobDefinition xmlns:jsdl="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdl" xmlns:jsdlfiletransfer="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdlfiletransfer" name="filetransfer">
  <jsdl:variables>
    <jsdl:stringVariable name="tws.jobstream.name">JOBS</jsdl:stringVariable>
    <jsdl:stringVariable name="tws.jobstream.id">JOBS</jsdl:stringVariable>
    <jsdl:stringVariable name="tws.job.name">SIADMIL_SEND_DUR-BR_TO_HOST_PALLICK_FT_7</jsdl:stringVariable>
    <jsdl:stringVariable name="tws.job.workstation">RIGID_1</jsdl:stringVariable>
    <jsdl:stringVariable name="tws.job.iawstz">202003310600</jsdl:stringVariable>
    <jsdl:stringVariable name="tws.job.promoted">NO</jsdl:stringVariable>
    <jsdl:stringVariable name="tws.job.resourcesForPromoted">10</jsdl:stringVariable>
    <jsdl:stringVariable name="tws.job.num">721932714</jsdl:stringVariable>
  </jsdl:variables>
  <jsdl:application name="filetransfer">
    <jsdlfiletransfer:filetransfer>
      <jsdlfiletransfer:uploadInfo>
        <jsdlfiletransfer:server>pallick.doit.wisc.edu</jsdlfiletransfer:server>
        <jsdlfiletransfer:localfile>/xfer/transfers/pallick/MILWADM/*.zip</jsdlfiletransfer:localfile>
        <jsdlfiletransfer:remotefile>/c/Users/xferuser/FDrive/MILWAUKEE/imagenow/Admissions/MIL_WAK_WSH_DUR-BR/</jsdlfiletransfer:remotefile>
        <jsdlfiletransfer:localCredentials>
          <jsdl:userName>maestro</jsdl:userName>
          <jsdl:password>{aes}xyNBNGlsKha+-actual-encrypted-password-hash-goes-here-E7HAz93N2e</jsdl:password>
        </jsdlfiletransfer:localCredentials>
        <jsdlfiletransfer:remoteCredentials>
          <jsdl:userName>xferuser_ft</jsdl:userName>
          <jsdl:password/>
        </jsdlfiletransfer:remoteCredentials>
        <jsdlfiletransfer:certificateGroup>
          <jsdlfiletransfer:keyStoreFilePath>/tws/doit/.ssh/id_rsa</jsdlfiletransfer:keyStoreFilePath>
          <jsdlfiletransfer:password/>
        </jsdlfiletransfer:certificateGroup>
        <jsdlfiletransfer:protocol>SSH</jsdlfiletransfer:protocol>
        <jsdlfiletransfer:transferMode>binary</jsdlfiletransfer:transferMode>
      </jsdlfiletransfer:uploadInfo>
    </jsdlfiletransfer:filetransfer>
  </jsdl:application>
  <jsdl:resources>
    <jsdl:orderedCandidatedWorkstations>
      <jsdl:workstation>8CD1581837A911EAAE6EE316E7B0E449</jsdl:workstation>
    </jsdl:orderedCandidatedWorkstations>
  </jsdl:resources>
</jsdl:jobDefinition>
= TWSRCMAP  :
= AGENT     : RIGID_1
= Job Number: 721932714
= Tue 03/31/2020 08:29:39 CDT
===============================================================
File Transfer Upload information
Remote File:/c/Users/xferuser/FDrive/MILWAUKEE/imagenow/Admissions/MIL_WAK_WSH_DUR-BR/MIL_WAK_20200331_031007_DUR-BR.zip
Local File:/xfer/transfers/pallick/MILWADM/MIL_WAK_20200331_031007_DUR-BR.zip
Remote User:xferuser_ft
Local User:maestro
Protocol:'SSH'
Total Transfer Time:2 seconds
===============================================================
= Exit Status           : 0
= Elapsed Time (hh:mm:ss) : 00:00:04
= Tue 03/31/2020 08:29:44 CDT
===============================================================