This document explains how to Use the IBM Workload Scheduler File Transfer Job definition XML to create/modify a File Transfer job type.
Below is an IBM Workload Scheduler File Transfer Job definition in its XML representation. This is how a File Transfer job is stored in IWS (much like EventRules are also XML).
This XML can be used in TWS/WebAdmin TextEdit mode to create and modify File Transfer jobs. The benefit of doing so is that the XML representation includes the KeyPath options for use of the SSH protocol even though the TWS/WebAdmin UI does not expose those fields in its File Transfer job UI.
CAVEAT The following is based on testing against a Windows 2016 VM running the BitVise SSH server. This BitVise SSH server provides a configuration option for a local bash.exe shell. If connecting to a different type of SSH-capable server, the file paths part of the information below may be different. A quick Google search shows that it is possible to install a Powershell version of OpenSSH on WindowsOS, but https://www.openssh.com/portable.html does not reveal a WindowsOS version available. This may be why applications like BitVise exist (?).
sh
shell after successful connection negotiation, and then runs the sftp command in that exec’d sh
shell<jsdlfiletransfer:localfile></jsdlfiletransfer:localfile>
and <jsdlfiletransfer:remotefile></jsdlfiletransfer:remotefile>
must be full paths from the root of the file system; e.g., where use of sendfiles.pl accepts a partial path like /MILWAUKEE/imagenow/Admissions/MIL_WAK_WSH_DUR-BR/
the File Transfer job requires a full path like /c/Users/xferuser/MILWAUKEE/imagenow/
(NOTE: this example path shows how MSYS2’s bash shell accesses the WindowsOS directory structure C:\Users\xferuser\MILWAUKEE\imagenow\
)<host>\f\path\to\file
; see item above)<jsdlfiletransfer:uploadInfo>
:
<jsdlfiletransfer:localfile>
denotes the source file(s)<jsdlfiletransfer:remotefile>
denotes the target folder<jsdlfiletransfer:downloadInfo>
:
<jsdlfiletransfer:remotefile>
denotes the source file(s)<jsdlfiletransfer:localfile>
denotes the target folder<jsdl:password></jsdl:password>
; on saving the job, IWS will encrypt the password with {aes}
and from that point on, one sees something like {aes}
xyNBNGlsKha+
-actual-encrypted-password-hash-goes-here-
E7HAz93N2e
in the password configuration<jsdlfiletransfer:certificateGroup></jsdlfiletransfer:certificateGroup>
section is only needed if using PubKey Authentication<jsdlfiletransfer:keyStoreFilePath></jsdlfiletransfer:keyStoreFilePath>
item should contain the full path from root to the private key file that matches the public key file stored/configured on the remote system (e.g., if the user maestro public id_rsa.pub
file is stored on the remote system for trusted connection negotiation, then the full path to the private key on RIGID or WINGRA goes here: /tws/doit/.ssh/id_rsa
)id_rsa
or id_dsa
private keys by confirming with the remote system which public key is used on it<jsdlfiletransfer:remoteCredentials>
password is not required if using PubKey Auth<jsdlfiletransfer:localCredentials>
sections; it should be true that the maestro credentials are what is correct for all jobs on RIGID and WINGRAThis example XML uses <jsdlfiletransfer:uploadInfo>
. For file downloads that XML data node is <jsdlfiletransfer:downloadInfo>
.
[dynamic_workstation]#[job_name]
TASK
<?xml version="1.0" encoding="UTF-8"?>
<jsdl:jobDefinition xmlns:jsdl="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdl" xmlns:jsdlfiletransfer="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdlfiletransfer" name="filetransfer">
<jsdl:application name="filetransfer">
<jsdlfiletransfer:filetransfer>
<jsdlfiletransfer:uploadInfo>
<jsdlfiletransfer:server>[remote.host.dns.name]</jsdlfiletransfer:server>
<jsdlfiletransfer:localfile>[/full/path/to/local/source/files]</jsdlfiletransfer:localfile>
<jsdlfiletransfer:remotefile>[/full/path/to/remote/target/folder]</jsdlfiletransfer:remotefile>
<jsdlfiletransfer:localCredentials>
<jsdl:userName>maestro</jsdl:userName>
<jsdl:password>{aes}[xyNBNGlsKha+-actual-encrypted-password-hash-goes-here-E7HAz93N2e]</jsdl:password>
</jsdlfiletransfer:localCredentials>
<jsdlfiletransfer:remoteCredentials>
<jsdl:userName>[remote_user_uid]</jsdl:userName>
<jsdl:password/>[remote_user_pwd]</jsdl:password>
</jsdlfiletransfer:remoteCredentials>
<jsdlfiletransfer:certificateGroup>
<jsdlfiletransfer:keyStoreFilePath>/tws/doit/.ssh/id_rsa</jsdlfiletransfer:keyStoreFilePath>
<jsdlfiletransfer:password/>
</jsdlfiletransfer:certificateGroup>
<jsdlfiletransfer:protocol>SSH</jsdlfiletransfer:protocol>
<jsdlfiletransfer:transferMode>binary</jsdlfiletransfer:transferMode>
</jsdlfiletransfer:uploadInfo>
</jsdlfiletransfer:filetransfer>
</jsdl:application>
</jsdl:jobDefinition>
DESCRIPTION "FT Move files to host PALLICK for conversion"
RECOVERY STOP
Follows is an IWS log entry for a successful run of a PubKey Auth SSH File Transfer Job type. Note that the TASK line, which in WINDOWS/UNIX job types shows the command/script executed, here shows runtime variables provided by IWS (which are not part of the File Transfer job type configuration) between <jsdl:variables>
and </jsdl:variables>
, with the complete File Transfer job XML configuration between <jsdl:application name="filetransfer"> and </jsdl:application>.
===============================================================
= JOB : RIGID_1#JOBS[(0600 03/31/20),(JOBS)].SIADMIL_SEND_DUR-BR_TO_HOST_PALLICK_FT_7
= TASK : <?xml version="1.0" encoding="UTF-8"?>
<jsdl:jobDefinition xmlns:jsdl="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdl" xmlns:jsdlfiletransfer="http://www.ibm.com/xmlns/prod/scheduling/1.0/jsdlfiletransfer" name="filetransfer">
<jsdl:variables>
<jsdl:stringVariable name="tws.jobstream.name">JOBS</jsdl:stringVariable>
<jsdl:stringVariable name="tws.jobstream.id">JOBS</jsdl:stringVariable>
<jsdl:stringVariable name="tws.job.name">SIADMIL_SEND_DUR-BR_TO_HOST_PALLICK_FT_7</jsdl:stringVariable>
<jsdl:stringVariable name="tws.job.workstation">RIGID_1</jsdl:stringVariable>
<jsdl:stringVariable name="tws.job.iawstz">202003310600</jsdl:stringVariable>
<jsdl:stringVariable name="tws.job.promoted">NO</jsdl:stringVariable>
<jsdl:stringVariable name="tws.job.resourcesForPromoted">10</jsdl:stringVariable>
<jsdl:stringVariable name="tws.job.num">721932714</jsdl:stringVariable>
</jsdl:variables>
<jsdl:application name="filetransfer">
<jsdlfiletransfer:filetransfer>
<jsdlfiletransfer:uploadInfo>
<jsdlfiletransfer:server>pallick.doit.wisc.edu</jsdlfiletransfer:server>
<jsdlfiletransfer:localfile>/xfer/transfers/pallick/MILWADM/*.zip</jsdlfiletransfer:localfile>
<jsdlfiletransfer:remotefile>/c/Users/xferuser/FDrive/MILWAUKEE/imagenow/Admissions/MIL_WAK_WSH_DUR-BR/</jsdlfiletransfer:remotefile>
<jsdlfiletransfer:localCredentials>
<jsdl:userName>maestro</jsdl:userName>
<jsdl:password>{aes}xyNBNGlsKha+-actual-encrypted-password-hash-goes-here-E7HAz93N2e</jsdl:password>
</jsdlfiletransfer:localCredentials>
<jsdlfiletransfer:remoteCredentials>
<jsdl:userName>xferuser_ft</jsdl:userName>
<jsdl:password/>
</jsdlfiletransfer:remoteCredentials>
<jsdlfiletransfer:certificateGroup>
<jsdlfiletransfer:keyStoreFilePath>/tws/doit/.ssh/id_rsa</jsdlfiletransfer:keyStoreFilePath>
<jsdlfiletransfer:password/>
</jsdlfiletransfer:certificateGroup>
<jsdlfiletransfer:protocol>SSH</jsdlfiletransfer:protocol>
<jsdlfiletransfer:transferMode>binary</jsdlfiletransfer:transferMode>
</jsdlfiletransfer:uploadInfo>
</jsdlfiletransfer:filetransfer>
</jsdl:application>
<jsdl:resources>
<jsdl:orderedCandidatedWorkstations>
<jsdl:workstation>8CD1581837A911EAAE6EE316E7B0E449</jsdl:workstation>
</jsdl:orderedCandidatedWorkstations>
</jsdl:resources>
</jsdl:jobDefinition>
= TWSRCMAP :
= AGENT : RIGID_1
= Job Number: 721932714
= Tue 03/31/2020 08:29:39 CDT
===============================================================
File Transfer Upload information
Remote File:/c/Users/xferuser/FDrive/MILWAUKEE/imagenow/Admissions/MIL_WAK_WSH_DUR-BR/MIL_WAK_20200331_031007_DUR-BR.zip
Local File:/xfer/transfers/pallick/MILWADM/MIL_WAK_20200331_031007_DUR-BR.zip
Remote User:xferuser_ft
Local User:maestro
Protocol:'SSH'
Total Transfer Time:2 seconds
===============================================================
= Exit Status : 0
= Elapsed Time (hh:mm:ss) : 00:00:04
= Tue 03/31/2020 08:29:44 CDT
===============================================================