jscp, jtool scp

jscp lets you copy files using the SCP or SFTP protocol. You can call it as jscp or jtool scp. When used from within a process, jscp can retrieve private key passphrases and user passwords from RunMyJobs login and passphrase credential objects.

jscp adheres to the following RFCs.

  • rfc4251: The Secure Shell (SSH) Protocol Architecture
  • rfc4252: The Secure Shell (SSH) Authentication Protocol
  • rfc4253: The Secure Shell (SSH) Transport Layer Protocol
  • rfc4254: The Secure Shell (SSH) Connection Protocol

Note: For installation instructions, see jtool.

Syntax

Copy 
jscp  [-h|-?|-help] [-l <loglevel>] [-f <logfile>] <command>

  Option           Default                Purpose
  ---------------- ---------------------- -----------------------------------
  -h|-?|-help                             Show this help and exit
  -l <loglevel>    ${JCS_LOGLEVEL:-info}  Set the logging level; defaults to environment variable ${JCS_LOGLEVEL}
                                          or info when the environment variable is not set.
  -f <logfile>     ${JCS_LOGFILE:-stderr} Log to file instead of stderr; defaults to environment variable
                                          ${JCS_LOGFILE} or stderr when the environment variable is not set.
  <[-]command>...                         Command(s) passed to jscp

where command represents one of the following:

[options] [user@]host:source target
[options] source [source...] [user@]host:target
[options] -ls [user@]host:filespec

Options:
  -V              print version information and exit
  -pgpfp          print PGP key fingerprints and exit
  -p              preserve file attributes
  -q              quiet, don't show statistics
  -r              copy directories recursively
  -v              show verbose messages
  -load sessname  load settings from saved session
  -P port         connect to specified port
  -u user         connect with specified username
  -pw passw       login with specified password
  -1 -2           force use of particular SSH protocol version
  -4 -6           force use of IPv4 or IPv6
  -C              enable compression
  -i key          private key file for authentication
  -[no]agent      en/disable use of authentication agent
  -[no]batch      en/disable all interactive prompts
  -unsafe         allow server-side wildcards (DANGEROUS)
  -storekey       store a new or updated server key in cache
  -sftp           force use of SFTP protocol
  -scp            force use of SCP protocol

SSH Host Keys

jscp does not use the ${HOME}/.ssh/known_hosts file. jtool scp reads the server authentication host key from ${HOME}/.jtool/sshhostkeys (UNIX) or HKEY_CURRENT_USER\Software\Redwood\Jtool\SshHostKeys (Windows).

You can run jtool scp interactively with the -nobatch argument in order to store the remote host key in ${HOME}/.jtool/sshhostkeys (UNIX) or HKEY_CURRENT_USER\Software\Redwood\Jtool\SshHostKeys (Windows). This must be done for every remote host to which jtool scp connects.

When running in a process, SSH_ASKPASS will default to jgetcredential. Therefore, during user authentication, ssh will attempt to retrieve any required private key passphrase or user password from RunMyJobs.

jtool scp Host Keys

jtool SSH host keys are stored in the following format.

In ${HOME}/.jtool/sshhostkeys:

Copy 
<keytype>@<port>:<host> <key>

In HKEY_CURRENT_USER\Software\Redwood\Jtool\SshHostKeys:

Name Type Data
<keytype>@<port>:<host> REG_SZ <key>
  • <keytype: The public key algorithm.
  • <port>: The port the OpenSSH-compatible server is listening on.
  • <host>: The hostname or IP address in square brackets.
  • <key>: A public key in PuTTY format.

You can use PuTTYGen to convert between OpenSSH and PuTTY public key formats.

OpenSSH Host Keys

The format of ${HOME}/.ssh/known_hosts in OpenSSH_7.4p1 uses the following syntax (provided HashKnownHosts is set to no ):

Copy 
<host>:<port>[,<host>:<port>]* <keytype> <key>
  • <host>: The hostname or IP address in square brackets.
  • <port>: The port the OpenSSH-compatible server is listening on.
  • <keytype}: The public key algorithm.
  • <key>: The public key in OpenSSH format.

For example:

Copy 
[pr1.example.com]:2222,[10.0.4.59]:2222,[10.0.4.57]:2222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKjLzSTr4DafbLQQPO+S+dGjLOoQue7z8BCMfICda012PkpGcS+97YaCE7rg0RzoFW4SPz0wpN6gFM05g6Fsomo=

Using PuTTYGen, you can convert these keys to PuTTY syntax and simply copy them into the sshhostkeys file or create a String Value in the Windows registry as outlined above.

Configuring Certificate Authentication

For increased security, you can opt for key-based authentication coupled with a passphrase stored in a RunMyJobs passphrase credential. In order to verify that the correct credential was used, you can use the -l debug switch with jscp, or JCS_LOGLEVEL=debug with ssh.

Falling Back to Username/Password Authentication

If the SSH server refuses to authenticate the client using a private key and passphrase, you can fall back to username/password authentication using a credential of type login, as follows:

Copy 
jtool scp -i <private keyfile> <otherargs> || jtool scp -u <username> <otherargs>

The jtool scp binary uses the PuTTY implementation for keys and certificates, so you must install PuTTY and PuTTYgen to generate the keys and certificates. Note that these tools are not needed on the systems where you run jtool scp. Only the keyfiles are required.

Note that these instructions are provided for your convenience and use a simple key with 4096 bits. Numerous resources are available online with more in-depth information on how to create more secure keys.

Generating a PuTTY Keyfile

After you issue the command to generate a key using PuTTYgen, you may need to move your mouse for a minute or two, until you see +++. The passphrase you enter should ideally have multiple words. Store the passphrase in a credential in RunMyJobs. The Username of the credential should be set to the value of the comment field (-C).

Copy 
$ puttygen -t rsa -b 4096 -C "jdoe" -o jdoe.ppk
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++
Enter passphrase to save key:
Re-enter passphrase to verify:

Here is the resulting public/private key pair.

Copy 
$ cat example.ppk
PuTTY-User-Key-File-2: ssh-rsa
Encryption: aes256-cbc
Comment: example
Public-Lines: 12
AAAAB3NzaC1yc2EAAAABJQAAAgEAg5ESdWYAARvMTdXkZSEsdjcB1H6fxsqUipwY
rJTaHKjGU0VvB/LdIWlSVx65vIMEtPatbK99QhczuAWoX95gIDmIRoyuCXVxp6Vi
zpVRpLv9+ni3GRm39RKiPvWLXZ+8opAU73cpwFONoJQ+3oW7n/TuGfibLJEXUch5
W3zZ4lyJQ+xMIPUcdVswOxD3a3O6HuSU94z7AwHEhOE4ABHMrlBE1t6wx00cs0E5
eKfvGxt+ExjJp6X+Jv81nFFCrj5A+V508JqicoLLRLLCkxfFGiFEjVLinxh46+HJ
8nwE3PRjTvt0sCS/q2Sz3sHF9pYC89/k+BD5YU0KVnN9hZZZpaH0JfH0AIDzOMtT
O5ymn/zE38JQEM0LbuJ8YM7bg/O1wrlnUtqkHqn29OrB6y75VgPaw26cEbDSDpo/
W1d3/HFIWifp8z0SJ7jPp0Sidle/7B+VWc/Vv3ZbBhDSp6/0LUDH8CtV1aJaLhUL
1U2D3yufZLMIUvwABOOipKI4Zmpdu5Y5h1nZf251o9GSD3nhkh0CQdQdw1lqGcOq
p71NmAWAm6A+RD3dCoCqOlaEL/OcoJLizYPoMLUfsBwSAjQ0mNC8m9vw8HR8I0Os
2V7BNvICk3p5fiYfWbsSIBB9fkyt3VHRJRlTA7+mdzfIEfoeIy5ixqmZ6ZjD8oB6
c4sGPa8=
Private-Lines: 28
4Oe1LWcx9uvMxP+g00Yqhy+hODG9FLaTpB5zAROKsdQiNdbD6Brqwd4CPoefT4qp
Ehv5QPy4Iei/9Bt8EKCsZM7gQ5u9LFmoa6H0+IsnxcIxddETrXkBfuiiQ4g6wvar
V1o/jvSh35sJv9r1aAiMxpZEK2xN7iubhBwJ2YI57+vYPSWu3LQkQgvDpTyD0Xq/
JZKMmizz887JD+Nd58V/yfwtxla1B3kKXAwwhqYYShG5n/EpBCdoHP3ion+/FDms
J0DR1Sd8mo/IbWPd2Lhszlhcsty24uqfo2Nq8RQPJCZZZFcIsgpv2EHk5PwI3+Zh
o92m6VrBtCm/Gy03kRedT4FyYppqKZiSwvkgz9QAqP9vCa2IQGJlchjEkg06/Z13
j5NW4zHzhhACavvkRZfadsnSeAFv9D9//TAbrov2Tx20PEqiTMItvgrg8KdgT7gY
Lh/STLcSJuuI2q3AX9UzMRGdcvtQj2Cgk3KAWWWUrNzXqyTt8wg/ZmIhMtTogDVO
naIXbpukq46dyRzMJxkMHeQLJu9Wj3uQLySqXm2OUp5p1ry5k6+xvq2Cdp0lWQXn
CFOhYVpdiN9O6aV1v7b7icBit9TNqflLqZCclqNhrT/Gx1DVptMCoMCZHZ6Ft24N
mIhxbVrDKw5AlFGZpxT1/18iIcI/7cn+SLFweG9Ke1T4mxEPC9knbKi2k5YerkJ5
M2DOEXi7Ki8tpPByD59Dc7vuoybTYkVxuZCL/kAzZkdzuMJHq5MTchXdBt27p3sP
V5W9CE1+bUjhfipG32vcR6CE1Ih2mi0ZCcW8A6foBTCLjjHJNylfYXY2cXMLGEBv
oH7TgC7iIQ0Fo1G/RI9IsUYcAmluuh27pdJSH3kIx9ie2M6kXPoKRAqTrG89Ey1O
CQMFxlmmAIQ9q7JRlMPK6GG/gM9lYrBWcGwnJCpSiQvahWLOHJsDVbDZDQShsSwH
JAbmipsRnwAdcTCjNWwY1RPalHxAA5li55Mbc/5mHme6Xm4SQKsbzf2MmbD5zUFj
kdV2S/RhpDAGk3ribawh1cjA5wFmYMSnQzXVJDQrUeZEIUSbVq81RbWCNEkTwOWL
F4zMGuwqETKKIZkYHktWySST9p0C5af3sla3aVeUhkPc3Ou/bRRCqkMnBr/GCWsl
L7ezDoJpsMebydu16YQ4q2fkgpSATECpiAOajfXknjeNiR1EGcy1wLIT1QtgFdUv
CPbR3cELGv0raeCp2ba2dg8EeVi6k8cNXEV5sV+d1WPVnPIP8N3t/QTwUrj2KYIf
vTT4hcMxBUKKkBZ4v8eW8m1Uel3D9czP3RGZFZX/jm5/1oZCoNcnbfvdslz7uktx
f5HdWQHOhJgAN6DY+MpuDDnk/ybdfQkih0AXnPu9NjyJzSoNUYR9mDzts4OulpzT
6rCnsIdGCchdMM2D3zEtARgutlNCM5lITtF7jq+nR693t5/FMo+kRoP8HwzmyHTo
dS2sz4oYj4uSnLYiyfPu90jB/1HRTp9uWwOvJLzvhBI1HbhX/6FdP+JC81p4uthj
5wCBUG+DHEOmJtNMgksIS5oxF7VMcd7xn4VjMPPob3I6fIHYqxwqjd3g6mFFAEnj
Vm9wI/J57u5qflk5m0DsnvYV0tdSwb54w6TQqiE4eVP7L/bk4dZchTCZCjR07ub0
VkhpAg5kPI4vNACafAoPjo2fBuEPSiKi/Z/6MyJbhsl5gf9Jpvsp1X8MR5jEmi98
nBf25NlIFi7sdmiWOXXM7A==
Private-MAC: 35095f101ZZZ826154023d94dcZZZ4479159bbb6

Create a passphrase credential in RunMyJobs to store the passphrase of your certificate.

Installing a PuTTY Key in SSH Server

When you install a PuTTY key in yhe ~/.ssh/authorized_keys file of an SSH server, copy the public key part to the remote SSH server.

Copy 
AAAAB3NzaC1yc2EAAAABJQAAAgEAg5ESdWYAARvMTdXkZSEsdjcB1H6fxsqUipwY
rJTaHKjGU0VvB/LdIWlSVx65vIMEtPatbK99QhczuAWoX95gIDmIRoyuCXVxp6Vi
zpVRpLv9+ni3GRm39RKiPvWLXZ+8opAU73cpwFONoJQ+3oW7n/TuGfibLJEXUch5
W3zZ4lyJQ+xMIPUcdVswOxD3a3O6HuSU94z7AwHEhOE4ABHMrlBE1t6wx00cs0E5
eKfvGxt+ExjJp6X+Jv81nFFCrj5A+V508JqicoLLRLLCkxfFGiFEjVLinxh46+HJ
8nwE3PRjTvt0sCS/q2Sz3sHF9pYC89/k+BD5YU0KVnN9hnfrpaH0JfH0AIDzOMtT
O5ymn/zE38JQEM0LbuJ8YM7bg/O1wrlnUtqkHqn29OrB6y75VgPaw26cEbDSDpo/
W1d3/HFIWifp8z0SJ7jPp0Sidle/7B+VWc/Vv3ZbBhDSp6/0LUDH8CtV1aJaLhUL
1U2D3yufZLMIUvwABOOipKI4Zmpdu5Y5h1nZf251o9GSD3nhkh0CQdQdw1lqGcOq
p71NmAWAm6A+RD3dCoCqOlaEL/OcoJLizYPoMLUfsBwSAjQ0mNC8m9vw8HR8I0Os
2V7BNvICk3p5fiYfWbsSIBB9fkyt3VHRJRlTA7+mdzfIEfoeIy5ixqmZ6ZjD8oB6
c4sGPa8=

You must prepend ssh-rsa to the public key (as specified in the keyfile), and the text must be on one line.

Copy 
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAgEAg5ESdWYAARvMTdXkZSEsdjcB1H6fxsqUipwYrJTaHKjGU0VvB/LdIWlSVx65vIMEtPatbK99QhczuAWoX[...]c4sGPa8=

Note that the keys above are for illustration purposes only and will not work.

Creating a Passphrase Credential

Use the following values to create a RunMyJobs passphrase credential.

  • Real User: jdoe (as specified in the comment of the key)
  • Password: <passphrase_as_specified_during_key_generation>
  • Protocol: passphrase
  • Endpoint: pr1.example.com

Troubleshooting

jtool scp will log the following.

  • secure.plib - Wrong passphrase This means the server accepted the keyfile, but the passphrase was incorrect. In this case, enable debugging and make sure the correct credential is being picked up by jtool scp.
  • secure.plib - Server refused our key This means the server does not know the public key. Inspect the .ssh/authorized_keys file and make sure the key is on one line and starts with the correct keyword (ssh-rsa in the example above) followed by a space. You may have to specify the username (for example, jdoe@pr1.example.com:/tmp/target_file.txt).

Advanced Settings

The jscp tool allows you to load PuTTY sessions settings using the -load parameter. jscp uses the following locations for configuration:

Windows

  1. PuTTY defaults.
  2. HKEY_CURRENT_USER\Software\Redwood\Jtool\Sessions\Default%20Settings: Windows default session settings.
  3. HKEY_CURRENT_USER\Software\Redwood\Jtool\Sessions\<session>: Windows session settings.

UNIX

  1. PuTTY defaults.
  2. ${HOME}/.jtool/sessions/Default%20Settings: UNIX default settings.
  3. ${HOME}/.jtool/sessions/<session>: UNIX session settings.

Session-specific settings override the Default%20Settings and PuTTY defaults.

Copy 
$ jscp -load

You can use the PuTTY settings as stored in the registry (Windows) or in ${HOME}/.putty/sessions (UNIX). Note, however, that a number of settings (such as terminal-emulation settings) do not affect jscp. The jscp tool will use the defaults for any unspecified settings.

The following settings are of particular interest.

Copy 
KEY=ecdh,dh-gex-sha1,dh-group14-sha1,rsa,WARN,dh-group1-sha1
Ciphers=aes,chacha20,blowfish,3des,WARN,arcfour,des
SSHManualHostKeys=ecdsa-sha2-nistp256@22:devinfra=nistp256,0x8af668653d17234635708a93323e70314e74910ecab4c0c4b8897203505f0a8b,0x15bcd4d9c9d9e09e32342bdcebf508bcea566eeeeeb3954b37f2d82c3c4f3b1b

In the Windows registry, you can create String values named, for example KEX, and Value is what comes after the = sign.

Consider the following registry entry:

Copy 
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Redwood\Jtool\Sessions\Custom]
"KEX"="ecdh,dh-group14-sha1,rsa,WARN,dh-group1-sha1,dh-gex-sha1"

This is equivalent to a file named ${HOME}/.jtool/sessions/Custom containing the following on UNIX:

Copy 
KEY=ecdh,dh-group14-sha1,rsa,WARN,dh-group1-sha1,dh-gex-sha1

To use session settings, specify the name of the session.

Copy 
jscp -load Custom -i mykey.ppk myFile.txt jdoe@pr1.example.com:

Tip: If you created a settings file/registry entry with PuTTY and you want to use that with jscp, you must replace any spaces in the session name with %20 (if PuTTY has not already done this for you). You can also export the registry entry and replace SimonTatham\PuTTY with Rewdood\Jtool.

Examples

Authenticating Using Login Credentials

Assume you have created a login credential with the following values.

  • Real User: jdoe
  • Password: ****
  • Protocol: login
  • Endpoint: *.example.com

Create a definition with the three mandatory parameters remoteUser, remoteHost, and path. The source of the definition is as follows:

Copy 
jtool scp -ls ${remoteUser}@${remoteHost}:${path}

Authenticating With Keys and Passphrases

Assume you created the following passphrase credential:

  • Real User: jdoe (as specified in the comment of the key)
  • Password: <passphrase_as_specified_during_key_generation>
  • Protocol: passphrase
  • Endpoint: pr1.example.com

Create a definition with two mandatory parameters, remoteUser and remoteHost, as well as the following source:

Copy 
jtool scp -i jdoe.ppk -storekey -ls ${remoteUser}@${remoteHost}:.

Output:

Copy 
drwxr-xr--    5 example staff         170 May 25 17:47 Applications
drwx------    8 example staff         272 Aug 20 12:26 Desktop
drwx------    7 example staff         238 May 23 15:46 Documents
drwx------   38 example staff        1292 May 22 18:47 Downloads
drwx------   45 example staff        1530 May 25 17:47 Library
drwx------   10 example staff         340 May 25 17:26 Movies
drwx------    4 example staff         136 May 22 18:37 Music
drwx------    6 example staff         204 May 25 13:42 Pictures
drwxr-xr-x    6 example staff         204 Dec 30  2014 Public

Retrieving Files with the SFTP Protocol

Assume you have a file on an SFTP server and you want to retrieve it. You have created the following login credential:

  • Real User: jdoe
  • Password: ****
  • Protocol: login
  • Endpoint: *.example.com

Create a definition with two mandatory parameters, remoteUser and remoteHost (set to jdoe and pr1.example.com, respectively), as well as the following source:

Copy 
jtool scp -sftp c:\MyFiles\myfile.txt ${remoteUser}@${remoteHost}:MyRemoteFiles/myfile

This example uses the SFTP protocol to retrieve the file.

You can specify the password as well (not recommended):

Copy 
jtool scp -sftp -pw myUnsafePassword c:\MyFiles\myfile.txt ${remoteUser}@${remoteHost}:MyRemoteFiles/myfile

Troubleshooting

If you are connecting to an SSH/SFTP server for the first time, and the host key is not cached, you will get an error like the following.

Copy 
ERROR 2023-07-27 17:35:14,486 CET [73199-jscp] secure.plib - The server's host key is not cached. You have no guarantee that the server is the computer you think it is.
ERROR 2023-07-27 17:35:14,486 CET [73199-jscp] secure.plib - The server's ssh-ed25519 key fingerprint is: ssh-ed25518 256 58:4e:bd:76:1f:11:33:cd:3b:c7:a1:78:b1:6b:7a:f7
ERROR 2023-07-27 17:35:14,486 CET [73199-jscp] secure.plib - Use the -storekey argument to accept this key in batch mode
ERROR 2023-07-27 17:35:14,486 CET [73199-jscp] secure.plib - Disconnected: Aborted at host key verification
FATAL 2023-07-27 17:35:14,486 CET [73199-jscp] secure.scp - Lost connection

You must verify the fingerprint of the host key, then add the host key to the system's cache. One way to do this is to run the jscp command once more with the -storekey option.

Example

Assume you initially ran this command from the previous example, and received the above error.

Copy 
C:\ > jtool scp c:\MyFiles\myfile.txt ${remoteUser}@${remoteHost}:MyRemoteFiles/myfile

You must add -storekey to the command at least once. After that, commands should no longer require it.

Copy 
C:\ > jtool scp -storekey c:\MyFiles\myfile.txt ${remoteUser}@${remoteHost}:MyRemoteFiles/myfile
INFO  2023-07-27 14:43:03,880 Etc/GMT-1 [12292-jscp] secure.plib - Stored host key for pr1.example.com
INFO  2023-07-27 14:43:03,880 Etc/GMT-1 [12292-jscp] secure.plib - myfile.txt                 copied 811 bytes
INFO  2023-07-27 14:43:03,980 Etc/GMT-1 [12292-jscp] secure.plib - Server sent command exit status 0

C:\ > jtool scpi c:\MyFiles\myfile.txt ${remoteUser}@${remoteHost}:MyRemoteFiles/myfile2
INFO  2023-07-27 14:44:03,880 Etc/GMT-1 [12294-jscp] secure.plib - myfile.txt                 copied 811 bytes
INFO  2023-07-27 14:44:03,980 Etc/GMT-1 [12294-jscp] secure.plib - Server sent command exit status 0