Overview
The SFTP protocol is very thorough in verifying data integrity. A file transfer involves taking a portion of the file, referred to as a “packet” (typically 32KiB), encrypting it, adding a signed checksum, and delivering the packet to the SFTP Server. Then, it waits for a verification of that packet from the server before proceeding to deliver the next packet.
This thoroughness is great for data integrity, but it can result in very slow transfers since each packet has to wait for a response before proceeding. The slow-down this imposes will only increase as the latency between the client and server increases, since the round-trip time between the data packet and the acknowledgement is longer.
To overcome this challenge, the SFTP protocol allows for multiple packets to be delivered even while waiting for acknowledgement packets to come back. By sending more packets at a time while waiting for acknowledgements (referred to as “outstanding packets”), the protocol is able to use more of the available bandwidth. This concept of sending multiple outstanding packets while awaiting acknowledgements is referred to as “pipelining”.
Diplomat MFT SFTP Pipelining
In Diplomat MFT 8.1.1 through 9.0, the SFTP pipelining process uses a fixed value of 16 for the maximum number of outstanding packets. While this works well in the general case, over high bandwidth and/or high latency connections, this can result in performance that does not achieve good bandwidth utilization. For example, on a 500MBps line with modest latency, a very large file transfer might only achieve 140Mbps.
Starting with Diplomat MFT 9.1 (Build 20220824), the default value for outstanding packets has been increased to 32. This can greatly improve the performance over those high latency and/or high bandwidth scenarios.
Configuring Max Outstanding Packets
Starting with Diplomat MFT 9.1 (Build 20220824), the Diplomat MFT administrator can configure the maximum outstanding packets for the SFTP protocol, or for any individual transaction. This is achieved through the use of a file called “diplomat.custom.properties” which must be placed into the path:
C:\ProgramData\Coviant Software\Diplomat-j\
NOTE: if you have specified a custom DiplomatData path using environment variables (per this KB article), then place the “diplomat.custom.properties” file there.
The contents of this file contain name/value pairs, where the name specifies which at which level you are overriding the Max Outstanding Packets value for SFTP. You can specify a value for all SFTP transfers, or at the Transaction Type level (Inbound, Outbound, or Synchronization), or at a specific transaction level.
Please see the attached file for an example of setting these values.
Notes
- If you are running a TRIAL version of Diplomat MFT, the path portion “Diplomat-j” should be replaced by “Diplomat-trial”
- The values in “diplomat.custom.properties” are loaded into memory the first time they are used by Diplomat MFT. Any changes to the “diplomat.custom.properties” after that time require a service restart to take effect.
- There is no upper limit on the number of outstanding packets you can supply; however, the larger the number, the more memory the file transfer will consume. When calculating memory use, a good rule of thumb is that Diplomat MFT will use 32KiB for each outstanding packet. So a “maxOutsandingPackets” value of “256” can use up to 256 x 32 KiB = 8,192 KiB = 8MiB for outsanding packet storage during a file transfer.
- If you are specifying the name of a Tranasction that has spaces within the name, prefix each space by the “\” character within the “diplomat.custom.properties” file, like:
ssh.maxoutstandingpackets.sftp.Outbound.Transaction\ With\ A\ Space\ In \The\ Name=128