Java implementation of SFTP upload and download files and problems encountered

Java implementation of SFTP upload and download files and problems encountered

Recently, JSch is used to operate the upload and download of SFTP files. This article records a packaged tool class and two problems actually encountered.

SFTP (Secure File Transfer Protocol) generally refers to the SSH file transfer protocol (SSH File Transfer Protocol), which uses encrypted transmission of authentication information and data, so compared to FTP, SFTP is very secure but the transmission efficiency is much lower.

JSch ( Java Secure Channel ) is a pure Java implementation of SSH2, which allows you to connect to an SSH server, and can use port forwarding, X11 forwarding, file transfer, etc.

SFTP tools

Add related package dependencies to the pom.xml file

< Dependency > < the groupId > com.jcraft </the groupId > < the artifactId > JSch </the artifactId > < Version > 0.1.55 </Version > </dependency > copy the code

SFTP tools, providing file upload and download functions

public class SftpClient { public boolean downloadFile (SftpConfig sftpConfig, SftpDownloadRequest request) { Session session = null ; ChannelSftp channelSftp = null ; try { session = getSession(sftpConfig); channelSftp = getChannelSftp(session); String remoteFileDir = getRemoteFileDir(request.getRemoteFilePath()); String remoteFileName = getRemoteFileName(request.getRemoteFilePath()); //Check if the file exists on SFTP if (!isFileExist(channelSftp, remoteFileDir, remoteFileName, request.getEndFlag())) { return false ; } //Switch to the SFTP file directory channelSftp.cd(remoteFileDir); //Download the file File localFile = new File(request.getLocalFilePath()); FileUtils.createDirIfNotExist(localFile); FileUtils.deleteQuietly(localFile); channelSftp.get(remoteFileName, request.getLocalFilePath()); return true ; } catch (JSchException jSchException) { throw new RuntimeException( "sftp connect failed:" + JsonUtils.toJson(sftpConfig), jSchException); } catch (SftpException sftpException) { throw new RuntimeException( "sftp download file failed:" + JsonUtils.toJson(request), sftpException); } finally { disconnect(channelSftp, session); } } public void uploadFile (SftpConfig sftpConfig, SftpUploadRequest request) { Session session = null ; ChannelSftp channelSftp = null ; try { session = getSession(sftpConfig); channelSftp = getChannelSftp(session); String remoteFileDir = getRemoteFileDir(request.getRemoteFilePath()); String remoteFileName = getRemoteFileName(request.getRemoteFilePath()); //Switch to the SFTP file directory cdOrMkdir(channelSftp, remoteFileDir); //upload files channelSftp.put(request.getLocalFilePath(), remoteFileName); if (StringUtils.isNoneBlank(request.getEndFlag())) { channelSftp.put(request.getLocalFilePath() + request.getEndFlag(), remoteFileName + request.getEndFlag()); } } catch (JSchException jSchException) { throw new RuntimeException( "sftp connect failed: " + JsonUtils.toJson(sftpConfig), jSchException); } catch (SftpException sftpException) { throw new RuntimeException( "sftp upload file failed: " + JsonUtils.toJson(request), sftpException); } finally { disconnect(channelSftp, session); } } private Session getSession (SftpConfig sftpConfig) throws JSchException { Session session; JSch jsch = new JSch(); if (StringUtils.isNoneBlank(sftpConfig.getIdentity())) { jsch.addIdentity(sftpConfig.getIdentity()); } if (sftpConfig.getPort() <= 0 ) { //default port session = jsch.getSession(sftpConfig.getUser(), sftpConfig.getHost()); } else { //Specify port session = jsch.getSession(sftpConfig.getUser(), sftpConfig.getHost(), sftpConfig.getPort()); } if (StringUtils.isNoneBlank(sftpConfig.getPassword())) { session.setPassword(sftpConfig.getPassword()); } session.setConfig( "StrictHostKeyChecking" , "no" ); session.setTimeout( 10 * 1000 ); //Set the timeout time to 10s session.connect(); return session; } private ChannelSftp getChannelSftp (Session session) throws JSchException { ChannelSftp channelSftp = (ChannelSftp) session.openChannel( "sftp" ); channelSftp.connect(); return channelSftp; } /** * Does the SFTP file exist? * true: exists; false: does not exist */ private boolean isFileExist (ChannelSftp channelSftp, String fileDir, String fileName, String endFlag) throws SftpException { if (StringUtils.isNoneBlank(endFlag)) { if (!isFileExist(channelSftp, fileDir, fileName + endFlag)) { return false ; } } else { if (!isFileExist(channelSftp, fileDir, fileName)) { return false ; } } return true ; } /** * Does the SFTP file exist? * true: exists; false: does not exist */ private boolean isFileExist (ChannelSftp channelSftp, String fileDir, String fileName) throws SftpException { if (!isDirExist(channelSftp, fileDir)) { return false ; } Vector vector = channelSftp.ls(fileDir); for ( int i = 0 ; i <vector.size(); ++i) { ChannelSftp.LsEntry entry = (ChannelSftp.LsEntry) vector.get(i); if (fileName.equals(entry.getFilename())) { return true ; } } return false ; } /** * Does the directory exist on sftp * true: exists; false: does not exist */ private boolean isDirExist (ChannelSftp channelSftp, String fileDir) { try { SftpATTRS sftpATTRS = channelSftp.lstat(fileDir); return sftpATTRS.isDir(); } catch (SftpException e) { return false ; } } private void cdOrMkdir (ChannelSftp channelSftp, String fileDir) throws SftpException { if (StringUtils.isBlank(fileDir)) { return ; } for (String dirName: fileDir.split(File.separator)) { if (StringUtils.isBlank(dirName)) { dirName = File.separator; } if (!isDirExist(channelSftp, dirName)) { channelSftp.mkdir(dirName); } channelSftp.cd(dirName); } } private String getRemoteFileDir (String remoteFilePath) { int remoteFileNameindex = remoteFilePath.lastIndexOf(File.separator); return remoteFileNameindex ==- 1 ? "" : remoteFilePath.substring( 0 , remoteFileNameindex); } private String getRemoteFileName (String remoteFilePath) { int remoteFileNameindex = remoteFilePath.lastIndexOf(File.separator); if (remoteFileNameindex ==- 1 ) { return remoteFilePath; } String remoteFileName = remoteFileNameindex ==- 1 ? remoteFilePath : remoteFilePath.substring(remoteFileNameindex + 1 ); if (StringUtils.isBlank(remoteFileName)) { throw new RuntimeException( "remoteFileName is blank" ); } return remoteFileName; } private void disconnect (ChannelSftp channelSftp, Session session) { if (channelSftp != null ) { channelSftp.disconnect(); } if (session != null ) { session.disconnect(); } } } Copy code

SFTP connection configuration

public class SftpConfig { /** * sftp server address */ private String host; /** * sftp server port */ private int port; /** * sftp server login user name */ private String user; /** * sftp server login password * Choose one of password and private key */ private String password; /** * Private key file * Choose one of private key and password */ private String identity; } Copy code

File upload request

public class SftpUploadRequest { /** * Local full file name */ private String localFilePath; /** * Full file name on sftp */ private String remoteFilePath; /** * Document completion mark * Optional */ private String endFlag; } Copy code

File download request

public class SftpDownloadRequest { /** * Full file name on sftp */ private String remoteFilePath; /** * Local full file name */ private String localFilePath; /** * Document completion mark * Optional */ private String endFlag; } Copy code

SftpException: Failure

When multiple tasks upload files at the same time, some tasks will fail to upload, and the error message is as follows:

Caused by: com.jcraft.jsch.SftpException: Failure at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java: 2873 ) ~[jsch- 0.1 .55 .jar!/:?] com.jcraft.jsch.ChannelSftp.mkdir AT (ChannelSftp.java: 2182 ) ~ [jsch- 0.1 .55 ! .jar/:?] Copy the code

I searched it online ( winscp.net/eng/docs/sf... ), there are several possibilities for Failure error:

  • A file with the same name exists when the file is renamed;
  • Created an existing folder;
  • The disk is full

As can be seen from the third line of the error message, it should have hit the second possibility: an existing folder was created.

Look at the logic of the cdOrMkdir function of the SftpClient class above. When the directory exists, enter the directory; otherwise, the directory will be created. The path of SFTP upload file is: bizType/{yyyyMMdd}/{dataLabel}/biz.txt. The dataLabel value of different tasks is different, there will be concurrency problems here:

  1. Task A judges that the bizType/20210101 directory does not exist;
  2. Task B judges that the bizType/20210101 directory does not exist;
  3. A task creates the bizType/20210101 directory;
  4. When task B created the bizType/20210101 directory, an error was reported because the directory was created by task A;

Solution: Change the path of SFTP upload files to bizType/{dataLabel}/{yyyyMMdd}/biz.txt so that the file paths of different tasks no longer conflict.

JSchException

When multiple tasks download files at the same time, some tasks will fail to download, and the error message is as follows:

Caused by: com.jcraft.jsch.JSchException: channel is not opened. at com.jcraft.jsch.Channel.sendChannelOpen(Channel.java: 765 ) ~[jsch- 0.1 .55 .jar!/:?] com.jcraft.jsch.Channel.connect AT (Channel.java: 151 ) ~ [jsch- 0.1 .55 ! .jar/:?] Copy the code

At first I suspected that it was still a concurrency problem. I searched the Internet. It may be that the number of SSH terminal connections in the system is too small. This parameter is configured in/etc/ssh/sshd_config. Feeling that this should not be the reason, so I re-looked at the error code:

protected void sendChannelOpen () throws Exception { Session _session = getSession(); if (!_session.isConnected()) { throw new JSchException( "session is down" ); } Packet packet = genChannelOpenPacket(); _session.write(packet); int retry = 2000 ; long start = System.currentTimeMillis(); long timeout = connectTimeout; if (timeout != 0L ) retry = 1 ; synchronized ( this ) { while ( this .getRecipient() ==- 1 && _session.isConnected() && retry> 0 ) { if (timeout> 0L ) { if ((System.currentTimeMillis()-start)> timeout) { retry = 0 ; continue ; } } try { long t = timeout == 0L ? 10L : timeout; this .notifyme = 1 ; wait(t); } catch (java.lang.InterruptedException e) { } finally { this .notifyme = 0 ; } retry--; } } if (!_session.isConnected()) { throw new JSchException( "session is down" ); } if ( this .getRecipient() ==- 1 ) { //timeout throw new JSchException( "channel is not opened." ); } if ( this .open_confirmation == false ) { //SSH_MSG_CHANNEL_OPEN_FAILURE throw new JSchException( "channel is not opened." ); } connected = true ; } Copy code

It can be seen from lines 38 to 39 that the cause of the error is a timeout. It turns out that the timeout period set at the beginning is too short:

channelSftp.connect ( 1000 ); //Set the timeout 1s duplicated code

Solution: Increase the timeout period or use the default value.

channelSftp.connect(); copy code