Skip to main content
Version: 4.14.9

FTP component

ftp-component

The FTP component is able to connect to FTP servers (FTP, SFTP, FTPS) to write or read files. It can act either as an outbound (writing files to FTP) or inbound component (reading files from FTP). When it is not the first component in a flow it acts as an outbound component (writing files to FTP) by default.

Basic configuration

The FTP component has the following basic configuration options:

Force inbound

This option is only available when the FTP component is not the first component in a flow. It overrules the default outbound (writing files) behaviour and forces the component into inbound (reading files) mode.

reading multiple files

Every time the FTP component polls the server it will create a list of all the available files. It will pick one to process, adhering to the Order by setting. This will happen as fast as the files can be downloaded until the list has been completed. Each file is treated as a seperate message that continues to follow your flow.

The body of the message before the FTP component will be overridden with the content of the consumed file. Headers with information about the file will be set, for instance filename and filesize. Refer to the Camel File - Message headers documentation for an extensive explanation.

adding files while processing

The FTP component will wait the specified Delay time before polling the server again after reading (multiple) files. Files that are added while finishing the previous list won't be processed until this next poll of the server.

Protocol

Specify the protocol of the server.

Options

  • SFTP (default)
  • FTPS
  • FTP
note
  • Authentication to SFTP servers can be provided by specifying a password and/or an SSH private key file.
  • FTPS supports FTP over TLS up to version 1.2.

Host

URL or IP address to the FTP server, e.g. ftp.example.org or 149.210.209.32.

Port

Specify the FTP port if the server uses non-default ports. When left empty the defaults are FTP: 21, SFTP: 22.

Username

The username to log in on the FTP server.

Password

The password to log in on the FTP server.

Private Key

Either specify the location of an SSH private key file (e.g. uploaded with the File component) or directly upload a file using the upload button on the right.

note
  • Only available when the Protocol is set to SFTP.
  • The private key file needs to be of the format OpenSSH.
  • The uploaded key is identical for the Test and Production environment.

Path

The path to the target directory. In outbound mode this is the directory where the file(s) are written to. In inbound mode this is the directory where files are being read from.

trailing slash

Don't include a / at the end of the path, the component is not able to write files to the server when you do.

File Name

Only pick up files with this filename in inbound mode or set the file name when outbound. You can't use wildcards in this field.

note

Can't be used in combination with Include and/or Exclude.

regex

You can use regular expressions in inbound mode with the Include and/or Exclude settting.

File Type

Specify the file(s) type.

Options

  • Text (default)
  • Binary
producing large text files (>10MB)

If you set the File Type to binary when working with large text files Dovetail doesn't have to read all the file's content into memory to validate the charset. It increases performance and can prevent errors parsing the file later in the flow.

Charset

Specify the encoding of the file.

File Exist

Specify what to do if a file with the same name already exists.

Options

  • Override (default)
  • Append
  • Ignore
  • Fail
note

Only available when the component is in outbound mode.

Delay

The amount of time (in milliseconds) to wait before the next poll of the file/directory.

note

Only available when the component is set to inbound mode.

Advanced configuration

The FTP component has the following advanced configuration options:

Passive Mode

Specify whether FTP should operate in active or passive mode.

Options

  • Yes (default)
  • No

Disconnect

Whether or not to disconnect from the FTP server right after use. It will reconnect when the component is used again.

Options

  • Yes (default)
  • No
note

Only available when the component is set to inbound mode.

Auto Create

Automatically create missing directories in the file path.

Options

  • Yes (default)
  • No
note

Only available when the component is set to outbound mode.

Order by

Specify the order in which files are read from the server.

Options

  • Default (no guaranteed order)
  • File name (ascending) (A-Z)
  • File name (descending (Z-A)
  • Modified date (ascending) (old-new)
  • Modified date (descending) (new-old)
note

Only available when the component is set to inbound mode.

Maximum batch size

Maximum number of files processed on each poll. When left empty (default) all files to be read will be processed at once, as quickly as possible.

note
  • If the batch functionality is not working as expected try setting Passive mode to Yes.
  • Only available when the component is set to inbound mode.

Recursive

Specify if only files in the specified path (No) should be read or if files in all subdirectories of the path should be read too (Yes).

Options

  • Yes
  • No (default)
infinite loop

If this option is set to Yes and you want to move and/or move failed files into a subdirectory after processing, make sure that these folders are hidden (prepended with a .). Hidden folders are excluded from polling and you don't end up with an infinite loop.

Read more about to how set up Recursion and Excluding directories

note

Only available when the component is set to inbound mode.

Delete file

If set to Yes, files will be deleted after processing.

Options

  • Yes
  • No (default)
note
  • When set to Yes, the Move to option is ignored and hidden.
  • Only available when the component is set to inbound mode.

Move to

The relative path of the directory where files are moved to after processing. The default is set to .dovetail.

Use ../[folder_name] or ../../[folder_name] to move up folders.

hidden folders/files

Specify a path that starts without a . (dot) if your FTP server doesn't allow the creation of 'hidden' directories or files.

note
  • Only used and shown when Delete file is set to No.
  • Only available when the component is set to inbound mode.

Move failed to

The relative path to the directory where files are moved to when they fail processing. The default is set to .error.

Use ../[folder_name] or ../../[folder_name] to move up folders.

hidden folders/files

Specify a path that starts without a . (dot) if your FTP server doesn't allow the creation of 'hidden' directories or files.

note

Only available when the component is set to inbound mode.

Read lock

If set to Changed, the component only consumes the files if it has exclusive read-lock on the file (i.e. the file is not in-progress or being written). It will wait until the file lock is granted.

Options

  • None (default)
  • Changed
using read lock: changed
  • Requires write permissions on the FTP server.
  • Uses file length or file modification timestamp to detect whether the file is currently being changed or not. It will at least use 1 second to check this and there will also be some time between each check, so reading files will be slower.
  • Requires a lock on the latest file until a new file has been written to the FTP server. It will not be processed until a new file has been dropped on the FTP server.
remarks

Only available when the component is set to inbound mode.

Include

Include files if the filename matches the specified regular expression (regex) pattern.

note
  • Can't be used in combination with File name.
  • Only available when the component is set to inbound mode.
  • Only works for filenames, not folder structures.

Exclude

Exclude files if the filename matches the specified regular expressions (regex) pattern.

note
  • Can't be used in combination with File name.
  • Only available when the component is set to inbound mode.
  • Only works for filenames, not folder structures.
create and evaluate regular expressions (regex)

A great resource to create and evaluate regex is Regex 101.
Make sure to set Flavour to Java 8 in the left column when creating regex for Dovetail.

Initial delay

The amount of time (in milliseconds) to wait before the 'polling' of files/directories starts.

note

Only available when the component is set to inbound mode.

Recursion and excluding directories

Imagine you have a directory structure as shown below, where 'mainfolder' is in the root of the folder structure and there's a file in both folders.

mainfolder
main.xml

subfolder
sub.xml

We want to:

  • Read files from both the 'mainfolder' and 'testfolder'.
  • Have a .dovetail folder for processed files.
  • Have a .error folder for failed files.

Make sure the FTP component is in inbound mode and:

After polling and processing the files the folder structure would look like this:

mainfolder

.dovetail
main.xml
.error
this folder is only created and filled if there are files that failed to process
subfolder
.dovetail
sub.xml
.error
this folder is only created and filled if there are files that failed to process

Notice:

  • Both files got processed and placed in seperate .dovetail folders, relative to the folder the files were read from.
  • If no files are added the next trigger to poll the FTP server results in no action because the hidden .dovetail and .error folders are ignored.
  • If there are files that fail to process they would be placed in their respective .error folder.
caution

This only works if you move (failed) files into hidden folders. Hidden folders are automatically excluded when polling the FTP server.

Reading from folders with a lot of files

When the FTP component is in inbound mode it can't pick up and process multiple files at the same time. Every time the FTP component pols the server it will create a list of all the available files and pick one to process, adhering to the Order by setting. It does this as quickly as possible, but consuming a folder with large amounts of files can still take more time than expected.

You can improve performance by dividing files over multiple subfolders and use different FTP components to 'poll' each subfolder seperately.

first in first out (fifo)

Order by and Maximum batch size can be combined to achieve specific behaviours such as FIFO. For example when you set Order by to Modified date (descending) and Maximum batch size to 1.

Last update on May 15, 2024