Best practices

This guide contains best practices for designing and running flows.

Designing flows

Keep your messages clean

Within a flow a message can grow in size, because of added headers and data in the body. It is good practice to clean up your messages as much as possible, especially before communicating with endpoints.

• Use the Remove Headers component to clean up headers you are not using anymore.
• Clean up the body whenever it is not applicable anymore, for example with an empty JSON ({}) or XML (<root/>).

Use variables

It's important to understand the different kinds of variables within Dovetail and how to use them to your advantage. Read more about Using variables in Dovetail.

Flow properties

It is good practice to use Flow properties for efficient configuration of settings in your flow, for example the base url of an API. Using Flow properties improves the scalability of your integration solution, making it much easier to copy similar solutions to different clients with only minor edits.

Tenant variables

It is good practice to use Tenant variables for data that is used in multiple flows or to change the functionality of a flow without having to edit and reinstall it.

Data formats

In application integration you encounter a big variety in data formats. It is good practice to:

• Use standardised data formats like XML or JSON over other formats like CSV, Excel, ASCII.
• Use text based data formats, instead of binary.
• Keep the size of messages lower than 10 MB.

Protocols

In general it's recommend to use HTTPS endpoints over other endpoint protocols like SFTP, email (SMTP) or file. HTTPS uses internet standards and scales well. For scalability also a RabbitMQ endpoint is provided.

Document your flows

Use the Notes, Component notes and Version remark to document how your flow works. It will greatly improve your understanding of a flow, especially over time. Your future self thanks you in advance!

Flow versions

Limit the number of Flow Versions to maintain application responsiveness and delete old versions when possible.

Encoding

When working with text data, it's essential to understand the encoding being used to ensure that characters are interpreted correctly. UTF-8 has become the standard for web content due to its versatility and compatibility with ASCII. Other encodings are still relevant in certain contexts, particularly for legacy systems and specific language requirements.

It is good practice to use the Encoder component to transform the encoding of a message to UTF-8 in flows for optimal compatibility with Dovetail components.

Pretty vs Raw HTTP responses

Be mindful that tools like Postman may alter responses for readability, possibly changing the encoding and/or data format layout 'under the hood'. Ensure you are viewing 'raw' responses for accuracy. Keep in mind that this 'raw' response is the response that Dovetail gets.

Scripts

It is good practice to minimise the use of the Script component. While it gives a lot of flexibility it also:

• Makes it more difficult or impossible for us to support your integration solution.
• Takes more resources on the server.
• Updates to the component and/or it's underlying libraries can impact on how the Script functions. Leading to:
  • Testing, and possibly extra development, when Dovetail updates.
  • No guarantees regarding backwards and/or forwards compatibility of Script code between Dovetail versions.

Prefer Groovy over JavaScript whenever possible. Groovy, like Camel, runs in the JVM (Java Virtual Machine) while Javascript runs via Node.js (and therefore has a higher impact on performance).

Running flows

Flow performance

To optimise the performance of your flow it is good practice to:

• Keep the number of steps in a flow as low as possible.
• Keep the number of linked flows as low as possible.
• Keep the number of high impact steps as low as possible.
• Prefer the Exchange pattern One way over Request reply.
• Use the Transport Synchronous by default. Avoid using Transport type Queues within flows whenever possible. Messages are stored on a broker when set to Queues, this makes processing them slower (it's used to keep memory consumption low).
• Disable tracing in production (see Trace Configuration)

Trace configuration

Transactions are stored in a database on the server your Dovetail instance is running on. Avoid enabling tracing on flows that process large files and/or headers and set the time to the smallest value you can to make sure the server doesn't run out of disk space.

Keep in mind that saving a copy of each message can impact the performance of your flows. Disable tracing on flows that process large quantities of data and/or large files. Implement a Management by Exception solution to monitor these types of flows.

Exception handling

Use a Management by Exception principle and define the expected response of an endpoint. All unexpected responses can be logged and/or handled functionally from within your flow. Refer to our Management by Exception guide for an extensive tutorial on how to implement this.

Good integration solutions

To summarize it's a good practice to:

• Make integrations between two endpoints (systems) instead of multiple endpoints (Keep it simple).
• Use scalable protocols/technologies like HTTPS, JMS, AMQP over not scalable ones like SFTP, File or Email.
• Use structure dataformats (XML or JSON) of fixed small size. This makes it easy to process messages in a predictable way.
• Keep as much of the application logic outside the integration platform.
• Not retrieve a new set of data when the previous set isn't done processing yet. Indications of this happening are increasing pending exchanges and/or instances running out of memory.

Last update on Feb 14, 2024