1. Contents
  2. Guides
  3. Channels and tasks
  4. Setting up data sources

Setting up data sources

This guide sets TRADElube up to fetch your supplier's file itself: on a fixed schedule and without anybody uploading it by hand. Two things are needed for that: a data source, which says where the file sits and how TRADElube signs in there, and an endpoint on the task, which connects that source with a data format. You set both up here. The fields in detail are under data sources (reference).

The reverse direction works the same way: via FTP and SFTP, TRADElube can also deliver an exported file. The steps are the same, only the task is then an export.

Prerequisites

  • The server's login details: host name, user name and password, plus the directory the files sit in. For HTTP the address instead and, if needed, the sign-in.
  • A file-based task with a finished data format, from importing products from a file for instance.

Step by step

  1. Create the data source. Open Settings and there Data Sources. New opens a menu with the three types FTP, SFTP and HTTP. The type is fixed after creation. If the supplier offers both FTP and SFTP, take SFTP, which encrypts the transfer. HTTP is the type for fetching by address, but it can only read. An export needs FTP or SFTP.

  2. Enter the login details. Give the source a name that identifies the supplier, because that is the name you pick later on the endpoint. Enter host, port, directory, user name and password; for HTTP the base address and, where needed, the sign-in.

  3. Test the connection. For FTP and SFTP the Connect button sits above the fields. Use it right away: it reports within two seconds whether sign-in and directory are right, and a click on the message shows the server's answer. Without this check a typo only shows up when the scheduled execution comes back empty.

  4. Set the endpoint on the task. Open the task and, in its Configuration section, the Endpoints list. An endpoint connects three entries: the data format, the data source and the file pattern. Choose format and source, and check the preset File Pattern: in it, {Name} stands for the name of the data format and * for any sequence of characters. The supplier's file has to match this pattern, otherwise the execution does not pick it up. What the remaining fields do is in the reference.

  5. Set the schedule. Set the task's Performance field to an interval, Nightly for instance. From then on every execution looks into the directory, reads what matches the pattern, and renames the processed file so the next one does not read it again. It stays on the server as evidence of what was delivered.

Checking the result

Put a file into the directory and start the task once through Execute instead of waiting for the schedule. Three things show that the route is in place:

  • The execution ends without errors, and the traces show the transfers of the rows read.
  • On the server the file has been renamed following the processed-file pattern.
  • A second start without a new file reads nothing, because the renamed one drops out of the pattern.

Frequent pitfalls

  • The file pattern does not match. {Name} is the name of the data format, not of the file. If the format is called "Article list CSV" and the file pricelist.csv, the preset pattern does not match. Adjust the pattern in step 4 to the actual file name.
  • The same file is to be read again. The execution from step 5 has renamed the processed file and thereby made it invisible to the next run. Put the file into the directory again or rename it back. There is no hidden memory, only the pattern.
  • A data source alone does nothing. The source from step 1 only takes effect once an endpoint names it, as in step 4. The reverse holds too: uploading by hand through Execute needs no endpoint at all.
  • After a password change at the supplier you change exactly one place, the login details from step 2. All tasks whose endpoints name it are working again from the next execution.

Further reading