Geeks With Blogs

News
Charles Young

The BizTalk Server 2004 FTP adapter offers a 'guaranteed delivery' feature, the details of which are not documented by Microsoft.  The mechanism is very straightforward, however, and involves the use of a temporary folder.   Files are staged by BizTalk in this folder before being moved to the destination folder.   BizTalk supports this for both Send and Receive handlers.  To use this feature, simply set the "Temporary Folder" property of the FTP adapter.   For Send ports, this will be the name of an FTP folder on the server, rather than a local folder.

The main advantage of using this feature is that retries can be performed in the event of a transport failure, with multiple copies of the same file being sent to the temporary folder.   However, at most, only one copy of the file will ever be moved to the destination folder or receive location.   The use of staging eliminates the possibility (in theory) of the FTP server (Send port) or BizTalk (Receive port) receiving multiple copies of the same file.   Another advantage is that you can ensure that a file is received in entirety, or not at all.   FTP allows files to be partially transferred, and supports transfer recovery where a partially completed transfer can be re-started from the point of failure (FTP regets and reputs).   While this is a great feature in some scenarios, it is potentially a killer in BizTalk, because it can lead to BizTalk believing it has received or sent a file, when in fact the file has only been partially transferred.   The use of a temporary staging folder allows a file to be fully transported before being transferred to its destination folder or receive location.   This is particularly useful when transporting large files or transporting files over un-reliable networks.

You do need to be very careful when using the feature on a receive port.  You configure the temporary folder at the host level via the BTS Administrative console.   If you have multiple servers in your BizTalk group, and run multiple instances of you host across those boxes, each instance will point to the same temporary folder.  In this case, things go badly wrong.   Unlike the File adapter, instances of the native FTP adapter do not lock or rename files in order to prevent other instances from processing the same input.   Hence, bad things happen.   It is possible to have multiple instances of the same message delivered to the MessageBox, even though you are using the 'guaranteed deliver' feature.  In addition, when one instance of the adapter moves a file from the temporary folder, a second instance then fails when it tries to do the same thing, and cannot find the file.   Always isolate your FTP adapter in a separate host, and always run just one instance for a receive handler.   One idea might be to assign your FTP-receiving host to multiple servers, but only enable one of the host instances.   You could then use WMI to automate a fail-over to another instance in the event of a failure.

Here is an extract from the FTP log showing exactly what commands the Send adapter sends to an external FTP server on a successful transfer.  In this case we are FTPing a file called {0d41d46a-f73a-4675-b7d2-51ffea0983e3}.xml to ftp://ftpServer:21/Import.   Note how a temporary copy of the file is staged in the temporary folder (ftp://ftpServer:21/ImportStaging) and then moved to the destination folder once BizTalk knows it has been staged.

USER developer1
PASS
CWD ImportStaging
created /ImportStaging/{01d8b961-d37d-9513-f09d49454be7}
RNFR {01d8b961-d37d-9513-f09d49454be7}
RNTO /Import/{0d41d46a-f73a-4675-b7d2-51ffea0983e3}.xml
QUIT

 In English, this translates as follows:

  • Establish an authenticated connection to the FTP server by sending a username and password
  • Request the server to change its working directory to the staging (transfer) directory
  • FTP the file to the working directory under a temporary file name
  • When a response is received indicating that the file has been created on the server, request the server to move the transfer file to the destination folder and rename it
  • Terminate the connection
Posted on Tuesday, August 10, 2004 10:03 AM BizTalk Server 2004/2006 | Back to top


Comments on this post: BizTalk Server 2004: FTP guaranteed delivery

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Thanks for that Charles, good to know.

I have been running into a problem using the FTP adapter, I'm not sure if it's a bug, or a 'feature'. I posted it on the newsgroup last week, but have not had any replies.

The post text is below.

Cheers,

Alan


I have a message hub that clients integrate with over FTP.
Each client has their own account, and I am using user isolation for FTP on
IIS 6.
My file structure is as follows:

FtpRoot
LocalUser
Cust1
In
Out
Cust2
In
Out

I have a receive port, and a receive location for Cust1.

Server - MyBox
UserName - Cust1
Folder - In
etc.

When I try to add a second rcv location for Cust 2

Server - MyBox
UserName - Cust2
Folder - In
etc.

I get the error box:

Error - Address should be unique.

Address 'ftp://MyBox:21/In/*.xml is already present blah blah blah...

As the two locations will connect to the server as different users, the FTP
server will route them to the correct directory, even though the address is
the same.

I was wondering if this could be a bug, as the FTP adapter should be able to
handle this, and it seems the best way to set up the FTP server for multiple
customers.

Also, if this is the case, could I set the folder to In2, export a binding
file, edit it to set the filder back to In, then import the file. This may
set the correct values, but may cause problems if the rcv locations are
edited.
Left by Alan on Aug 12, 2004 3:12 PM

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Thanks for sharing this with us. It works very good.
I have one question, how is that guid generated ?
Left by Patrick Wellink on Sep 17, 2004 3:46 PM

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Hi,

Thanks for the info that you have described about FTP adapter. I'm seeing a problem with FTP adapter with BIzTalk 2004. In my case, I'm going to get some files from client's FTP location(Source) to my local directory(destination/receive location) for processing. I've specified the parameters to FTP adapter. Same files are downloading continuously(it looks like infinite loop) and it never ending process. Of course some of my files are bigger i.e. 500 MB zip files. Are there any suggestion to resolve the issue?

Thanks a lot
-Hari
Left by Hari on Oct 03, 2004 3:04 AM

# re: BizTalk Server 2004: FTP guaranteed delivery (HARI)
Requesting Gravatar...
Hari-

Have you verified the permissions on the folder that you are using? The account which the BizTalk Host is using needs to have permisions to delete the file once it has copied it, otherwise after it copies the file, its still there as a "new" file.

John
Left by Callaway on Jan 27, 2005 6:20 PM

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Charles,

Thanks for that, it almost works on our biztalk implementation. Somehow the unix ftp server gives me an error message if the RNTO command is run with a / at the beginning. Using your example:

RNFR {01d8b961-d37d-9513-f09d49454be7}
RNTO /Import/{0d41d46a-f73a-4675-b7d2-51ffea0983e3}.xml

Or if I login manually:

ren {01d8b961-d37d-9513-f09d49454be7} /Import/{0d41d46a-f73a-4675-b7d2-51ffea0983e3}.xml

Both fail with a 550 rename: No such file or directory. error.

If I login manually and enter (no first / in destination):

ren {01d8b961-d37d-9513-f09d49454be7} Import/{0d41d46a-f73a-4675-b7d2-51ffea0983e3}.xml

Then it works. Has anyone ever run into this or know a way around it?

Thank you,
Thiago
Left by Thiago Almeida on Feb 10, 2005 3:58 AM

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Thiago, I have the same problem, but no solution for it!

I found that this work in manual ftp
ren {98274093...} ../import/{002984032}

but biztalk gives me the same error us you got.

Does anyone has solution for this ?

Is it related to destination ftp server software ?

Tomi
Left by Tomi on Mar 10, 2005 9:59 AM

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Founded a solution for the temporary folder problem with starting slash!

In my case it was needed to give full address of the destination folder like:
/home/user/import
Left by Tomi on Apr 13, 2005 1:55 PM

# Using the BizTalk FTP Adapter
Requesting Gravatar...
Recently as I have been learning more about BizTalk I decided to explore the FTP adapter that comes packaged...
Left by Adam Beal BizTalk, .NET 2.0, and on Nov 17, 2005 5:21 PM

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Hi All,

We are also facing the similar problem while using FTP Adapter.

We have 2 Biztalk Boxes in production and have 2 seprate host for all our receive ports.All our receive ports runs under seprate host and not the one provided by microsoft default "Microsoft Biztalk server application".All our receive ports which is used for downloading the files from FTP server are having the Temporary Path mentioned for guaranteed delivery. It is not done at host level rather on receive location level but the temp path is same and there is temp path folder created in both the biztalk boxes.

Now while downloading the file which is of only 82Bytes , some time we find that biztalk picks and process only 0 KB file which is causing a hell lot of problem for our business.

This does not happen always, but I feel sometime when file is just dropped, one of the host picks up and start processing but that time file is not even copied fully to the FTP Server which causes it to process the 0KB file.

Any one has similar expr or any idea why this is happening.
Left by Sachin on Jan 01, 2006 8:06 PM

# re: BizTalk Server 2004: FTP guaranteed delivery
Requesting Gravatar...
Hello Charles,

Our experience with the FTP Adapter or the combination FTP & BizTalk is not very positive.

An architecture with FTP on receive side is not scalable which is a huge downsize. To make it scalable you should manually assign handlers for each receive location. so when putting a new BizTalk server in the architecture you must reconfigure all FTP Receive locations to spread the load over all BizTalk Server.

that's why we try to avoid the FTP protocol and let the customer FTP to our FTP Gateway, while we will then pickup the message using the File Adapter (thru a file share).

Left by Stijn Degrieck on Jan 05, 2006 4:17 AM

Your comment:
 (will show your gravatar)


Copyright © Charles Young | Powered by: GeeksWithBlogs.net