Mule’s file connector allows us to read and write files to and from the underlying file system. Conveniently, we can also use stream large files rather than reading the whole lot in but what is the difference in behaviour in each case?
If we had to read a whole file into memory before processing it, large files could cripple our Mule application and this is why streaming is so useful. The file connector will stream files by default so this is out-of-the-box functionality.
On the other hand, if you want to read in small files, then you might want to read the whole file in at once. Many people do not realise that since streaming is enabled by default, even small files will be streamed. Unless this is disabled, the following sequence of events will happen:
1. Mule creates a new stream based on the file and reads the whole file in (since it is small)
2. The stream is passed along through the Mule services. Until the stream is closed, the original file is still available and would not have been moved to the moveToDirectory (or deleted).
3. The pollingFrequency kicks in and Mule tries to read in the same file again causing an error or causing the same message to be re-processed.
Moral of this blog post: If you do not want to stream files, disable this by configuring your file connector accordingly: