Home All Groups Group Topic Archive Search About

FileSystemWatcher advice required please

Author
9 Feb 2006 4:16 PM
jimmyfishbean
Hi,

I have a VB.Net Windows Service that monitors a varying number of
directories (that are read from an INI file).  I set up a
FileSystemWatcher object to monitor each directory that is read from
the INI file and add each object to a collection.

The purpose of my service is to monitor the directories for any XML
files that are dropped into them.  When a file is dropped, it is
firstly renamed by appending the extension '.lock' to the name of the
XML file.  Secondly, the file is then processed by sending the contents
to a Web Service.  Once the file has been completely processed and a
return value is received from the Web Service, the file is moved from
the directory to a 'Processed Files' folder (that is not monitored by a
FileSystemWatcher object).

I call a sub to process each directory, which loops through all files
in the directory (at one moment in time) and calls the web service for
each file.

When the service starts, any XML files in the 'monitored' are picked up
and processed without any erros/trouble.  The problem starts if any
'new' XML files are dropped into the monitored directories when the
service is currently busy processing.  I call the same sub that loops
through all files in a directory.  Of course, there will be files
present in both calls to the sub, causing errors.

How can I tackle this?  Is it reasonable to use a timer (I do not
currently use one). I did add a timer, but the number of handles and
memory kept increasing all the time (even when there was nothing to
process).

I am not concerned about the deletion of files or modification or
access.  Simply addition/creation.

Any advice/guidance is greatly appreciated.  Thanks.

Jimmy

Code snippet:

Private Sub setup()
  ...
   While l <= UBound(aInputDirs)
      Dim fsw As New FileSystemWatcher
      fsw.Path = aInputDirs(l)
      fsw.Filter = "*.xml"
      fsw.IncludeSubdirectories = False
      fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite
      AddHandler fsw.Changed, AddressOf OnChanged
      fsw.EnableRaisingEvents = True

      watcherCollection.Add(fsw)
      l = l + 1
   End While

   l = 0
   While l < UBound(aInputDirs)
      ProcessDir(l)
      l = l + 1
   End While
   ...
End Sub


    Private Shared Sub OnChanged(ByVal source As Object, ByVal e As
System.IO.FileSystemEventArgs)

        'Dim strDir As String = e.Name

        '1. work out the directory that has been changed (using the
full path/filename : e.Name)
        Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\")
        Dim strDir As String = Left(e.FullPath, intLastSlash)


        If Not bBusyProcessing Then
            If Not e.Name = "XMLtoSOAP.ini" Then
                If (File.GetAttributes(e.FullPath) And
FileAttributes.Directory) = FileAttributes.Directory Then
                    'Directory changed, created, or deleted.
                Else
                    'File changed, created, or deleted.
                    Dim i As Integer = 0

                    Do While i < UBound(aInputDirs)
                        If aInputDirs(i) = strDir Then
                            ProcessDir(i)
                            Exit Do
                        End If

                        i = i + 1
                    Loop
                End If
            Else
                'setup()
            End If
        End If

    End Sub

    Public Sub ProcessDir(ByRef lElement As Integer)
        ' Find all files in this directory and try to process them.
        Dim fso As New Scripting.FileSystemObject
        Dim oFile As Scripting.File
        Dim strOrigFile As String
        Dim stream As Scripting.TextStream
        Dim bFileAlreadyOpen As Boolean
        Dim cProcessedFiles As New Collection
        Dim bMoreToProcess, bProcessedOne As Boolean

        bMoreToProcess = True

        While bMoreToProcess = True
            bProcessedOne = False
            For Each oFile In fso.GetFolder(aInputDirs(lElement)).Files

                bBusyProcessing = True

                'Make sure this file hasn't already been tried - if the
item is not in the collection an error is returned
                On Error Resume Next
                If cProcessedFiles.Count() <> 0 And
cProcessedFiles.Item(oFile.Name) <> oFile.Name Then
                    On Error GoTo 0
                    bFileAlreadyOpen = False
                    strOrigFile = oFile.Name
                    'Rename the file to see if it is still locked by
any process
                    On Error GoTo FileStillOpen

                    'create a copy of the original file to revert back
to if in changerootnode mode
                    If bChangeRootNode Then
                        If Not fso.FolderExists(aInputDirs(lElement) &
"\Temp") Then
                            fso.CreateFolder(aInputDirs(lElement) &
"\Temp")
                        End If

                        fso.CopyFile(oFile.Path, aInputDirs(lElement) &
"\Temp\" & oFile.Name)
                        sCurrFile = aInputDirs(lElement) & "\Temp\" &
oFile.Name
                    End If

                    fso.MoveFile(oFile.Path, oFile.Path & ".lock")
                    On Error GoTo 0
                    If Not bFileAlreadyOpen Then
                        ProcessFile(lElement, strOrigFile)
                    End If
                    'Store the file in a collection for exclusion in
the lower loop
                    cProcessedFiles.Add(strOrigFile, strOrigFile)
                    bProcessedOne = True
                End If

                bBusyProcessing = False
            Next oFile
            bMoreToProcess = bProcessedOne
        End While

        fso = Nothing

        Exit Sub

FileStillOpen:
        bFileAlreadyOpen = True
        Resume Next

    End Sub

'NOTE: Sub ProcessFile calls the web service

Author
9 Feb 2006 5:55 PM
Kevin Spencer
Okay, you have a FileSystemWatcher which, I assume, is monitoring the
"Created" event. When a file is created, the FileSystemEventArgs passed with
the event contains the path of the file created. You can use this
information to move the file to the new location. No need to create a list
of files to move. Process each one at a time.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
We got a sick zebra a hat,
you ultimate tuna.


<jimmyfishb***@yahoo.co.uk> wrote in message
Show quote
news:1139501783.714521.6360@o13g2000cwo.googlegroups.com...
> Hi,
>
> I have a VB.Net Windows Service that monitors a varying number of
> directories (that are read from an INI file).  I set up a
> FileSystemWatcher object to monitor each directory that is read from
> the INI file and add each object to a collection.
>
> The purpose of my service is to monitor the directories for any XML
> files that are dropped into them.  When a file is dropped, it is
> firstly renamed by appending the extension '.lock' to the name of the
> XML file.  Secondly, the file is then processed by sending the contents
> to a Web Service.  Once the file has been completely processed and a
> return value is received from the Web Service, the file is moved from
> the directory to a 'Processed Files' folder (that is not monitored by a
> FileSystemWatcher object).
>
> I call a sub to process each directory, which loops through all files
> in the directory (at one moment in time) and calls the web service for
> each file.
>
> When the service starts, any XML files in the 'monitored' are picked up
> and processed without any erros/trouble.  The problem starts if any
> 'new' XML files are dropped into the monitored directories when the
> service is currently busy processing.  I call the same sub that loops
> through all files in a directory.  Of course, there will be files
> present in both calls to the sub, causing errors.
>
> How can I tackle this?  Is it reasonable to use a timer (I do not
> currently use one). I did add a timer, but the number of handles and
> memory kept increasing all the time (even when there was nothing to
> process).
>
> I am not concerned about the deletion of files or modification or
> access.  Simply addition/creation.
>
> Any advice/guidance is greatly appreciated.  Thanks.
>
> Jimmy
>
> Code snippet:
>
> Private Sub setup()
>  ...
>   While l <= UBound(aInputDirs)
>      Dim fsw As New FileSystemWatcher
>      fsw.Path = aInputDirs(l)
>      fsw.Filter = "*.xml"
>      fsw.IncludeSubdirectories = False
>      fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite
>      AddHandler fsw.Changed, AddressOf OnChanged
>      fsw.EnableRaisingEvents = True
>
>      watcherCollection.Add(fsw)
>      l = l + 1
>   End While
>
>   l = 0
>   While l < UBound(aInputDirs)
>      ProcessDir(l)
>      l = l + 1
>   End While
>   ...
> End Sub
>
>
>    Private Shared Sub OnChanged(ByVal source As Object, ByVal e As
> System.IO.FileSystemEventArgs)
>
>        'Dim strDir As String = e.Name
>
>        '1. work out the directory that has been changed (using the
> full path/filename : e.Name)
>        Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\")
>        Dim strDir As String = Left(e.FullPath, intLastSlash)
>
>
>        If Not bBusyProcessing Then
>            If Not e.Name = "XMLtoSOAP.ini" Then
>                If (File.GetAttributes(e.FullPath) And
> FileAttributes.Directory) = FileAttributes.Directory Then
>                    'Directory changed, created, or deleted.
>                Else
>                    'File changed, created, or deleted.
>                    Dim i As Integer = 0
>
>                    Do While i < UBound(aInputDirs)
>                        If aInputDirs(i) = strDir Then
>                            ProcessDir(i)
>                            Exit Do
>                        End If
>
>                        i = i + 1
>                    Loop
>                End If
>            Else
>                'setup()
>            End If
>        End If
>
>    End Sub
>
>    Public Sub ProcessDir(ByRef lElement As Integer)
>        ' Find all files in this directory and try to process them.
>        Dim fso As New Scripting.FileSystemObject
>        Dim oFile As Scripting.File
>        Dim strOrigFile As String
>        Dim stream As Scripting.TextStream
>        Dim bFileAlreadyOpen As Boolean
>        Dim cProcessedFiles As New Collection
>        Dim bMoreToProcess, bProcessedOne As Boolean
>
>        bMoreToProcess = True
>
>        While bMoreToProcess = True
>            bProcessedOne = False
>            For Each oFile In fso.GetFolder(aInputDirs(lElement)).Files
>
>                bBusyProcessing = True
>
>                'Make sure this file hasn't already been tried - if the
> item is not in the collection an error is returned
>                On Error Resume Next
>                If cProcessedFiles.Count() <> 0 And
> cProcessedFiles.Item(oFile.Name) <> oFile.Name Then
>                    On Error GoTo 0
>                    bFileAlreadyOpen = False
>                    strOrigFile = oFile.Name
>                    'Rename the file to see if it is still locked by
> any process
>                    On Error GoTo FileStillOpen
>
>                    'create a copy of the original file to revert back
> to if in changerootnode mode
>                    If bChangeRootNode Then
>                        If Not fso.FolderExists(aInputDirs(lElement) &
> "\Temp") Then
>                            fso.CreateFolder(aInputDirs(lElement) &
> "\Temp")
>                        End If
>
>                        fso.CopyFile(oFile.Path, aInputDirs(lElement) &
> "\Temp\" & oFile.Name)
>                        sCurrFile = aInputDirs(lElement) & "\Temp\" &
> oFile.Name
>                    End If
>
>                    fso.MoveFile(oFile.Path, oFile.Path & ".lock")
>                    On Error GoTo 0
>                    If Not bFileAlreadyOpen Then
>                        ProcessFile(lElement, strOrigFile)
>                    End If
>                    'Store the file in a collection for exclusion in
> the lower loop
>                    cProcessedFiles.Add(strOrigFile, strOrigFile)
>                    bProcessedOne = True
>                End If
>
>                bBusyProcessing = False
>            Next oFile
>            bMoreToProcess = bProcessedOne
>        End While
>
>        fso = Nothing
>
>        Exit Sub
>
> FileStillOpen:
>        bFileAlreadyOpen = True
>        Resume Next
>
>    End Sub
>
> 'NOTE: Sub ProcessFile calls the web service
>
Author
10 Feb 2006 1:59 PM
jimmyfishbean
Thanks for that Kevin.  I have tried your suggestion.

However, if I drop more than 1 file into the DROP folders, only the
first file is passed to the OnChanged sub.  Therefore I can only move 1
file (unless I call a sub that moves all files from the DROP folder to
a temp folder).  Am I correct to assume that only 1 event is triggered
regardless of the number of files dropped into the directory?  It would
be a lot easier/better if a separate event was triggered for each file.

If I try to move all files in the DROP folder in the OnChanged sub,
only 1 gets moved (as the application thinks that only one file is
there at the time the event was triggered).  Do I need to use a timer
here to periodically check the fodler?  I do not want to, and thought
the whole purpose of the filesystemwatcher was to provide functionality
that would allow each and every file droppped to trigger an event.  I
take it I have it all so wrong...

My new OnChanged sub:

    Private Shared Sub OnChanged(ByVal source As Object, ByVal e As
System.IO.FileSystemEventArgs)
        Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\")
        Dim strDir As String = Left(e.FullPath, intLastSlash)
        Dim fso As New Scripting.FileSystemObject
        Dim oFile As Scripting.File
        Dim file As File

        If Not e.Name = "XMLtoSOAP.ini" Then
            If (File.GetAttributes(e.FullPath) And
FileAttributes.Directory) = FileAttributes.Directory Then
                'Directory changed, created, or deleted.
            Else
                'File changed, created, or deleted.
                'move the files to a temp location
                If Not Directory.Exists(strDir & "\Temp") Then
                    Directory.CreateDirectory(strDir & "\Temp")

                    If File.Exists(strDir & "\Temp\" & e.Name) Then
                        file.Delete(strDir & "\Temp\" & e.Name)
                    End If

                    'now move the file(s)
                    For Each oFile In fso.GetFolder(strDir).Files
                        oFile.Move(strDir & "\Temp\" & e.Name)
                    Next
                End If
            End If
        End If
    End Sub

The problem is that I never know how many files will be dropped into
the DROP folder at one time.  There may be just a single one, there may
be 20 and ther may be 20, followed by another 30 straight afterwards.

Thanks.
Jimmy
Author
10 Feb 2006 2:28 PM
jimmyfishbean
The sub should read:

    Private Shared Sub OnChanged(ByVal source As Object, ByVal e As
System.IO.FileSystemEventArgs)
        Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\")
        Dim strDir As String = Left(e.FullPath, intLastSlash)
        Dim fso As New Scripting.FileSystemObject
        Dim oFile As Scripting.File
        Dim file As File

        On Error Resume Next

        If Not e.Name = "XMLtoSOAP.ini" Then
            If (file.GetAttributes(e.FullPath) And
FileAttributes.Directory) = FileAttributes.Directory Then
                'Directory changed, created, or deleted.
            Else
                'File changed, created, or deleted.
                'move the files to a temp location
                If Not Directory.Exists(strDir & "\Temp") Then
                    Directory.CreateDirectory(strDir & "\Temp")

                End If

                If file.Exists(strDir & "\Temp\" & e.Name) Then
                    file.Delete(strDir & "\Temp\" & e.Name)
                End If

                'now move the file(s)
                For Each oFile In fso.GetFolder(strDir).Files
                    oFile.Move(strDir & "\Temp\" & e.Name)
                Next
            End If
        End If

    End Sub

Cheers.
Author
10 Feb 2006 3:04 PM
Kevin Spencer
Hi Jimmy,

Why are you using OnChange? If you remember my first reply, I told you that
I assumed you were using the OnCreated event. This is because you said that
the files were being "dropped" into the directory. This causes a Create
event to occur, and that should be the only event that you need to listen
for.

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
We got a sick zebra a hat,
you ultimate tuna.


<jimmyfishb***@yahoo.co.uk> wrote in message
Show quote
news:1139579975.215832.79630@g43g2000cwa.googlegroups.com...
> Thanks for that Kevin.  I have tried your suggestion.
>
> However, if I drop more than 1 file into the DROP folders, only the
> first file is passed to the OnChanged sub.  Therefore I can only move 1
> file (unless I call a sub that moves all files from the DROP folder to
> a temp folder).  Am I correct to assume that only 1 event is triggered
> regardless of the number of files dropped into the directory?  It would
> be a lot easier/better if a separate event was triggered for each file.
>
> If I try to move all files in the DROP folder in the OnChanged sub,
> only 1 gets moved (as the application thinks that only one file is
> there at the time the event was triggered).  Do I need to use a timer
> here to periodically check the fodler?  I do not want to, and thought
> the whole purpose of the filesystemwatcher was to provide functionality
> that would allow each and every file droppped to trigger an event.  I
> take it I have it all so wrong...
>
> My new OnChanged sub:
>
>    Private Shared Sub OnChanged(ByVal source As Object, ByVal e As
> System.IO.FileSystemEventArgs)
>        Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\")
>        Dim strDir As String = Left(e.FullPath, intLastSlash)
>        Dim fso As New Scripting.FileSystemObject
>        Dim oFile As Scripting.File
>        Dim file As File
>
>        If Not e.Name = "XMLtoSOAP.ini" Then
>            If (File.GetAttributes(e.FullPath) And
> FileAttributes.Directory) = FileAttributes.Directory Then
>                'Directory changed, created, or deleted.
>            Else
>                'File changed, created, or deleted.
>                'move the files to a temp location
>                If Not Directory.Exists(strDir & "\Temp") Then
>                    Directory.CreateDirectory(strDir & "\Temp")
>
>                    If File.Exists(strDir & "\Temp\" & e.Name) Then
>                        file.Delete(strDir & "\Temp\" & e.Name)
>                    End If
>
>                    'now move the file(s)
>                    For Each oFile In fso.GetFolder(strDir).Files
>                        oFile.Move(strDir & "\Temp\" & e.Name)
>                    Next
>                End If
>            End If
>        End If
>    End Sub
>
> The problem is that I never know how many files will be dropped into
> the DROP folder at one time.  There may be just a single one, there may
> be 20 and ther may be 20, followed by another 30 straight afterwards.
>
> Thanks.
> Jimmy
>
Author
14 Feb 2006 12:41 PM
jimmyfishbean
Hi Kevin,

I am monitoring just for the OnCreated event as you have suggested:

   fsw.IncludeSubdirectories = False
   fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite
   fsw.NotifyFilter = System.IO.NotifyFilters.FileName
   fsw.NotifyFilter = System.IO.NotifyFilters.LastAccess

   AddHandler fsw.Changed, AddressOf OnCreated
   AddHandler fsw.Created, AddressOf OnCreated 'not recommended see the
link at top this while loop

In my OnCreated sub, I try to move all files that have been dropped
into the DROP folder to a different folder where they will be
processed.  I would expect all files to be moved:

                If Not Directory.Exists(strDir & "\Temp") Then
                    Directory.CreateDirectory(strDir & "\Temp")
                End If

                If File.Exists(strDir & "\Temp\" & e.Name) Then
                    File.Delete(strDir & "\Temp\" & e.Name)
                End If

                'now move the file(s)
                For Each oFile In fso.GetFolder(strDir).Files
                    oFile.Move(strDir & "\Temp\" & e.Name)
                    bFileMoved = True
                Next

However, only one file gets moved to the Temp folder.  It seems as if
the OnCreated event gets called when the first file is identified as
being created, and when I try to move all files in the folder, the
filesystem thinks there is only the one file (i.e. the one that
triggered the event).  Therefore, I have had to implement a timer that
will read the DROP folder, and move the remaining files periodically.
To me this is defeating the objective of the FileSystemWatcher.

To make matters worse ;P, the memory/no of handles used  by my
application is forever increasing.  I am closing objects and setting
them to nothing, as well as calling GC.Collect() in each sub.

Is this really how the FileSystemWatcher works?  Can you please suggest
a more elegant/efficient way of achieving my aim?  Much appreciated.

Jimmy
Author
14 Feb 2006 1:23 PM
Kevin Spencer
Hi Jimmy,

The Created event occurs once per each file created. In addition, I see that
you're attaching the same event handler to the Changed event. This was not
what I suggested. To quote my earlier replies:

First reply:

"When a file is created, the FileSystemEventArgs passed with the event
contains the path of the file created. You can use this
information to move the file to the new location.No need to create a list of
files to move. **Process each one at a time.**"

Second reply:

"Why are you using OnChange? If you remember my first reply, I told you that
I assumed you were using the OnCreated event. This is because you said that
the files were being "dropped" into the directory. This causes a Create
event to occur, and that should be the only event that you need to listen
for."

You know, my mother used to tell me about people who loved her recipes for
one thing or another. But they would come back and tell her that when they
used her recipe, it just didn't taste as good as my mother's. Invariably,
she found out that they had "improvised" a bit on her recipe, and that this
was the reason it didn't taste the same!

I'm thinking that perhaps you are misunderstanding something about these
files being dropped into the folder, because of a question you asked earlier
in this thread:

"Am I correct to assume that only 1 event is triggered regardless of the
number of files dropped into the directory?  It would
be a lot easier/better if a separate event was triggered for each file."

At the time you were asking about the Change event, but I believe you
misunderstand the idea of "dropping multiple files" in general. When you
drop multiple files into a folder, they are created one at a time. It
happens so fast that it may look like it's doing more than one at a time,
but it's not.

Also, multiple events are fired during different types of IO operations.
Take a note of this Note from the SDK:

"Common file system operations might raise more than one event. For example,
when a file is moved from one directory to another, several OnChanged and
some OnCreated and OnDeleted events might be raised. Moving a file is a
complex operation that consists of multiple simple operations, therefore
raising multiple events. Likewise, some applications (for example,
anti-virus software) might cause additional file system events that are
detected by FileSystemWatcher."

Your objective is to handle as few as necessary to get the job done.
Otherwise, you may have one event handler trying to perform the same IO
operation as another "at the sem time," and run into beaucoups trouble.

So, here's what I want you to do. Handle *only* the Created event. Don't
worry about NotifyFilter. For the OnChange event handler, the default
(LastWrite | FileName | DirectoryName) is perfect. In your OnChange
override, simply move the file specified in the FileSystemEventArgs.FullPath
member passed. The OnChange event handler method will be called for *each*
file dropped into the folder. Remember, *one* at a time!

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
We got a sick zebra a hat,
you ultimate tuna.


<jimmyfishb***@yahoo.co.uk> wrote in message
Show quote
news:1139920893.901524.19210@g47g2000cwa.googlegroups.com...
> Hi Kevin,
>
> I am monitoring just for the OnCreated event as you have suggested:
>
>   fsw.IncludeSubdirectories = False
>   fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite
>   fsw.NotifyFilter = System.IO.NotifyFilters.FileName
>   fsw.NotifyFilter = System.IO.NotifyFilters.LastAccess
>
>   AddHandler fsw.Changed, AddressOf OnCreated
>   AddHandler fsw.Created, AddressOf OnCreated 'not recommended see the
> link at top this while loop
>
> In my OnCreated sub, I try to move all files that have been dropped
> into the DROP folder to a different folder where they will be
> processed.  I would expect all files to be moved:
>
>                If Not Directory.Exists(strDir & "\Temp") Then
>                    Directory.CreateDirectory(strDir & "\Temp")
>                End If
>
>                If File.Exists(strDir & "\Temp\" & e.Name) Then
>                    File.Delete(strDir & "\Temp\" & e.Name)
>                End If
>
>                'now move the file(s)
>                For Each oFile In fso.GetFolder(strDir).Files
>                    oFile.Move(strDir & "\Temp\" & e.Name)
>                    bFileMoved = True
>                Next
>
> However, only one file gets moved to the Temp folder.  It seems as if
> the OnCreated event gets called when the first file is identified as
> being created, and when I try to move all files in the folder, the
> filesystem thinks there is only the one file (i.e. the one that
> triggered the event).  Therefore, I have had to implement a timer that
> will read the DROP folder, and move the remaining files periodically.
> To me this is defeating the objective of the FileSystemWatcher.
>
> To make matters worse ;P, the memory/no of handles used  by my
> application is forever increasing.  I am closing objects and setting
> them to nothing, as well as calling GC.Collect() in each sub.
>
> Is this really how the FileSystemWatcher works?  Can you please suggest
> a more elegant/efficient way of achieving my aim?  Much appreciated.
>
> Jimmy
>
Author
14 Feb 2006 3:01 PM
Michael D. Ober
Here's a complete VB 2005 application that monitors a directory tree and
processes files that are ready to process.  I defined ready to process as a
non-zero byte size file that hasn't had a change in file size for at least
one minute.  I think I have indented the correctly - to verify the indents,
copy into a VB 2005 project and use the IDE's indenter.

Mike.

'============================================================

Option Compare Text
Option Explicit On
Option Strict On

Imports System.IO
Imports System.Threading

Module ArchiveImportLoader
Private ArchiveImport As String
Private FilesToProcess As ProcessFiles
Private fsw As FileSystemWatcher
Private Declare Function GetConsoleWindow Lib "kernel32.dll" () As IntPtr
Private Declare Function ShowWindow Lib "user32.dll" (ByVal hwnd As IntPtr,
ByVal nCmdShow As Int32) As Int32
Private Const SW_SHOWMINNOACTIVE As Int32 = 7

Public Sub Main()
  Dim NoVersion As New Collection
  logs.WriteLog(My.Application.Info.Title & " Starting")
  Console.Title = AppName()
  Thread.CurrentThread.Name = "MAIN"
  ' Read environment
  ArchiveImport = OSInterface.iniWrapper.ReadString("ArchiveManager",
"ImportRoot", "wakefield.ini")
  Dim ArchiveRoot As String =
OSInterface.iniWrapper.ReadString("ArchiveManager", "Root", "wakefield.ini")
  For Each v As String In
Split(OSInterface.iniWrapper.ReadString("ArchiveManager", "NoVersion",
"wakefield.ini"), "|")
    NoVersion.Add("*" & v)
  Next
  ' Ensure the import directory exists
  WriteLog("Creating File System Watcher on '" & ArchiveImport & "'")
  My.Computer.FileSystem.CreateDirectory(ArchiveImport)
  ' Start the worker thread and configure the FSW subsystem
  FilesToProcess = New ProcessFiles(ArchiveImport, ArchiveRoot, NoVersion)
  ConfigureFSW()
  ' Configuration done; Minimize the window
  WriteLog("Initialization Complete")
  Thread.Sleep(New TimeSpan(0, 0, 5))
  ShowWindow(GetConsoleWindow(), SW_SHOWMINNOACTIVE)
  ' Go to sleep, but periodically wakeup and check for missed files
  Dim SleepPeriod As TimeSpan = New TimeSpan(0, 15, 0)
  Do While True
    Console.WriteLine(Now.ToString & vbTab & Thread.CurrentThread.Name & ":
Checking for Missed Files")
    ProcessTree()
    Thread.Sleep(SleepPeriod)
  Loop
End Sub

Private Sub ProcessTree()
  Try
    My.Computer.FileSystem.CreateDirectory(ArchiveImport)
    For Each fName As String In
My.Computer.FileSystem.GetFiles(ArchiveImport,
FileIO.SearchOption.SearchAllSubDirectories)
      FilesToProcess.Add(fName)
    Next
  Catch ex As Exception
    WriteLog(ex.ToString)
  End Try
End Sub

Private Sub ConfigureFSW()
  WriteLog("ConfigureFSW: Configure File System Watcher on '" &
ArchiveImport & "'")
  My.Computer.FileSystem.CreateDirectory(ArchiveImport)
  If Not fsw Is Nothing Then
    WriteLog(vbTab & "Lost Previous File System Watcher system context,
removing from application")
    With fsw
      .EnableRaisingEvents = False
      RemoveHandler .Changed, AddressOf OnFSWChanged
      RemoveHandler .Error, AddressOf OnFSWError
    End With
    fsw = Nothing
  End If
  WriteLog(vbTab & "Create New File System Watcher on '" & ArchiveImport &
"'")
  fsw = New FileSystemWatcher(ArchiveImport, "*")
  With fsw
    .NotifyFilter = NotifyFilters.Size
    .IncludeSubdirectories = True
    AddHandler .Changed, AddressOf OnFSWChanged
    AddHandler .Error, AddressOf OnFSWError
    .EnableRaisingEvents = True
  End With
End Sub

Private Sub OnFSWChanged(ByVal sender As Object, ByVal e As
FileSystemEventArgs)
  ' Specify what is done when a file is changed, created, or deleted.
  On Error Resume Next
  Thread.CurrentThread.Name = "FSWChanged"
  FilesToProcess.Add(e.FullPath)
End Sub

Private Sub OnFSWError(ByVal sender As Object, ByVal e As ErrorEventArgs)
  On Error Resume Next
  Thread.CurrentThread.Name = "FSWError"
  WriteLog(TypeName(sender) & vbNewLine & e.ToString)
  ConfigureFSW()
  ProcessTree()
End Sub

Public Sub WriteLog(ByVal msg As String)
  On Error Resume Next
  Dim tName As String = Thread.CurrentThread.Name
  If tName <> "" Then msg = tName & ": " & msg
  Console.WriteLine(Now.ToString & vbTab & msg)
  logs.WriteLog(msg)
End Sub
End Module

Public Class ProcessFiles
  Inherits Collections.ObjectModel.KeyedCollection(Of String, FileToMove)

Private ThreadLock As New SynchronizationContext
Private NewFile As New AutoResetEvent(False)
' Set up the WorkerThread last as the sync objects need to exist
Private WorkerThread As New Thread(AddressOf Worker)

' One object can be passed to a new worker thread when it starts; the
FileControl class is that object
Private Class FileControl
  Public ImportRoot As String
  Public ArchiveRoot As String
  Public NoVersion As New Collection
End Class

Protected Overrides Function GetKeyForItem(ByVal item As FileToMove) As
String
  Return item.Key
End Function

Public Sub New(ByVal CopyFrom As String, ByVal CopyTo As String, ByVal
cNoVersions As Collection)
  Dim fc As New FileControl
  WriteLog("Creating new ProcessFiles container.")
  With fc
  .ImportRoot = CopyFrom
  .ArchiveRoot = CopyTo
  For Each str As String In cNoVersions
    .NoVersion.Add(str, str)
  Next
  End With
  WorkerThread.IsBackground = True ' Allow the program to exit at any time
  WorkerThread.Name = "Mover" ' Used to track log entries
  WorkerThread.Start(fc)
End Sub

Public Shadows Sub Add(ByVal File As String)
  Try
    Dim fm As New FileToMove(File)
    SyncLock ThreadLock
      If Not Contains(fm.Key) Then
        MyBase.Add(fm)
        WriteLog("Found '" & File & "'")
      End If
    End SyncLock
  Catch ex As Exception
    WriteLog(ex.ToString)
  End Try
  ' Always trigger this just in case there are files waiting for movement
  NewFile.Set()
End Sub

Private Sub Worker(ByVal obj As Object)
  Dim fc As FileControl = CType(obj, FileControl)
  Dim FilesRemaining As String
  Dim LocalServer As String = My.Computer.Name
  ' The target file names will always be in UNC format. Local Server will be
used to replace the actual targ
  ' for the local copy that will get backed up to tape during incremental
backups.
  Do While Left$(LocalServer, 1) = "\"
    LocalServer = Mid$(LocalServer, 2)
  Loop
  LocalServer = "\\" & TrailSlash(LocalServer) ' Format is now
"\\<servername>\"
  ' This thread never exits
  Do While True
  ' Wait for a file to be added to the list
  NewFile.WaitOne()
  Do While Count > 0
    FilesRemaining = ""
    Dim WaitTime As TimeSpan = FileToMove.OneMinute
    Dim index As Integer = Count - 1
    Try
      Do While index >= 0
        Dim f As FileToMove = Item(index)
        Dim msg As String = ""
        Try
          If f.SizeStable Then
            Dim FileBase As String = Mid$(f.FileName, InStrRev(f.FileName,
"\") + 1)
            Dim DestFile As String = Replace(f.FileName, fc.ImportRoot &
"\", fc.ArchiveRoot & "\")
            Dim UseVersioning As Boolean = True
            For Each template As String In fc.NoVersion
              If FileBase Like template Then
                UseVersioning = False
                Exit For
              End If
            Next
            If UseVersioning AndAlso File.Exists(DestFile) Then
              Dim i As Integer = InStrRev(DestFile, ".")
              Dim FileExt As String = ""
              If i > 0 Then
                FileBase = Left$(DestFile, i - 1)
                FileExt = Mid$(DestFile, i)
              Else
               FileBase = DestFile
             End If
             i = 0
             Do
               i += 1
               DestFile = FileBase & "_" & i.ToString & FileExt
             Loop While File.Exists(DestFile)
           End If
           ' Move the source file
           Try
             ' Copy to the local archives first; always override the
destination
             Dim LocalDest As String = Replace(DestFile, "\\hermes\",
LocalServer)
             Try
                My.Computer.FileSystem.CopyFile(f.FileName, LocalDest)
             Catch ex As Exception
               logs.WriteLog("Unable to copy '" & f.FileName & "' to '" &
LocalDest & "'")
             End Try
             My.Computer.FileSystem.MoveFile(f.FileName, DestFile, True)
             msg &= vbNewLine & "Moving '" & f.FileName & "' => '" &
DestFile
             SyncLock ThreadLock
                Remove(f.Key)
             End SyncLock
           Catch ex As Exception
              msg &= vbNewLine & "Exception Processing: " & f.FileName &
vbNewLine & ex.ToString
              If FilesRemaining <> "" Then FilesRemaining &= vbNewLine
              FilesRemaining &= f.FileName
           End Try
         Else
           If FilesRemaining <> "" Then FilesRemaining &= vbNewLine
           FilesRemaining &= f.FileName
         End If
      Catch ex As FileNotFoundException
         SyncLock ThreadLock
            Remove(f.FileName)
         End SyncLock
      Catch ex As Exception
         If FilesRemaining <> "" Then FilesRemaining &= vbNewLine
         FilesRemaining &= f.FileName
         msg &= vbNewLine & "Exception Processing: " & f.FileName &
vbNewLine & ex.ToString
      Finally
        If f.NextCheck < WaitTime Then WaitTime = f.NextCheck
      End Try
      If msg <> "" Then WriteLog(msg)
      ' Decrement index to get the next item. This always works as deleting
an item will leave the bottom index the same
      index -= 1
    Loop
  Catch ex As Exception
    WriteLog(ex.ToString)
  End Try
  If Count > 0 Then
    WriteLog("Files Left: " & Count.ToString & vbNewLine & FilesRemaining)
    WriteLog("Sleep Until: " & (Now + WaitTime).ToString)
    Thread.Sleep(WaitTime)
    WriteLog("Sleep Done: " & Now.ToString)
  End If
  Loop
Loop
End Sub
End Class

Public Class FileToMove
Private mFileName As String
Private utcLastChecked As Date
Private mLastSize As Long = 0
Public Shared ReadOnly OneMinute As TimeSpan = New TimeSpan(0, 1, 0)
Private Shared ReadOnly OneHour As TimeSpan = New TimeSpan(1, 0, 0)

Public Sub New(ByVal FileName As String)
  mFileName = FileName
  UpdateSize()
  utcLastChecked =
My.Computer.FileSystem.GetFileInfo(mFileName).LastWriteTime.ToUniversalTime
End Sub

Public ReadOnly Property Key() As String
  Get
     Return mFileName
  End Get
End Property

Public ReadOnly Property FileName() As String
  Get
    Return mFileName
  End Get
End Property

Private Shared ReadOnly Property utcNow() As Date
  Get
    Return Now.ToUniversalTime
  End Get
End Property

Private Sub UpdateSize()
  UpdateSize(My.Computer.FileSystem.GetFileInfo(mFileName).Length)
End Sub

Private Sub UpdateSize(ByVal NewSize As Long)
  mLastSize = NewSize
  utcLastChecked = utcNow
End Sub

Public ReadOnly Property SizeStable() As Boolean
  Get
    If mLastSize = 0 Then
      If utcNow -
My.Computer.FileSystem.GetFileInfo(mFileName).CreationTime.ToUniversalTime >
OneHour Then
        My.Computer.FileSystem.DeleteFile(mFileName)
        Throw New FileNotFoundException("File still empty after one hour")
      End If
      UpdateSize()
      Return False
    End If
    Dim CurrentSize As Long =
My.Computer.FileSystem.GetFileInfo(mFileName).Length
    If mLastSize = CurrentSize AndAlso _
      (utcNow - utcLastChecked) > OneMinute AndAlso _
      (utcNow -
My.Computer.FileSystem.GetFileInfo(mFileName).LastWriteTime.ToUniversalTime)
> OneMinute Then

      Return True
    Else
      UpdateSize(CurrentSize)
      Return False
    End If
  End Get
End Property

Public ReadOnly Property NextCheck() As TimeSpan
  Get
    Dim ts As TimeSpan = utcNow - utcLastChecked
    Debug.Print(ts.ToString)
    If ts > OneMinute Then Return New TimeSpan(0)
    Debug.Print((OneMinute - ts).ToString)
    Return OneMinute - ts
  End Get
End Property
End Class

'===================================================================
<jimmyfishb***@yahoo.co.uk> wrote in message
Show quote
news:1139920893.901524.19210@g47g2000cwa.googlegroups.com...
> Hi Kevin,
>
> I am monitoring just for the OnCreated event as you have suggested:
>
>    fsw.IncludeSubdirectories = False
>    fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite
>    fsw.NotifyFilter = System.IO.NotifyFilters.FileName
>    fsw.NotifyFilter = System.IO.NotifyFilters.LastAccess
>
>    AddHandler fsw.Changed, AddressOf OnCreated
>    AddHandler fsw.Created, AddressOf OnCreated 'not recommended see the
> link at top this while loop
>
> In my OnCreated sub, I try to move all files that have been dropped
> into the DROP folder to a different folder where they will be
> processed.  I would expect all files to be moved:
>
>                 If Not Directory.Exists(strDir & "\Temp") Then
>                     Directory.CreateDirectory(strDir & "\Temp")
>                 End If
>
>                 If File.Exists(strDir & "\Temp\" & e.Name) Then
>                     File.Delete(strDir & "\Temp\" & e.Name)
>                 End If
>
>                 'now move the file(s)
>                 For Each oFile In fso.GetFolder(strDir).Files
>                     oFile.Move(strDir & "\Temp\" & e.Name)
>                     bFileMoved = True
>                 Next
>
> However, only one file gets moved to the Temp folder.  It seems as if
> the OnCreated event gets called when the first file is identified as
> being created, and when I try to move all files in the folder, the
> filesystem thinks there is only the one file (i.e. the one that
> triggered the event).  Therefore, I have had to implement a timer that
> will read the DROP folder, and move the remaining files periodically.
> To me this is defeating the objective of the FileSystemWatcher.
>
> To make matters worse ;P, the memory/no of handles used  by my
> application is forever increasing.  I am closing objects and setting
> them to nothing, as well as calling GC.Collect() in each sub.
>
> Is this really how the FileSystemWatcher works?  Can you please suggest
> a more elegant/efficient way of achieving my aim?  Much appreciated.
>
> Jimmy
>
>
Author
14 Feb 2006 3:55 PM
jimmyfishbean
Hi Kevin,

Thanks for the guidance (and patience).  I'm on the right track now
thanks to you.  Think the explanantion with the recipes did it!!

Despite only monitoring 2 directories, my application/service seems to
use up to 25 - 30 MB of memory usage (the handles go up to 250 +).  Is
this a common burden with the FileSystemWatcher, or is it likely to be
due to inefficient code?  The VB.Net application I have created has
actually been upgraded from VB6 to .Net.  The VB6 application used
several directory monitoring API calls (i.e.
FindFirstChangeNotification, FindNextChangeNotification,
MsgWaitForMultipleObjects, etc).  When processing the same number of
files as the VB.Net application, the memory usage would reach no more
than 13 MB (with 150 + handles).  Thanks once again Kevin.

Hi Mike,

Thanks for the code.  Before I deploy code from your application, may I
ask roughly how much memory the application uses and whether it reaches
the high levels (30MB) of my (inefficient ;P) windows service
application?  Cheers,

Jimmy
Author
14 Feb 2006 3:58 PM
jimmyfishbean
May I point out that despite upgrading my VB6 project to VB.Net, I was
unable to get the directory monitoring API calls to work, hence the use
of the FileSystemWatcher component.  If someone has been able to make
use of the API calls then please let me know.  Thanks.

Jimmy
Author
14 Feb 2006 5:10 PM
Michael D. Ober
I'm not sure you really want to use the API for this.  The FSW in .NET 2.0
appears to be a "smart" wrapper around the API.  By "smart" I mean that it
checks to ensure the file that triggered the API calls meets the file
template requirements you gave the FSW object before returning.  FSW also
resets the API for the next event so you don't have to remember to do this
yourself.  (The VB 2005 code I posted was originally written in VC++ 6 and
not having to do these two items dumped a page of code.)

Mike.

<jimmyfishb***@yahoo.co.uk> wrote in message
Show quote
news:1139932733.209081.217820@o13g2000cwo.googlegroups.com...
> May I point out that despite upgrading my VB6 project to VB.Net, I was
> unable to get the directory monitoring API calls to work, hence the use
> of the FileSystemWatcher component.  If someone has been able to make
> use of the API calls then please let me know.  Thanks.
>
> Jimmy
>
>
Author
14 Feb 2006 5:51 PM
Kevin Spencer
Hi Jimmy,

The .Net Framework tends to use a lot more memory than a VB6 or C++ native
app would. This is because of the built-in memory management. As long as you
are disposing anything that needs disposing (such as the FileSystemWatcher
class, when you stop monitoring), you shouldn't worry about it. Garbage
Collection will take care of it. The .Net Framework tends to cache for a
long time, but it is very reliable. I'm assuming you're only using one
instance of the FileSystemWatcher?

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
We got a sick zebra a hat,
you ultimate tuna.


<jimmyfishb***@yahoo.co.uk> wrote in message
Show quote
news:1139932733.209081.217820@o13g2000cwo.googlegroups.com...
> May I point out that despite upgrading my VB6 project to VB.Net, I was
> unable to get the directory monitoring API calls to work, hence the use
> of the FileSystemWatcher component.  If someone has been able to make
> use of the API calls then please let me know.  Thanks.
>
> Jimmy
>
Author
14 Feb 2006 6:31 PM
Michael D. Ober
Kevin,

Jimmy will need two instances of the FileSystemWatcher since he is
monitoring two directories.  Unless the directories are under a common
parent, in which case he can monitor the parent.  In either case, the
footprint shouldn't be a big as he is seeing.

Mike Ober.

Show quote
"Kevin Spencer" <kevin@DIESPAMMERSDIEtakempis.com> wrote in message
news:e4$OV9YMGHA.2416@TK2MSFTNGP15.phx.gbl...
>
> Hi Jimmy,
>
> The .Net Framework tends to use a lot more memory than a VB6 or C++ native
> app would. This is because of the built-in memory management. As long as
you
> are disposing anything that needs disposing (such as the FileSystemWatcher
> class, when you stop monitoring), you shouldn't worry about it. Garbage
> Collection will take care of it. The .Net Framework tends to cache for a
> long time, but it is very reliable. I'm assuming you're only using one
> instance of the FileSystemWatcher?
>
> --
> HTH,
>
> Kevin Spencer
> Microsoft MVP
> .Net Developer
> We got a sick zebra a hat,
> you ultimate tuna.
>
>
> <jimmyfishb***@yahoo.co.uk> wrote in message
> news:1139932733.209081.217820@o13g2000cwo.googlegroups.com...
> > May I point out that despite upgrading my VB6 project to VB.Net, I was
> > unable to get the directory monitoring API calls to work, hence the use
> > of the FileSystemWatcher component.  If someone has been able to make
> > use of the API calls then please let me know.  Thanks.
> >
> > Jimmy
> >
>
>
>
Author
14 Feb 2006 8:30 PM
Kevin Spencer
Hi Michael,

I should have said "1 per drop directory."

--
HTH,

Kevin Spencer
Microsoft MVP
..Net Developer
We got a sick zebra a hat,
you ultimate tuna.


Show quote
"Michael D. Ober" <ober***@.alum.mit.edu.nospam> wrote in message
news:%231Ty$TZMGHA.2336@TK2MSFTNGP12.phx.gbl...
>
> Kevin,
>
> Jimmy will need two instances of the FileSystemWatcher since he is
> monitoring two directories.  Unless the directories are under a common
> parent, in which case he can monitor the parent.  In either case, the
> footprint shouldn't be a big as he is seeing.
>
> Mike Ober.
>
> "Kevin Spencer" <kevin@DIESPAMMERSDIEtakempis.com> wrote in message
> news:e4$OV9YMGHA.2416@TK2MSFTNGP15.phx.gbl...
>>
>> Hi Jimmy,
>>
>> The .Net Framework tends to use a lot more memory than a VB6 or C++
>> native
>> app would. This is because of the built-in memory management. As long as
> you
>> are disposing anything that needs disposing (such as the
>> FileSystemWatcher
>> class, when you stop monitoring), you shouldn't worry about it. Garbage
>> Collection will take care of it. The .Net Framework tends to cache for a
>> long time, but it is very reliable. I'm assuming you're only using one
>> instance of the FileSystemWatcher?
>>
>> --
>> HTH,
>>
>> Kevin Spencer
>> Microsoft MVP
>> .Net Developer
>> We got a sick zebra a hat,
>> you ultimate tuna.
>>
>>
>> <jimmyfishb***@yahoo.co.uk> wrote in message
>> news:1139932733.209081.217820@o13g2000cwo.googlegroups.com...
>> > May I point out that despite upgrading my VB6 project to VB.Net, I was
>> > unable to get the directory monitoring API calls to work, hence the use
>> > of the FileSystemWatcher component.  If someone has been able to make
>> > use of the API calls then please let me know.  Thanks.
>> >
>> > Jimmy
>> >
>>
>>
>>
>
>
>
Author
16 Feb 2006 10:56 AM
jimmyfishbean
Kevin, Mike,

My app is now working ;P, and as efficient as it possibly can.  Thanks
guys.

Jimmy
Author
14 Feb 2006 5:05 PM
Michael D. Ober
On a server that has been up since Feb 3rd, the code I posted has run for 10
minutes total and is currently using about 11Mb of VM allocation and a
physical memory footprint of 1 Mb.  Numbers are from the server's Task
Manager and event viewer.

Mike.

<jimmyfishb***@yahoo.co.uk> wrote in message
Show quote
news:1139932541.134321.136200@f14g2000cwb.googlegroups.com...
> Hi Kevin,
>
> Thanks for the guidance (and patience).  I'm on the right track now
> thanks to you.  Think the explanantion with the recipes did it!!
>
> Despite only monitoring 2 directories, my application/service seems to
> use up to 25 - 30 MB of memory usage (the handles go up to 250 +).  Is
> this a common burden with the FileSystemWatcher, or is it likely to be
> due to inefficient code?  The VB.Net application I have created has
> actually been upgraded from VB6 to .Net.  The VB6 application used
> several directory monitoring API calls (i.e.
> FindFirstChangeNotification, FindNextChangeNotification,
> MsgWaitForMultipleObjects, etc).  When processing the same number of
> files as the VB.Net application, the memory usage would reach no more
> than 13 MB (with 150 + handles).  Thanks once again Kevin.
>
> Hi Mike,
>
> Thanks for the code.  Before I deploy code from your application, may I
> ask roughly how much memory the application uses and whether it reaches
> the high levels (30MB) of my (inefficient ;P) windows service
> application?  Cheers,
>
> Jimmy
>
Author
14 Feb 2006 6:32 PM
Michael D. Ober
Let me clarify - the code I posted started when the server came up the
evening of the 3rd and has used 10 minutes of CPU time since then.

Mike Ober.

Show quote
"Michael D. Ober" <ober***@.alum.mit.edu.nospam> wrote in message
news:%23NymDkYMGHA.2528@TK2MSFTNGP12.phx.gbl...
>
>
> On a server that has been up since Feb 3rd, the code I posted has run for
10
> minutes total and is currently using about 11Mb of VM allocation and a
> physical memory footprint of 1 Mb.  Numbers are from the server's Task
> Manager and event viewer.
>
> Mike.
>
> <jimmyfishb***@yahoo.co.uk> wrote in message
> news:1139932541.134321.136200@f14g2000cwb.googlegroups.com...
> > Hi Kevin,
> >
> > Thanks for the guidance (and patience).  I'm on the right track now
> > thanks to you.  Think the explanantion with the recipes did it!!
> >
> > Despite only monitoring 2 directories, my application/service seems to
> > use up to 25 - 30 MB of memory usage (the handles go up to 250 +).  Is
> > this a common burden with the FileSystemWatcher, or is it likely to be
> > due to inefficient code?  The VB.Net application I have created has
> > actually been upgraded from VB6 to .Net.  The VB6 application used
> > several directory monitoring API calls (i.e.
> > FindFirstChangeNotification, FindNextChangeNotification,
> > MsgWaitForMultipleObjects, etc).  When processing the same number of
> > files as the VB.Net application, the memory usage would reach no more
> > than 13 MB (with 150 + handles).  Thanks once again Kevin.
> >
> > Hi Mike,
> >
> > Thanks for the code.  Before I deploy code from your application, may I
> > ask roughly how much memory the application uses and whether it reaches
> > the high levels (30MB) of my (inefficient ;P) windows service
> > application?  Cheers,
> >
> > Jimmy
> >
>
>
>
>

AddThis Social Bookmark Button