|
dev
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
FileSystemWatcher advice required pleaseI have a VB.Net Windows Service that monitors a varying number of directories (that are read from an INI file). I set up a FileSystemWatcher object to monitor each directory that is read from the INI file and add each object to a collection. The purpose of my service is to monitor the directories for any XML files that are dropped into them. When a file is dropped, it is firstly renamed by appending the extension '.lock' to the name of the XML file. Secondly, the file is then processed by sending the contents to a Web Service. Once the file has been completely processed and a return value is received from the Web Service, the file is moved from the directory to a 'Processed Files' folder (that is not monitored by a FileSystemWatcher object). I call a sub to process each directory, which loops through all files in the directory (at one moment in time) and calls the web service for each file. When the service starts, any XML files in the 'monitored' are picked up and processed without any erros/trouble. The problem starts if any 'new' XML files are dropped into the monitored directories when the service is currently busy processing. I call the same sub that loops through all files in a directory. Of course, there will be files present in both calls to the sub, causing errors. How can I tackle this? Is it reasonable to use a timer (I do not currently use one). I did add a timer, but the number of handles and memory kept increasing all the time (even when there was nothing to process). I am not concerned about the deletion of files or modification or access. Simply addition/creation. Any advice/guidance is greatly appreciated. Thanks. Jimmy Code snippet: Private Sub setup() ... While l <= UBound(aInputDirs) Dim fsw As New FileSystemWatcher fsw.Path = aInputDirs(l) fsw.Filter = "*.xml" fsw.IncludeSubdirectories = False fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite AddHandler fsw.Changed, AddressOf OnChanged fsw.EnableRaisingEvents = True watcherCollection.Add(fsw) l = l + 1 End While l = 0 While l < UBound(aInputDirs) ProcessDir(l) l = l + 1 End While ... End Sub Private Shared Sub OnChanged(ByVal source As Object, ByVal e As System.IO.FileSystemEventArgs) 'Dim strDir As String = e.Name '1. work out the directory that has been changed (using the full path/filename : e.Name) Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\") Dim strDir As String = Left(e.FullPath, intLastSlash) If Not bBusyProcessing Then If Not e.Name = "XMLtoSOAP.ini" Then If (File.GetAttributes(e.FullPath) And FileAttributes.Directory) = FileAttributes.Directory Then 'Directory changed, created, or deleted. Else 'File changed, created, or deleted. Dim i As Integer = 0 Do While i < UBound(aInputDirs) If aInputDirs(i) = strDir Then ProcessDir(i) Exit Do End If i = i + 1 Loop End If Else 'setup() End If End If End Sub Public Sub ProcessDir(ByRef lElement As Integer) ' Find all files in this directory and try to process them. Dim fso As New Scripting.FileSystemObject Dim oFile As Scripting.File Dim strOrigFile As String Dim stream As Scripting.TextStream Dim bFileAlreadyOpen As Boolean Dim cProcessedFiles As New Collection Dim bMoreToProcess, bProcessedOne As Boolean bMoreToProcess = True While bMoreToProcess = True bProcessedOne = False For Each oFile In fso.GetFolder(aInputDirs(lElement)).Files bBusyProcessing = True 'Make sure this file hasn't already been tried - if the item is not in the collection an error is returned On Error Resume Next If cProcessedFiles.Count() <> 0 And cProcessedFiles.Item(oFile.Name) <> oFile.Name Then On Error GoTo 0 bFileAlreadyOpen = False strOrigFile = oFile.Name 'Rename the file to see if it is still locked by any process On Error GoTo FileStillOpen 'create a copy of the original file to revert back to if in changerootnode mode If bChangeRootNode Then If Not fso.FolderExists(aInputDirs(lElement) & "\Temp") Then fso.CreateFolder(aInputDirs(lElement) & "\Temp") End If fso.CopyFile(oFile.Path, aInputDirs(lElement) & "\Temp\" & oFile.Name) sCurrFile = aInputDirs(lElement) & "\Temp\" & oFile.Name End If fso.MoveFile(oFile.Path, oFile.Path & ".lock") On Error GoTo 0 If Not bFileAlreadyOpen Then ProcessFile(lElement, strOrigFile) End If 'Store the file in a collection for exclusion in the lower loop cProcessedFiles.Add(strOrigFile, strOrigFile) bProcessedOne = True End If bBusyProcessing = False Next oFile bMoreToProcess = bProcessedOne End While fso = Nothing Exit Sub FileStillOpen: bFileAlreadyOpen = True Resume Next End Sub 'NOTE: Sub ProcessFile calls the web service Okay, you have a FileSystemWatcher which, I assume, is monitoring the
"Created" event. When a file is created, the FileSystemEventArgs passed with the event contains the path of the file created. You can use this information to move the file to the new location. No need to create a list of files to move. Process each one at a time. -- Show quoteHTH, Kevin Spencer Microsoft MVP ..Net Developer We got a sick zebra a hat, you ultimate tuna. <jimmyfishb***@yahoo.co.uk> wrote in message news:1139501783.714521.6360@o13g2000cwo.googlegroups.com... > Hi, > > I have a VB.Net Windows Service that monitors a varying number of > directories (that are read from an INI file). I set up a > FileSystemWatcher object to monitor each directory that is read from > the INI file and add each object to a collection. > > The purpose of my service is to monitor the directories for any XML > files that are dropped into them. When a file is dropped, it is > firstly renamed by appending the extension '.lock' to the name of the > XML file. Secondly, the file is then processed by sending the contents > to a Web Service. Once the file has been completely processed and a > return value is received from the Web Service, the file is moved from > the directory to a 'Processed Files' folder (that is not monitored by a > FileSystemWatcher object). > > I call a sub to process each directory, which loops through all files > in the directory (at one moment in time) and calls the web service for > each file. > > When the service starts, any XML files in the 'monitored' are picked up > and processed without any erros/trouble. The problem starts if any > 'new' XML files are dropped into the monitored directories when the > service is currently busy processing. I call the same sub that loops > through all files in a directory. Of course, there will be files > present in both calls to the sub, causing errors. > > How can I tackle this? Is it reasonable to use a timer (I do not > currently use one). I did add a timer, but the number of handles and > memory kept increasing all the time (even when there was nothing to > process). > > I am not concerned about the deletion of files or modification or > access. Simply addition/creation. > > Any advice/guidance is greatly appreciated. Thanks. > > Jimmy > > Code snippet: > > Private Sub setup() > ... > While l <= UBound(aInputDirs) > Dim fsw As New FileSystemWatcher > fsw.Path = aInputDirs(l) > fsw.Filter = "*.xml" > fsw.IncludeSubdirectories = False > fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite > AddHandler fsw.Changed, AddressOf OnChanged > fsw.EnableRaisingEvents = True > > watcherCollection.Add(fsw) > l = l + 1 > End While > > l = 0 > While l < UBound(aInputDirs) > ProcessDir(l) > l = l + 1 > End While > ... > End Sub > > > Private Shared Sub OnChanged(ByVal source As Object, ByVal e As > System.IO.FileSystemEventArgs) > > 'Dim strDir As String = e.Name > > '1. work out the directory that has been changed (using the > full path/filename : e.Name) > Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\") > Dim strDir As String = Left(e.FullPath, intLastSlash) > > > If Not bBusyProcessing Then > If Not e.Name = "XMLtoSOAP.ini" Then > If (File.GetAttributes(e.FullPath) And > FileAttributes.Directory) = FileAttributes.Directory Then > 'Directory changed, created, or deleted. > Else > 'File changed, created, or deleted. > Dim i As Integer = 0 > > Do While i < UBound(aInputDirs) > If aInputDirs(i) = strDir Then > ProcessDir(i) > Exit Do > End If > > i = i + 1 > Loop > End If > Else > 'setup() > End If > End If > > End Sub > > Public Sub ProcessDir(ByRef lElement As Integer) > ' Find all files in this directory and try to process them. > Dim fso As New Scripting.FileSystemObject > Dim oFile As Scripting.File > Dim strOrigFile As String > Dim stream As Scripting.TextStream > Dim bFileAlreadyOpen As Boolean > Dim cProcessedFiles As New Collection > Dim bMoreToProcess, bProcessedOne As Boolean > > bMoreToProcess = True > > While bMoreToProcess = True > bProcessedOne = False > For Each oFile In fso.GetFolder(aInputDirs(lElement)).Files > > bBusyProcessing = True > > 'Make sure this file hasn't already been tried - if the > item is not in the collection an error is returned > On Error Resume Next > If cProcessedFiles.Count() <> 0 And > cProcessedFiles.Item(oFile.Name) <> oFile.Name Then > On Error GoTo 0 > bFileAlreadyOpen = False > strOrigFile = oFile.Name > 'Rename the file to see if it is still locked by > any process > On Error GoTo FileStillOpen > > 'create a copy of the original file to revert back > to if in changerootnode mode > If bChangeRootNode Then > If Not fso.FolderExists(aInputDirs(lElement) & > "\Temp") Then > fso.CreateFolder(aInputDirs(lElement) & > "\Temp") > End If > > fso.CopyFile(oFile.Path, aInputDirs(lElement) & > "\Temp\" & oFile.Name) > sCurrFile = aInputDirs(lElement) & "\Temp\" & > oFile.Name > End If > > fso.MoveFile(oFile.Path, oFile.Path & ".lock") > On Error GoTo 0 > If Not bFileAlreadyOpen Then > ProcessFile(lElement, strOrigFile) > End If > 'Store the file in a collection for exclusion in > the lower loop > cProcessedFiles.Add(strOrigFile, strOrigFile) > bProcessedOne = True > End If > > bBusyProcessing = False > Next oFile > bMoreToProcess = bProcessedOne > End While > > fso = Nothing > > Exit Sub > > FileStillOpen: > bFileAlreadyOpen = True > Resume Next > > End Sub > > 'NOTE: Sub ProcessFile calls the web service > Thanks for that Kevin. I have tried your suggestion.
However, if I drop more than 1 file into the DROP folders, only the first file is passed to the OnChanged sub. Therefore I can only move 1 file (unless I call a sub that moves all files from the DROP folder to a temp folder). Am I correct to assume that only 1 event is triggered regardless of the number of files dropped into the directory? It would be a lot easier/better if a separate event was triggered for each file. If I try to move all files in the DROP folder in the OnChanged sub, only 1 gets moved (as the application thinks that only one file is there at the time the event was triggered). Do I need to use a timer here to periodically check the fodler? I do not want to, and thought the whole purpose of the filesystemwatcher was to provide functionality that would allow each and every file droppped to trigger an event. I take it I have it all so wrong... My new OnChanged sub: Private Shared Sub OnChanged(ByVal source As Object, ByVal e As System.IO.FileSystemEventArgs) Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\") Dim strDir As String = Left(e.FullPath, intLastSlash) Dim fso As New Scripting.FileSystemObject Dim oFile As Scripting.File Dim file As File If Not e.Name = "XMLtoSOAP.ini" Then If (File.GetAttributes(e.FullPath) And FileAttributes.Directory) = FileAttributes.Directory Then 'Directory changed, created, or deleted. Else 'File changed, created, or deleted. 'move the files to a temp location If Not Directory.Exists(strDir & "\Temp") Then Directory.CreateDirectory(strDir & "\Temp") If File.Exists(strDir & "\Temp\" & e.Name) Then file.Delete(strDir & "\Temp\" & e.Name) End If 'now move the file(s) For Each oFile In fso.GetFolder(strDir).Files oFile.Move(strDir & "\Temp\" & e.Name) Next End If End If End If End Sub The problem is that I never know how many files will be dropped into the DROP folder at one time. There may be just a single one, there may be 20 and ther may be 20, followed by another 30 straight afterwards. Thanks. Jimmy The sub should read:
Private Shared Sub OnChanged(ByVal source As Object, ByVal e As System.IO.FileSystemEventArgs) Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\") Dim strDir As String = Left(e.FullPath, intLastSlash) Dim fso As New Scripting.FileSystemObject Dim oFile As Scripting.File Dim file As File On Error Resume Next If Not e.Name = "XMLtoSOAP.ini" Then If (file.GetAttributes(e.FullPath) And FileAttributes.Directory) = FileAttributes.Directory Then 'Directory changed, created, or deleted. Else 'File changed, created, or deleted. 'move the files to a temp location If Not Directory.Exists(strDir & "\Temp") Then Directory.CreateDirectory(strDir & "\Temp") End If If file.Exists(strDir & "\Temp\" & e.Name) Then file.Delete(strDir & "\Temp\" & e.Name) End If 'now move the file(s) For Each oFile In fso.GetFolder(strDir).Files oFile.Move(strDir & "\Temp\" & e.Name) Next End If End If End Sub Cheers. Hi Jimmy,
Why are you using OnChange? If you remember my first reply, I told you that I assumed you were using the OnCreated event. This is because you said that the files were being "dropped" into the directory. This causes a Create event to occur, and that should be the only event that you need to listen for. -- Show quoteHTH, Kevin Spencer Microsoft MVP ..Net Developer We got a sick zebra a hat, you ultimate tuna. <jimmyfishb***@yahoo.co.uk> wrote in message news:1139579975.215832.79630@g43g2000cwa.googlegroups.com... > Thanks for that Kevin. I have tried your suggestion. > > However, if I drop more than 1 file into the DROP folders, only the > first file is passed to the OnChanged sub. Therefore I can only move 1 > file (unless I call a sub that moves all files from the DROP folder to > a temp folder). Am I correct to assume that only 1 event is triggered > regardless of the number of files dropped into the directory? It would > be a lot easier/better if a separate event was triggered for each file. > > If I try to move all files in the DROP folder in the OnChanged sub, > only 1 gets moved (as the application thinks that only one file is > there at the time the event was triggered). Do I need to use a timer > here to periodically check the fodler? I do not want to, and thought > the whole purpose of the filesystemwatcher was to provide functionality > that would allow each and every file droppped to trigger an event. I > take it I have it all so wrong... > > My new OnChanged sub: > > Private Shared Sub OnChanged(ByVal source As Object, ByVal e As > System.IO.FileSystemEventArgs) > Dim intLastSlash As Integer = e.FullPath.LastIndexOf("\") > Dim strDir As String = Left(e.FullPath, intLastSlash) > Dim fso As New Scripting.FileSystemObject > Dim oFile As Scripting.File > Dim file As File > > If Not e.Name = "XMLtoSOAP.ini" Then > If (File.GetAttributes(e.FullPath) And > FileAttributes.Directory) = FileAttributes.Directory Then > 'Directory changed, created, or deleted. > Else > 'File changed, created, or deleted. > 'move the files to a temp location > If Not Directory.Exists(strDir & "\Temp") Then > Directory.CreateDirectory(strDir & "\Temp") > > If File.Exists(strDir & "\Temp\" & e.Name) Then > file.Delete(strDir & "\Temp\" & e.Name) > End If > > 'now move the file(s) > For Each oFile In fso.GetFolder(strDir).Files > oFile.Move(strDir & "\Temp\" & e.Name) > Next > End If > End If > End If > End Sub > > The problem is that I never know how many files will be dropped into > the DROP folder at one time. There may be just a single one, there may > be 20 and ther may be 20, followed by another 30 straight afterwards. > > Thanks. > Jimmy > Hi Kevin,
I am monitoring just for the OnCreated event as you have suggested: fsw.IncludeSubdirectories = False fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite fsw.NotifyFilter = System.IO.NotifyFilters.FileName fsw.NotifyFilter = System.IO.NotifyFilters.LastAccess AddHandler fsw.Changed, AddressOf OnCreated AddHandler fsw.Created, AddressOf OnCreated 'not recommended see the link at top this while loop In my OnCreated sub, I try to move all files that have been dropped into the DROP folder to a different folder where they will be processed. I would expect all files to be moved: If Not Directory.Exists(strDir & "\Temp") Then Directory.CreateDirectory(strDir & "\Temp") End If If File.Exists(strDir & "\Temp\" & e.Name) Then File.Delete(strDir & "\Temp\" & e.Name) End If 'now move the file(s) For Each oFile In fso.GetFolder(strDir).Files oFile.Move(strDir & "\Temp\" & e.Name) bFileMoved = True Next However, only one file gets moved to the Temp folder. It seems as if the OnCreated event gets called when the first file is identified as being created, and when I try to move all files in the folder, the filesystem thinks there is only the one file (i.e. the one that triggered the event). Therefore, I have had to implement a timer that will read the DROP folder, and move the remaining files periodically. To me this is defeating the objective of the FileSystemWatcher. To make matters worse ;P, the memory/no of handles used by my application is forever increasing. I am closing objects and setting them to nothing, as well as calling GC.Collect() in each sub. Is this really how the FileSystemWatcher works? Can you please suggest a more elegant/efficient way of achieving my aim? Much appreciated. Jimmy Hi Jimmy,
The Created event occurs once per each file created. In addition, I see that you're attaching the same event handler to the Changed event. This was not what I suggested. To quote my earlier replies: First reply: "When a file is created, the FileSystemEventArgs passed with the event contains the path of the file created. You can use this information to move the file to the new location.No need to create a list of files to move. **Process each one at a time.**" Second reply: "Why are you using OnChange? If you remember my first reply, I told you that I assumed you were using the OnCreated event. This is because you said that the files were being "dropped" into the directory. This causes a Create event to occur, and that should be the only event that you need to listen for." You know, my mother used to tell me about people who loved her recipes for one thing or another. But they would come back and tell her that when they used her recipe, it just didn't taste as good as my mother's. Invariably, she found out that they had "improvised" a bit on her recipe, and that this was the reason it didn't taste the same! I'm thinking that perhaps you are misunderstanding something about these files being dropped into the folder, because of a question you asked earlier in this thread: "Am I correct to assume that only 1 event is triggered regardless of the number of files dropped into the directory? It would be a lot easier/better if a separate event was triggered for each file." At the time you were asking about the Change event, but I believe you misunderstand the idea of "dropping multiple files" in general. When you drop multiple files into a folder, they are created one at a time. It happens so fast that it may look like it's doing more than one at a time, but it's not. Also, multiple events are fired during different types of IO operations. Take a note of this Note from the SDK: "Common file system operations might raise more than one event. For example, when a file is moved from one directory to another, several OnChanged and some OnCreated and OnDeleted events might be raised. Moving a file is a complex operation that consists of multiple simple operations, therefore raising multiple events. Likewise, some applications (for example, anti-virus software) might cause additional file system events that are detected by FileSystemWatcher." Your objective is to handle as few as necessary to get the job done. Otherwise, you may have one event handler trying to perform the same IO operation as another "at the sem time," and run into beaucoups trouble. So, here's what I want you to do. Handle *only* the Created event. Don't worry about NotifyFilter. For the OnChange event handler, the default (LastWrite | FileName | DirectoryName) is perfect. In your OnChange override, simply move the file specified in the FileSystemEventArgs.FullPath member passed. The OnChange event handler method will be called for *each* file dropped into the folder. Remember, *one* at a time! -- Show quoteHTH, Kevin Spencer Microsoft MVP ..Net Developer We got a sick zebra a hat, you ultimate tuna. <jimmyfishb***@yahoo.co.uk> wrote in message news:1139920893.901524.19210@g47g2000cwa.googlegroups.com... > Hi Kevin, > > I am monitoring just for the OnCreated event as you have suggested: > > fsw.IncludeSubdirectories = False > fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite > fsw.NotifyFilter = System.IO.NotifyFilters.FileName > fsw.NotifyFilter = System.IO.NotifyFilters.LastAccess > > AddHandler fsw.Changed, AddressOf OnCreated > AddHandler fsw.Created, AddressOf OnCreated 'not recommended see the > link at top this while loop > > In my OnCreated sub, I try to move all files that have been dropped > into the DROP folder to a different folder where they will be > processed. I would expect all files to be moved: > > If Not Directory.Exists(strDir & "\Temp") Then > Directory.CreateDirectory(strDir & "\Temp") > End If > > If File.Exists(strDir & "\Temp\" & e.Name) Then > File.Delete(strDir & "\Temp\" & e.Name) > End If > > 'now move the file(s) > For Each oFile In fso.GetFolder(strDir).Files > oFile.Move(strDir & "\Temp\" & e.Name) > bFileMoved = True > Next > > However, only one file gets moved to the Temp folder. It seems as if > the OnCreated event gets called when the first file is identified as > being created, and when I try to move all files in the folder, the > filesystem thinks there is only the one file (i.e. the one that > triggered the event). Therefore, I have had to implement a timer that > will read the DROP folder, and move the remaining files periodically. > To me this is defeating the objective of the FileSystemWatcher. > > To make matters worse ;P, the memory/no of handles used by my > application is forever increasing. I am closing objects and setting > them to nothing, as well as calling GC.Collect() in each sub. > > Is this really how the FileSystemWatcher works? Can you please suggest > a more elegant/efficient way of achieving my aim? Much appreciated. > > Jimmy > Here's a complete VB 2005 application that monitors a directory tree and
processes files that are ready to process. I defined ready to process as a non-zero byte size file that hasn't had a change in file size for at least one minute. I think I have indented the correctly - to verify the indents, copy into a VB 2005 project and use the IDE's indenter. Mike. '============================================================ Option Compare Text Option Explicit On Option Strict On Imports System.IO Imports System.Threading Module ArchiveImportLoader Private ArchiveImport As String Private FilesToProcess As ProcessFiles Private fsw As FileSystemWatcher Private Declare Function GetConsoleWindow Lib "kernel32.dll" () As IntPtr Private Declare Function ShowWindow Lib "user32.dll" (ByVal hwnd As IntPtr, ByVal nCmdShow As Int32) As Int32 Private Const SW_SHOWMINNOACTIVE As Int32 = 7 Public Sub Main() Dim NoVersion As New Collection logs.WriteLog(My.Application.Info.Title & " Starting") Console.Title = AppName() Thread.CurrentThread.Name = "MAIN" ' Read environment ArchiveImport = OSInterface.iniWrapper.ReadString("ArchiveManager", "ImportRoot", "wakefield.ini") Dim ArchiveRoot As String = OSInterface.iniWrapper.ReadString("ArchiveManager", "Root", "wakefield.ini") For Each v As String In Split(OSInterface.iniWrapper.ReadString("ArchiveManager", "NoVersion", "wakefield.ini"), "|") NoVersion.Add("*" & v) Next ' Ensure the import directory exists WriteLog("Creating File System Watcher on '" & ArchiveImport & "'") My.Computer.FileSystem.CreateDirectory(ArchiveImport) ' Start the worker thread and configure the FSW subsystem FilesToProcess = New ProcessFiles(ArchiveImport, ArchiveRoot, NoVersion) ConfigureFSW() ' Configuration done; Minimize the window WriteLog("Initialization Complete") Thread.Sleep(New TimeSpan(0, 0, 5)) ShowWindow(GetConsoleWindow(), SW_SHOWMINNOACTIVE) ' Go to sleep, but periodically wakeup and check for missed files Dim SleepPeriod As TimeSpan = New TimeSpan(0, 15, 0) Do While True Console.WriteLine(Now.ToString & vbTab & Thread.CurrentThread.Name & ": Checking for Missed Files") ProcessTree() Thread.Sleep(SleepPeriod) Loop End Sub Private Sub ProcessTree() Try My.Computer.FileSystem.CreateDirectory(ArchiveImport) For Each fName As String In My.Computer.FileSystem.GetFiles(ArchiveImport, FileIO.SearchOption.SearchAllSubDirectories) FilesToProcess.Add(fName) Next Catch ex As Exception WriteLog(ex.ToString) End Try End Sub Private Sub ConfigureFSW() WriteLog("ConfigureFSW: Configure File System Watcher on '" & ArchiveImport & "'") My.Computer.FileSystem.CreateDirectory(ArchiveImport) If Not fsw Is Nothing Then WriteLog(vbTab & "Lost Previous File System Watcher system context, removing from application") With fsw .EnableRaisingEvents = False RemoveHandler .Changed, AddressOf OnFSWChanged RemoveHandler .Error, AddressOf OnFSWError End With fsw = Nothing End If WriteLog(vbTab & "Create New File System Watcher on '" & ArchiveImport & "'") fsw = New FileSystemWatcher(ArchiveImport, "*") With fsw .NotifyFilter = NotifyFilters.Size .IncludeSubdirectories = True AddHandler .Changed, AddressOf OnFSWChanged AddHandler .Error, AddressOf OnFSWError .EnableRaisingEvents = True End With End Sub Private Sub OnFSWChanged(ByVal sender As Object, ByVal e As FileSystemEventArgs) ' Specify what is done when a file is changed, created, or deleted. On Error Resume Next Thread.CurrentThread.Name = "FSWChanged" FilesToProcess.Add(e.FullPath) End Sub Private Sub OnFSWError(ByVal sender As Object, ByVal e As ErrorEventArgs) On Error Resume Next Thread.CurrentThread.Name = "FSWError" WriteLog(TypeName(sender) & vbNewLine & e.ToString) ConfigureFSW() ProcessTree() End Sub Public Sub WriteLog(ByVal msg As String) On Error Resume Next Dim tName As String = Thread.CurrentThread.Name If tName <> "" Then msg = tName & ": " & msg Console.WriteLine(Now.ToString & vbTab & msg) logs.WriteLog(msg) End Sub End Module Public Class ProcessFiles Inherits Collections.ObjectModel.KeyedCollection(Of String, FileToMove) Private ThreadLock As New SynchronizationContext Private NewFile As New AutoResetEvent(False) ' Set up the WorkerThread last as the sync objects need to exist Private WorkerThread As New Thread(AddressOf Worker) ' One object can be passed to a new worker thread when it starts; the FileControl class is that object Private Class FileControl Public ImportRoot As String Public ArchiveRoot As String Public NoVersion As New Collection End Class Protected Overrides Function GetKeyForItem(ByVal item As FileToMove) As String Return item.Key End Function Public Sub New(ByVal CopyFrom As String, ByVal CopyTo As String, ByVal cNoVersions As Collection) Dim fc As New FileControl WriteLog("Creating new ProcessFiles container.") With fc .ImportRoot = CopyFrom .ArchiveRoot = CopyTo For Each str As String In cNoVersions .NoVersion.Add(str, str) Next End With WorkerThread.IsBackground = True ' Allow the program to exit at any time WorkerThread.Name = "Mover" ' Used to track log entries WorkerThread.Start(fc) End Sub Public Shadows Sub Add(ByVal File As String) Try Dim fm As New FileToMove(File) SyncLock ThreadLock If Not Contains(fm.Key) Then MyBase.Add(fm) WriteLog("Found '" & File & "'") End If End SyncLock Catch ex As Exception WriteLog(ex.ToString) End Try ' Always trigger this just in case there are files waiting for movement NewFile.Set() End Sub Private Sub Worker(ByVal obj As Object) Dim fc As FileControl = CType(obj, FileControl) Dim FilesRemaining As String Dim LocalServer As String = My.Computer.Name ' The target file names will always be in UNC format. Local Server will be used to replace the actual targ ' for the local copy that will get backed up to tape during incremental backups. Do While Left$(LocalServer, 1) = "\" LocalServer = Mid$(LocalServer, 2) Loop LocalServer = "\\" & TrailSlash(LocalServer) ' Format is now "\\<servername>\" ' This thread never exits Do While True ' Wait for a file to be added to the list NewFile.WaitOne() Do While Count > 0 FilesRemaining = "" Dim WaitTime As TimeSpan = FileToMove.OneMinute Dim index As Integer = Count - 1 Try Do While index >= 0 Dim f As FileToMove = Item(index) Dim msg As String = "" Try If f.SizeStable Then Dim FileBase As String = Mid$(f.FileName, InStrRev(f.FileName, "\") + 1) Dim DestFile As String = Replace(f.FileName, fc.ImportRoot & "\", fc.ArchiveRoot & "\") Dim UseVersioning As Boolean = True For Each template As String In fc.NoVersion If FileBase Like template Then UseVersioning = False Exit For End If Next If UseVersioning AndAlso File.Exists(DestFile) Then Dim i As Integer = InStrRev(DestFile, ".") Dim FileExt As String = "" If i > 0 Then FileBase = Left$(DestFile, i - 1) FileExt = Mid$(DestFile, i) Else FileBase = DestFile End If i = 0 Do i += 1 DestFile = FileBase & "_" & i.ToString & FileExt Loop While File.Exists(DestFile) End If ' Move the source file Try ' Copy to the local archives first; always override the destination Dim LocalDest As String = Replace(DestFile, "\\hermes\", LocalServer) Try My.Computer.FileSystem.CopyFile(f.FileName, LocalDest) Catch ex As Exception logs.WriteLog("Unable to copy '" & f.FileName & "' to '" & LocalDest & "'") End Try My.Computer.FileSystem.MoveFile(f.FileName, DestFile, True) msg &= vbNewLine & "Moving '" & f.FileName & "' => '" & DestFile SyncLock ThreadLock Remove(f.Key) End SyncLock Catch ex As Exception msg &= vbNewLine & "Exception Processing: " & f.FileName & vbNewLine & ex.ToString If FilesRemaining <> "" Then FilesRemaining &= vbNewLine FilesRemaining &= f.FileName End Try Else If FilesRemaining <> "" Then FilesRemaining &= vbNewLine FilesRemaining &= f.FileName End If Catch ex As FileNotFoundException SyncLock ThreadLock Remove(f.FileName) End SyncLock Catch ex As Exception If FilesRemaining <> "" Then FilesRemaining &= vbNewLine FilesRemaining &= f.FileName msg &= vbNewLine & "Exception Processing: " & f.FileName & vbNewLine & ex.ToString Finally If f.NextCheck < WaitTime Then WaitTime = f.NextCheck End Try If msg <> "" Then WriteLog(msg) ' Decrement index to get the next item. This always works as deleting an item will leave the bottom index the same index -= 1 Loop Catch ex As Exception WriteLog(ex.ToString) End Try If Count > 0 Then WriteLog("Files Left: " & Count.ToString & vbNewLine & FilesRemaining) WriteLog("Sleep Until: " & (Now + WaitTime).ToString) Thread.Sleep(WaitTime) WriteLog("Sleep Done: " & Now.ToString) End If Loop Loop End Sub End Class Public Class FileToMove Private mFileName As String Private utcLastChecked As Date Private mLastSize As Long = 0 Public Shared ReadOnly OneMinute As TimeSpan = New TimeSpan(0, 1, 0) Private Shared ReadOnly OneHour As TimeSpan = New TimeSpan(1, 0, 0) Public Sub New(ByVal FileName As String) mFileName = FileName UpdateSize() utcLastChecked = My.Computer.FileSystem.GetFileInfo(mFileName).LastWriteTime.ToUniversalTime End Sub Public ReadOnly Property Key() As String Get Return mFileName End Get End Property Public ReadOnly Property FileName() As String Get Return mFileName End Get End Property Private Shared ReadOnly Property utcNow() As Date Get Return Now.ToUniversalTime End Get End Property Private Sub UpdateSize() UpdateSize(My.Computer.FileSystem.GetFileInfo(mFileName).Length) End Sub Private Sub UpdateSize(ByVal NewSize As Long) mLastSize = NewSize utcLastChecked = utcNow End Sub Public ReadOnly Property SizeStable() As Boolean Get If mLastSize = 0 Then If utcNow - My.Computer.FileSystem.GetFileInfo(mFileName).CreationTime.ToUniversalTime > OneHour Then My.Computer.FileSystem.DeleteFile(mFileName) Throw New FileNotFoundException("File still empty after one hour") End If UpdateSize() Return False End If Dim CurrentSize As Long = My.Computer.FileSystem.GetFileInfo(mFileName).Length If mLastSize = CurrentSize AndAlso _ (utcNow - utcLastChecked) > OneMinute AndAlso _ (utcNow - My.Computer.FileSystem.GetFileInfo(mFileName).LastWriteTime.ToUniversalTime) > OneMinute Then Return TrueElse UpdateSize(CurrentSize) Return False End If End Get End Property Public ReadOnly Property NextCheck() As TimeSpan Get Dim ts As TimeSpan = utcNow - utcLastChecked Debug.Print(ts.ToString) If ts > OneMinute Then Return New TimeSpan(0) Debug.Print((OneMinute - ts).ToString) Return OneMinute - ts End Get End Property End Class '=================================================================== <jimmyfishb***@yahoo.co.uk> wrote in message Show quote news:1139920893.901524.19210@g47g2000cwa.googlegroups.com... > Hi Kevin, > > I am monitoring just for the OnCreated event as you have suggested: > > fsw.IncludeSubdirectories = False > fsw.NotifyFilter = System.IO.NotifyFilters.LastWrite > fsw.NotifyFilter = System.IO.NotifyFilters.FileName > fsw.NotifyFilter = System.IO.NotifyFilters.LastAccess > > AddHandler fsw.Changed, AddressOf OnCreated > AddHandler fsw.Created, AddressOf OnCreated 'not recommended see the > link at top this while loop > > In my OnCreated sub, I try to move all files that have been dropped > into the DROP folder to a different folder where they will be > processed. I would expect all files to be moved: > > If Not Directory.Exists(strDir & "\Temp") Then > Directory.CreateDirectory(strDir & "\Temp") > End If > > If File.Exists(strDir & "\Temp\" & e.Name) Then > File.Delete(strDir & "\Temp\" & e.Name) > End If > > 'now move the file(s) > For Each oFile In fso.GetFolder(strDir).Files > oFile.Move(strDir & "\Temp\" & e.Name) > bFileMoved = True > Next > > However, only one file gets moved to the Temp folder. It seems as if > the OnCreated event gets called when the first file is identified as > being created, and when I try to move all files in the folder, the > filesystem thinks there is only the one file (i.e. the one that > triggered the event). Therefore, I have had to implement a timer that > will read the DROP folder, and move the remaining files periodically. > To me this is defeating the objective of the FileSystemWatcher. > > To make matters worse ;P, the memory/no of handles used by my > application is forever increasing. I am closing objects and setting > them to nothing, as well as calling GC.Collect() in each sub. > > Is this really how the FileSystemWatcher works? Can you please suggest > a more elegant/efficient way of achieving my aim? Much appreciated. > > Jimmy > > Hi Kevin,
Thanks for the guidance (and patience). I'm on the right track now thanks to you. Think the explanantion with the recipes did it!! Despite only monitoring 2 directories, my application/service seems to use up to 25 - 30 MB of memory usage (the handles go up to 250 +). Is this a common burden with the FileSystemWatcher, or is it likely to be due to inefficient code? The VB.Net application I have created has actually been upgraded from VB6 to .Net. The VB6 application used several directory monitoring API calls (i.e. FindFirstChangeNotification, FindNextChangeNotification, MsgWaitForMultipleObjects, etc). When processing the same number of files as the VB.Net application, the memory usage would reach no more than 13 MB (with 150 + handles). Thanks once again Kevin. Hi Mike, Thanks for the code. Before I deploy code from your application, may I ask roughly how much memory the application uses and whether it reaches the high levels (30MB) of my (inefficient ;P) windows service application? Cheers, Jimmy May I point out that despite upgrading my VB6 project to VB.Net, I was
unable to get the directory monitoring API calls to work, hence the use of the FileSystemWatcher component. If someone has been able to make use of the API calls then please let me know. Thanks. Jimmy I'm not sure you really want to use the API for this. The FSW in .NET 2.0
appears to be a "smart" wrapper around the API. By "smart" I mean that it checks to ensure the file that triggered the API calls meets the file template requirements you gave the FSW object before returning. FSW also resets the API for the next event so you don't have to remember to do this yourself. (The VB 2005 code I posted was originally written in VC++ 6 and not having to do these two items dumped a page of code.) Mike. <jimmyfishb***@yahoo.co.uk> wrote in message Show quote news:1139932733.209081.217820@o13g2000cwo.googlegroups.com... > May I point out that despite upgrading my VB6 project to VB.Net, I was > unable to get the directory monitoring API calls to work, hence the use > of the FileSystemWatcher component. If someone has been able to make > use of the API calls then please let me know. Thanks. > > Jimmy > > Hi Jimmy,
The .Net Framework tends to use a lot more memory than a VB6 or C++ native app would. This is because of the built-in memory management. As long as you are disposing anything that needs disposing (such as the FileSystemWatcher class, when you stop monitoring), you shouldn't worry about it. Garbage Collection will take care of it. The .Net Framework tends to cache for a long time, but it is very reliable. I'm assuming you're only using one instance of the FileSystemWatcher? -- Show quoteHTH, Kevin Spencer Microsoft MVP ..Net Developer We got a sick zebra a hat, you ultimate tuna. <jimmyfishb***@yahoo.co.uk> wrote in message news:1139932733.209081.217820@o13g2000cwo.googlegroups.com... > May I point out that despite upgrading my VB6 project to VB.Net, I was > unable to get the directory monitoring API calls to work, hence the use > of the FileSystemWatcher component. If someone has been able to make > use of the API calls then please let me know. Thanks. > > Jimmy > Kevin,
Jimmy will need two instances of the FileSystemWatcher since he is monitoring two directories. Unless the directories are under a common parent, in which case he can monitor the parent. In either case, the footprint shouldn't be a big as he is seeing. Mike Ober. Show quote "Kevin Spencer" <kevin@DIESPAMMERSDIEtakempis.com> wrote in message news:e4$OV9YMGHA.2416@TK2MSFTNGP15.phx.gbl... > > Hi Jimmy, > > The .Net Framework tends to use a lot more memory than a VB6 or C++ native > app would. This is because of the built-in memory management. As long as you > are disposing anything that needs disposing (such as the FileSystemWatcher > class, when you stop monitoring), you shouldn't worry about it. Garbage > Collection will take care of it. The .Net Framework tends to cache for a > long time, but it is very reliable. I'm assuming you're only using one > instance of the FileSystemWatcher? > > -- > HTH, > > Kevin Spencer > Microsoft MVP > .Net Developer > We got a sick zebra a hat, > you ultimate tuna. > > > <jimmyfishb***@yahoo.co.uk> wrote in message > news:1139932733.209081.217820@o13g2000cwo.googlegroups.com... > > May I point out that despite upgrading my VB6 project to VB.Net, I was > > unable to get the directory monitoring API calls to work, hence the use > > of the FileSystemWatcher component. If someone has been able to make > > use of the API calls then please let me know. Thanks. > > > > Jimmy > > > > > Hi Michael,
I should have said "1 per drop directory." -- Show quoteHTH, Kevin Spencer Microsoft MVP ..Net Developer We got a sick zebra a hat, you ultimate tuna. "Michael D. Ober" <ober***@.alum.mit.edu.nospam> wrote in message news:%231Ty$TZMGHA.2336@TK2MSFTNGP12.phx.gbl... > > Kevin, > > Jimmy will need two instances of the FileSystemWatcher since he is > monitoring two directories. Unless the directories are under a common > parent, in which case he can monitor the parent. In either case, the > footprint shouldn't be a big as he is seeing. > > Mike Ober. > > "Kevin Spencer" <kevin@DIESPAMMERSDIEtakempis.com> wrote in message > news:e4$OV9YMGHA.2416@TK2MSFTNGP15.phx.gbl... >> >> Hi Jimmy, >> >> The .Net Framework tends to use a lot more memory than a VB6 or C++ >> native >> app would. This is because of the built-in memory management. As long as > you >> are disposing anything that needs disposing (such as the >> FileSystemWatcher >> class, when you stop monitoring), you shouldn't worry about it. Garbage >> Collection will take care of it. The .Net Framework tends to cache for a >> long time, but it is very reliable. I'm assuming you're only using one >> instance of the FileSystemWatcher? >> >> -- >> HTH, >> >> Kevin Spencer >> Microsoft MVP >> .Net Developer >> We got a sick zebra a hat, >> you ultimate tuna. >> >> >> <jimmyfishb***@yahoo.co.uk> wrote in message >> news:1139932733.209081.217820@o13g2000cwo.googlegroups.com... >> > May I point out that despite upgrading my VB6 project to VB.Net, I was >> > unable to get the directory monitoring API calls to work, hence the use >> > of the FileSystemWatcher component. If someone has been able to make >> > use of the API calls then please let me know. Thanks. >> > >> > Jimmy >> > >> >> >> > > > Kevin, Mike,
My app is now working ;P, and as efficient as it possibly can. Thanks guys. Jimmy On a server that has been up since Feb 3rd, the code I posted has run for 10
minutes total and is currently using about 11Mb of VM allocation and a physical memory footprint of 1 Mb. Numbers are from the server's Task Manager and event viewer. Mike. <jimmyfishb***@yahoo.co.uk> wrote in message Show quote news:1139932541.134321.136200@f14g2000cwb.googlegroups.com... > Hi Kevin, > > Thanks for the guidance (and patience). I'm on the right track now > thanks to you. Think the explanantion with the recipes did it!! > > Despite only monitoring 2 directories, my application/service seems to > use up to 25 - 30 MB of memory usage (the handles go up to 250 +). Is > this a common burden with the FileSystemWatcher, or is it likely to be > due to inefficient code? The VB.Net application I have created has > actually been upgraded from VB6 to .Net. The VB6 application used > several directory monitoring API calls (i.e. > FindFirstChangeNotification, FindNextChangeNotification, > MsgWaitForMultipleObjects, etc). When processing the same number of > files as the VB.Net application, the memory usage would reach no more > than 13 MB (with 150 + handles). Thanks once again Kevin. > > Hi Mike, > > Thanks for the code. Before I deploy code from your application, may I > ask roughly how much memory the application uses and whether it reaches > the high levels (30MB) of my (inefficient ;P) windows service > application? Cheers, > > Jimmy > Let me clarify - the code I posted started when the server came up the
evening of the 3rd and has used 10 minutes of CPU time since then. Mike Ober. Show quote "Michael D. Ober" <ober***@.alum.mit.edu.nospam> wrote in message news:%23NymDkYMGHA.2528@TK2MSFTNGP12.phx.gbl... > > > On a server that has been up since Feb 3rd, the code I posted has run for 10 > minutes total and is currently using about 11Mb of VM allocation and a > physical memory footprint of 1 Mb. Numbers are from the server's Task > Manager and event viewer. > > Mike. > > <jimmyfishb***@yahoo.co.uk> wrote in message > news:1139932541.134321.136200@f14g2000cwb.googlegroups.com... > > Hi Kevin, > > > > Thanks for the guidance (and patience). I'm on the right track now > > thanks to you. Think the explanantion with the recipes did it!! > > > > Despite only monitoring 2 directories, my application/service seems to > > use up to 25 - 30 MB of memory usage (the handles go up to 250 +). Is > > this a common burden with the FileSystemWatcher, or is it likely to be > > due to inefficient code? The VB.Net application I have created has > > actually been upgraded from VB6 to .Net. The VB6 application used > > several directory monitoring API calls (i.e. > > FindFirstChangeNotification, FindNextChangeNotification, > > MsgWaitForMultipleObjects, etc). When processing the same number of > > files as the VB.Net application, the memory usage would reach no more > > than 13 MB (with 150 + handles). Thanks once again Kevin. > > > > Hi Mike, > > > > Thanks for the code. Before I deploy code from your application, may I > > ask roughly how much memory the application uses and whether it reaches > > the high levels (30MB) of my (inefficient ;P) windows service > > application? Cheers, > > > > Jimmy > > > > > > |
|||||||||||||||||||||||