|
dev
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Large DatasetsHas anyone seen any documentation or white papers with suggestions on how
to handle large datasets. My task is to read in 10K records and then step through each one processing it's data. The processing involved fetching 3 other records that match the current, manipulate the data and then write out 5 records to a different database. DataReader is out of the question, because this whole precessing time could take 10 or 20 minutes, so it looks like a DataSet. My concern is what happens if a DataSet exceeds memory. Is it cached on disk in the swapfile or does .Net handle it's temp storage. I'd love to find some kind of guideline on how large can a DataSet get without causing calaterail damage in the environment. Thanks for any guidance. Bernie Hi Bernie,
Is server processing an option for you? (stored procedure or something) What databaes are we talking? -- Show quoteMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ "Bernie Hunt" <bh***@optonline.net> wrote in message news:Xns979B51A9D722Ebhuntoptonlinenet@207.46.248.16... > Has anyone seen any documentation or white papers with suggestions on how > to handle large datasets. My task is to read in 10K records and then step > through each one processing it's data. The processing involved fetching 3 > other records that match the current, manipulate the data and then write > out 5 records to a different database. > > DataReader is out of the question, because this whole precessing time > could > take 10 or 20 minutes, so it looks like a DataSet. My concern is what > happens if a DataSet exceeds memory. Is it cached on disk in the swapfile > or does .Net handle it's temp storage. I'd love to find some kind of > guideline on how large can a DataSet get without causing calaterail damage > in the environment. > > Thanks for any guidance. > > Bernie No, There are two applications where this applies. First the originating
database is an ODBC connection to a Faircom database. The vast majority of the customers using this will be running XP Pro with no server. The second application is for data conversion where the originating database could be just about anything, Pervasive, Advantage and dBase are the most common. Bernie Show quote > Is server processing an option for you? (stored procedure or something) > What databaes are we talking? > Bernie,
A dataset can be as large as the program leaves enough memory for it. Where the program can be AFAIK not larger than 800Mb. I hope this helps Cor Show quote "Bernie Hunt" <bh***@optonline.net> schreef in bericht news:Xns979B51A9D722Ebhuntoptonlinenet@207.46.248.16... > Has anyone seen any documentation or white papers with suggestions on how > to handle large datasets. My task is to read in 10K records and then step > through each one processing it's data. The processing involved fetching 3 > other records that match the current, manipulate the data and then write > out 5 records to a different database. > > DataReader is out of the question, because this whole precessing time > could > take 10 or 20 minutes, so it looks like a DataSet. My concern is what > happens if a DataSet exceeds memory. Is it cached on disk in the swapfile > or does .Net handle it's temp storage. I'd love to find some kind of > guideline on how large can a DataSet get without causing calaterail damage > in the environment. > > Thanks for any guidance. > > Bernie Hi Cor,
"Cor Ligthert [MVP]" <notmyfirstn***@planet.nl> wrote in message Where did you get this number from? Voodoo?news:OJHJ0p%23VGHA.4300@TK2MSFTNGP14.phx.gbl... > Bernie, > > A dataset can be as large as the program leaves enough memory for it. > Where the program can be AFAIK not larger than 800Mb. -- Miha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ > Miha Markic [MVP C#] What do you mean?> Where did you get this number from? Voodoo? > > -- Cor I mean where do you get the 800Mb limit from.
-- Show quoteMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ "Cor Ligthert [MVP]" <notmyfirstn***@planet.nl> wrote in message news:u8MA5l$VGHA.4900@TK2MSFTNGP12.phx.gbl... >> Miha Markic [MVP C#] >> Where did you get this number from? Voodoo? >> >> -- > What do you mean? > > Cor > >I mean where do you get the 800Mb limit from. From Herfried K. Wagner who gave this link> http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx Cor I could discuss here, but there are 64 bit CPUs out there and there is .net
2 that manages memory better. Ah, and if I recall correctly, there is a switch that lets you use more than 2GB memory for user appsin 32bit windows, too. And 800Mb is not accurate - it is just an assumption. -- Show quoteMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ "Cor Ligthert [MVP]" <notmyfirstn***@planet.nl> wrote in message news:ObK4PPBWGHA.6000@TK2MSFTNGP14.phx.gbl... > >I mean where do you get the 800Mb limit from. >> > > From Herfried K. Wagner who gave this link > > http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx > > Cor > >I could discuss here, but there are 64 bit CPUs out there and there is .net Did I write something else?>2 that manages memory better. >Ah, and if I recall correctly, there is a switch that lets you use more >than >2GB memory for user appsin 32bit windows, too. >And 800Mb is not accurate - it is just an assumption. Although the by you mentioned switch (the name is 3Gb switch) seems not affect this behaviour it stays aprox 800Mb per process. Cor "Cor Ligthert [MVP]" <notmyfirstn***@planet.nl> wrote in message " Where the program can be AFAIK not larger than 800Mb."news:%23EB4AAJWGHA.4924@TK2MSFTNGP05.phx.gbl... > >>I could discuss here, but there are 64 bit CPUs out there and there is >>.net >>2 that manages memory better. >>Ah, and if I recall correctly, there is a switch that lets you use more >>than >>2GB memory for user appsin 32bit windows, too. >>And 800Mb is not accurate - it is just an assumption. > > > Did I write something else? Note: 64bit CPUs can address terabytes of memory. -- Miha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ > Yes I wrote As Far As I Know.> " Where the program can be AFAIK not larger than 800Mb." > > Note: 64bit CPUs can address terabytes of memory. > Can you than give it for a 64bit CPU. Because this is not a memory behaviour but an OS behaviour and than even as it is written with the layer Net 1.x. However as you can enlighten me with some more precise figurs for a 64bit XP OS with Net 2.0 and/or Vista than I will be delighted. Cor Cor:
I did some extensive testing on this a couple of years back with the 1.1 ..Net Framework and console/web services. The /3GB switch *might* affect the amount of memory a .Net application can use. It really depends on the flavor of Operating System. For example, a test app on 32-bit Windows XP or Windows 2003 Standard Server (using /3GB) will only allocate an additional 900MB - 1200 MB of memory. But on 32-bit Windows 2003 Enterprise Server, using the /3GB switch will provide an application with the additional memory. Note, the /3GB switch is a double edged sword as it will limit any OS processes max memory size to use 1GB. Some useful info from the original thread (watch word wrap) - http://groups.google.com/group/microsoft.public.dotnet.framework.aspnet.w ebservices/tree/browse_frm/thread/cf6567aa5b6ee5d8/265abc31aa209df0?rnum= 1&hl=en&q=%2F3gb+clausius&_done=%2Fgroup% 2Fmicrosoft.public.dotnet.framework.aspnet.webservices%2Fbrowse_frm% 2Fthread%2Fcf6567aa5b6ee5d8%2F265abc31aa209df0%3Ftvc%3D1%26q%3D% 2F3gb+clausius%26hl%3Den%26#doc_265abc31aa209df0 Jeff Clausius SourceGear Show quote "Cor Ligthert [MVP]" <notmyfirstn***@planet.nl> wrote in news:#EB4AAJWGHA.4924@TK2MSFTNGP05.phx.gbl: > Although the by you mentioned switch (the name is 3Gb switch) seems > not affect this behaviour it stays aprox 800Mb per process. > > Cor > Jeff,
Thanks, Cor Show quote "Jeff Clausius" <je***@raegecruos.com> schreef in bericht news:Xns979C52B089958jeffcsourcegearcom@207.46.248.16... > Cor: > > I did some extensive testing on this a couple of years back with the 1.1 > .Net Framework and console/web services. The /3GB switch *might* affect > the amount of memory a .Net application can use. It really depends on > the flavor of Operating System. > > > For example, a test app on 32-bit Windows XP or Windows 2003 Standard > Server (using /3GB) will only allocate an additional 900MB - 1200 MB of > memory. But on 32-bit Windows 2003 Enterprise Server, using the /3GB > switch will provide an application with the additional memory. Note, the > /3GB switch is a double edged sword as it will limit any OS processes max > memory size to use 1GB. > > > Some useful info from the original thread (watch word wrap) - > > http://groups.google.com/group/microsoft.public.dotnet.framework.aspnet.w > ebservices/tree/browse_frm/thread/cf6567aa5b6ee5d8/265abc31aa209df0?rnum= > 1&hl=en&q=%2F3gb+clausius&_done=%2Fgroup% > 2Fmicrosoft.public.dotnet.framework.aspnet.webservices%2Fbrowse_frm% > 2Fthread%2Fcf6567aa5b6ee5d8%2F265abc31aa209df0%3Ftvc%3D1%26q%3D% > 2F3gb+clausius%26hl%3Den%26#doc_265abc31aa209df0 > > > Jeff Clausius > SourceGear > > > > "Cor Ligthert [MVP]" <notmyfirstn***@planet.nl> wrote in > news:#EB4AAJWGHA.4924@TK2MSFTNGP05.phx.gbl: > >> Although the by you mentioned switch (the name is 3Gb switch) seems >> not affect this behaviour it stays aprox 800Mb per process. >> >> Cor >> I could discuss here, but there are 64 bit CPUs out there and there is .net
2 that manages memory better. Ah, and if I recall correctly, there is a switch that lets you use more than 2GB memory for user appsin 32bit windows, too. And 800Mb is not accurate - it is just an assumption. -- Show quoteMiha Markic [MVP C#] RightHand .NET consulting & development www.rthand.com Blog: http://cs.rthand.com/blogs/blog_with_righthand/ "Cor Ligthert [MVP]" <notmyfirstn***@planet.nl> wrote in message news:ObK4PPBWGHA.6000@TK2MSFTNGP14.phx.gbl... > >I mean where do you get the 800Mb limit from. >> > > From Herfried K. Wagner who gave this link > > http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx > > Cor >
Show quote
"Bernie Hunt" <bh***@optonline.net> wrote in message 10,000 rows in a DataSet really isn't all that much. Just try it.news:Xns979B51A9D722Ebhuntoptonlinenet@207.46.248.16... > Has anyone seen any documentation or white papers with suggestions on how > to handle large datasets. My task is to read in 10K records and then step > through each one processing it's data. The processing involved fetching 3 > other records that match the current, manipulate the data and then write > out 5 records to a different database. > > DataReader is out of the question, because this whole precessing time > could > take 10 or 20 minutes, so it looks like a DataSet. My concern is what > happens if a DataSet exceeds memory. Is it cached on disk in the swapfile > or does .Net handle it's temp storage. I'd love to find some kind of > guideline on how large can a DataSet get without causing calaterail damage > in the environment. > David  |
|||||||||||||||||||||||
Other interesting topics