Handling very huge data

5 Jan 2005 5:25 PM

Makarand Keer

Hi All

Wish all of u Happy New Year

I am developing a application where in I retrieve data from Database which
can go up millions records, do some processing on that and update the same in
single transaction,
now with limited RAM say 500MB how should I handle this data (I know one
solution might be paging) so that my ASP.NET process does not fall short of
memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
recycled because memory consumption exceeded the 305 MB (60 percent of
available RAM). "

Regards
Makarand

5 Jan 2005 6:21 PM

Miha Markic [MVP C#]

Did you consider using a stored procedure for processing the data?

--
Miha Markic [MVP C#] - RightHand .NET consulting & development
SLODUG - Slovene Developer Users Group
www.rthand.com

Show quoteHide quote

"Makarand Keer" <MakarandK***@discussions.microsoft.com> wrote in message
news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
> in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
> of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

5 Jan 2005 6:51 PM

Makarand Keer

Miha
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand

Show quoteHide quote

"Miha Markic [MVP C#]" wrote:

> Did you consider using a stored procedure for processing the data?
>
> --
> Miha Markic [MVP C#] - RightHand .NET consulting & development
> SLODUG - Slovene Developer Users Group
> www.rthand.com
>
> "Makarand Keer" <MakarandK***@discussions.microsoft.com> wrote in message
> news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> > Hi All
> >
> > Wish all of u Happy New Year
> >
> > I am developing a application where in I retrieve data from Database which
> > can go up millions records, do some processing on that and update the same
> > in
> > single transaction,
> > now with limited RAM say 500MB how should I handle this data (I know one
> > solution might be paging) so that my ASP.NET process does not fall short
> > of
> > memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> > recycled because memory consumption exceeded the 305 MB (60 percent of
> > available RAM). "
> >
> > Regards
> > Makarand
>
>
>

5 Jan 2005 6:29 PM

Sahil Malik

This much data is not a good candidate for putting in a dataset or the likes
of dataset. Keep it in the database for processing if you can.
What is the compelling reason that you must pull it out of the db to work on
so much data?

- Sahil Malik
http://dotnetjunkies.com/weblog/sahilmalik

Show quoteHide quote

"Makarand Keer" <MakarandK***@discussions.microsoft.com> wrote in message
news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

5 Jan 2005 6:45 PM

Makarand Keer

Sahil
Thanx for prompt reply,
Thing is I get the data from database server, add feed this data to MS
RuleEngine for applying Business Rules and after that I take same data and
update it in database.
So I don't c any scope of doing all these things in database.
Hope this time I am more clear on my problem

Makarand

Show quoteHide quote

"Sahil Malik" wrote:

> This much data is not a good candidate for putting in a dataset or the likes
> of dataset. Keep it in the database for processing if you can.
> What is the compelling reason that you must pull it out of the db to work on
> so much data?
>
> - Sahil Malik
> http://dotnetjunkies.com/weblog/sahilmalik
>
>
>
> "Makarand Keer" <MakarandK***@discussions.microsoft.com> wrote in message
> news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> > Hi All
> >
> > Wish all of u Happy New Year
> >
> > I am developing a application where in I retrieve data from Database which
> > can go up millions records, do some processing on that and update the same
> in
> > single transaction,
> > now with limited RAM say 500MB how should I handle this data (I know one
> > solution might be paging) so that my ASP.NET process does not fall short
> of
> > memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> > recycled because memory consumption exceeded the 305 MB (60 percent of
> > available RAM). "
> >
> > Regards
> > Makarand
>
>
>

5 Jan 2005 10:20 PM

Sahil Malik

I'd recommend getting peices of the data rather than millions of rows
together. Maybe in another 4-5 years when we have terabytes of RAM and 64
bit CPUs my suggestion will change, but today this is what I'd recommend.

- Sahil Malik
http://dotnetjunkies.com/weblog/sahilmalik

Show quoteHide quote

"Makarand Keer" <MakarandK***@discussions.microsoft.com> wrote in message
news:BF23980B-0C35-4C6D-8496-426C5DDF051F@microsoft.com...
> Sahil
> Thanx for prompt reply,
> Thing is I get the data from database server, add feed this data to MS
> RuleEngine for applying Business Rules and after that I take same data and
> update it in database.
> So I don't c any scope of doing all these things in database.
> Hope this time I am more clear on my problem
>
> Makarand
>
> "Sahil Malik" wrote:
>
> > This much data is not a good candidate for putting in a dataset or the
likes
> > of dataset. Keep it in the database for processing if you can.
> > What is the compelling reason that you must pull it out of the db to
work on
> > so much data?
> >
> > - Sahil Malik
> > http://dotnetjunkies.com/weblog/sahilmalik
> >
> >
> >
> > "Makarand Keer" <MakarandK***@discussions.microsoft.com> wrote in
message
> > news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> > > Hi All
> > >
> > > Wish all of u Happy New Year
> > >
> > > I am developing a application where in I retrieve data from Database
which
> > > can go up millions records, do some processing on that and update the
same
> > in
> > > single transaction,
> > > now with limited RAM say 500MB how should I handle this data (I know
one
> > > solution might be paging) so that my ASP.NET process does not fall
short
> > of
> > > memory, right now it is giving me error as "aspnet_wp.exe (PID: 596)
was
> > > recycled because memory consumption exceeded the 305 MB (60 percent of
> > > available RAM). "
> > >
> > > Regards
> > > Makarand
> >
> >
> >

5 Jan 2005 6:30 PM

Patrice

I would try to do this in the DB using SQL...

Patrice

--

Show quoteHide quote

"Makarand Keer" <MakarandK***@discussions.microsoft.com> a écrit dans le
message de news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

5 Jan 2005 6:32 PM

Eric Barr

Makarand,

It sounds to me like you have to real options here.

1) Chunk thru the data. I would choose this if there is some specific
business logic that you cannot perform directly on the database layer.

2) Process the Data in the Database. I would choose this if you can
perform the logic on the database layer. If you can go this route it
will probably be the most efficient.

HTH,
-eric

Makarand Keer wrote:
Show quoteHide quote

> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand

6 Jan 2005 4:57 AM

Joyjit Mukherjee

Hi,

try using API server side cursors that would fetch you data in batches, so
that your app. will not block for that period, which greatly improves user
experience.

regards
Joyjit

Show quoteHide quote

"Makarand Keer" <MakarandK***@discussions.microsoft.com> wrote in message
news:30EE4383-9550-4070-AFF3-7446F03CF56F@microsoft.com...
> Hi All
>
> Wish all of u Happy New Year
>
> I am developing a application where in I retrieve data from Database which
> can go up millions records, do some processing on that and update the same
in
> single transaction,
> now with limited RAM say 500MB how should I handle this data (I know one
> solution might be paging) so that my ASP.NET process does not fall short
of
> memory, right now it is giving me error as "aspnet_wp.exe (PID: 596) was
> recycled because memory consumption exceeded the 305 MB (60 percent of
> available RAM). "
>
> Regards
> Makarand