Home All Groups Group Topic Archive Search About

Writing into a binary file simultaneously from multithreads...

Author
11 Apr 2006 11:21 AM
Steve B.
Hi,

I'm building an application that use a binary file format with a proprietary
format.
Files can grow up to several GBytes...

The mecanism of building the data is quite complex and I'd like to know if
multiple threads can -simultaneously- write into the data file...

For example, I know in advance thread one will write the first 100MB, thread
two 50 next MB, and so on. Can I write from multiple threads ?
What objects Have I to use ?
One FileStream and seeking as needing ? One FileStream and several
StreamWriters ? etc ...

Thanks in advance...
Steve

Author
11 Apr 2006 4:48 PM
Jon Skeet [C# MVP]
Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote:
> I'm building an application that use a binary file format with a proprietary
> format.
> Files can grow up to several GBytes...
>
> The mecanism of building the data is quite complex and I'd like to know if
> multiple threads can -simultaneously- write into the data file...
>
> For example, I know in advance thread one will write the first 100MB, thread
> two 50 next MB, and so on. Can I write from multiple threads ?
> What objects Have I to use ?
> One FileStream and seeking as needing ? One FileStream and several
> StreamWriters ? etc ...

I suspect there *are* ways of doing it from multiple threads, but it
feels like the kind of thing which could very easily go wrong. An
alternative which may well be simpler would be to have a single thread
which had a queue of write requests. The other threads would add jobs
to the queue, and then the single thread could process those requests.

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Author
12 Apr 2006 11:27 AM
Steve B.
After some tests, I found a solution which seems to work :

lock (outputStream)

{

outputStream.Seek(

dataOffset + numberObBytesWritten

SeekOrigin.Begin

);

outputStream.Write(

buffer,

0,

read

);

I'm not sure this is a very smart solution.
Your solution should look like this :

WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)...
That's it ?

Thanks,
Steve

"Jon Skeet [C# MVP]" <sk***@pobox.com> a écrit dans le message de news:
MPG.1ea5ee209c38794798d***@msnews.microsoft.com...
Show quote
> Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote:
>> I'm building an application that use a binary file format with a
>> proprietary
>> format.
>> Files can grow up to several GBytes...
>>
>> The mecanism of building the data is quite complex and I'd like to know
>> if
>> multiple threads can -simultaneously- write into the data file...
>>
>> For example, I know in advance thread one will write the first 100MB,
>> thread
>> two 50 next MB, and so on. Can I write from multiple threads ?
>> What objects Have I to use ?
>> One FileStream and seeking as needing ? One FileStream and several
>> StreamWriters ? etc ...
>
> I suspect there *are* ways of doing it from multiple threads, but it
> feels like the kind of thing which could very easily go wrong. An
> alternative which may well be simpler would be to have a single thread
> which had a queue of write requests. The other threads would add jobs
> to the queue, and then the single thread could process those requests.
>
> --
> Jon Skeet - <sk***@pobox.com>
> http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
> If replying to the group, please do not mail me too
Author
12 Apr 2006 5:14 PM
Jon Skeet [C# MVP]
Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote:
Show quote
> After some tests, I found a solution which seems to work :
>
> lock (outputStream)
> {
> outputStream.Seek(
> dataOffset + numberObBytesWritten
> SeekOrigin.Begin
> );
>
> outputStream.Write(
> buffer,
> 0,
> read
> );
>
> I'm not sure this is a very smart solution.

It's not particularly safe, as if you have two streams pointing at the
same file, they could still both update it simultaneously.

> Your solution should look like this :
>
> WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)...
> That's it ?

I wouldn't pass the stream - I'd create one WriterClass (or whatever)
per file I wanted to write to, and have it maintain the stream. Now,
admittedly you'd still need to make sure you didn't create two
WriterClasses for the same file, but that's probably easier to do than
the stream equivalent. (You could easily provide a factory method, in
fact.)

Note that an easier way of seeking is often to use the Position
property:

outputStream.Position = dataOffset+numberOfBytesWritten;

--
Jon Skeet - <sk***@pobox.com>
http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too
Author
13 Apr 2006 7:10 AM
Steve B.
Thanks,
I'll take a deep look into this mecanism.

Steve


"Jon Skeet [C# MVP]" <sk***@pobox.com> a écrit dans le message de news:
MPG.1ea745d2a50f720f98d***@msnews.microsoft.com...
Show quote
> Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote:
>> After some tests, I found a solution which seems to work :
>>
>> lock (outputStream)
>> {
>> outputStream.Seek(
>> dataOffset + numberObBytesWritten
>> SeekOrigin.Begin
>> );
>>
>> outputStream.Write(
>> buffer,
>> 0,
>> read
>> );
>>
>> I'm not sure this is a very smart solution.
>
> It's not particularly safe, as if you have two streams pointing at the
> same file, they could still both update it simultaneously.
>
>> Your solution should look like this :
>>
>> WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)...
>> That's it ?
>
> I wouldn't pass the stream - I'd create one WriterClass (or whatever)
> per file I wanted to write to, and have it maintain the stream. Now,
> admittedly you'd still need to make sure you didn't create two
> WriterClasses for the same file, but that's probably easier to do than
> the stream equivalent. (You could easily provide a factory method, in
> fact.)
>
> Note that an easier way of seeking is often to use the Position
> property:
>
> outputStream.Position = dataOffset+numberOfBytesWritten;
>
> --
> Jon Skeet - <sk***@pobox.com>
> http://www.pobox.com/~skeet   Blog: http://www.msmvps.com/jon.skeet
> If replying to the group, please do not mail me too

AddThis Social Bookmark Button