|
dev
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Writing into a binary file simultaneously from multithreads...Hi,
I'm building an application that use a binary file format with a proprietary format. Files can grow up to several GBytes... The mecanism of building the data is quite complex and I'd like to know if multiple threads can -simultaneously- write into the data file... For example, I know in advance thread one will write the first 100MB, thread two 50 next MB, and so on. Can I write from multiple threads ? What objects Have I to use ? One FileStream and seeking as needing ? One FileStream and several StreamWriters ? etc ... Thanks in advance... Steve Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote:
> I'm building an application that use a binary file format with a proprietary I suspect there *are* ways of doing it from multiple threads, but it > format. > Files can grow up to several GBytes... > > The mecanism of building the data is quite complex and I'd like to know if > multiple threads can -simultaneously- write into the data file... > > For example, I know in advance thread one will write the first 100MB, thread > two 50 next MB, and so on. Can I write from multiple threads ? > What objects Have I to use ? > One FileStream and seeking as needing ? One FileStream and several > StreamWriters ? etc ... feels like the kind of thing which could very easily go wrong. An alternative which may well be simpler would be to have a single thread which had a queue of write requests. The other threads would add jobs to the queue, and then the single thread could process those requests. -- Jon Skeet - <sk***@pobox.com> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet If replying to the group, please do not mail me too After some tests, I found a solution which seems to work :
lock (outputStream) { outputStream.Seek( dataOffset + numberObBytesWritten SeekOrigin.Begin ); outputStream.Write( buffer, 0, read ); I'm not sure this is a very smart solution. Your solution should look like this : WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)... That's it ? Thanks, Steve "Jon Skeet [C# MVP]" <sk***@pobox.com> a écrit dans le message de news: MPG.1ea5ee209c38794798d***@msnews.microsoft.com...Show quote > Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote: >> I'm building an application that use a binary file format with a >> proprietary >> format. >> Files can grow up to several GBytes... >> >> The mecanism of building the data is quite complex and I'd like to know >> if >> multiple threads can -simultaneously- write into the data file... >> >> For example, I know in advance thread one will write the first 100MB, >> thread >> two 50 next MB, and so on. Can I write from multiple threads ? >> What objects Have I to use ? >> One FileStream and seeking as needing ? One FileStream and several >> StreamWriters ? etc ... > > I suspect there *are* ways of doing it from multiple threads, but it > feels like the kind of thing which could very easily go wrong. An > alternative which may well be simpler would be to have a single thread > which had a queue of write requests. The other threads would add jobs > to the queue, and then the single thread could process those requests. > > -- > Jon Skeet - <sk***@pobox.com> > http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet > If replying to the group, please do not mail me too Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote:
Show quote > After some tests, I found a solution which seems to work : It's not particularly safe, as if you have two streams pointing at the > > lock (outputStream) > { > outputStream.Seek( > dataOffset + numberObBytesWritten > SeekOrigin.Begin > ); > > outputStream.Write( > buffer, > 0, > read > ); > > I'm not sure this is a very smart solution. same file, they could still both update it simultaneously. > Your solution should look like this : I wouldn't pass the stream - I'd create one WriterClass (or whatever) > > WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)... > That's it ? per file I wanted to write to, and have it maintain the stream. Now, admittedly you'd still need to make sure you didn't create two WriterClasses for the same file, but that's probably easier to do than the stream equivalent. (You could easily provide a factory method, in fact.) Note that an easier way of seeking is often to use the Position property: outputStream.Position = dataOffset+numberOfBytesWritten; -- Jon Skeet - <sk***@pobox.com> http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet If replying to the group, please do not mail me too Thanks,
I'll take a deep look into this mecanism. Steve "Jon Skeet [C# MVP]" <sk***@pobox.com> a écrit dans le message de news: MPG.1ea745d2a50f720f98d***@msnews.microsoft.com...Show quote > Steve B. <steve_beauge@com.msn_swap_com_and_msn> wrote: >> After some tests, I found a solution which seems to work : >> >> lock (outputStream) >> { >> outputStream.Seek( >> dataOffset + numberObBytesWritten >> SeekOrigin.Begin >> ); >> >> outputStream.Write( >> buffer, >> 0, >> read >> ); >> >> I'm not sure this is a very smart solution. > > It's not particularly safe, as if you have two streams pointing at the > same file, they could still both update it simultaneously. > >> Your solution should look like this : >> >> WriterClass.QueueWriteOperation(Stream s, int offset, byte[] buffer)... >> That's it ? > > I wouldn't pass the stream - I'd create one WriterClass (or whatever) > per file I wanted to write to, and have it maintain the stream. Now, > admittedly you'd still need to make sure you didn't create two > WriterClasses for the same file, but that's probably easier to do than > the stream equivalent. (You could easily provide a factory method, in > fact.) > > Note that an easier way of seeking is often to use the Position > property: > > outputStream.Position = dataOffset+numberOfBytesWritten; > > -- > Jon Skeet - <sk***@pobox.com> > http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet > If replying to the group, please do not mail me too |
|||||||||||||||||||||||