Hi,
I'm currently building a little web site for me and my friends where we
would like to be able to post big files (10Mb to 250Mb approx.) Right now
the ASP.Net UploadFile control can't do it as we are limited 1) by the
MaxRequestLength (4Mb) and 2) with big files (such as 250Mb) the server
crashes because it only uses memory to cache the upload...

Is there an easy way to develop an HTTP Module to handle the upload (links,
tutorials, ...)? I've seen components to do this on the net, but they are
too expensive for what we need to do and the budget we have (near 0.00$)...
I know we could use FTP (there are plenty of free server softwares out
there), but we could not use all the other features an asp.net has to offer
such as the ease of use and data binding and everything...  right now I
would like to be able to put these files into a database, which is possible
with the FileUpload control, but because of the "big files" requirement, I
just can't use it...

Anybody have ideas?

Thanks

ThunderMusic

The MaxRequestLength limit can be changed in the web.config file.

<configuration>
   <system.web>
      <httpRuntime maxRequestLength="40960" />
   </system.web>
</configuration>

Please see this document:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpgenref/html/gngrfHttpRuntimeSection.asp

Show quoteHide quote

I know this part...  We changed it, but there are 2 problems with this 1) it
opens the server for Denial Of Service attacks and 2)even if I put it to 2Gb
and upload a file of 200Mb, the server crashes because the FileUpload
control caches the request entirely in memory...

Thanks

ThunderMusic


Show quoteHide quote

I know you said your budget was $0 but I use a component called aspupload
that works great.  I think the single server license is $150.00.


Show quoteHide quote

it could be interesting...  I'll look for it on google... (or maybe you have
to link to the home page og the publisher?)

I'm currently trying a free solution that is under the LGPL called :
NeatUpload... I found it a little before noon (Eastern time USA) I'll see if
it does the job...  if it does, I'll use it, if it doesn't, I'll look for
your aspupload... ;)

thanks a lot...

ThunderMusic


Show quoteHide quote

http://www.aspupload.com/ is the site.  I think their main site is at
http://www.persits.com/

HTH,
Brian

Show quoteHide quote

I was working on a component to allow a "multi-post upload"; I had to
set it down for the moment in order to take care of more urgent work,
but I intend to get back on it next week.

It is a fairly simple server side script but it does require a client
side component that I currently have coded as a VB Application and
needs to be recoded as a downloadable .net component (it would be even
if this was a java applet, for cross browser compatability)

It was working, and the basic design is in place, but it could use some
code review and exception handling. It is pretty simple and should
allow for arbitrary small posts, and abitrarly large uploads, limitted
only by disk availability.

If anyone uses this, or improves it, please let me know so I can
benefit from the code review.

Basic concept:
1. client creates a connection and gets a hash from server
2. client reads file and post "postSize" bytes - server side opens file
associated with hash, seek() to offset sent by client and writes the
data sent. -repeat
3. client sends a completed signal.

There could be some optimization if you kept the file open between
posts, but that brings up the

The files are:
gigaUploadClient.vb - a class that implements the client
UploadServlet.aspx
UploadServlet.aspx.vb - implements the server side storage
uploadTester.vb - The code behind a form demonstrating how to use the
client.

uploadTester.vb:
Public Class uploadTester

    Private _uploader As gigaUploadClient

    Private Sub btnSend_Click(ByVal sender As System.Object, ByVal e As
System.EventArgs) Handles btnSend.Click
        btnSend.Enabled = False
        Dim postSize As Integer
        If Not Integer.TryParse(txtPostSizeKB.Text, postSize) Then
            MessageBox.Show("<" + txtPostSizeKB.Text + "> is not a
valid integer!")
            Return
        End If
        _uploader = New gigaUploadClient(txtFile.Text, txtUrl.Text,
postSize)
        Dim upldThrd As New Threading.Thread(AddressOf
_uploader.SendFile)
        upldThrd.Start()
    End Sub

    Private Sub btnUpdate_Click(ByVal sender As System.Object, ByVal e
As System.EventArgs) Handles btnUpdate.Click
        lblProgress.Text = _uploader.GetProgress + " : " +
Math.Round(_uploader.GetProgressFraction * 100).ToString + "%"
        btnSend.Enabled = Not _uploader.IsRunning()
    End Sub
End Class

********************************************************************************************
UploadServlet.aspx:
<%@ Page Language="VB" AutoEventWireup="false"
CodeFile="UploadServlet.aspx.vb" Inherits="_UploadServlet" %>

********************************************************************************************
Imports System
Imports System.IO

Partial Class _UploadServlet
    Inherits System.Web.UI.Page
    Private savePath As String
    Private Const readBlockSize As Integer = 10240

    Public Sub New()
        Dim settings As Configuration.AppSettingsReader = New
Configuration.AppSettingsReader()
        Try
            savePath = settings.GetValue("savePath", GetType(String))
        Catch
            savePath = "c:\temp\uploader"
        End Try
        Try
            If Not Directory.Exists(savePath) Then
                Directory.CreateDirectory(savePath)
            End If
        Catch ex As Exception

        End Try
    End Sub

    Protected Sub Page_Load(ByVal sender As Object, ByVal e As
System.EventArgs) Handles Me.Load
        ProcessUploadRequest()
    End Sub

    Private Sub restartUpload(ByVal fpath As String)
        '/* Find the seek position */
        Dim seekPos As Long = 0
        Try
            seekPos = Long.Parse(Request("offset"))
        Catch nex As Exception
            Response.Write("-1")
            Return
        End Try

        Dim fout As FileStream
        If seekPos > 0 Then
            '* Open a RandomAccessFile so that we can seek on it.
            fout = File.Create(fpath, FileMode.OpenOrCreate,
FileOptions.RandomAccess)
            'this seek call takes significant time when the file is
big, so only do
            'it if we get a seekposition which suggest a resume upload
            fout.Seek(seekPos, IO.SeekOrigin.Begin)
            'truncate the file to the restart position
            fout.Close()
            Response.Output.Write(seekPos.ToString)
            Return
        Else
            Response.Write("-1")
            Return
        End If
    End Sub

    Private Sub SpoolFile(ByVal fpath As String)

        Dim instr As Stream = Request.InputStream

        If (Not Directory.Exists(Path.GetDirectoryName(fpath))) Then
            Try
                Directory.CreateDirectory(Path.GetDirectoryName(fpath))
            Catch ex As Exception
            End Try
        End If

        Dim fout As FileStream
        If (File.Exists(fpath)) Then
            fout = File.OpenWrite(fpath)
            fout.Seek(0, SeekOrigin.End)
        Else
            fout = File.Create(fpath, FileMode.CreateNew)
        End If


        Dim b(readBlockSize) As Byte
        Dim i As Integer

        Do
            i = Request.InputStream.Read(b, 0, readBlockSize)
            fout.Write(b, 0, i)
            Diagnostics.Debug.Print(Text.Encoding.ASCII.GetString(b, 0,
i))
        Loop While (i = readBlockSize)
        'Response.Write(fout.Length)
        fout.Close()

    End Sub

    Public Sub ProcessUploadRequest()
        Try
            Response.ContentType = "text/html;charset=UTF-8"

            Dim hash As String = Request("hash")
            Dim cmd As String = Request("cmd")
            Dim fname As String = Request("fname")
            If "".Equals(fname) Or fname Is Nothing Then fname =
"downloadfile.dat"

            Diagnostics.Debug.Print("cmd ::" + cmd)
            Diagnostics.Debug.Print("fname ::" + fname)

            If hash Is Nothing Then hash = "0" +
Guid.NewGuid().ToString + "1"
            If ("").Equals(hash) Then hash = "0" +
Guid.NewGuid().ToString + "1"
            Dim fpath As String = Path.Combine(Path.Combine(savePath,
hash), fname)

            Diagnostics.Debug.Print("hash ::" + hash)

            Select Case cmd

                Case "getconnection"
                    Response.Output.Write(hash)
                    Response.Output.Close()
                    Return

                Case "getstatus"
                    'this is and extra step for the current
configuration
                    'but it is useful if we want to implement a resume
upload.
                    'this is not currently implemented.
                    Diagnostics.Debug.Print("writing file to " + fpath)

                    If (File.Exists(fpath)) Then
                        '* The file exists on the server. A previous
upload has failed.
                        '* report back how many bytes have been written
                        Response.Output.Write(New
FileInfo(fpath).Length())
                    Else
                        '* report back that zero bytes have been
written
                        Response.Output.Write("0")
                        Response.Output.Close()
                    End If

                Case "completeupload"
                    ' this is where we could do any clean up or file
moving that we want

                    Response.Output.Write("Success. Bytes written: " +
FileLen(fpath).ToString)
                    Return

                Case "restart"
                    restartUpload(fpath)
                    Response.Output.Close()
                    Return
                Case "spoolfile"
                    'this is the case of spooling the file
                    If (hash Is Nothing) Then
                        Response.Output.Write("null hash, null cmd is
not valid")
                    End If
                    SpoolFile(fpath)
                    Response.Output.Close()
                    Return
            End Select

        Catch e As Exception
            Response.Output.Write("failed")
            Response.Output.Write(e.Message)
            Diagnostics.Debug.Print(e.Message)
        End Try
    End Sub
End Class


***********************************************************************************************

Imports System
Imports System.IO
Imports System.Net

Public Class gigaUploadClient
    Private _cli As WebClient
    Private _hash As String
    Private _fileSize As Long
    Private _reader As IO.Stream
    Private _uploadUrl As String
    Private _sourceFile As String
    Private _postSize As Long
    Private _curPos As Long = 0
    Private _lockObj As New Object
    Private _isRunning As Boolean = False

    Public Sub New(ByVal sourceFile As String, ByVal uploadUrl As
String, ByVal postSize As Integer)
        _uploadUrl = uploadUrl
        _sourceFile = sourceFile
        _postSize = postSize
    End Sub


    Public Sub SendFile()
        _isRunning = True
        _cli = New WebClient()

        _cli.QueryString.Add("cmd", "getconnection")
        'cli.QueryString.Add("hash", "")
        _hash = _cli.DownloadString(_uploadUrl)

        'begin hack
        'trim off any whitespace. think this was because I used a
WriteLine() on the servlet. duh!
        _hash.TrimEnd(Constants.vbCrLf.ToCharArray)
        If _hash.EndsWith(Constants.vbCrLf) Then
            _hash = _hash.Substring(0, _hash.Length - 2)
        End If
        'end hack

        Dim fileOffset As Long = getStatus()
        If fileOffset > 0 Then
            restartTransfer(fileOffset)
        End If
        spoolFile()
        _isRunning = False
    End Sub

    Private Function getStatus() As Long
        _cli.QueryString.Clear()
        _cli.QueryString.Add("cmd", "getstatus")
        _cli.QueryString.Add("hash", _hash)
        _cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))

        Dim fileOffsetStr As String = _cli.DownloadString(_uploadUrl)
        Dim fileOffset As Long
        If Long.TryParse(fileOffsetStr, fileOffset) Then
            Return fileOffset
        Else
            Return 0
        End If
    End Function

    Private Sub restartTransfer(ByVal offset As Long)
        _cli.QueryString.Clear()
        _cli.QueryString.Add("cmd", "restart")
        _cli.QueryString.Add("hash", _hash)
        _cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))

        Dim fileOffsetStr As String = _cli.DownloadString(_uploadUrl)
        SyncLock _lockObj
            _curPos = offset
        End SyncLock
    End Sub

    Private Sub confirmFileFinished()
        _cli.QueryString.Clear()
        _cli.QueryString.Add("cmd", "completeupload")
        _cli.QueryString.Add("fname", Path.GetFileName(_sourceFile))
        _cli.QueryString.Add("hash", _hash)
        Dim resp As String = _cli.DownloadString(_uploadUrl)
    End Sub

    Private Sub spoolFile()
        Dim out As IO.Stream
        _fileSize = FileLen(_sourceFile)

        Const readBufferSize As Integer = 4096
        Dim b(readBufferSize) As Byte
        Dim postSize As Integer = _postSize * 1024
        Dim i As Integer
        Dim bytesSpooled As Long = 0

        _reader = File.OpenRead(_sourceFile)
        If _curPos > 0 Then
            _reader.Seek(_curPos, SeekOrigin.Begin)
        End If

        Do
            _cli.QueryString.Clear()
            _cli.QueryString.Add("cmd", "spoolfile")
            _cli.QueryString.Add("hash", _hash)
            _cli.QueryString.Add("fname",
Path.GetFileName(_sourceFile))
            _cli.Headers.Add("Content-Type",
"application/octet-stream")
            'don't write offset unless we need to resume an upload
            'cli.QueryString.Add("offset", curPos.ToString)
            out = _cli.OpenWrite(_uploadUrl)
            Dim bytesSpooledThisPost As Integer = 0
            Do
                i = _reader.Read(b, 0, readBufferSize)
                out.Write(b, 0, i)
                bytesSpooledThisPost += i
                bytesSpooled += i
                SyncLock _lockObj
                    _curPos = bytesSpooled
                End SyncLock
            Loop While (i = readBufferSize And bytesSpooledThisPost <
postSize)
            out.Close()
        Loop While (i = readBufferSize) ' i will be less then
readBufferSize when we reach the end of the file
    End Sub

    Public Function GetProgressFraction() As Double
        SyncLock _lockObj
            Return _curPos / _fileSize
        End SyncLock
    End Function

    Public Function GetProgress() As String
        SyncLock _lockObj
            Return Math.Round(_curPos / 1024).ToString + "kb of " +
Math.Round(_fileSize / 1024).ToString + "kb transfered"
        End SyncLock
    End Function

    Public Function IsRunning() As Boolean
        Return _isRunning
    End Function
End Class

Ok, piss-poor editing on that last post.
If anyone wants the files, let me know, and I can email them.

Please pass them along...  bhammer  at leas dot  com



Thanks,
Brian

Show quoteHide quote

Using ASP.NET 2.0's ImageMap Control
Serious Socket Programming Problems
Creating a remotable object like DCOM
unable to install framework 2
Threadpool and Sockets