|
dev
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Using SqlClient.SqlBulkCopy to insert CSV into database tableI'm using SqlClient.SqlBulkCopy to try and bulk copy a csv file into a database. I am getting the following error after calling the ..WriteToServer method. "The given value of type String from the data source cannot be converted to type bit of the specified target column." Now I know that this means that you cannot shove a string or other incompatible datatype into a bit field, however when examine the data going into the .WriteToServer method, it only contains 1's or 0's. Surely these qualify as acceptable values for a bit field!? At any rate, the bit column accepts nulls also, however even using nulls, it still errors out with the same messgae. Can someone please give me some fresh ideas on this one? Here's my code so far.... ' Declare the streamreader objects and read the CSV file into memory Dim strFilePath As String = "c:\temp\tblDateLine.hms" Dim sr As System.IO.StreamReader = New System.IO.StreamReader(strFilePath) Dim strImportCSVFile As String = sr.ReadToEnd ' Destroy the Streamreader objects sr.Close() sr.Dispose() ' Split into rows Dim strDelimiter As String = "\n" & vbCrLf Dim strInputFileRows() As String strInputFileRows = Split(strImportCSVFile, strDelimiter) ' Split into columns Dim dt As DataTable = New DataTable Dim strInputFileColumns() As String ' Create the columns in the data table that correspond to the number of columns in the input file strInputFileColumns = Split(strInputFileRows(0), "|,") ' Just the first row only For Each s As String In strInputFileColumns dt.Columns.Add(New DataColumn) Next ' Now split the rows into columns and add them to the data table Dim row As DataRow Dim finalLine As String = "" For Each line As String In strInputFileRows row = dt.NewRow finalLine = line.Replace(Convert.ToString(vbCr), "") 'finalLine = line.Replace("|,|", "|,NULL") finalLine = line.Replace("|,|", "|,") If Microsoft.VisualBasic.Right(finalLine, 1) = "|" Then finalLine = Microsoft.VisualBasic.Left(finalLine, Microsoft.VisualBasic.Len(finalLine) - 1) row.ItemArray = Split(finalLine, "|,") dt.Rows.Add(row) Next ' Do the bulk copy m_strOLEDBConnectionString = BuildOLEDBConnectionString(g_udtDatabaseSetupOpts.strBlankDatabaseName) Dim sqlSQLServerBulkCopy As System.Data.SqlClient.SqlBulkCopy = New System.Data.SqlClient.SqlBulkCopy(m_strOLEDBConnectionString, SqlClient.SqlBulkCopyOptions.KeepNulls) ' Or System.Data.SqlClient.SqlBulkCopyOptions.TableLock Or SqlClient.SqlBulkCopyOptions.Default) sqlSQLServerBulkCopy.BatchSize = dt.Rows.Count sqlSQLServerBulkCopy.DestinationTableName = "tblDateLine" sqlSQLServerBulkCopy.WriteToServer(dt) sqlSQLServerBulkCopy.Close() ================ Here's a sample of the CSV I'm using 2004-01-01 23:59:59.000|,2004-01-01 00:00:00.000|,9.0|,2004-01-01 08:00:00.000|,2004-01-01 17:00:00.000|,0|\n The row delimiter is |\n & crlf The column delimeter is |, Many thanks and best regards, Rod Issue #1 (check first): Check your datatable, esp. the types on the
datatable, while in debug break mode. You can set up the table to have specific types by adding to the columns collection prior to adding rows, which is wise (explicit programming). In this manner, the bit column becomes a boolean and you have to translate from 0|1 to false|true, but you are already ripping rows. Note that a strongly typed dataset with a single datatable will uncover these errors as soon as you try to load. I am not sure on your split, as dual delimeters can be problematic with a split. | is adequate to separate 99.9% of all files without the ,. I am not sure this is cause problems for you, but examining the data table will tell. You might find that you are dealing with data, empty, data, empty, etc. now instead of data, data, etc. If you require dual delimiters a Regex split might be a better option. -- Show quoteGregory A. Beamer MVP; MCP: +I, SE, SD, DBA http://gregorybeamer.spaces.live.com ******************************************** Think outside the box! ******************************************** "Rod" <rod.w***@gmail.com> wrote in message news:1165329770.021830.41560@79g2000cws.googlegroups.com... > OK, I'm completely stumped on this one. > > I'm using SqlClient.SqlBulkCopy to try and bulk copy a csv file into a > database. I am getting the following error after calling the > .WriteToServer method. > > "The given value of type String from the data source cannot be > converted to type bit of the specified target column." > > Now I know that this means that you cannot shove a string or other > incompatible datatype into a bit field, however when examine the data > going into the .WriteToServer method, it only contains 1's or 0's. > Surely these qualify as acceptable values for a bit field!? > > At any rate, the bit column accepts nulls also, however even using > nulls, it still errors out with the same messgae. > > Can someone please give me some fresh ideas on this one? Here's my > code so far.... > > ' Declare the streamreader objects and read the CSV file into > memory > Dim strFilePath As String = "c:\temp\tblDateLine.hms" > Dim sr As System.IO.StreamReader = New > System.IO.StreamReader(strFilePath) > Dim strImportCSVFile As String = sr.ReadToEnd > > ' Destroy the Streamreader objects > sr.Close() > sr.Dispose() > > ' Split into rows > Dim strDelimiter As String = "\n" & vbCrLf > Dim strInputFileRows() As String > strInputFileRows = Split(strImportCSVFile, strDelimiter) > > ' Split into columns > Dim dt As DataTable = New DataTable > Dim strInputFileColumns() As String > > ' Create the columns in the data table that correspond to the > number of columns in the input file > strInputFileColumns = Split(strInputFileRows(0), "|,") ' Just > the first row only > For Each s As String In strInputFileColumns > dt.Columns.Add(New DataColumn) > Next > > ' Now split the rows into columns and add them to the data > table > Dim row As DataRow > Dim finalLine As String = "" > For Each line As String In strInputFileRows > row = dt.NewRow > finalLine = line.Replace(Convert.ToString(vbCr), "") > 'finalLine = line.Replace("|,|", "|,NULL") > finalLine = line.Replace("|,|", "|,") > If Microsoft.VisualBasic.Right(finalLine, 1) = "|" Then > finalLine = Microsoft.VisualBasic.Left(finalLine, > Microsoft.VisualBasic.Len(finalLine) - 1) > row.ItemArray = Split(finalLine, "|,") > dt.Rows.Add(row) > Next > > ' Do the bulk copy > m_strOLEDBConnectionString = > BuildOLEDBConnectionString(g_udtDatabaseSetupOpts.strBlankDatabaseName) > Dim sqlSQLServerBulkCopy As System.Data.SqlClient.SqlBulkCopy = > New System.Data.SqlClient.SqlBulkCopy(m_strOLEDBConnectionString, > SqlClient.SqlBulkCopyOptions.KeepNulls) ' Or > System.Data.SqlClient.SqlBulkCopyOptions.TableLock Or > SqlClient.SqlBulkCopyOptions.Default) > > sqlSQLServerBulkCopy.BatchSize = dt.Rows.Count > sqlSQLServerBulkCopy.DestinationTableName = "tblDateLine" > sqlSQLServerBulkCopy.WriteToServer(dt) > sqlSQLServerBulkCopy.Close() > > > ================ > > Here's a sample of the CSV I'm using > > 2004-01-01 23:59:59.000|,2004-01-01 00:00:00.000|,9.0|,2004-01-01 > 08:00:00.000|,2004-01-01 17:00:00.000|,0|\n > > The row delimiter is |\n & crlf > The column delimeter is |, > > Many thanks and best regards, > > Rod > Hi Gregory,
Thanks for having a look at my code and responding. I know what your'e saying about creating explicit data types in the data table - I'm sure that would help, however it's something that sounds like a really big hassle. I'm trying to convert VB6 code that is using SQLDMO and the bulk copy over to .net. The thing is, the DMO code just works as is - no problems at all. The .net code seems to be so much more complex for this type of thing - seems like a step backwards almost? This code is used to package and deploy a database. Under the old DMO code, all I had to do was match up the database table name with it's corresponding CSV file and everything worked great during the bulk copy. If I were to explicitly type the column types on the data table, I would have to look up those values from the live database itself. .....OK, perhaps it could be done..I'll look into it.... Regarding the delimeters, so far they seems to be working OK - I've had no problems at all with them. Thanks anyway. Cowboy (Gregory A. Beamer) wrote: Show quote > Issue #1 (check first): Check your datatable, esp. the types on the > datatable, while in debug break mode. You can set up the table to have > specific types by adding to the columns collection prior to adding rows, > which is wise (explicit programming). In this manner, the bit column becomes > a boolean and you have to translate from 0|1 to false|true, but you are > already ripping rows. Note that a strongly typed dataset with a single > datatable will uncover these errors as soon as you try to load. > > I am not sure on your split, as dual delimeters can be problematic with a > split. | is adequate to separate 99.9% of all files without the ,. I am not > sure this is cause problems for you, but examining the data table will tell. > You might find that you are dealing with data, empty, data, empty, etc. now > instead of data, data, etc. If you require dual delimiters a Regex split > might be a better option. > > -- > Gregory A. Beamer > MVP; MCP: +I, SE, SD, DBA > http://gregorybeamer.spaces.live.com > > ******************************************** > Think outside the box! > ******************************************** I'm a bit puzzled as to why you're creating a local data table first. Can't
you simply BCP directly from the CSV to a SQL table? -- Show quote____________________________________ William (Bill) Vaughn Author, Mentor, Consultant Microsoft MVP INETA Speaker www.betav.com/blog/billva www.betav.com Please reply only to the newsgroup so that others can benefit. This posting is provided "AS IS" with no warranties, and confers no rights. __________________________________ Visit www.hitchhikerguides.net to get more information on my latest book: Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) ----------------------------------------------------------------------------------------------------------------------- "Rod" <rod.w***@gmail.com> wrote in message news:1165329770.021830.41560@79g2000cws.googlegroups.com... > OK, I'm completely stumped on this one. > > I'm using SqlClient.SqlBulkCopy to try and bulk copy a csv file into a > database. I am getting the following error after calling the > .WriteToServer method. > > "The given value of type String from the data source cannot be > converted to type bit of the specified target column." > > Now I know that this means that you cannot shove a string or other > incompatible datatype into a bit field, however when examine the data > going into the .WriteToServer method, it only contains 1's or 0's. > Surely these qualify as acceptable values for a bit field!? > > At any rate, the bit column accepts nulls also, however even using > nulls, it still errors out with the same messgae. > > Can someone please give me some fresh ideas on this one? Here's my > code so far.... > > ' Declare the streamreader objects and read the CSV file into > memory > Dim strFilePath As String = "c:\temp\tblDateLine.hms" > Dim sr As System.IO.StreamReader = New > System.IO.StreamReader(strFilePath) > Dim strImportCSVFile As String = sr.ReadToEnd > > ' Destroy the Streamreader objects > sr.Close() > sr.Dispose() > > ' Split into rows > Dim strDelimiter As String = "\n" & vbCrLf > Dim strInputFileRows() As String > strInputFileRows = Split(strImportCSVFile, strDelimiter) > > ' Split into columns > Dim dt As DataTable = New DataTable > Dim strInputFileColumns() As String > > ' Create the columns in the data table that correspond to the > number of columns in the input file > strInputFileColumns = Split(strInputFileRows(0), "|,") ' Just > the first row only > For Each s As String In strInputFileColumns > dt.Columns.Add(New DataColumn) > Next > > ' Now split the rows into columns and add them to the data > table > Dim row As DataRow > Dim finalLine As String = "" > For Each line As String In strInputFileRows > row = dt.NewRow > finalLine = line.Replace(Convert.ToString(vbCr), "") > 'finalLine = line.Replace("|,|", "|,NULL") > finalLine = line.Replace("|,|", "|,") > If Microsoft.VisualBasic.Right(finalLine, 1) = "|" Then > finalLine = Microsoft.VisualBasic.Left(finalLine, > Microsoft.VisualBasic.Len(finalLine) - 1) > row.ItemArray = Split(finalLine, "|,") > dt.Rows.Add(row) > Next > > ' Do the bulk copy > m_strOLEDBConnectionString = > BuildOLEDBConnectionString(g_udtDatabaseSetupOpts.strBlankDatabaseName) > Dim sqlSQLServerBulkCopy As System.Data.SqlClient.SqlBulkCopy = > New System.Data.SqlClient.SqlBulkCopy(m_strOLEDBConnectionString, > SqlClient.SqlBulkCopyOptions.KeepNulls) ' Or > System.Data.SqlClient.SqlBulkCopyOptions.TableLock Or > SqlClient.SqlBulkCopyOptions.Default) > > sqlSQLServerBulkCopy.BatchSize = dt.Rows.Count > sqlSQLServerBulkCopy.DestinationTableName = "tblDateLine" > sqlSQLServerBulkCopy.WriteToServer(dt) > sqlSQLServerBulkCopy.Close() > > > ================ > > Here's a sample of the CSV I'm using > > 2004-01-01 23:59:59.000|,2004-01-01 00:00:00.000|,9.0|,2004-01-01 > 08:00:00.000|,2004-01-01 17:00:00.000|,0|\n > > The row delimiter is |\n & crlf > The column delimeter is |, > > Many thanks and best regards, > > Rod > Hi Bill,
Thanks for your input here. The reason I'm using a local data table is because I'm trying to use the .net framework to do this. In the good old VB6 days, I simply used SQL DMO to do this and everything was simply, elegant and sweet. It just worked going straight from a CSV to the database. Now it seems that you cannot do this. Perhaps I've missed something - I'm relatively new to the .net framework, and it seems as if there are quite a few ways to connect to a database. I've tried looking everywhere on the net for examples of this type of code and there doesn't seem to be much at all. Thanks again Bill. I heard you speak at VSLive in Sydney back in 2001. Really enjoyed your style. Good stuff. Best regards, Rod William (Bill) Vaughn wrote: Show quote > I'm a bit puzzled as to why you're creating a local data table first. Can't > you simply BCP directly from the CSV to a SQL table? > > -- > ____________________________________ > William (Bill) Vaughn > Author, Mentor, Consultant > Microsoft MVP > INETA Speaker > www.betav.com/blog/billva > www.betav.com > Please reply only to the newsgroup so that others can benefit. > This posting is provided "AS IS" with no warranties, and confers no rights. > __________________________________ > Visit www.hitchhikerguides.net to get more information on my latest book: > Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) > and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) > ----------------------------------------------------------------------------------------------------------------------- > One of the SqlBulkCopy examples in my new book is a CSV import. I know it's
tricky to setup but it works. This new 2.0 class is designed to call the SQLSMO layer underneath the covers--that replaces SQL DMO. -- Show quote____________________________________ William (Bill) Vaughn Author, Mentor, Consultant Microsoft MVP INETA Speaker www.betav.com/blog/billva www.betav.com Please reply only to the newsgroup so that others can benefit. This posting is provided "AS IS" with no warranties, and confers no rights. __________________________________ Visit www.hitchhikerguides.net to get more information on my latest book: Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) ----------------------------------------------------------------------------------------------------------------------- "Rod" <rod.w***@gmail.com> wrote in message news:1165360509.264783.142100@80g2000cwy.googlegroups.com... > Hi Bill, > > Thanks for your input here. The reason I'm using a local data table is > because I'm trying to use the .net framework to do this. In the good > old VB6 days, I simply used SQL DMO to do this and everything was > simply, elegant and sweet. It just worked going straight from a CSV to > the database. Now it seems that you cannot do this. Perhaps I've > missed something - I'm relatively new to the .net framework, and it > seems as if there are quite a few ways to connect to a database. I've > tried looking everywhere on the net for examples of this type of code > and there doesn't seem to be much at all. > > Thanks again Bill. I heard you speak at VSLive in Sydney back in 2001. > Really enjoyed your style. Good stuff. > > Best regards, > > Rod > > William (Bill) Vaughn wrote: >> I'm a bit puzzled as to why you're creating a local data table first. >> Can't >> you simply BCP directly from the CSV to a SQL table? >> >> -- >> ____________________________________ >> William (Bill) Vaughn >> Author, Mentor, Consultant >> Microsoft MVP >> INETA Speaker >> www.betav.com/blog/billva >> www.betav.com >> Please reply only to the newsgroup so that others can benefit. >> This posting is provided "AS IS" with no warranties, and confers no >> rights. >> __________________________________ >> Visit www.hitchhikerguides.net to get more information on my latest book: >> Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) >> and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) >> ----------------------------------------------------------------------------------------------------------------------- >> > Thanks Bill - good to hear that you've got a new book out. I'll check
it out. In the meantime, is the method that I described how you would recommend. Basically, Read CSV into a data table (using Streamreader) Bulk copy the data table to the database table (using SQLBulkCopy) Is this about it? Thanks, Rod. William (Bill) Vaughn wrote: Show quote > One of the SqlBulkCopy examples in my new book is a CSV import. I know it's > tricky to setup but it works. > This new 2.0 class is designed to call the SQLSMO layer underneath the > covers--that replaces SQL DMO. > > -- > ____________________________________ > William (Bill) Vaughn > Author, Mentor, Consultant > Microsoft MVP > INETA Speaker > www.betav.com/blog/billva > www.betav.com > Please reply only to the newsgroup so that others can benefit. > This posting is provided "AS IS" with no warranties, and confers no rights. > __________________________________ > Visit www.hitchhikerguides.net to get more information on my latest book: > Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) > and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) > ----------------------------------------------------------------------------------------------------------------------- > > "Rod" <rod.w***@gmail.com> wrote in message > news:1165360509.264783.142100@80g2000cwy.googlegroups.com... > > Hi Bill, > > > > Thanks for your input here. The reason I'm using a local data table is > > because I'm trying to use the .net framework to do this. In the good > > old VB6 days, I simply used SQL DMO to do this and everything was > > simply, elegant and sweet. It just worked going straight from a CSV to > > the database. Now it seems that you cannot do this. Perhaps I've > > missed something - I'm relatively new to the .net framework, and it > > seems as if there are quite a few ways to connect to a database. I've > > tried looking everywhere on the net for examples of this type of code > > and there doesn't seem to be much at all. > > > > Thanks again Bill. I heard you speak at VSLive in Sydney back in 2001. > > Really enjoyed your style. Good stuff. > > > > Best regards, > > > > Rod > > > > William (Bill) Vaughn wrote: > >> I'm a bit puzzled as to why you're creating a local data table first. > >> Can't > >> you simply BCP directly from the CSV to a SQL table? > >> > >> -- > >> ____________________________________ > >> William (Bill) Vaughn > >> Author, Mentor, Consultant > >> Microsoft MVP > >> INETA Speaker > >> www.betav.com/blog/billva > >> www.betav.com > >> Please reply only to the newsgroup so that others can benefit. > >> This posting is provided "AS IS" with no warranties, and confers no > >> rights. > >> __________________________________ > >> Visit www.hitchhikerguides.net to get more information on my latest book: > >> Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) > >> and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) > >> ----------------------------------------------------------------------------------------------------------------------- > >> > > Ah, I would suggest that if you can open a DataReader against the CSV to
create a DataTable, you'll be able to pass it directly to a SQL Server table. -- Show quote____________________________________ William (Bill) Vaughn Author, Mentor, Consultant Microsoft MVP INETA Speaker www.betav.com/blog/billva www.betav.com Please reply only to the newsgroup so that others can benefit. This posting is provided "AS IS" with no warranties, and confers no rights. __________________________________ Visit www.hitchhikerguides.net to get more information on my latest book: Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) ----------------------------------------------------------------------------------------------------------------------- "Rod" <rod.w***@gmail.com> wrote in message news:1165371775.860057.183620@f1g2000cwa.googlegroups.com... > Thanks Bill - good to hear that you've got a new book out. I'll check > it out. > > In the meantime, is the method that I described how you would > recommend. > > Basically, > > Read CSV into a data table (using Streamreader) > Bulk copy the data table to the database table (using SQLBulkCopy) > > Is this about it? > > Thanks, > > Rod. > > William (Bill) Vaughn wrote: >> One of the SqlBulkCopy examples in my new book is a CSV import. I know >> it's >> tricky to setup but it works. >> This new 2.0 class is designed to call the SQLSMO layer underneath the >> covers--that replaces SQL DMO. >> >> -- >> ____________________________________ >> William (Bill) Vaughn >> Author, Mentor, Consultant >> Microsoft MVP >> INETA Speaker >> www.betav.com/blog/billva >> www.betav.com >> Please reply only to the newsgroup so that others can benefit. >> This posting is provided "AS IS" with no warranties, and confers no >> rights. >> __________________________________ >> Visit www.hitchhikerguides.net to get more information on my latest book: >> Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) >> and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) >> ----------------------------------------------------------------------------------------------------------------------- >> >> "Rod" <rod.w***@gmail.com> wrote in message >> news:1165360509.264783.142100@80g2000cwy.googlegroups.com... >> > Hi Bill, >> > >> > Thanks for your input here. The reason I'm using a local data table is >> > because I'm trying to use the .net framework to do this. In the good >> > old VB6 days, I simply used SQL DMO to do this and everything was >> > simply, elegant and sweet. It just worked going straight from a CSV to >> > the database. Now it seems that you cannot do this. Perhaps I've >> > missed something - I'm relatively new to the .net framework, and it >> > seems as if there are quite a few ways to connect to a database. I've >> > tried looking everywhere on the net for examples of this type of code >> > and there doesn't seem to be much at all. >> > >> > Thanks again Bill. I heard you speak at VSLive in Sydney back in 2001. >> > Really enjoyed your style. Good stuff. >> > >> > Best regards, >> > >> > Rod >> > >> > William (Bill) Vaughn wrote: >> >> I'm a bit puzzled as to why you're creating a local data table first. >> >> Can't >> >> you simply BCP directly from the CSV to a SQL table? >> >> >> >> -- >> >> ____________________________________ >> >> William (Bill) Vaughn >> >> Author, Mentor, Consultant >> >> Microsoft MVP >> >> INETA Speaker >> >> www.betav.com/blog/billva >> >> www.betav.com >> >> Please reply only to the newsgroup so that others can benefit. >> >> This posting is provided "AS IS" with no warranties, and confers no >> >> rights. >> >> __________________________________ >> >> Visit www.hitchhikerguides.net to get more information on my latest >> >> book: >> >> Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) >> >> and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) >> >> ----------------------------------------------------------------------------------------------------------------------- >> >> >> > > |
|||||||||||||||||||||||