|
dev
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
SqlDecimal calculations in .NET 2.0I'm trying to do some simple calculations with System.Data.SqlDecimal on ..NET 2.0: SqlDecimal one = new SqlDecimal(1); SqlDecimal seven = new SqlDecimal(7); SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven); The precision of the result in oneBySeven is way below my expectations. I get 0.142857, but I expected something like 0.1428571428571428571428571429 or even more precise. Additional values reported by the debugger: // Data[0] Data[1] Data[2] Data[3] Precision Scale Value // one 0x00000001 0x00000000 0x00000000 0x00000000 1 0 {1} // seven 0x00000007 0x00000000 0x00000000 0x00000000 1 0 {7} // oneBySeven 0x00022e09 0x00000000 0x00000000 0x00000000 7 6 {0.142857} Any explanations? Thanks! Beat <DIV>"Beat Bucheli" <beat.buch***@timeware.ch> wrote in
Show quote message news:u4IiqK01GHA.4484@TK2MSFTNGP02.phx.gbl...</DIV>> Hello, Sure. This explains all:> > I'm trying to do some simple calculations with System.Data.SqlDecimal on > .NET 2.0: > > SqlDecimal one = new SqlDecimal(1); > SqlDecimal seven = new SqlDecimal(7); > SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven); > > The precision of the result in oneBySeven is way below my expectations. I > get 0.142857, but I expected something like 0.1428571428571428571428571429 > or even more precise. Additional values reported by the debugger: > > // Data[0] Data[1] Data[2] Data[3] Precision > Scale Value > // one 0x00000001 0x00000000 0x00000000 0x00000000 1 > 0 {1} > // seven 0x00000007 0x00000000 0x00000000 0x00000000 1 > 0 {7} > // oneBySeven 0x00022e09 0x00000000 0x00000000 0x00000000 7 > 6 {0.142857} > > Any explanations? Thanks! > Beat > Precision, Scale, and Length (Transact-SQL) http://msdn2.microsoft.com/en-us/library/ms190476.aspx Given decimals e1 and e1, with precision p1 and p2 and scale s1 and s2 respectively, the quotient (e1 / e2) will have precision = p1 - s1 + s2 + max(6, s1 + p2 + 1) and scale = max(6, s1 + p2 + 1) In a table or in TSQL a decimal type is declared with a precision and scale. When you use this .NET constructor the precision and scale are inferred from the constructor argument. If you want more precision in the quotient you can specify more precision in the arguments: SqlDecimal one = SqlDecimal.ConvertToPrecScale(new SqlDecimal(1), 20, 12); SqlDecimal seven = SqlDecimal.ConvertToPrecScale(new SqlDecimal(7), 20, 12); ; SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven); Console.WriteLine(oneBySeven); David |
|||||||||||||||||||||||