Home All Groups Group Topic Archive Search About

SqlDecimal calculations in .NET 2.0

Author
13 Sep 2006 2:41 PM
Beat Bucheli
Hello,

I'm trying to do some simple calculations with System.Data.SqlDecimal on
..NET 2.0:

SqlDecimal one = new SqlDecimal(1);
SqlDecimal seven = new SqlDecimal(7);
SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven);

The precision of the result in oneBySeven is way below my expectations. I
get 0.142857, but I expected something like 0.1428571428571428571428571429
or even more precise. Additional values reported by the debugger:

//             Data[0]     Data[1]     Data[2]     Data[3]     Precision
Scale  Value
// one         0x00000001  0x00000000  0x00000000  0x00000000  1          0
{1}
// seven       0x00000007  0x00000000  0x00000000  0x00000000  1          0
{7}
// oneBySeven  0x00022e09  0x00000000  0x00000000  0x00000000  7          6
{0.142857}

Any explanations? Thanks!
Beat

Author
13 Sep 2006 3:38 PM
David Browne
<DIV>&quot;Beat Bucheli&quot; &lt;beat.buch***@timeware.ch&gt; wrote in
Show quote
message news:u4IiqK01GHA.4484@TK2MSFTNGP02.phx.gbl...</DIV>> Hello,
>
> I'm trying to do some simple calculations with System.Data.SqlDecimal on
> .NET 2.0:
>
> SqlDecimal one = new SqlDecimal(1);
> SqlDecimal seven = new SqlDecimal(7);
> SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven);
>
> The precision of the result in oneBySeven is way below my expectations. I
> get 0.142857, but I expected something like 0.1428571428571428571428571429
> or even more precise. Additional values reported by the debugger:
>
> //             Data[0]     Data[1]     Data[2]     Data[3]     Precision
> Scale  Value
> // one         0x00000001  0x00000000  0x00000000  0x00000000  1
> 0 {1}
> // seven       0x00000007  0x00000000  0x00000000  0x00000000  1
> 0 {7}
> // oneBySeven  0x00022e09  0x00000000  0x00000000  0x00000000  7
> 6 {0.142857}
>
> Any explanations? Thanks!
> Beat
>

Sure.  This explains all:

Precision, Scale, and Length (Transact-SQL)
http://msdn2.microsoft.com/en-us/library/ms190476.aspx


Given decimals e1 and e1, with precision p1 and p2 and scale s1 and s2
respectively, the quotient (e1 / e2)
will have precision =  p1 - s1 + s2 + max(6, s1 + p2 + 1) and scale = max(6,
s1 + p2 + 1)


In a table or in TSQL a decimal type is declared with a precision and scale.
When you use this .NET constructor the precision and scale are inferred from
the constructor argument.  If you want more precision in the quotient you
can specify more precision in the arguments:

    SqlDecimal one = SqlDecimal.ConvertToPrecScale(new SqlDecimal(1), 20,
12);
    SqlDecimal seven = SqlDecimal.ConvertToPrecScale(new SqlDecimal(7), 20,
12); ;
    SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven);

    Console.WriteLine(oneBySeven);

David

AddThis Social Bookmark Button