converting/formatting a float value rounds up incorrectly.

Last post 11-10-2009 3:49 PM by dtsob75. 5 replies.

Sort Posts:

  • converting/formatting a float value rounds up incorrectly.

    11-05-2009, 12:14 PM
    • Member
      143 point Member
    • dtsob75
    • Member since 09-17-2002, 5:10 AM
    • Tulsa, OK
    • Posts 67

    We have a value coming from an Oracle database, thru web svcs that has the data type of float (over zealous dba).  We need to display that value on a web page.  We originally had a generic conversion routine:

       public static decimal get_Decimal(object objIn)
            {
                decimal d = 0;
                if (objIn == null)
                { return d; }
                decimal.TryParse(objIn.ToString(), out d);         
                return d;
            }

    This worked fine until we had a value of -15856966.  The TryParse failed and returned a 0. (The internal value was -1.58569E7, or something close to that.)  Of course, 0 would not satisfy the users!!!

    We then tried:

    float f = (float)objIn;

    d = (decimal)f;

    This returns -15856970, so it is rounding up and is 4 off.

    I then tried to format the value for display purposes directly from the float:

    string  d = String.Format( "{0:#,###,###,###.##;(#,###,###,###.##)}", this._nonLaborDollarsQy );

    where _nonLaborDollarsQy is the float value.  This returns -15,856,970.00, which, as you can see is still rounded up.

    I am not sure where to go from here.  I am sure the users are not going to be too happy with the rounded number.

    Anyone have any ideas?

    Thanks in advance for your help.

     

     

    -Joel WZ
  • Re: converting/formatting a float value rounds up incorrectly.

    11-05-2009, 3:05 PM
    • Member
      254 point Member
    • LuizFicer
    • Member since 03-13-2009, 4:45 AM
    • Lima-Peru
    • Posts 82

    if you are using oracle and have some values decimal whit  E over you will get and error

    i think that you need save round values with oracle for dont get error 

    Dont Forget mark Answer if This Help

    Luis Antonio Cuadra Collaton
    LuizFicer - MCTS
    My Site under Construction : peruasp - NET peru

  • Re: converting/formatting a float value rounds up incorrectly.

    11-05-2009, 3:45 PM
    • Member
      188 point Member
    • binli0114
    • Member since 09-26-2006, 12:31 AM
    • Sydney
    • Posts 48

    Hi

    I would recommend you to use System.Convert.ToDecimal(objIn) to convert the objIn to Decimal value. System.convert provides some intelligent roundings in numerical conversions, however the extra care taken by the System.Convert methods does come at a performance cost.

    The alternative is to used Decimal.Round methods when you wanna do the decimal places rounding.

    ------------------------------------------------------------
    I LOVE THIS GAME
  • Re: converting/formatting a float value rounds up incorrectly.

    11-09-2009, 12:57 PM
    • Member
      143 point Member
    • dtsob75
    • Member since 09-17-2002, 5:10 AM
    • Tulsa, OK
    • Posts 67

     I had tried the Convert.ToDecimal(objIn) and the String.Format methods, and they all worked find until the number got above 10,000,000, for example 15856966.  When it did, they returned a rounded number of 15856970.

    What I finally got to work was:

    decimal d = 0;
    float f = (float)objIn;
    double dblNbr = Convert.ToDouble( f );
    d = Convert.ToDecimal( dblNbr );

    This is pretty crazy. I am not sure if there is a bug in .Net or what.


     

    -Joel WZ
  • Re: converting/formatting a float value rounds up incorrectly.

    11-09-2009, 7:22 PM
    • Contributor
      4,994 point Contributor
    • Paul Linton
    • Member since 04-30-2008, 3:16 AM
    • Posts 872

    You are working down in the murky, least significant bits of the float.  I think you will find that your float values are actually correct, because there is no way that just converting to a double can invent more bits of precision.  The problem is with the way the float is displayed, either in the debugger or via some form of Response.Write.  The default format string, I think, is "G", General which does not show all the stored digits of a float!  Try using the "Round-trip" format specifier when you display the float value.  Something like

    string formattedNumber = string.Format("{0:R}", f);

    where f is from line 2 of your code ( the cast objIn).  I suspect you will see that formattedNumber is correct.  You can get more information on the "R" specifier at http://msdn.microsoft.com/en-us/library/dwhawy9k.aspx#RFormatString

    Got a c# problem? Try .NET Book Zero from Charles Petzold, it's a free pdf.
  • Re: converting/formatting a float value rounds up incorrectly.

    11-10-2009, 3:49 PM
    • Member
      143 point Member
    • dtsob75
    • Member since 09-17-2002, 5:10 AM
    • Tulsa, OK
    • Posts 67

    I failed to mention that this application is kind of a hodge-podge of platforms.  The db is Oracle, the middle tier is written in java, and the web ui is .Net.  I can see the value in the database and it is -15856966. However, when the TryParse executed it returned zero on the value that was passed thru the web svc (which also was verified to return the -1586966).  The standard ToString returned -1.58569E7.  I need to display the data in the UI, but when I do the display formatting String.Format( "{0:#,###,###,###.##;(#,###,###,###.##)}", it actually displays -15,856,970. So, not sure about the rounding issue.  However, the Format using the "R" does return the proper value. 

    -Joel WZ
Page 1 of 1 (6 items)