Double/float to long integer

bastl · 06-02-2006, 07:52 AM

Hi to all.
I'v a problem on double conversion to long int.
Test this function:

Code:

void test( double d )
  {
  long int x;
  
  x = int( d * 2 * 100 );
  KMessageBox::messageBox ( 0, KMessageBox::Information, QString("x =  %1").arg(x), "Info ");
  }

At inserting 0.03 for d the result is 5 but should be 0.03 * 2 * 100 = 6 - any solution ?

I'm using:
gcc-3.4.4
CPU AMD Athlon 6,4 ,mmx,mmxext,3dnow,3dnowext

Nylex · 06-02-2006, 08:00 AM

A quick fix would be to multiply d by 100 before converting:

Code:

d *= 100;
x = int(d * 2);

There's probably a better way, though.

graemef · 06-02-2006, 08:45 AM

Are you sure that d = 0.03?
If it had the value 0.029999 then the integer result would indeed be 5.

bastl · 06-02-2006, 09:53 AM

Yes the value d is a KDoubleNumInput and I give in 0.03 and with<br>

Code:

KMessageBox::messageBox ( 0, KMessageBox::Information, QString("x = %1\nd = %2").arg(x).arg(d), "Info ")

I get it confirmed.
But I see at the moment that the factor 2 is in a unsigned int variable so the code should be :

Code:

void test( double d )
  {
  long int x;
  unsigned int ui=2;

  x = int( d * ui * 100 );
  KMessageBox::messageBox ( 0, KMessageBox::Information, QString("x = %1\nd = %2").arg(x).arg(d), "Info ")
 }

I tested all, no it doesn't matter what the position is or whatever you use ( floor, ceil, nearest )
the result is 5 and should be 6.

jim mcnamara · 06-02-2006, 10:24 AM

It's very simple. Start reading here:
http://en.wikipedia.org/wiki/IEEE_754
this site is fun to play with:
http://www.vinc17.org/research/fptest.en.html

.03 cannot be represented correctly and precisely in Floating Point. Period. If you actually dump the contents of the variable and examine it you will see. Try gdb to examine your FP variables. But read about IEEE 754 someplace before you start or it won't make sense.

graemef is absolutely correct.

bastl · 06-02-2006, 11:19 AM

Yes I know about the failure of floating point numbers - that's way I need to convert them.
But this is a failure not in some last positions this is a failure at the first position.
You know that that is not an error of floating point numbers. They are precise in a range of many digits and value and that's were my error occurs. To convert 6.0000 to 6 (int) is no problem but to round 6.4995 to 6 can be one. With floating point numbers you have a deviation maybe of 1/10exp9 but not as big as 20% of your number.

graemef · 06-02-2006, 12:26 PM

Okay let us assume that .03 is held as .0299999, when this number is multiplied by 2 the sum will be .0599998, which may actually be held as .0599999 but when this is multiplied by 100 it will give 5.99999. Converting this to an integer through straight type casting will give 5. In all this calculation the error is negligible.

graemef · 06-02-2006, 12:41 PM

Something for you to ponder...

Code:

void test( double d )
{
  long int x;
  unsigned int ui=2;

  x = long( d * (ui * 100) );
  std::cout << x << std::endl;

  x = long( d * (ui * 10000000) );
  std::cout << x << std::endl;

  double dd = d * ui * 100;
  x = int(dd);
  std::cout << dd << std::endl;
  std::cout << x << std::endl;
}

int main()
{
  test(.03);
  return 0;
}

The results are:

Code:

jim mcnamara · 06-02-2006, 12:46 PM

Since he doesn't want to examine what is in RAM, then it doesn't matter. He won't listen.

graemef · 06-02-2006, 02:37 PM

...but at least you appreciate it

bastl · 06-02-2006, 05:59 PM

Ah here it is:

Code:

double dd = d * ui * 100;
  x = int(dd);

and you guess that the result is 6, but not in my case - it's 5 !!
You can use ceil too, the result is 5 in my case.
Or something like this:

Code:

double dd = d * ui * 10000;
  x = int(dd)/100;

The result is 5 !!
And my FPU is set to round nearest,
so in every case it should be rounded to 6.
Now I want to know if this error is an error of my CPU/FPU type or if others have
this problem too. Then it can be an error of gcc and glibc.
The joke is that QT/KDE prints dd correctly (6.00), but if I convert this value to
integer ( int(dd) ) it is 5 !!!
GCC uses the FPU command FRNDINT (round to integer) for "int(dd)" by default, and now I guess that my
FPU does this work wrong on every third floating point number (0.03, 0,06, 0,09, 0,12...).

At the moment I do the conversion like this:
write the floating point number to a string and read it back as integer is as fast as the FPU command FRNDINT.

graemef · 06-02-2006, 07:58 PM

What do you get when you don't use FRNDINT?

bastl · 06-03-2006, 03:51 AM

To make it clear, FRNDINT is not something off c++ or a function, it's a FPU command and has the machine code D9 FC, c++ only presents it to you as int() or long().
Instead of FRNDINT you can use any rounding function written in your favorite language and the result will be exactly what you wanted to get - 6 .
But it's a very doubtful situation if int() or long() don't convert right to integers.

ppanyam · 06-03-2006, 06:07 AM

I always do it this way

long x = (long)(d * 2 * 100 + 0.5);

This will give desired results almost all cases.

Only those floating point numbers are dealt in exact, which can be repreesented perfectly as

a1/2 + a2/2**2 + a3/2**3 ......a23/2**23

where a1 to a23 are either 1 or 0. For example .75 is ok coz it is 1/2+1/4.

Panyam Prabhakar

graemef · 06-03-2006, 07:47 AM

Quote:

Originally Posted by bastl

To make it clear, FRNDINT is not something off c++ or a function, it's a FPU command and has the machine code D9 FC, c++ only presents it to you as int() or long().

I am aware that FRNDINT is an FPU instruction. So why don't you tell us exactly how you are using it. If you are assuming that the typecasting int() is using it then yes you are correct and it is using it with the rounding flag set to towards zero, hence giving the results in a consistent manner.

If you are using it in a different way then tell us, how else can we assist?