Main Board > Language

Rounding

(1/1)

Marty:
Why are these answers different - Pvx+ V9.00

->floatingpoint
->round off
->set_param 'rn'=0
->set_param 'nr'
->set_param 'rs'=0
->? (2/6374)-(1/6557.5)
-.152497140678612276E-03
->a=(2/6374)-(1/6557.5)
->? a
 .1612775690797811975E-03
->

chrisk:

Print followed by parenthesis is interpreted as you passing a channel number. That means the first part of your formula (2/6374) is evaluated to zero. So the remainder of the print ‘-(1/6557.5)’ is output to screen.

You can fix this by putting a 0+ in front of the formula:

->? 0+(2/6374)-(1/6557.5)   
 .1612775690797811975E-03   

Marty:
Thanks Chris.

Navigation

[0] Message Index