Skip to content

Unify coerce prec calculation#394

Merged
tompng merged 1 commit intoruby:masterfrom
tompng:coerce_prec_refactor_fix
Jul 29, 2025
Merged

Unify coerce prec calculation#394
tompng merged 1 commit intoruby:masterfrom
tompng:coerce_prec_refactor_fix

Conversation

@tompng
Copy link
Copy Markdown
Member

@tompng tompng commented Jul 29, 2025

Fixes #388
Unify coerce logic in add/sub/mult/div/coerce/cmp.
There are some differences. Unify them to div's precision calculation because add/sub/mult's calculation coerce Rational with low precision.

# div
BigDecimal(1) / (3/7r) # Same as `BigDecimal(1) / BigDecimal(3/7r, 0)`
# => 0.233333333333333333333333333333332555555555555556e1 # ok
BigDecimal(1).div(3/7r, 50) # Same as `BigDecimal(1).div(BigDecimal(3/7r, 50), 50)`
# => 0.23333333333333333333333333333333333333333333333333e1 # ok

# add, sub, mult
BigDecimal(1) + (3/7r)
# before: 0.1428571429e1 (too low)
# after: 0.142857142857142857142857142857143e1 
BigDecimal(1).add(3/7r, 50)
# before: 0.1428571429e1 (too low)
# after: 0.14285714285714285714285714285714285714285714285714e1

Cmp with rational will change. The original behavior was weird.

10/3r == BigDecimal(10/3r, prec)
# before: always false
# after: always false

1/3r == BigDecimal(1/3r, prec)
# before: true when prec == 9*n (n>=1)

# after: true when prec == 9*n (prec >= 2*double_fig)

@tompng tompng force-pushed the coerce_prec_refactor_fix branch from 064052a to 263ceb9 Compare July 29, 2025 18:49
Use div's precision calculation because add/sub/mult's calculation coerce Rational with low precision.
`a.add(rational, prec)` will now coerce Rational with `Max(prec, 2*BIGDECIMAL_DOUBLE_FIGURES)` precision.
Minimum precision(2*BIGDECIMAL_DOUBLE_FIGURES) is the minimum precision of BigDecimal(rational, 0)
@tompng tompng force-pushed the coerce_prec_refactor_fix branch from 263ceb9 to 9a37497 Compare July 29, 2025 19:11
@tompng tompng merged commit d458433 into ruby:master Jul 29, 2025
79 checks passed
@tompng tompng deleted the coerce_prec_refactor_fix branch July 29, 2025 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rational to BigDecimal conversion precision too low in add, sub and mult

1 participant