Reading results: display and precision

This page explains how Varlig shows the answer beside each line: how many decimal places you see, when large numbers are shortened, how money is rounded, what each kind of answer looks like, and how exact the numbers behind the answers are. Read it from the top as a guide, or jump to a table when an answer looks unexpected. The limits on what a single line can do are listed near the end.

Stored values and displayed answers

Every line works in two steps. First the calculation runs and keeps the full value. Then the answer is formatted so it’s easy to read, which can mean rounding it. The rounding only affects what you see.

When a later line uses a name, it uses the full value, not the rounded digits:

calcCopy

share = 250 / 3 gives 83.3333333333

share * 3 gives 250

If you type the digits you see into a new line, you get the rounded number instead. Use the name to keep full precision.

Decimal places

Answers show up to ten decimal places, with the last digit rounded. Trailing zeros are left off, so you see 0.75 rather than 0.7500000000.

calcCopy

2 / 3 gives 0.6666666667

1 / 8 gives 0.125

Conversions through an ingredient’s density show at most nine significant digits, the precision of the density itself, so 1 cup sugar in grams shows 200 g. The stored value keeps every digit.

To show a different number of decimal places, change the default in Settings, where the calculation options accept 0 to 15. To change a single line, use the answer formatting options in that line’s context menu. Both change only the display.

To round the value itself, write to 2 dp. The rounded value is what later lines use. Rounding goes half away from zero, so 2.345 to 2 dp is 2.35 and round(-2.5) is -3.

calcCopy

exact = 2 / 3 gives 0.6666666667

approx = 2 / 3 to 2 dp gives 0.67

exact * 3 gives 2

approx * 3 gives 2.01

For more ways to round, see Rounding.

Money amounts

Money follows the rules for its currency rather than the ten-decimal default:

• Rounded to the currency’s precision. Most currencies show two decimal places, yen shows none and Kuwaiti dinars show three. The currency reference lists them all.
• Trailing zeros are kept. $15 shows as $15.00.
• Small positive amounts keep their digits. An amount under one cent, such as $0.004, isn’t rounded to $0.00.
• Prices per unit keep up to four decimal places as you wrote them, so a tariff of £0.075/kWh or a pump price of £1.499/L shows in full. Amounts written in pence or cents do the same, so 6.2p shows £0.062 and 24.5p/kWh shows £0.245/kWh, while a total worked out from them comes to the penny: 148.9p/L * 45 L is £67.01. With more decimals, a price under £1 shows four significant digits (£0.12345/kWh shows £0.1235/kWh), and other prices, such as £8/kg / 3, are rounded to the penny.
• Cryptocurrencies show up to ten decimal places, like plain numbers.

A line that assigns money to a name shows the amount the same way as any other line. The name keeps the full value, so later lines that use it stay exact.

calcCopy

£250 / 3 gives £83.33

bill = £250 / 3 gives £83.33

bill * 3 gives £250.00

¥1,250 / 4 gives ¥313

£0.075/kWh gives £0.075/kWh

£8/kg / 3 gives £2.67/kg

The currency options in Settings let you turn off rounding to currency precision, show codes instead of symbols (15.00 USD rather than $15.00), or drop trailing zeros ($15 rather than $15.00).

Large and small numbers

From 100,000 upwards, plain numbers are shortened with a letter: k for thousand, M for million, G for billion (a thousand million), T for trillion and P for quadrillion. Very large and very small numbers switch to scientific notation.

Money is shown in full, to the penny, below a million, so a shortened figure never hides part of an amount you pay or earn: £199,999.98 stays £199,999.98. From a million, money is shortened too, with B where plain numbers use G, and at most three decimal places.

calcCopy

250000 gives 250k

£99,999.99 * 2 gives £199,999.98

$1234567.89 gives $1.235M

8,100,000,000 gives 8.1G

$1,234,567,890 gives $1.235B

0.0000001 gives 1e-7

The letter changes only the display. 1.5 million is stored as 1,500,000 and shows as 1.5M, and 3bn shows as 3G. Below 100,000, answers use thousands separators (99,999).

A number with a unit is always written out in full, because a letter next to a unit would read like a prefix: 175,200 Wh could otherwise look like kilowatt-hours. Only values above 10¹⁷ or below 0.000001 switch to scientific notation.

calcCopy

175200 Wh gives 175,200 Wh

1 light year in km gives 9,460,730,472,580.8 km

In Settings you can turn off the short forms, which writes numbers out in full, or turn off thousands separators.

Kinds of answer

Each kind of value has its own format. This table shows one or two examples of each.

A few formats depend on today’s date. A date in the current year leaves the year off, and dates close to today can show as a relative word:

calcAnswers depend on today’s date or live dataCopy

March 12 gives March 12

tomorrow at 1am gives Tomorrow at 1:00 AM

Two other answers are worth knowing:

• No answer and no error means the line has nothing to show: a comment, a blank line, a function definition, or an unfinished line such as total =.
• pending… means the line needs data that isn’t available yet.

How exact answers are

Exact arithmetic

Whole numbers, decimals and fractions are stored exactly. Adding, subtracting, multiplying, dividing and raising to whole-number powers keep them exact, including fractions such as ⅓ that have no exact decimal form. Comparisons with == check the exact values, with no rounding.

calcCopy

0.1 + 0.2 == 0.3 gives true

1/3 + 1/3 + 1/3 == 1 gives true

Square roots are exact when the root can be written exactly, and otherwise accurate to 38 significant digits (the extra digits are cut off, not rounded). Standard deviation uses the same square root. gcd and lcm are exact, never negative, and work on fractions without rounding them first.

calcCopy

sqrt(2.25) gives 1.5

sqrt(1e-64) gives 1e-32

gcd(12, 18) gives 6

Approximate results

Some operations can’t be done exactly, so they use binary floating point, which is accurate to a limited number of digits:

• trigonometry, logarithms and exponentials
• powers that aren’t whole numbers, such as 2^0.5, and scientific literals with a fractional exponent, such as 1e2.5
• some unit conversions
• solving equations, numerical derivatives and integrals
• matrix decompositions and eigenvalues
• functions of complex numbers beyond basic arithmetic

Because == compares exact values, two approximate results can look the same and still not be equal. Compare the difference with a small tolerance instead:

calcCopy

sqrt(2)^2 gives 2

sqrt(2)^2 == 2 gives false

abs(sqrt(2)^2 - 2) < 0.000000001 gives true

Choose a tolerance that suits the size of your numbers. Ten decimal places on screen don’t guarantee ten correct decimal places, especially for difficult numerical problems.

In lists, matrices and complex numbers, a part that is only floating-point round-off, far smaller than the values around it, shows as 0. That keeps answers such as a matrix decomposition or exp(i * pi) readable. The stored value keeps its digits, so the same tolerance check applies:

calcCopy

exp(i * pi) gives -1

abs(exp(i * pi) + 1) < 0.000000001 gives true

Number size

• Decimal numbers are stored as a whole number of up to 2¹²⁸−1 (about 38 digits) multiplied by a power of ten from 10⁻¹²⁸ to 10¹²⁷. Numbers beyond 1e127 still work while their digits fit.
• Fractions can have a numerator and denominator of up to 4,096 bits each.
• Adding a tiny number to a huge one can lose the tiny part: 1e30 + 0.001 shows 1e30.
• Going out of range when multiplying or raising to a power shows Error: ∞, whether the result is too large or too small. The calculation never wraps round to a wrong value.

calcCopy

1e30 + 0.001 gives 1e30

1e100 * 1e100 gives Error: ∞

Limits

Each line has to stay within these limits. Hitting one shows an error on that line only, usually Error: calculation limit, and the rest of the note keeps working. You never get a partial sum or an unfinished result presented as an answer. To fix it, make the range, matrix, nesting or problem smaller.

Varlig Calc doesn’t currently support general symbolic algebra, indefinite integrals, systems of non-linear equations, historical cryptocurrency prices, dice notation such as 2d6, arithmetic with uncertainties such as 10 ± 0.5, or chained comparisons such as 1 < x < 5. For more on how the advanced maths features work, see Advanced maths.

Putting it together

Here’s a trip budget that uses most of the rules on this page:

calcCopy

flights = £1,284.60 gives £1,284.60

hotel = £96.50 * 4 gives £386.00

total = flights + hotel gives £1,670.60

each = total / 3 gives £556.87

each * 3 gives £1,670.60

each / total as % gives 33.3333333333%

each / total as fraction gives ⅓

total to 0 dp gives £1,671.00

total * 1000 gives £1.671M

• each = total / 3 shows the share rounded to pence, while the name keeps the full value.
• each * 3 uses that full value, so it comes back to exactly £1,670.60 rather than £1,670.61.
• The percentage isn’t money, so it shows ten decimal places. As a fraction it’s exact.
• to 0 dp rounds the value itself, and the trailing zeros stay because it’s money.
• Multiplying by 1,000 passes 1,000,000, so the answer is shortened to £1.671M.