bash free reports wrong memory

Roken · 2019-02-06 22:02:53

I have 16Gb DDR4 installed on my MD, Asus ROG, UEFI.

Issuing

free -h

reports

              total        used        free      shared  buff/cache   available
Mem:           15Gi       1.2Gi        13Gi       102Mi       1.1Gi        14Gi

Where's the extra 1Gb?

BIOS reports two 8192Mb sticks, and memtest86 from a USB stick reports them correctly. What happens in a booted system that 1Gb is missing?

frostschutz · 2019-02-06 22:18:45

so... what does it say without -h?

Also cat /proc/meminfo

Also (perhaps) dmesg | grep memory where I have this

[    0.085036] Reserving Intel graphics memory at [mem 0xdba00000-0xdf9fffff]

and such things.

graysky · 2019-02-06 22:20:14

Shared memory for onboard video?
EDIT: saw in your sig external nvidia card.

Last edited by graysky (2019-02-06 22:20:50)

Ropid · 2019-02-06 22:33:33

It seems PCI devices can reserve memory (or address space?) for their own use. That part of memory is then not available to the OS. I think you can see it in the output of "lspci -vv".

I'm not sure I read the output of "lspci -vv" right, but I'm thinking it's these kind of lines here:

$ lspci -vv | grep -i 'memory at'
...
	Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at f0000000 (64-bit, prefetchable) [size=2M]
...

I just came up with the following to try to sum all those values up:

$ lspci -vv | perl -nE '/memory at/i and /\[size=(.*?)\]/ and say $1' | numfmt --from iec | perl -nE '$size += $_ }{ say $size'
271163664

If I now look at the output in bytes of "free" and add that number to it, I get pretty close to 16 G:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       2.2Gi       4.1Gi       1.1Gi       9.2Gi        11Gi
Swap:          15Gi          0B        15Gi

$ free -b
              total        used        free      shared  buff/cache   available
Mem:    16777646080  2402045952  4444409856  1195761664  9931190272 12833669120
Swap:   17179865088           0 17179865088

$ bc -l <<< '(16777646080 + 271163664) / 2^30'
15.87794138491153717041

Trilby · 2019-02-07 00:14:43

16GB is very close to 15GiB: 16 * 1000^3 / 1024^3 = 14.9

edit: oops, this was basically covered in the previous post.

Last edited by Trilby (2019-02-07 00:15:51)

seth · 2019-02-07 08:36:21

Tempting, but

Roken wrote:

BIOS reports two 8192Mb sticks

(and this kind of HDD marketing is rather untypical for RAM)

=> post dmesg

mpan · 2019-02-07 11:49:09

It takes some RAM to hold the kernel and, on top of that, the kernel need keep some free RAM for itself to ensure that it will not run out of it.

See:

dmesg | grep -i 'Memory: [0-9]\+K/[0-9]\+K available'

The “reserved” is probably where your RAM is gone.

A side note: decimal prefixes are not HDD marketing. It is the other way around. Everything uses decimal prefixes, but RAM sizes are usually a power of two. For convenience their sizes were, however, still using decimal prefixes, since the error from approximation was low (just few percent for early memories).

seth · 2019-02-07 12:59:38

On a sidenote: bytes were originally never counted in SI but in 2^n. At some point HDD vendors started to sell disks w/ SI numbers (because they're bigger - they can tell what they want but there's no way they would have done that the other way round) and users all across the planet were upset because widows, linux and every existing OS reported "less" size than they paid for.
The problem didn't exist for baud rates, because modem speeds were so unimpressive that they were instantly marketed in bits/s (and SI)

I'm actually old enough to remember that shit…
Great. Now I'm depressed.

mich41 · 2019-02-07 14:20:34

This is a recurring topic, complete with the flamewars over whether RAM is decimal-cheated or not (it's not) and whether decimal-cheating is cheating or not (it is, seth explains why).
https://bbs.archlinux.org/viewtopic.php?id=204078

15Gi

First of all, the real number probably is 15xxx MB and we don't really know how much.
Secondly, I believe 1/64 of all memory is eaten up by the mem_map array which describes the contents of the rest, so write it off.

[    0.000000] On node 0 totalpages: 521758
[    0.000000] free_area_init_node: node 0, pgdat 80986cc0, node_mem_map f680e020
[    0.000000]   Normal zone: 3824 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 489372 pages, LIFO batch:31
[    0.000000]   HighMem zone: 32386 pages, LIFO batch:7

Here the number is 1/128 because a 32-bit system.

Then you may lose 64 megs on AMD systems.

[    0.000000] AGP: Node 0: aperture [bus addr 0x7f02000000-0x7f03ffffff] (32MB)
[    0.000000] Aperture beyond 4GB. Ignoring.
[    0.000000] AGP: Your BIOS doesn't leave an aperture memory hole
[    0.000000] AGP: Please enable the IOMMU option in the BIOS setup
[    0.000000] AGP: This costs you 64MB of RAM
[    0.000000] AGP: Mapping aperture over RAM [mem 0xd4000000-0xd7ffffff] (65536KB)

Don't get too excited about Intel, unless you think bounce buffers are better than GART for accessing 32bit DMA PCI devices. If you do, the "GART IOMMU" can be disabled, by the way.

I presume kernel code and static data aren't counted above too, so correct for those.

[    0.000000] Memory: 2059280K/2087032K available (6238K kernel code, 509K rwdata, 2012K rodata, 548K init, 320K bss, 27752K reserved, 0K cma-reserved, 129544K highmem)

Some mem is reserved by BIOS. I'm not sure if all ranges listed as reserved here must be RAM, however. I think some may be MMIO. But most are, most of the time.

[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e6000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000dfe8ffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000dfe90000-0x00000000dfe9dfff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000dfe9e000-0x00000000dfedffff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000dfee0000-0x00000000dfefffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fff00000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000217ffffff] usable

Memory permanently assigned to the integrated GPU, if any, isn't counted above. So add that separately if you have an iGPU.

At this point your count should be within megabytes of the expected amount (16384 MiB for the OP).

I once naively tried to read through dmesg and account for every single kB. Impossible without understanding kernel internals.

seth wrote:

At some point HDD vendors started to sell disks w/ SI numbers

The exact point was 40GB IIRC. No vendor did it before, every vendor does it since. Total coincidence driven by a desire to abide by international metrology standards.

seth wrote:

The problem didn't exist for baud rates, because modem speeds were so unimpressive that they were instantly marketed in bits/s (and SI)

Nah, that's tinfoil hat territory IMO. RS232 comes from times when even byte being 8 bits wasn't entirely a sure thing. I think nobody cared, they just picked some round number of bits per second that was easy to write on a spec sheet.
But PC disks are a different story.

Roken · 2019-02-07 20:19:24

Thanks, all. I just tested again and am getting a slightly different result. If I bc, I'm now getting 15.66 Gb. I guess it's probably the PCI bus reservation.

@mpan - reserved is only 8K, so I'm guessing not that.

Arch Linux

#1 2019-02-06 22:02:53

bash free reports wrong memory

#2 2019-02-06 22:18:45

Re: bash free reports wrong memory

#3 2019-02-06 22:20:14

Re: bash free reports wrong memory

#4 2019-02-06 22:33:33

Re: bash free reports wrong memory

#5 2019-02-07 00:14:43

Re: bash free reports wrong memory

#6 2019-02-07 08:36:21

Re: bash free reports wrong memory

#7 2019-02-07 11:49:09

Re: bash free reports wrong memory

#8 2019-02-07 12:59:38

Re: bash free reports wrong memory

#9 2019-02-07 14:20:34

Re: bash free reports wrong memory

#10 2019-02-07 20:19:24

Re: bash free reports wrong memory

Board footer