fglrx crashing with large RAM (4GB): How I fixed my MTRR

[hubick]

So I upgraded my system RAM from 2 to 4 gigabytes, and then X refused to start

It took me a lot of researching to figure out what was going on, and to finally fix it - by fuddling with my MTRR settings. There is not a lot of MTRR documentation out there (I even read the source code from the Linux Kernel), so I thought I would post what I came up with in the hopes it might help others.

If you aren't familiar with MTRR's, I will quote the kernel documentation:

On Intel P6 family processors (Pentium Pro, Pentium II and later)
the Memory Type Range Registers (MTRRs) may be used to control
processor access to memory ranges. This is most useful when you have
a video (VGA) card on a PCI or AGP bus. Enabling write-combining
allows bus write transfers to be combined into a larger transfer
before bursting over the PCI/AGP bus. This can increase performance
of image write operations 2.5 times or more.

You can see your MTRR regions by doing a "cat /proc/mtrr" from the command line. These regions are how your system tells the CPU what memory is available, if it can be cached (in L1/L2, for performance), etc.

There are some things to note:

- Your system's memory consists of normal physical 'main' memory, as well as memory address ranges used for talking to your devices.
- A section of 'main' memory is used for your AGP aperature... which the video card uses as additional video memory. It actually can use more system memory than that, but can only directly access the aperature memory on it's own... other data must be swapped/paged in and out of the aperature space by your CPU.
- Your video driver will attempt to use one or more "write-combining" MTRR region entries for talking to the AGP aperature and it's video memory. It will attempt to create write-combining MTRR region entries if they don't already exist (creates them on X startup, removes them on shutdown).
- You can see what memory ranges are used by the devices in your system by doing a "lspci -v".
- Some device's memory, such as your video device's, should be marked as 'write-combining' for performance, where the CPU doesn't do writes to memory instantly, but batches them up together.
- Some device's memory *must* be marked as 'uncachable', where writes must not be cached by the CPU and must be written to main memory immediately. If your system has devices like this marked as cacheable, your system could/will crash.
- Your main 'working' memory is usually covered by a 'write-back' MTRR, which can be cached similar to write-combining, but not combined into batches.
- Your system has 8 MTRR region registers.
- MTRR region sizes must be a power of 2!!!
- You will not see an MTRR region for each device - a region can cover some or all of your memory address space, and overlap and be used by several devices, thoughall those devices must support that MTRR region's cache policy.
- If you mark all your system memory, or memory for a cachable device, as 'uncachable', then your system will become *REALLY* slow (though it should still work).
- MTRR regions can not overlap, except for that an "uncachable" region can be contained *within* a cacheable (write-back, write-combining) one.
- Removing regions in the wrong order can lock your system up... I think a case of this is when removing a 'parent' region whithout first removing the uncachable 'child' it contains.

I don't know what my MTRR table looked like before I upgraded my RAM, but looking at the output of the "dmesg" command, the fglrx driver was now trying to create a write-combining MTRR region entry where an uncachable one already existed. In addition to this, my whole main memory was marked as write-back, meaning no write-combined regions could be created anywhere. So I basically needed to write a script to delete the whole thing and build up a fresh table. My problem was marking the things that needed to be cachable and uncachable as such, but getting the regions laid out in powers of two! The problem was, with 4 gigs of RAM, to get everything the way I needed it, I would have ran out of regions! In the end I got close enough that my system seems to work again, with my additional RAM.

The last problem was that once I knew what commands I needed to execute, where to do it from! I decided to create a batch file, and have it executed early in the boot process by the init program:

#!/bin/sh
# Fixup /proc/mtrr
#
# chkconfig: 12345 0 99
# description: Fixup /proc/mtrr
#
# Save this file as /etc/init.d/mtrrfixup and
# use "chkconfig --add mtrrfixup"
# to have this script run by init when the system starts.
#
# The default mtrr table after boot looks like the following:
# reg00: base=0xd8000000 (3456MB), size= 128MB: uncachable, count=1
# reg01: base=0xe0000000 (3584MB), size= 512MB: uncachable, count=1
# reg02: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
# reg03: base=0x100000000 (4096MB), size= 512MB: write-back, count=1
# reg04: base=0x120000000 (4608MB), size= 128MB: write-back, count=1
# reg05: base=0xd7f80000 (3455MB), size= 512KB: uncachable, count=1
#
# The main problem is entry 1, which covers
# the AGP aperature and Video Card memory.
# Being present and uncachable, it prevents
# the fglrx driver module from using/creating a
# write-combined region for the video card.
# The second problem is entry 2, covering all
# our ram, which prevents us from creating
# any new sections which aren't uncachable.
#
# We have a maximum of 7 mtrr registers.
# All sizes must be a power of two.
# The start and end of any page must
# have the same 'upper bits'.

# First remove all existing values...
# Order matters - wrong order locks system hard!
echo "disable=2" >| /proc/mtrr
echo "disable=5" >| /proc/mtrr
echo "disable=1" >| /proc/mtrr
echo "disable=3" >| /proc/mtrr
echo "disable=4" >| /proc/mtrr
echo "disable=0" >| /proc/mtrr

# Now create the right ones...

# Main Memory: 0-2048,2048-3072,3072-3328,3328-3456
# These are powers of two, they get progressively smaller
# so we can get right up to the system device page below.
echo "base=0x00000000 size=0x80000000 type=write-back" >| /proc/mtrr
echo "base=0x80000000 size=0x40000000 type=write-back" >| /proc/mtrr
echo "base=0xC0000000 size=0x10000000 type=write-back" >| /proc/mtrr
echo "base=0xD0000000 size=0x08000000 type=write-back" >| /proc/mtrr

# System Devices: 3456-3584
echo "base=0xD8000000 size=0x08000000 type=uncachable" >| /proc/mtrr

# AGP Aperature: 3584-3712
echo "base=0xE0000000 size=0x08000000 type=write-combining" >| /proc/mtrr

# Video Card: 3712-3840
echo "base=0xE8000000 size=0x08000000 type=write-combining" >| /proc/mtrr

# High memory area: 4096-5096
echo "base=0x100000000 size=0x40000000 type=write-back" >| /proc/mtrr

Anyhow, after a lot of trial and error (and system lockups, slowdowns, and brokenness) creating that script, I got my system working. Hopefully it can serve as an additional example for others having problems, who need to set up their MTRR regions manually.

[hubick]

I will also provide a link to the other major thread on this issue: Fix for mtrr "overlapping" error.

[traveling]

Thanks for sharing this.