Excellent Article on Memory Limitations of 32-Bit Windows

[Dr. Zhivago]

This goes beyond any other explanation I have ever seen. The limitation seems to be imposed by MS themselves. Highly technical article, with full explanations and proof of concepts.

Though machines with 4GB are not yet the typical purchase for home or business use, they are readily available from major manufacturers and it won’t be long before they are the typical purchase. But there are problems. You don’t have to stand for long in a computer shop to hear a sales assistant talk of 4GB as some sort of limit for 32-bit operating systems, and it won’t be long before this sales patter develops into outright promotion of 64-bit Windows as the only way to get past this limit. Some sense of this can be seen already in manufacturers’ advertising materials, as in the following fine print from Dell:

The total amount of available memory will be less than 4GB. The amount less depends on the actual system configuration. To fully utilise 4GB or more of memory requires a 64-bit enabled processor and 64-bit operating system, available on selected systems only.

Let me stress now that I do not complain about Dell’s statement. Its first two sentences are correct for all 32-bit editions of Windows Vista exactly as configured by Microsoft and installed by Dell. In the last sentence, I might quibble that the talk of a 64-bit processor is superfluous since the machine on offer does have such a processor, but otherwise the sentence is correct because of the word fully. Yet although Dell’s statement is true, it is not the whole truth: there is something that Microsoft does not tell you, and perhaps does not tell Dell.

That 32-bit editions of Windows Vista are limited to 4GB is not because of any physical or technical constraint on 32-bit operating systems. The 32-bit editions of Windows Vista all contain code for using physical memory above 4GB. Microsoft just doesn’t license you to use that code.

Well, to say it that way is perhaps to put words in Microsoft’s mouth. I say the restriction to 4GB is a licensing issue because that’s how Microsoft’s programmers evidently have thought of it. The 4GB limit is retrieved from the registry by calling a function named ZwQueryLicenseValue, which is itself called from an internal procedure which Microsoft’s published symbol files name as MxMemoryLicense. If you remove this check for the licensed memory limit then a restriction to 4GB is demonstrably not enforced by other means. Yet I must admit that I have not found where Microsoft says directly that 32-bit Windows Vista is limited to 4GB only by licensing. The supposed License Agreement doesn’t even mention the word memory. What, really, is going on?

Source: Geoff Chappell

[caveman-jim]

sans the obvious 'Microsoft is screwing you! They should give you their IP for free!' comments, an excellent article.

If you were fortunate enough to have 4GB in a machine for running a client version of Windows up to and including Windows XP SP1, and you had memory remapping so that you had the use of 4GB in total, then you will have faced an unfortunate side-effect when upgrading to Windows XP SP2: you will have bought a downgrade of how much RAM Microsoft permits you to use. How well did Microsoft disclose this side-effect in advance?

It was disclosed eventually. The Knowledge Base article The amount of RAM reported by the System Properties dialog box and the System Information tool is less than you expect after you install Windows XP Service Pack 2 reports that enabling PAE in Windows XP SP2 does not allow the use of memory above 4GB. It also talks of changes to the HAL. What actually was changed so strains credibility that I had to be goaded into studying it. As much as anyone talks of 32-bit drivers that simply assume a 32-bit physical address space, it will forever remain that the extreme example of such an assumption is what Microsoft itself coded into the HAL for Windows XP SP2.

Where the Knowledge Base article talks of “unlimited map registers”, what it means is that the HAL itself assumes double buffering is never required for any DMA operations on a device that can handle 32-bit physical addresses.10 In other words, having seen that some 32-bit drivers assume all physical addresses fit 32 bits and thereby think to escape some coding obligations regarding double buffering for their 32-bit device, Microsoft deliberately adopted the same faulty assumption into the HAL, i.e., into arguably the second most critical executable in all of Windows. A company that can do such a thing is clearly not thinking primarily of its product’s technical integrity.

He has obviously not had a lot of experience white-boxing PC's in the early XP days.

consider Windows Server 2008. For the loader and kernel in Windows Vista SP1 (and, by the way, for the overwhelming majority of all executables), the corresponding executable in Windows Server 2008 is exactly the same, byte for byte. Yet Microsoft sells 32-bit Windows Server 2008 for use with as much as 64GB of memory. Does Microsoft really mean to say that when it re-badges these same executables as Windows Vista SP1, they suddenly acquire an architectural limit of 4GB? Or is it that a driver for Windows Server 2008 is safe for using with memory above 4GB as long as you don’t let it interact with the identical executables from Windows Vista SP1?

Many companies will not support server instances using non-Server certified hardware and drivers. Those that use a unified installer and driver, such as intel, broadcom etc. have the resources to make sure their product performs as expected. Others, such as, say, Creative do not and will not support their hardware and drivers on Server 2008 and vice versa- you can't get support for Server 2008 if you use non-supported hardware and drivers. It's all about licensing, Microsoft will admit to that.

[Dr. Zhivago]

Yah, it is a bit MS bashing. But what do you mean by the white-boxing statement you made?

[Android1]

Yah, it is a bit MS bashing. But what do you mean by the white-boxing statement you made?

He is referring to PCs built using off-the-shelf parts. Great article, by the way!

[Dr. Zhivago]

He is referring to PCs built using off-the-shelf parts. Great article, by the way!

I know that, but I'm not following WHY he made that statement.

[Dr. Zhivago]

By the way, here is a table listing all the versions of Windows that support PAE and how much physical memory they support:

Link

[Android1]

I know that, but I'm not following WHY he made that statement.

Oh, I see. I was asking myself the same question.

[caveman-jim]

I remember many a white-box build on 2000 & XP that once you loaded drivers was flaky as hell. You had a choice of the original driver which performed like crap and didnt have all the features or the newest driver which fell over every 10 minutes. From his comments he doesn't understand why people cringe when talking about Windows drivers, or why people credit Apples success in part to their closed hardware platform.

I think most of the reason MS instituted WHQL and memory model limiting was to increase performance and reliability to counter claims from Apple.

[Uber_Tiny]

I remember many a white-box build on 2000 & XP that once you loaded drivers was flaky as hell. You had a choice of the original driver which performed like crap and didnt have all the features or the newest driver which fell over every 10 minutes. From his comments he doesn't understand why people cringe when talking about Windows drivers, or why people credit Apples success in part to their closed hardware platform.

I think most of the reason MS instituted WHQL and memory model limiting was to increase performance and reliability to counter claims from Apple.

I agree with your intuitive statement.

Good article too.

[Dr. Zhivago]

I remember many a white-box build on 2000 & XP that once you loaded drivers was flaky as hell. You had a choice of the original driver which performed like crap and didnt have all the features or the newest driver which fell over every 10 minutes. From his comments he doesn't understand why people cringe when talking about Windows drivers, or why people credit Apples success in part to their closed hardware platform.

I think most of the reason MS instituted WHQL and memory model limiting was to increase performance and reliability to counter claims from Apple.

I agree with your intuitive statement.

Good article too.

Me 2. I say the same thing myself quite often. For some reason I wasn't making the connection with what you quoted and your statement.

Thanks for the explanation Jim!

Glad you enjoyed the article Uber.

[damicatz]

That article was a load of hogwash. Microsoft is not out to get you by limiting PAE on 32-bit consumer-level Windows. What possible reason would they have to provide a license that's good for 32-bit and 64-bit versions of Windows but then limit 32-bit Windows to 4 gigabytes of address space for "licensing reasons"? If they wanted to make you pay more to use more RAM, they would have charged extra for the 64-bit version.

1.Every computer has what's known as a physical address space. An address space is a collection of memory addresses. For the purpose of this discussion, the physical (memory) address space of a standard 32-bit system (or a 64-bit system running in legacy mode; that is, running a 32-bit operating system) contains roughly 4 gigabytes worth of addresses.

2.For the purpose of this discussion, one physical address corresponds to a byte (or 8 bits) of data. Therefore, every byte of RAM needs its own address.

3.In addition, other things need physical address space as well. For example, your video card uses physical address space. Every byte of RAM you have on your video card requires an equivalent amount of physical address space. In addition, there are a number of legacy holdovers that take up address space. If you have an X-Fi model with the onboard memory, that requires address space as well.

4.A lot of hardware needs physical addresses (I/O Addresses) in order to enable communication between the CPU and the hardware. With the x86, there is actually a separate address space for this but it is only 64,000 addresses in size (and this drops down to 16,000 usable addresses if you use 32-bit addresses). When you run out of space here, you have to do something called memory-mapped I/O where you use addresses from the address space used by RAM. This combined with #3 eats out of your physical address space and limits the total amount of RAM your system can see.

5.Most recent (<10 years old) IA32 processors (32-bit processors) have support for something known as PAE.

6.With PAE, additional address lines are added to the CPU and when PAE is enabled, the CPU can gain the ability to use 36-bit physical addresses.

7.On the other side of the fence we have virtual address spaces. Pretty much every modern operating system abstracts programs from the hardware. In the DOS days, programs dealt directly with the BIOS and directly with the physical address space.

8.Virtual address spaces allow each program to have it's own address space separate from other programs. When a program needs to store something in physical memory, the operating system maps virtual addresses in the processes's address space into the physical address space. In addition, this virtual address space is used for other things such as memory-mapped file IO where you map out a file into your address space and access it as if it was part of memory.

9.Virtual addresses can be redirected to almost anywhere on the local system, which is what allows things like a page file and memory-mapped file IO. The program is unaware of this; to the program all it sees is one address space. The operating system takes care of the work of mapping things transparently.

10.On a 32-bit x86 processor and a 64-bit x86 processor running in legacy mode, all virtual address spaces use 32-bit virtual addresses.

11.All enabling PAE does is increase the size of the physical addresses to 36-bits. Virtual addresses remain the same size.

12.In Windows, a program's virtual address space is split into two. The lower 2gb of the address space is available to a process and the upper 2gb of space is reserved for the kernel. In programming terms, this means that memory addresses with the upper (32nd) bit clear are reserved for use by the program and those that use the 32nd bit are reserved for the kernel.

13.This behavior can be changed by using the /3gb switch. This takes 1 gigabyte of the upper 2 gigabytes (one half of the 32nd bit) and allocates it the program. This is generally not a great idea because the upper half is used for things like accessing video memory and (especially if you have video card(s) with lots of RAM) can break things.

By default, Windows will limit the amount of address space a 32-bit program can use to 2 gigabytes, even with the /3gb switch. Because with 2 gigabytes of available address space, Windows ignores the upper bit on an address, dummy programmers decided that it would be a great idea to use the upper bit as a storage place without regards to what might happen in the future. If programs written like this start suddenly getting addresses that use the upper bit, it can cause breakage.

In order for Windows to allow the program to go above 2 gigabytes, it has to be linked with the largeaddressaware flag.

14.PAE is only used in Windows XP to enable the NX-bit (due to it's location in the page table). Windows XP will keep the physical address space at 32-bits.

15.This is not due to licensing or Microsoft being greedy. This is done because of driver issues. A lot of less than competent programmers hard-coded their drivers with the assumption that the physical address space would always be the same size as the virtual address space. When drivers that aren't expecting 36-bit addresses get them, unexpected consequences can result. Using Geoff's kernel patch WILL MOST LIKELY BREAK THINGS.

The reason why Windows Server fully supports PAE is because drivers for Windows Server undergo a much more extensive validation process to make sure they don't do stupid things that would cause them to break if the memory address size changed.

16.#15 not withstanding, if PAE is used (on a supporting operating system), all it does is make it so that the operating system can use more RAM. Remember that the virtual address spaces are still 32-bits and you still have #12 and #13 to deal with (regarding the 2 gigabyte limit).

17.Contrary to what the article says, if a program wants to make use of more than 2 (or 3) gigabytes of RAM, it has to use an API called Address Windowing Extension (AWE). This is basically a software equivalent to the expanded memory of yore. Essentially, while the 32-bit program cannot use more than 2 (or 3) gb of memory at a time, it can swap parts of it's address space in and out on demand to use more than 2-3gb total (but never more than 2-3gb at one time).

18.Pretty much no consumer-level program is going to be coded with AWE support because it's a difficult API to use and it adds overhead.

All of this nonsense is eliminated by migrating to a 64-bit operating system. Quite frankly, Microsoft's biggest mistake was releasing a 32-bit version of Windows 7. It's time to force the issue. It's very irritating when you have an app like Google Earth, that uses gobs of memory, and that crashes due to running out of address space because it's 32-bit and not large address aware.

If your computer isn't 32-bit, then chances are, it's not powerful enough that you should be running Windows 7. The state of 64-bit Windows right now is a joke. Besides the decision to use an LLP64 data model, there is almost no 64-bit software (and there are still companies that release new hardware without 64-bit drivers). Even Microsoft has still released software without 64-bit support (how long did it take for Windows Home Server to get full 64-bit support?).

[caveman-jim]

Did you just decide to reword the article? He says himself it'll break things, not that it's a way to circumvent the licensing.

[damicatz]

Did you just decide to reword the article? He says himself it'll break things, not that it's a way to circumvent the licensing.

I did not reword his article. I corrected it.

He implies that Microsoft is abusing their monopoly and being anti-competitive by limiting memory on 32-bit Windows. This is patently absurd.

He also states that PAE doesn't require applications to be recoded to benefit from it; this is not true. For full effect, an application needs to use AWE in order to use more than 2/3gb of RAM as would be provided to PAE. Otherwise, it's still limited to the same amount of RAM as it would be without PAE. The only benefit without AWE-coding is that the combined total usage of RAM amongst all processes can exceed 4 gigabytes (and if you find yourself in that situation frequently, you should be using a 64-bit operating system instead.)

He then goes on to state various conspiracy theories under transparacy about why he thinks evil Microsoft won't let you use more than 4 gigabytes of RAM on a 32-bit OS.

Sorry but I don't buy it. Again, Microsoft has no reason to limit 32-bit Windows to 4 gigabytes of physical address space for "anticompetetive" reasons when that same license gives you the right to use the equivilant 64-bit version of Windows and go well above 4gb of address space.

[Dr. Zhivago]

Did you just decide to reword the article? He says himself it'll break things, not that it's a way to circumvent the licensing.

That's what it seems like. All the stuff he wrote in his post is covered in the article...

[caveman-jim]

I did not reword his article. I corrected it.

He implies that Microsoft is abusing their monopoly and being anti-competitive by limiting memory on 32-bit Windows. This is patently absurd.

He also states that PAE doesn't require applications to be recoded to benefit from it; this is not true. For full effect, an application needs to use AWE in order to use more than 2/3gb of RAM as would be provided to PAE. Otherwise, it's still limited to the same amount of RAM as it would be without PAE. The only benefit without AWE-coding is that the combined total usage of RAM amongst all processes can exceed 4 gigabytes (and if you find yourself in that situation frequently, you should be using a 64-bit operating system instead.)

He then goes on to state various conspiracy theories under transparacy about why he thinks evil Microsoft won't let you use more than 4 gigabytes of RAM on a 32-bit OS.

Sorry but I don't buy it. Again, Microsoft has no reason to limit 32-bit Windows to 4 gigabytes of physical address space for "anticompetetive" reasons when that same license gives you the right to use the equivilant 64-bit version of Windows and go well above 4gb of address space.

Ah, so you haven't read the thread either. okey-dokey.

[HAL10000]

I think most of the reason MS instituted WHQL and memory model limiting was to increase performance and reliability to counter claims from Apple.

It was more like developers regularly took "liberties" in writing PC drivers and doing such things as direct resource access (totally dead now because of nx bit and ASLR).

[caveman-jim]

True, but that's not contrary to my speculation either, could be a cumulative effect of both (weighting is a matter of debate, though).

[Geoff Chappell]

17.Contrary to what the article says, if a program wants to make use of more than 2 (or 3) gigabytes of RAM, it has to use an API called Address Windowing Extension (AWE).

That's plainly unfair. At best you can say the article is silent about how a program may use more memory than fits in its virtual address space. You have no grounds at all for saying "contrary to what the article says". In no sense does the article say that PAE lets a program use more memory than fits in its virtual address space.

The article's subject is that 32-bit Windows (and Windows programs) can use _physical_ memory above 4GB. Nowhere does the article talk about individual programs using more than 2-3GB of RAM. It's just not the article's subject.

The article even has it in highlights that 32-bit Windows can't give more than 3GB (of linear address space, in the context of the containing paragraph) to any one process. Did you miss that? Or are you being mischievous?

All of this nonsense is eliminated by migrating to a 64-bit operating system.

The article explicitly does not advocate one way or the other. What it aims to do is put on record that 32-bit Windows Vista has all the code for using physical memory above 4GB by 32-bit Windows Vista but is restrained from using it. (Indeed, Windows Vista is the first 32-bit Windows for which this is true.) Whether you care is up to you. Many take the view that there's no harm and therefore no foul. Many others feel a bit deceived - and not just by Microsoft.

Microsoft is not out to get you by limiting PAE on 32-bit consumer-level Windows. What possible reason would they have to provide a license that's good for 32-bit and 64-bit versions of Windows but then limit 32-bit Windows to 4 gigabytes of address space for "licensing reasons"? If they wanted to make you pay more to use more RAM, they would have charged extra for the 64-bit version.

You do read into things a lot.

The article does not say Microsoft is "out to get" anyone or to "make you pay more to use more RAM". What it does say is that Microsoft "is not exactly forward in describing this mechanism by which 32-bit Windows Vista is restricted to 4GB" and that some non-technical consideration must be involved since technical reasons do not themselves explain the implementation.

The article speculates that the non-technical consideration is marketing, i.e., that a perceived 4GB limit to 32-bit operating systems may help the faster take-up of 64-bit Windows. I should think it obvious that Microsoft and much of the industry sees this faster take-up as desirable in and of itself. Nothing in the article suggests that Microsoft makes more money directly from this faster take-up.

What is suggested is an attitude in which anything that helps move the masses is welcome, and not just by Microsoft, even if it comes with a sleight of hand about what 32-bit Windows actually can do - or could, with suitable license data. That Microsoft makes the take-up easier, e.g., by providing a 64-bit upgrade as nearly free, in no way invalidates this point and may actually enhance it.

For what financial benefit this might bring Microsoft, the article at best speculates that it "brings forward a cycle of upgrades" as users quite naturally decide that having changed to a 64-bit operating system, they may as well get 64-bit applications (rather than keep their old 32-bit applications). But that's about it.

15.This is not due to licensing or Microsoft being greedy.

As with "licensing reasons" above and "due to licensing" here, I can't imagine what either could mean. The article's claim about licensing is that it provides the _mechanism_ of the restriction, which claim is proven beyond debate by the explanations (of the algorithm for deciding what memory is usable) and the demonstration (i.e., disable the licensing mechanism and nothing stops Windows from trying to use memory above 4GB).

This is done because of driver issues. A lot of less than competent programmers hard-coded their drivers with the assumption that the physical address space would always be the same size as the virtual address space. When drivers that aren't expecting 36-bit addresses get them, unexpected consequences can result.

So the article says, too.

However, the article also attempts to evaluate what danger this presents in practice. It reasons that as much as driver issues ever were good for _forcibly_ restricting consumer versions of 32-bit Windows to memory below 4GB, they're highly implausible for Windows _Vista_.

If you're going to get 64-bit Windows Vista running on a computer with memory above 4GB, then 32-bit Windows Vista will almost certainly use memory above 4GB without trouble (though you may have to track down updated 32-bit drivers that are contemporaneous with the 64-bit versions).

Using Geoff's kernel patch WILL MOST LIKELY BREAK THINGS.

Maybe, but at least I provide the means to evaluate what Microsoft says about those driver issues. That's what the patch is for. Without it, all you can do is recite Microsoft. You'd think people all through the industry would be suspicious of reciting something they can't test - but it turns out that few are.

[Geoff Chappell]

He implies that Microsoft is abusing their monopoly and being anti-competitive by limiting memory on 32-bit Windows.

That's not exactly what's said. They've been (possibly) anti-competitive by suppressing competition in a subsidiary industry by lying about which editions of 32-bit Windows have code to use memory above 4GB.

That Microsoft (and others) blame driver writers for not having tested a case that Microsoft itself long portrayed as applicable only to exotic server versions is at best distasteful.

Online documents that used to cite just a few server versions as supporting PAE have been silently modified in the last couple of years to widen the applicability.

I don't know - and the article doesn't say - that any of this is actionable. But it certainly is all fishy.

He also states that PAE doesn't require applications to be recoded to benefit from it; this is not true.

Of course it's true - and not just because the "it" is explicitly a "more-than-32-bit physical address space".

A larger physical address space brings significant benefits to an application in terms of avoiding the use of a paging file to meet the application's need for memory in competition with other applications and with the system itself. This is covered in the article.

I suspect many users are within a factor of 2 for the total use of physical memory to reach 4GB but are a factor of 10-20 from having any one application use 2GB. Being able to use more physical memory in total may be more important to them than having any one application use more than 2-3GB. The article notes that users who face the latter limit would get no value from 32-bit Windows being permitted to use memory above 4GB.

For full effect, an application needs to use AWE in order to use more than 2/3gb of RAM as would be provided to PAE.

Benefit without full effect?

This too is covered in the article, e.g., where it differentiates from the "fully utilise" in Dell's fine print.

He then goes on to state various conspiracy theories under transparacy about why he thinks evil Microsoft won't let you use more than 4 gigabytes of RAM on a 32-bit OS.

Not conspiracy theories but implications. The article doesn't have much to say about intent, except to suggest that Microsoft will have recognised that under-documenting may be strategically convenient.

Are you just one of these people who think that criticism of Microsoft implies a belief that Microsoft is evil?

Again, Microsoft has no reason to limit 32-bit Windows to 4 gigabytes of physical address space for "anticompetetive" reasons when that same license gives you the right to use the equivilant 64-bit version of Windows and go well above 4gb of address space.

There you go again reading intent into it. Anti-trust laws (and, for that matter, trade practices laws about misrepresentation) can be transgressed without intent - and without benefit.

Whether the practice that I say may have been anti-competitive is actionable or not, it precedes by many years this ability to upgrade from 32-bit to 64-bit on the same license key.

[Android1]

Posts approved, Geoff. Welcome to Rage3D.

[AluminumHaste]

I doubt he will be here long, they never do stick around.

It seems to me a lot of work and headaches to get 32bit Windows to use more than 4GB or memory. When offered a chance to use 64bit Win7 and Vista I jumped on the bandwagon. I look forward to being able to install up to 12GB in my laptop without having to resort to user-made patches.

If this is Microsofts way of pushing all to 64bit, then I hope they try a lot harder.

[javiermq]

Well, isnt the reason for having a 32bit release of windows 7 the same one for having released win95 with 16bit dos compatibility, the reason being keeping as much backward compatibility with users' older programs as possible? Probably also, because most computers still don't come with 4gb RAM, 2gb seems pretty good for everyday use.