Ah thats the hard bit. The app I want to spy on isn't written by me. I'm only doing the API wrapper DLL. The app gets a pointer to the framebuffer from the DLL, and the app then AFAIK trys to create DX surfaces (DX3 BTW) in the memory. Under NT it's obviously broke as you can't access other process' memory, and > 2GB is protected. I'm currently working out how to best go about bypassing windows memory protect. I think a memory driver is probably the only option.
However I need to know what the failing function is so I can read parameters etc. All I ever see is error number 80004001. When I step through it in the debugger all I get is ASM. Not even function entry points with the function name. Usually debug versions of DLLs give you that. I've currently installed DX9 and the DX8 SDK went walkies. So I'm going to continue work on win98 (assuming the same problem doesn't occur).
What I don't know is if the "framebuffer" is actually the real framebuffer or just a huge DX surface, or even a chunk of memory in the API DLL process?
At the point where DX tries to create the texture surfaces in this non-existant memory apperture, it has already set the resolution, and gone into full screen mode. So, I wonder if it has or wants full access to the video memory? Although when I'm debugging I alt-tab the app back to the desktop. Surely that would screw up the framebuffer? Argghhh!
To much thinking have to go sleep for a while.
I prefer to think the "framebuffer" is just an aperture in the DLL process where the game puts the texture surfaces. Then passes the surface pointer to the API when it wants to load a texture. The DLL then does the appropriate magic with the texture. That makes the most sense. The last option I have is to stuff the memory in a closet somewhere and not use any textures from the game. A dev\null would be nice!
Interestingly Win9x puts memory mapped files into memory space above 2GB this is the same system memory region for all processes. Below 2GB is the processes own space that nothing else can see, this is where NT apparently puts created Memory mapped files. If I could get the game and DLL to run in win9x compatibility mode, maybe a memory mapped file would go into the > 2GB region. Then I could open a map to the file using the games process handle from the DLL. Problems I envisage is the file being paged out to disk, can the file be accessed from the game by pointer only? Can't use file seek etc. Just a pointer and direct write to memory. Don't even know if any of that is possible!
<Euan enters long coma now>