The only thing i can think of, right now, where GPU drivers would have a impact in loading times, is if they are "optimizing" shaders behind the scenes for the game, but this usually happens only one time and is mostly the first time you run the game. Imo most of the time long loading times is caused by many other factors, the game engine itself, some engines, like idtech 4 for example, have no asset streaming so they load everything a level needs into memory at level start, that is why it takes time to start a map but then restarting is very fast, because everything is there already, of course HDD/SSD's speeds have a say in this, is also very important the type of assets the game uses, for example a simple human readable file takes more time to process then a binary non human readable file, other things like, texture amounts, texture resolutions, geometry complexity, shader complexity, etc, al of this has effect on loading times and sometimes even quit time.
edit: its seems you found that drivers where indeed causing this particular problem, glad you solved it.