Well, that's pretty much how everything in 3d works. Basically, the camera is at (0,0,0) and is looking along the Z axis. This never changes. You also don't want to change your geometry according to position, would be very cumbersome. So, you have a matrix which transforms every vertex you pass to the API into the cameras space.
Basically, like this in OpenGL:
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glRotatef(0,0,1, angleZ); // Rotate around Z
glRotatef(1,0,0, angleX); // Rotate around X
glRotatef(0,1,0, angleY); // Rotate around Y
glTranslatef(-cameraX, -cameraY, -cameraZ);
From now on you can just pass vertices as they are in your world and it'll be as though the camera has moved. Personally, I just setup the matrix like that and think of it as the camera has moved, it's easier that way.