I have a similar problem, I'm a n00b in 3D design, but im following a Digital Tutors tutorial on car modeling, and I'm working on a 2010 Camaro.
I cant change the settings when it’s set to Nitrous. It displays that it’s installed, but shows “Nitrous (Direct 3D 9.0 - Nvidia GeForce GTX 460 SE)” and the configuration button is locked. and although I have DirectX 11 installed, I cant get it to use it.
So I selected “Direct 3D” and that works. I was able to change the configuration. I set the background texture size to 1024, ticked "match bitmap size as closely as possible" ok'd and exited. But when I reopen Max, the settings have gone back to the unticked position!
The reference pic is still blurred and I'm at a loss on what to do next.