Hey man, im pretty new to Sub-D myself(ish) but try doing what i have illustrated in the image, the smoothing may be from the n-gons not having any support, and going a bit ape-shit :P
If that doesnt do it, you may need to chamfer the edges that run around the holes, or add in control edges around the outside, but try what i have illustrated first.
