The more I play around with it, the more I think it's the material I currently have set up that's causing it.
I have 3 layered textures set up to a blinn (1 for color, 1 for spec, and 1 for normal). Each one of those layered textures has 2 texture files each that I linked together in 2 UV sets. Please let me know if this is the right way to go to about it or if what I can do to fix it.