I keep seeing this question come up time and time again, what causes waviness? How do we fix it? I feel the reason it keeps coming up, is a fundamental lack of understanding.

To get to the root of the problem, we have to look at how normals are averaged, and thus, how the actual rays that are doing the "baking" are cast, and in what direction.

I whipped up a quick test scene here, to show a couple different types of "waviness".

First off, the highpoly, notice that the right two meshes are the same, and the left 6 are also the same, the big difference being a slope instead of a hard 90 degree angle at the ridges.


Now, we'll take a look at the lowpoly meshes. From the averaged vertex normals here we can see the beginnings of our problems, the vertex normals here are far from flat, so any and all differences between the high and the low will become exaggerated. You should be able to learn to spot potential ray trace problems, "waviness" or "skewed details" simply by looking at the lowpoly mesh normals.


And the resulting bake;

Quickly we see that even a gentle slope with our ridged details produces a much different affect, and much more acceptable waviness. A1 and B1 are really worst case examples, and you should design and model to avoid these sort of shapes, unless you have the geometry budget to accurately match the shape of the high.

A3 shows how we can account for "skewing" type details, by adding control loops on either side of the shape, B4 shows what happens when we do not do this.

B1-B4 basically show that we simply do not have enough geometry, and unless this object was going to be seen very small on screen, it simply looks terrible. No amount of tweaks or repainting or cage hacks will fix these meshes, they are hopeless. At least from the side view, from 3/4s view, the waviness actually helps to make certain areas more presentable.


As we break down what the mesh normals are doing, and as a result, the direction the rays are "looking" when we bake the mesh, we see the problem. Along these hard angles the rays are being cast out at a 45 degree angle from the surface.


Now when we look at the difference between the high and the low, the problem becomes very obvious. The greater the difference in curvature, the more problems you're going to have.


Now, the other common problem, skewed detail; is really the exact same problem, just with a different type of geometry.

Here we see that adding these supporting loops flattens out the mesh normals along the cylinder, resulting in a nice, even bake. Without the loops, the normals are averaged along the length of the mesh, and the projection is variable, resulting in "skewed" details along the length.


So, the next time you go looking for a "software fix" or a "workaround" to these problems, first stop and think: Is my geometry to blame? Am I able to simply match the low to the high more accurately? Can I simplify the design to work in a more reliable manner?

Because every software hack that we do, cage tweak, every time we paint the "errors" out in photoshop, every time we do a bunch of different bakes and combine them, we must remember that:

A. The work must be redone every time we change the mesh.
B. The work must be redone every time your boss requests you to change the mesh.
C. The work must be redone by some other poor slob, when you're unable to do it and they're forced to, requiring them to entirely debug/rework your model back to an acceptable state, or attempt to preform the same magic voodoo dance you had to perform to get a good bake. - This one here is key, you owe it to your fellow man to properly understand this stuff.



Oh, and another thing. Even if your lowpoly's normals look like A, your Cage/Projection mesh normals look like B. That is unless of course your bake is set up poorly(either using "offset" in max, match using "surface normals" in maya, or using the ray distance in XN instead of a proper, welded cage). If your cage is not averaged, you will get seams on all of your hard edges, as the normals will be facing perpendictular along those edges, and cause gaps in the projection.