Need help baking normal maps using Maya

Psychotic_Mike · Feb 2013

Hi, so after reading EarthQuakes post on making sense of hard edges, uvs, normal maps, vertex counts & Understanding averaged normals and ray projection/Who put waviness in my normal map? I decided to re-bake a normal map of model I created some time ago but never managed to get good baking results.
The thing is I use Maya for baking normal maps and I havent seen any tutorials/pipelines out there on how to correctly bake normals using either Maya's transfer maps options or a Maya to Xnormal guide to correctly know how to make accurate cages.
Below you will see my meshes and the results I got from baking in Maya and xnormals. Hope somebody can give me an explanation I can understand

Stupid question could the low res mesh be used as a cage in xnormals as long as it covers accurately the high res version? - wouldn't that give the same results as just using the low res mesh in xnormals (no cage imported just a cage created in xnormal)

I sincerely hoped Maya's average normals would give a perfect result as I matched as closely as possible the high res version.
I know my high res mesh is not perfect (lots of 90 degree angles) ideally I would make the edges more bevelled so it would be easier to calculate the normals around the edges but at the time I didn't knew that.
I purposely broke the uvs anywhere I had 90 degree angles on my low res mesh so that I could get the best results possible. (let me know if I'm wrong about this).

Questions:
Is my geometry to blame?
How can I make explicit projections work in xnomal? what should I changfe on my low res mesh?
How can I make a cage in Maya an export it in Xnormals? (can I use the low res version as it covers the high res?

Preview of the low and high res meshes

Here I show which edge loops are hard and which are soft.

Next I show the uv's packed on the 0 to 1 space. with edge padding to avoid bleeding.

Results baked in Maya (average projection) and xnormal (explicit projection)

I know why this issue (gaps) happens I would appreciate a guide to understand what to do to get better edge results with explicit mesh normals for projections.

A huge thank you to anyone who can help out!
Thanks in advance!!!!

AlecMoody · Feb 2013

What exactly is wrong with your results baked in maya? There is something funky going on with dark shading at the top of the largest polygon but aside from that it looks fine.

My xnormal custom cages workflow looks like this:

*Open the xnormal 3d viewer
*Check the show cage box on the left
*expand the cage by dragging the slider
*save the mesh as a new file
*DO NOT agree to automatically assign meshes (attempts to resave and reassign your highpoly mesh for some terrible reason)
*Say yes to the prompt about skipping large meshes if it comes up
*load the new file in your lowpoly meshes and check the "use cage" button. Xnormal's OVB files contain both the base lowpoly and the cage mesh in one file.

working this way will create an averaged projection. The other option is to import a custom file by right clicking on the lowpoly and finding the option called something like "import cage". I have had problems with xnormal rejecting files in the past and my undestanding is that this works best with sbm files. For the custom cage to work correctly you need to have no smoothing splits in the cage mesh and you need to have the "use cage" box checked.

Psychotic_Mike · Feb 2013

Thanks for the reply!!!
Its just that I get these smudged edges (yellow lines) that do no give a sharp continuity to my normal map.
Unless that is the way normal maps are supposed to look I compare my stuff with tutorials that I see from Max and the results suck!
The normals from Xnormal have a better gradient distribution and more crisper results, while Mayas seem smudged, it seems like there is a lighting issue or something.

will def try your way and post the reults

CheeseOnToast · Feb 2013

Here's my Maya to Xnormal workflow using custom cages. It should work every time.

1. Export your highpoly mesh as an .OBJ file. An export straight from Zbrush is fine.

2. Export your lowpoly mesh from maya as an .OBJ file. Check that the .obj export settings have both normals and smoothing checked "on" before you export.

3. Duplicate the lowpoly, switch to component mode and select all the verts of the duplicate. Then do edit mesh>transform component. Drag on the BLUE transform handle to inflate the mesh along its vertex normals. Make it just big enough to cover the highpoly mesh completely. Then export it as an OBJ file. This is your custom cage.

4. In Xnormal, assign your highpoly and lowpoly meshes. In the lowpoly list, right click on the line that your mesh is in, then choose to "browse external cage file". Browse to your custom cage and load it. You'll get a warning dialogue about making sure it matches the lowpoly exactly. Accept it, then you're ready to bake.

5. Bake the normal map, making sure you have a decent bit of edge padding.

6. That's it

Note that baking with a cage won't stop skewing. It will, however, get you a much "flatter", cleaner normal map with little gradiation. To stop skewing, you'll need to add some more support geometry to your low poly.

Psychotic_Mike · Feb 2013

Cool

Component mode is F9 right?
Also will I have to smooth the normlas of the entire cage before I export likeAlecMoody suggested?

Psychotic_Mike · Feb 2013

I tried it it was just what I was looking for a million thanks guys!!! (I cant stress how much you guys helped me) I still have some issues but I think I need to recalibrate my cage. I am more pleased with the results now.

1. shows the high res (grey) low res (Yellow) cage (Red)
2,3,4. are the lines that I think are caused by the cage as I said it needs to be recalibrated!!!!

Psychotic_Mike · Feb 2013

here is a final result with a quick texture applied

Psychotic_Mike · Feb 2013

only downside to this is because of the too many uv seams I doubled my vertex numbers. Need to learn to model efficiently.
Again thanks

EarthQuake · Feb 2013

Yeah, with the crazy amount of hard edges/uv splits you have here, you could have simply beveled edges and went with less uv islands in a lot of places, which would have made your bake more straight forward as well.

BARDLER · Feb 2013

EarthQuake wrote: »

Yeah, with the crazy amount of hard edges/uv splits you have here, you could have simply beveled edges and went with less uv islands in a lot of places, which would have made your bake more straight forward as well.

Yea do this. It will not only give you far less UV islands, which would be easier to texture. It will also make the low poly look a lot better.

Psychotic_Mike · Feb 2013

Yeah thanks for that, I only went into the extreme side of things cause I thought that anywhere you have a 90 degree angle there should be a hard edge therefore a uv split.

EarthQuake · Feb 2013

Another thing that can really help, you mention 90 degree angles etc, a great way to avoid that is to simply not model everything with 90 degree angles in your highpoly. If you plan your highpoly so that it will be friendly to your low, with more gradual angles you can really help yourself out.

Heres an example:

With A, the way you modeled it, you've got all these blocky little shapes that you have to match up in your low, and in turn either use a lot of supporting geometry/bevels/etc, or lots of hard edges.

Now with B, we turn some of these blocky shapes into curved sloped trims. You still get visual interest out of these sort of shapes, but you can use much more forgiving angled geometry in your lowpoly. You don't need to do it exactly like this, its just a really quick example.

This is one aspect I feel people really overlook, you have plan it all out and be aware of the type of lowpoly geometry you'll need to support while designing and modeling your high res.

Here is another example form the waviness thread:

EarthQuake wrote: »

Alright, so I took a little time to do a more practical example. You guys have probably seen me write "model for your low" before, so I'm going to show what I mean by that.

First the high:
You'll see A is similar to the worst case example again, but is a very common type of model to see. Lots of straight edges, little regard given to how well this would translate to a low. B and C are the same(to show two different low methods), you can see here lots of low-friendly slopes, on the problem areas like the front of the barrel that we would likely add extra geometry to help keep it from skewing I've modeled a nice fat bevel there, so our extra geo will actually go towards the quality of the model.

Now, onto the low. A cool thing about the "more friendly" model is that it is actually less geometry, and is more interesting. We can add a little more geo to the larger cylindrical shapes(which again will help waviness) and STILL come out under the tri count of A. For everyone who always thinks you've gotta use 2x more geometry to get good bakes, noooo, you just need to understand the issues and plan accordingly.

Bakes, besides the obvious stuff, take note of how the high and low "bullet hole" shape is model in the front of the barrel, in the high we have a nice fat bevel in the center, and in the low we've got that nice support bevel as well, this results in a much better bake, in the A example, where the high has a simple indent and no supporting geo in the low, this area barely shows up.

Also notable, while B/C are clearly better, both of these look fine at most angles aside from a completely straight side view, so, try not to be so anal about minor waviness.

Ok, so we see that modifying our design enables us to get better bakes with less geometry, but not only in regards to waviness.

The "friendly" mesh(s) here:
1. Look better
2. Have less waviness
3. Have less skewed details
4. Use a simpler uv map with less uv island, this easier to work with + better performance
5. Use less extreme normals, thus resulting in less potential smoothing errors, which again allows us to have a simpler uv map, as we do not need to detach as many uv islands to work well with hard edges - This one is pretty huge.

So, even if all things were equal, and you could get the same/better results as far as waviness on A as we do on B/C, it simply isn't worth your time making models like this, for the variety of other advantages that proper understanding and planing can bring to your mesh.

Psychotic_Mike · Feb 2013

Thanks for replying, yeah the time I modelled the mesh no one told me that the high res geometry played an important role. It wasn't till I read your post that I finally understood that. You really opened my eyes! Thanks!!!

jibberishballr · Mar 2013

CheeseOnToast wrote: »

Here's my Maya to Xnormal workflow using custom cages. It should work every time.

1. Export your highpoly mesh as an .OBJ file. An export straight from Zbrush is fine.

2. Export your lowpoly mesh from maya as an .OBJ file. Check that the .obj export settings have both normals and smoothing checked "on" before you export.

3. Duplicate the lowpoly, switch to component mode and select all the verts of the duplicate. Then do edit mesh>transform component. Drag on the BLUE transform handle to inflate the mesh along its vertex normals. Make it just big enough to cover the highpoly mesh completely. Then export it as an OBJ file. This is your custom cage.

4. In Xnormal, assign your highpoly and lowpoly meshes. In the lowpoly list, right click on the line that your mesh is in, then choose to "browse external cage file". Browse to your custom cage and load it. You'll get a warning dialogue about making sure it matches the lowpoly exactly. Accept it, then you're ready to bake.

5. Bake the normal map, making sure you have a decent bit of edge padding.

6. That's it Note that baking with a cage won't stop skewing. It will, however, get you a much "flatter", cleaner normal map with little gradiation. To stop skewing, you'll need to add some more support geometry to your low poly.

I'd like to try this as I have had terrible luck getting my nice looking high res sculpts into Maya, with just my high/low OBJ's and Xnormal. The problem is I can't know how big to make the low poly mesh with the transform component option, since I can't get my high res model into Maya. When you get into the multiple millions, even with a good system, Maya does not like it. So I can't really create the cage without just guessing with the low poly and no high poly present.

jibberishballr · Mar 2013

EarthQuake wrote: »

Heres an example:

How would you go about with hard surface models requiring hard crisp angles?

Need help baking normal maps using Maya

Replies