I'm also confused about the cage, but more specifically about how I go about making a good cage and how to recognize a bad cage. I've always just done distance based in xNormal and it's worked, but of course with a lot of tweaking. What's the best way to do my cage?

There aren't really any rules for a "good cage", chances are if you need to worry about having a "good cage" you've got a bad lowpoly model. Some people like to spend a lot of time tweaking the cage, when they should just make sure they have a solid lowpoly and understand how averaged normals work, and how that affects the ray projection.

Using the distance based method in xnormal isn't a cage persay, as your hard edges are breaking the cage etc.

I'm sure I could give you some tips on what makes a good cage, but that is essentially useless information. Once you understand the technical aspect of raytraced projection and how it relates to your lowpoly meshnormals/cage, the rest should be straight forward.

Originally Posted by EarthQuake

Once you understand the technical aspect of raytraced projection and how it relates to your lowpoly meshnormals/cage, the rest should be straight forward.

I think this is what I don't fully understand yet.. I will refer to my previous question again.



You said that even if you put in smoothing groups and stuff and the meshnormals look like A, the cage will still look like B. So the cage averages all normals which also having everything in one smoothing group does, so whats the point of using hard edges instead of just one smoothgroup for everything? This got me totally confused, after I thought I understand it >_< Especially after watching Racer445's Sci-Fi prop tutorial because there he uses smoothgroups everywhere and almost no bevels. Could you elaborate on that again, for a newbie? Thanks!

Originally Posted by hamzaaa

I think this is what I don't fully understand yet.. I will refer to my previous question again.



You said that even if you put in smoothing groups and stuff and the meshnormals look like A, the cage will still look like B. So the cage averages all normals which also having everything in one smoothing group does, so whats the point of using hard edges instead of just one smoothgroup for everything? This got me totally confused, after I thought I understand it >_< Especially after watching Racer445's Sci-Fi prop tutorial because there he uses smoothgroups everywhere and almost no bevels. Could you elaborate on that again, for a newbie? Thanks!

When using a proper cage, your lowpoly meshnormals (hardedges, smoothing groups, whatever you want to call it) are ignored. This enables you to project an entirely seamless bake.

If you're using a non-averaged cage, you're using the mesh normals of the lowpoly, and the vertex' are broken along your hard edges, this means you get gaps in your projection anywhere you have hard edges.

So, when using a cage, you can use bevels, smoothing groups, etc it doesn't actually matter. The cage has averaged normals, regardless of your lowpoly mesh normals.

The point of using hard edges/smoothing groups should never be to fix ray projection errors(waviness, skewed details etc) as all that does is create more problems(seams from the gap) but is often used to lesson smoothing errors and other similar artifacts.

Thanks for your answer, now I understand!

Thanks for the replies. I do have a few more questions, if you can bear with me.

I understand (the basics of) how the raytracing works and why hard edges cause issues with it. But, from my understanding when you apply a normal map to the low poly the normals of the loware ignored and replaced by those of the normal map. So, why bother having multiple smoothing groups at all? Perhaps I'm just blindly overlooking something here. For all of my models, I've been baking in xNormal with averaged but based on a distance that works for the model (making sure the distances aren't too far out and cause errors). When I bring the normal map back into Max or Unreal or Source, the models look fine with one smoothing group and the map applied.

Am I just getting lucky?

Hopefully I'm getting something right.

If it looks fine, you shouldn't worry about it then.
Mine weren't looking alright. Everything set to 1 smoothing group lead to huge gradients over flat surfaces, which looked crappy using viewport shaders.
If your models are organic or higher poly, without flatter areas, that become less visible. But in my case, I had to split the low poly in many different smoothing groups, as if it was supposed to be vertex lit in an older engine. Worked fine in the end.

But yeah, if you can get away with 1 smoothing group, just do it and be glad that it works!

Using uniform ray distances will never get you a very accurate bake and in most cases you are much better off using a cage because they offer much more control

The only reason you would want to use smoothing groups is if the tangent basis of the baker doesn't match the engine that you are rendering the final model with.

The colours and gradients in the normal map are there to represent the differences between the low and high poly models and if the tangent basis doesn't match then these colours and gradients cant be translated accurately into correct lighting, which is why you can get the smoothing errors (which are simply the visual difference between the tangent basis of the baked map and the renderer)
Adding smoothing groups reduces the gradients and in doing so, also reduces the errors because there is less chance of an incorrect translation happening as the areas with smoothing groups tend to be flatter and resulting values tend to be much closer to 128,128,255 (in addition to other factors)

In xNormal, you always want to bake using the final low poly model with it's normals intact (as exported) as averaging the normals in xNormal is different to the averaged cage that EQ is talking about. If you are averaging the normals in xNormal and then using that bake on a mesh with different normals, then the bake wont match the LP mesh, which will cause errors.

Exported normals Vs Av. Normals in Marmoset (on the exported normals LP)


Averaged normals


Exported normals


Difference map


xNormal requires that the topology of the cage matches the LP mesh 100% and doesn't have an averaged cage, so using it's inbuilt cage feature doesn't work because when the cage is expanded, gaps are also expanded.



The only work around i have found is to make a duplicate of the LP mesh for use as an external cage and then apply the smoothing groups after the cage has been expanded in your 3d application, which seems to reduce any gaps in the bake so that they are pretty much non existent.

Hope that helps and sorry for all the pics!

http://dl.dropbox.com/u/2057427/Poly...PC_SG_Test.zip

Thanks for the in-depth explanations!

The thing is that I've been keeping it all at one smoothing group in Max, so the normals are the same for baking in xNormal. The thing I'm not doing is breaking my models into smoothing groups, at all. I've been putting them all to 1, importing into xNormal and baking with the distance rays.

I get exactly how a cage is important (at least I do now), but for all my work so far there haven't been issues. My question is that if my smoothing groups are always set to averaged across applications, why should it matter or why should I bother with smoothing groups.

I've read a bit about using hard edges and an averaged cage, but what difference does it make from smooth edges and an averaged cage?

Maybe you've answered this but if so it's gone over my head, so sorry for my ignorance and thanks for all the help!

If you are using a single smoothing group and the tangent basis of the baker matches the engine that you are rendering the final model with, then there is no need to use smoothing groups, other than for texturing reasons (it makes for a cleaner normal map that is easier to texture in some cases)

The cages Max and xNormal work differently because xNormal doesn't use averaged cages, but if you are only using 1 smoothing group then it doesn't matter because there are no breaks in the meshes normals and therefore, no gaps to worry about.
As EQ said, with an averaged cage in Max, there is no difference in using a model with smoothing groups or no smoothing groups because the cage ignores the normals anyway

metalliandy, lots of good information there. Have to respectfully disagree with some things though, please don't hate me

Originally Posted by metalliandy

Using uniform ray distances will never get you a very accurate bake

As I can remember, we've always( in the history of mankind and the dinosaurs) done that and don't experience inaccurate bakes. Using custom cages can significantly reduce your efficiency and cause much frustration. If you require the use of cages it's usually because the lowpoly mesh has problem areas. Fix the lowpoly instead of using a cage.

Originally Posted by metalliandy

The only reason you would want to use smoothing groups is if the tangent basis of the baker doesn't match the engine that you are rendering the final model with.

Harsh vertex normal changes, as you get on mechanical objects, still require the use of smoothing groups, even with a synchronized renderer, in both Max and Maya.

Here's an example image, you see the diagonal visual artifact from the screw in the top left corner there, in the bottom image (this object was made unnecessarily with 1 SG as a test).

Keep in mind that there's no reason not to use SGs at your UV splits. Since they can help a lot with visual quality you might as well do it.

Per: he's referring to using the cage as apposed to the "offset" method in max, which break edges when using SG's. He is correct in what he's say. You're mistaken in thinking that he's suggesting using a Custom cage tweaked vrs a simple cage setup. But he's talking about a proper averaged cage, vrs the broken edge projection of the offset thing, or basic ray distance setting in xnormal etc. Unless you're trying to say something else and I misread.

[edit] NM I see his "control" comment below and yeah, agree with per here, if you've gotta do any custom cage work, you should really be looking to fix your mesh, not your cage. Tweaking cages needs to be redone any time you change your mesh, so its really just a bad idea.

Now as far as SG's, yeah there really is no reason NOT to use smoothing groups. Use a quick script to set all your uv borders as hard edges/SG in Max or Maya, there is absolutely no drawback to performance to do this, no matter if you're using a synced display or not. The only drawback is that you have to actually use the proper cage, but really, being able to visibly see the cage as apposed to guessing a ray distance value is a much easier way to work anyway.

No problem Perna, it's always good to be challenged

I would like to explain myself a little better as perhaps my wording wasn't super awesome.

Originally Posted by metalliandy

Using uniform ray distances will never get you a very accurate bake and in most cases you are much better off using a cage because they offer much more control

By this I meant that you can run into issues like projection problems when using the minimum/maximum uniform ray distances on things that are close together and cant be exploded, such as fingers for example. Perhaps accurate wasn't the best terminology
You can use things like blockers, but i always find it easier to use a cage. I still use uniform projection for things like textures though.

I 100% agree on the smoothing groups comment though. There is no reason not to use them on SG splits (i actually released a Blender 2.5 script for this a few days ago ^^) but as i don't use Max or Maya, i cant really comment on the use qualified normals for them
Also, I 100% agree on the pros. of using smoothing groups for texturing reasons too.

With Blender, xNormal and Marmoset there are a couple of very small errors with the mesh above (top corner) but nothing really major.

xNormal baked Normal in Marmoset (single SG)


xNormal baked Normal (single SG)


I made the test mesh above to see how well Blender could handle single smoothing groups with its internal renderer and baker, since they have recently implemented a new tangent basis it seems they did a really good job in getting everything synced up (GLSL viewport shader, Baker and the Rendering engine)

Top row is HP, bottom is LP (single SG)


Blender baked Normal (single SG)


@EQ,
Just noticed your post!
Yea, you pretty much hit the nail on the head with what i was trying to say
Thanks for the translation!

Gold mine of information here guys

I'm trying to understand this stuff as thoroughly as possible, but I think I'm still missing some things. Sorry if it's already been explained and I missed it.

Does it matter if you use one smoothing as opposed to multiple smoothing groups with UV splits? Is it a matter or personal preference, or are there specific instances when each method would be better suited?

I've been talking to oniram about his recent SMG, where he baked with one smoothing group and ended up with great results. I then tried it with a recent model I worked on and ended up with worse results than I got from using smoothing groups with UV splits.

Originally Posted by Sean VanGorder

Gold mine of information here guys

I'm trying to understand this stuff as thoroughly as possible, but I think I'm still missing some things. Sorry if it's already been explained and I missed it.

Does it matter if you use one smoothing as opposed to multiple smoothing groups with UV splits? Is it a matter or personal preference, or are there specific instances when each method would be better suited?

I've been talking to oniram about his recent SMG, where he baked with one smoothing group and ended up with great results. I then tried it with a recent model I worked on and ended up with worse results than I got from using smoothing groups with UV splits.

Well the real question you need to ask is why would you ever need to avoid using smoothing groups/hard edges? If you can come up with the answer for this question, it should be apparent.

If we're using a proper averaged cage(as has been covered here extensively) there will be visual no gain to using 1 smoothing group. On the technical side, your vertex's are already being double along your UV borders, so using hard edges there adds zero performance loss. So why this obsessive need to worry about having only 1 SG?

This whole thing comes from some programmer telling someone 1 SG is better, back in like 2001 or whatever, and everyone repeating that advice. But really, the conclusion is that using hard edges/smoothing groups along your uv boarders will never be a bad thing.


Now, a few good reasons TO use hard edges on your uv borders:
1. If your engine isn't synced, it can significantly reduce smoothing errors.
2. Even if your engine is synced, it can avoid artifacts related to texture resolution. EI: your triangle is too small and the require gradiant needed to counter the smoothing is impossible to record in your normal map.
3. Extracting "detail maps" from crazybump or xnormal is a more straight forward process, you worry less about artifacts from extreme gradiation.
4. Texture compression generally is going to produce better results with less gradiation.

So yeah, the question shouldn't be which is better when, it should be why would I ever want to use 1SG?


Now a lot of people, upon hearing this ask me: "Why do I care about synced normals then?" Well, the answer is relatively simple.

Even if you're using hard edges, your end result can be significantly better with synced normals(3ps Quality mode, Maya, etc), so much so that you can even take old models, baked with hard edges etc and simply turn the feature on and see a notable improvement.

When you have synced normals, you can also simply worry much less about uvs and hard edges. Without sync'd normals, you may need to use a lot more hard edges and split UV islands to get a bake without any smoothing errors. Generally with a sync'd workflow, you can simply model naturally, not worry about excessive bevels, or excessive hard edges/uv splits, just model in a sensible manner and use a script to add hard edges where your natural UVs seams are to ensure an even higher level of quality, and you're good to go.

Hoever, this boils down to a matter of Sync'd or no, not 1 SG or no.


On a complex mesh, you could see 2-3x the vertex count(either with excessive beveling, or excessive hard edges) to get the same results with a sync'd bake.

Ah, thanks for the extensive response EQ, that clears some things up. Much appreciated

Originally Posted by EarthQuake

[edit] NM I see his "control" comment below and yeah, agree with per here, if you've gotta do any custom cage work, you should really be looking to fix your mesh, not your cage. Tweaking cages needs to be redone any time you change your mesh, so its really just a bad idea.

I don't mean custom cage work
By control, I meant that using using uniform ray distance/offset is much more of a guessing game than using a standard non tweaked cage and the results can often leave something to be desired.
When making a cage I just use use Blenders equivalent of the push modifier with no other edits.

Originally Posted by metalliandy

By this I meant that you can run into issues like projection problems when using the minimum/maximum uniform ray distances on things that are close together and cant be exploded, such as fingers for example. Perhaps accurate wasn't the best terminology
You can use things like blockers, but i always find it easier to use a cage. I still use uniform projection for things like textures though.

Generally for these kind of things, like fingers, the trick is to sample based on surface direction (xNormal has had this for quite a while). When you do that, you'll only sample the next finger if the ray travels all the way through it, which is quite a distance. If you don't do it, you'll sample the side of the second finger that's closest to the first finger and you'll get the artifacts you mention. I'm sure you're aware of that, just dropping it in there for anyone following the chatter.

Yeah, when you say "control", to my understanding that means you are using a custom cage, because the only way to control it is to tweak it, and if you tweak it, then it's custom. 3ds Max causes some confusion with terms here, because even if you just set a constant distance, it's still called a "cage", which is a term I'd only use for when you're customizing it. Fixed distance can be/is done 100% programmatically, except 3ds chooses to visualize it as a "cage". Don't know how this is in, say, blender.

Originally Posted by metalliandy

By control, I meant that using using uniform ray distance/offset is much more of a guessing game than using a standard non tweaked cage

Yeah, definitely an issue of terminology here. Technically a non-tweaked cage is not a cage per se, since it does not need to be stored. The "cage" doesn't have to exist; it's actually just a single "uniform ray distance/offset" as you say. You can visualize it as a cage, but it doesn't need to be stored as a separate mesh (that 3ds does this is just non-ideal programming on the part of 3ds)

The best way of defining the distinction should probably be:

-if your ray target position parameter is a single value, you're not using a cage
-if your ray target position parameters are per-vertex, you're using a cage, with the extra work required to maintain those parameters.

Of course, if you're in a situation where you have no choice but to use a cage, then you just have no choice, but it's definitely inefficient and should be avoided wherever possible

Originally Posted by perna

Generally for these kind of things, like fingers, the trick is to sample based on surface direction (xNormal has had this for quite a while). When you do that, you'll only sample the next finger if the ray travels all the way through it, which is quite a distance. If you don't do it, you'll sample the side of the second finger that's closest to the first finger and you'll get the artifacts you mention. I'm sure you're aware of that, just dropping it in there for anyone following the chatter.

I didn't know this actually.
How is it enabled in xNormal?

Originally Posted by perna

Yeah, when you say "control", to my understanding that means you are using a custom cage, because the only way to control it is to tweak it, and if you tweak it, then it's custom. 3ds Max causes some confusion with terms here, because even if you just set a constant distance, it's still called a "cage", which is a term I'd only use for when you're customizing it. Fixed distance can be/is done 100% programmatically, except 3ds chooses to visualize it as a "cage". Don't know how this is in, say, blender.

Ahh right, I see what you mean. I have always viewed these a totally separate ray casting methods, so I can see what you are getting at now.

Blender uses Distance and Bias settings with numerical input in which you set the maximum distance from the active object to the target with the bias allowing you to set a preference to objects that are further away. Most of the time it seems to get nice results, but its not super intuitive (initially, at least :P).
I do all my baking xNormal though, as it generally provides more detailed results.

Originally Posted by perna

Yeah, definitely an issue of terminology here. Technically a non-tweaked cage is not a cage per se, since it does not need to be stored. The "cage" doesn't have to exist; it's actually just a single "uniform ray distance/offset" as you say. You can visualize it as a cage, but it doesn't need to be stored as a separate mesh (that 3ds does this is just non-ideal programming on the part of 3ds)

The best way of defining the distinction should probably be:

-if your ray target position parameter is a single value, you're not using a cage
-if your ray target position parameters are per-vertex, you're using a cage, with the extra work required to maintain those parameters.

Of course, if you're in a situation where you have no choice but to use a cage, then you just have no choice, but it's definitely inefficient and should be avoided wherever possible

Ok awesome
My only issue is that when I use the max/min ray distance in xNormal, it often bakes incorrectly and breaks the edges, but this doesn't happen when I use a cage (non tweaked) in xNormal. Any ideas why this is happening? Maybe the sample based on surface direction setting you talked about will fix it?

Thanks for the detailed post Perna. Always appreciated

Originally Posted by metalliandy

I didn't know this actually.
How is it enabled in xNormal?

Sounds like "Discard back-faces hits" to me.

Originally Posted by SpeCter

Sounds like "Discard back-faces hits" to me.

Maybe. I always have this checked and I still get bad results though :/

This thread is awesome!
Can someone post a script for 3ds Max (2012) which allows me to apply hard edges on UV seams?

Originally Posted by metalliandy

I didn't know this actually.
How is it enabled in xNormal?
Ok awesome
My only issue is that when I use the max/min ray distance in xNormal, it often bakes incorrectly and breaks the edges, but this doesn't happen when I use a cage (non tweaked) in xNormal. Any ideas why this is happening? Maybe the sample based on surface direction setting you talked about will fix it?

Thanks for the detailed post Perna. Always appreciated

In Xnormal, when you're just using the ray distance, it uses the vertex normals of your lowpoly mesh, so you get gaps along your hard edges.

In max, if you use the "offset" method, it does the same thing.

So to me, when I say cage, I mean a proper averaged cage, not a simple ray distance. I don't mean a manually tweaked cage, just a proper averaged cage. But thats getting a bit semantical, I don't think we need two different terms for if you've manually tweaked your cage or not. You've got an averaged cage, or you've got a distance based raycast from your lowpoly mesh normals, these are the important things to identify. To me, using a cage certainly doesn't mean you're going in there and tweaking it to avoid errors and things like that, its differntiating between the two obvious methods that the software gives you.

It is all a "cage" in a sense, but I think Per's explanation of terminology is a bit confusing there.

In max, you have:

A. Offset Method
B. Cage

In Xnormal:

A. Ray distance
B. Cage

In Maya:

A. Surface normals
B. Geometry normals(averaged)


Anyway, these are the important things to understand, "offset" in max will cause gaps, ray distance in xnormal will cause gaps, surface normals in Maya will cause gaps.

Weather or not you're manually tweaking your cage has no bearing on the technique used to project, and that is the important factor here, that you're using a proper averaged cage.

In most apps, what is presented to the user is; you're either using the cage or you're not, and that, to the user, is what causes or solves the hard edge gap problem, its the most basic and simple way to refer to it. You should be using the cage, not the basic ray distance. We can talk about the technical side behind it etc, but the user isn't seeing that, so its not particularly relevant.

So to say "Are you using a cage or a ray distance?" is a good question to ask, as almost everyone will have some basic understanding here.

In a sense, Maya operates on the User level the closest to what Per is explaining, its all just a ray distance, and you have a choice between averaged normals on the projection "envelope"(what maya calls it lol) or using the lowpoly vertex normals.

In max its more complex and much less ambiguous, as the two methods not only have completely different UIs, but have different behavior as well. In max tweaking the cage will affect normal direction as well as distance, in maya it will only effect distance. XN is the same as max in this regard as well, a completely different interface and behavior for using a "cage" or not.

hamzaaa - http://www.polycount.com/forum/showt...39#post1105439

Originally Posted by Sean VanGorder

hamzaaa - http://www.polycount.com/forum/showt...39#post1105439

Muchas gracias ;)