First off, I am terribly sorry if this has been answered before, I did search but I couldn't really find any.

I'm creating a SCAR-L weapon and I'm having some baking issues. As you can see from the image below the baked normal map looks like it is sort of angled. The lowpoly is all soften edges (One smoothing group), Ive tried both surface and geometry normals in the transfer maps dialog and modifying the cage, adding extra vertical edgeloops on lowpoly for extra support, I'm all out of ideas. I am pretty sure its an easy fix.

Thanks for any help!

I believe it is because you are using a cage, so the rays are casting from the angles of the cage. You usually get this when your cage isn't tight enough to your mesh. Maya doesn't call it a cage, but it is the setting "geometry" and "surface" in the normal map calculation type. Other programs allow you to actually manually create a cage (Maya might too), while Maya by default just increases the cage distance as a setting.

You can try reducing your cage distance, create a custom tighter cage, or worst case, you can do 2 bakes, 1 with the geometry setting and 1 using the surface method instead. The cage will give you the nice beveled edges, and the surface will get you accurate details, so you can combine them both in photoshop and get the best of both worlds.

Actually Ive made my cage as tight as it can be as you can see in the image below. Ive tried baking in xNormal too with the same angled result. I recall having a similar problem before which I solved by changing from geometry to surface, or the other way around, but that didn't help this time.
Ive also tried baking in both Object and World space without any luck.

Also I don't get any significant change when baking in either geometry or surface

Here's an image with my cage in perspective, side and front view

Harden the edges around the ends, it'll change how the rays are cast. I wouldn't bother using a casting cage for this, it should be ok without.

Just harden those edges that make up the UV borders of the end caps, it should bake fine after that.

Originally Posted by MoP

Harden the edges around the ends, it'll change how the rays are cast. I wouldn't bother using a casting cage for this, it should be ok without.

Just harden those edges that make up the UV borders of the end caps, it should bake fine after that.

This ...

... and if this problem is still there just try to bake only the face that has the problem seperately and merge back in Photoshop.

Originally Posted by PredatorGSR

I believe it is because you are using a cage, so the rays are casting from the angles of the cage. You usually get this when your cage isn't tight enough to your mesh. Maya doesn't call it a cage, but it is the setting "geometry" and "surface" in the normal map calculation type. Other programs allow you to actually manually create a cage (Maya might too), while Maya by default just increases the cage distance as a setting.

You can try reducing your cage distance, create a custom tighter cage, or worst case, you can do 2 bakes, 1 with the geometry setting and 1 using the surface method instead. The cage will give you the nice beveled edges, and the surface will get you accurate details, so you can combine them both in photoshop and get the best of both worlds.

It isn't "because he is using a cage" it is how averaged normals work. It has *nothing* to do with cage distance, cage distance only becomes a problem when it gets much too large and starts collapsing in on itself.

The 2 bake method will work, but is pretty hacky.

Originally Posted by MoP

Harden the edges around the ends, it'll change how the rays are cast. I wouldn't bother using a casting cage for this, it should be ok without.

Just harden those edges that make up the UV borders of the end caps, it should bake fine after that.

NO. Simply N to the O.

C'mon Mop, doing this will create nasty obvious seams around the hard edges, you break the cage and lose detail on the edges.

The best thing to do is *understand the problem*, and the problem is that, around these hard angles(the front/end caps) the normals get averaged, this means when the mesh projects out, its ray direction is skewed.

The most simple and obvious fix is to add a little more geometry so that the rays aren't pointing in such a skewed direction. This can be as simple as cutting in just a few edges near where your details are(i would experiment with this method, as it adds a very small amount of geo), or just add an extra edge loop on both ends of your mesh.

Originally Posted by Guriamo

... and if this problem is still there just try to bake only the face that has the problem seperately and merge back in Photoshop.

Sorry, this is just horrible advice.

Originally Posted by pumbaa

adding extra vertical edgeloops on lowpoly for extra support

This is the solution, the problem is you're simply doing it wrong. You can get rid of all of those cuts in the center, the problem is that the 2 extra cuts on either end need to be very close to the ends of the mesh, right now they're like 1/5th into the model, and the detail on the very ends still gets skewed because of it. Make your cuts before your floating detail on either end and the problem will go away.

Meh, my solution is fine, granted it's not the "perfect" solution but it is perfectly acceptable, the seams will be minimal if you do it right (even not noticeable at all most likely), and it will mean you can optimise the lowpoly more if you want (you won't need the supporting edges at all).

But yeah the above solution that EQ posted will work. Just move those supporting edge loops right out to the sides of the lowpoly mesh and adjust the UVs to match.

Well, I actually prefer seperating the shells, never had any problems with seams using this method
... but it depends on the model. I tend to use in for architechture where i have as less polies as possible.

Thats also when it works best to kist bake the plane that give you the problem seperately. Its just a quick way to get the best result, especially for interchangeable parts on a texture.

Thanks a lot guys, Ive tried MoPs first method, and while it worked on the floating geo, it didnt work on the indent parts. So what I tried next was to recreate the indents as floating geo, I figured since it did the trick for the flaoting screws, it might work here too but this didnt do the trick either.

I then tried EQ's method and this created some seam like artefact along the supporting edges.

I just cant seem to get the indent parts right, they're still slightly skewed towards the top.

While Guriamo's method might work, I find it quite troublesome atm so I'll leave it to last, thanks though

Originally Posted by EarthQuake

[...]
The most simple and obvious fix is to add a little more geometry so that the rays aren't pointing in such a skewed direction. This can be as simple as cutting in just a few edges near where your details are(i would experiment with this method, as it adds a very small amount of geo), or just add an extra edge loop on both ends of your mesh.
[...]

Can't one just remove the extra edges after the bake to get rid of the extra geo aggain? Cosidereing you don't mess up the UVs with it, which should realy be doable.

Did you try changing your smoothing groups based off you UV clusters... this might help the baking process so your averaged normals aren't so extreme since it wouldn't be trying to average the entire model.

OR

Try adding 3-4 equal spaced horizontal edge loops around that problem section... after the bake just remove them

Pumba, make sure you're rebaking after making changes to the geometry. If you want, upload your maya file with high+low and I will have a look.

Edit: Another cut, horizontal this time, above those details should fix the "pointing up" issue.

Issue #2(the "seam" type issue) is caused by a lack of resolution. You can:
A. Move the edges a little farther from the end
B. Use hard edges(not cage spliting hard edges, just regular old hard edges, difference being the projection type in maya... "match using", you'll want the default method)

Jet_Pilot: once you remove geometry the mesh normals change, thus requiring you to rebake, in most cases. The mesh could potentially be set up in a way that it would be a small change, but it still might be noticeable.

@THE 5: Yeah, I hope so, they seem unnecessary, but if they're a must for the normal map to work correct I guess they're not..

@Jet_Pilot: Do you mean making every uv border to a hard edge?

@EarthQuake: Hmm yeah, I just kind of realized that the cages geometry doesn't update whenever I edit the lowpoly mesh geometry. I guess renewing the cage geo is included in the rebaking process, not only just pressing "Bake". Is it ok to remodel the cage manually or do I have to Clear the lowpoly mesh from the Transfer Maps window and re-add it (This requires some editing to the cage as well, to encapsulate the floating geo)?

That horizontal cut worked excellent. However the seam at the end does not fully disappear when adding extra resolution and if I were to move the edge closest to the end inwards the screw will be distorted :/
I've sent you a pm EarthQuake. In the following image you can see my transfer maps settings.

Well, this became a huge report. Thanks for all the feedback guys, much appreciated.

If all the detail is floating geo, an easy way to do what Predator suggested with combining the two renders is to render the floaters separately. And when baking them, add an alpha to your output maps.

Ok, so I wanted to write up a little here about general workflows, and why I am so adamant about the methods I use.

Generally speaking, I am morally apposed to using solutions that cause more problems, that in turn need extra hacks to fix(like breaking the edges on the cage), and solutions that just cause more work like doing multiple bakes. When it comes down to a complex mesh, and specifically doing this stuff in a professional environment, all these little tweaks add up. Think about it, when your art director comes over and requests a change, you've gotta rebake your stuff, and redo all this extra little fiddly work. I would rather just fix the problem at the source any day of the week. Sure doing proper ray correction geometry is going to add more tris, but these days triangles aren't that expensive, and as long as you're getting your art created within the allotted geometry count, what is the problem? Its not like you're going to make the game run 50% faster if you save 200 tris on a FPV weapon, hell, you wont see a difference in performance if you trim 2000 tris on a FPV weapon, with a modern game on modern hardware.


So lets run down pros/cons of different methods:

Fixing the geometry itself
Pros:
It just works, no need to fiddle with hacky "fixes"
No need to redo your little hax every time a revision is requested
General quality of mesh tends to be improved(can use the extra geo to help silhouette in some cases.
Normal map artifacts lessened, which is good for: A. Resolution based smoothing errors, B. Compression and C. Generating "detail" maps from your NM in Crazybump.

Cons:
Will use a bit more geometry



Breaking hard edges on the cage
Pros:
Will use a bit less geometry

Cons:
Will create noticeable seams on an asset viewed up close(like a FPV gun)
Countering the seams(Mop suggested making the low slightly smaller than the high) is hacky and predictable
Forcing hard edges to split on the cage means, you can't use hard edges anywhere else on your model without also getting seams there, and hard edges are very good to use to limit resolution based smoothing errors(as shown above) so on top of all this, you'll probably end up doing multiple bakes and composting them



Multiple bakes:
Pros:
Will use a bit less geometry

Cons:
Require excessive repetitive rework for any change made to the mesh
Takes 2x as long to bake your maps


At the end of the day, if you need to use the absolute bare min amount of geometry, I would suggesting looking into hard edges/multiple bakes/etc, otherwise it just isn't productive in the grand scheme of things. Now if this was just one specific problem you'll encounter once in your life, sure, do whatever. But this is an extremely common problem that you will run into on *almost every mechanical model you ever make*. So its best just to have a good way to deal with it.

Pumbaa, looks like you got the skewed projection issues solved, really all you need to do is run this script now to set hard edges via uv islands:

http://www.polycount.com/forum/showthread.php?t=52722

As long as your uvs are set up well(IE: the end cap is on its own island and not welded to the side of the other bit) this will fix these little errors.

As far as resetting cages in maya;
A. I'm generally importing meshes in from modo, so the cage is created new when I import
B. Maya is weird and the cage updating likely gets broken. In your settings make sure "delete after bake" is enabled.
C. Maya's cages doesn't work the same way as max(in max you can fix *some* projection issues by tweaking the cage) so there really is no reason to manually edit the cage.

Actually multiple bakes wouldnt take too much time when changing something..

just bake the part you changed.. I've done that in a professional environment countless times to quick fix meshes, and it didnt take me longer to bake or change anything compared to others who do it different. I found it even handy to have bakes from certain pieces as seperate layers in photoshop.

but this method is only advisable on very specific meshes and workflows. Not suitable for everything or everyone.

Guiramo: I'de prefer to never knowingly use work-flows and procedures that will add more work and more time to asset creation, i guess that's a personal thing but it doesn't seem that crazy to me. =P

Your last sentence I agree with.


If:
A. you'll never have to rework it and
B. you want the absolute min amount of geo at all costs

It can work, however information posted in tech talk has a way of repeating itself, you know the telephone game, someone says "oh just do multiple bakes" and we've got people repeating this advice over and over(without really understanding why). So its best to look at solutions that cover a wide variety of situations when giving advice, if suitable.


PS: Consider this, you get a request to change the UV layout to give more space to a specific area, chances are you will have to rework the entire layout to pack everything back in, and of course redo your bakes entirely. If you're do extra stuff somewhere else, you'll have to do all of that again too. This is a fairly realistic and common situation where it would take a significant amount of time.

fair enough..

but i dont say bake it in millions of steps.. mostly i use like 3 or 4 steps, thats not really hard to redo if needed... failed to mention that i use it when i get pieces that are really not coming out right no matter what...

trust me, I rebaked lots of objects to fit in a new UV layout or things like that, but there is absolutely no problem in any way to fix something quick instead of tweaking the mesh/cage tousands of time and test bake it every time.

Thats exactly why i started using it for quick fixes...as you say, time constraints are always a problem, thats why i started to use the method


Also if you have complex normal maps consisting of multiple objects (like a car interior for example with lots of intersecting meshes) you wouldn't bake it all together at once also, or would you?

Yeah I would do a key-framed exploded bake. Frame one normal, frame 10 low/high exploded to prevent any intersecting errors. 1 bake, no BS. More than anything this will be done for areas that need to animate, or akward shapes that need to be exploded for other reasons, generally speaking, I try to model in such a way that intersecting isn't a huge issue, and that means keeping meshes pretty solid and not an excessive amount of intersections to begin with.

For me, doing modeling work is very quick, doing quick tweaks, cutting in a few extra verts takes a matter of seconds, its faster than setting up a custom bake, baking normals, baking AO, baking RGB, masking/compostiting them back together in PS. Test bakes are also an essential part of the process, i dont know how anyone would get by without doing a quick testbake to check for errors, sooo many things can be quickly fixed by doing a proper test bake.

Also, when you understand how normals smooth you'll be able to simply look at a mesh, understand where your floaters/detail areas are and cut in the required geo, this will all happen during the lowpoly modeling process when you're used to it, so its not a big task to do at the end of production. You cut in a few extra bits to polish it off after your test bake(again which you should be doing regardless, to check for a variety of issues).

Again i always prefer to be proactive, not plan to make fixes at the end. I believe a lot of problems can be solved by proper planing and workflows *before* they even become issues.

One last thing, i'm not trying to say that doing separate maps and combining is an impossible task or it will take you all day, just that to me, it isn't an efficient way to work, i've done all that stuff in the past, but have since seen the light and moved on to better ways of working. =)

I've gone from workflows that were downright terrible, resulting in all sorts of errors and intersectings etc etc and manually painting them out in PS, to baking out multiple maps at different ray distances, different projection types, combining them all together etc. What I do now is so rewarding, you press the bake button and then you're done, its great!

Hey EQ, one thing regarding the separate bakes vs single bakes bothers me:
if you want to explode a mesh for the single bake method, it's kind of hard to do this if the moving parts are not separate objects, especially if the HP parts have to move along easily. In Max you can't have separate objects write to a single texture file, if I'm correct, so you still end up with separate maps per object that need to be combined. Is the answer to this simply to use something else than Max ?

Xoliul: I'm interested in seeing a real-world case of that actually happening.

edit: Just to elaborate: A case that isn't super-easy to get around with a small amount of planning.

Xoliul: In that case you'de simply keyframe each chuck by vertex. Unless i'm missing something

(by that i mean keyframe in sub-object mode)

Well if you do that, then first of all, all your keyframes for each chunk will be condensed into one single keyframe for that object, which doesn't appeal to me that much in case something would go wrong with it. Secondly, to get your HP to transform the same way, you'd have to apply the exact same transformations, so that means copy/pasting the values (slow and tedious!), instead of simply linking them if they were separate objects.
It seems kind of fiddly to do it this way, imo? The chunks could easily get misaligned (if you're c/p'ing values), and it gets more annoying to fix this with the single per vertex keyframe.

Can Maya actually have multiple objects bake to a single texture ?

Originally Posted by Xoliul

you'd have to apply the exact same transformations, so that means copy/pasting the values (slow and tedious!)

No, you just use standard values, which can be used absolutely, incrementally or even scripted.

Originally Posted by Xoliul

It seems kind of fiddly to do it this way, imo?

This is why I asked for a real-world example. After baking a ridiculous amount of assets I don't see what the big issue is that you're running into?