Normal map baking, horrid smoothing...

[martynball]

Hey guys, i'm trying to bake normal maps for a model, just a section of a model anyway for testing. Now to catch all of the highpoly detail im under the impression - after reading loads of tutorials and articles on how normals work - that you need to give the model 1 smoothing group.

Now, due to there being 90 degree angles, and 1 smoothing group. The smoothing is going to look rather bad. But I thought that this is hidden when normal maps are applied. But as you can see below, the horrid smoothing is there..
http://dl.dropbox.com/u/63494571/Models/Help/crappyShading.jpg

Normals (they have errors for some reasons :S but one problem at a time):
http://dl.dropbox.com/u/63494571/Models/Help/scope_lowNormalsMap.jpg

The highpoly is the outer coating of this scope:
http://dl.dropbox.com/u/63494571/Models/Halo%204/BR55%20HB/h4_br_prog9.jpg

[AlecMoody]

Hey guys, i'm trying to bake normal maps for a model, just a section of a model anyway for testing. Now to catch all of the highpoly detail im under the impression - after reading loads of tutorials and articles on how normals work - that you need to give the model 1 smoothing group.



The information you have been reading is false/incomplete. What software are you actually baking in?

[sprunghunt]

Why are you modeling this piece separate to the rest of the scope? You won't need most of the polygons that are causing the smoothing problems.

[martynball]

Yes I will, the polygons that are causing the smoothing errors are the 90 degree angles, they are unavoidable. Only thing I can think of is to chamfer the edges...

[Baltic Beazt]

You don't need to chamfer those edges.
Hard edges have to be on separate smoothing groups and UV islands have to be split.

Like in this image:
http://i.imgur.com/YgC0V.png

Nice battle rifle btw.

[sprunghunt]

Yes I will, the polygons that are causing the smoothing errors are the 90 degree angles, they are unavoidable. Only thing I can think of is to chamfer the edges...

I didnt say that was the only problem you have. What is the image of the mesh with the normal map from?

[martynball]

Thanks Baltic Beazt, and its from the scope sprunghunt

[sprunghunt]

Thanks Baltic Beazt, and its from the scope sprunghunt

I meant what engine/ renderer is it in? What did you use to bake the map?

[martynball]

Oooh, it's in 3ds max. Using mental ray

[Computron]

there's a long explanation but if you got any money, I recommend the normal map baking video series on 3dMotive by Alec Moody(IIRC), he also posts on polycount. Thats the easiest most comprehensive tutorial without having to read 15 pages of the 'who put waviness in my normals' thread.

Your edges are way to tight for the normal map anyway, try to loosen them up like mine:

Halo 4 Battle Rifle (WIP) (http://www.polycount.com/forum/showthread.php?t=96854)

[sprunghunt]

Oooh, it's in 3ds max. Using mental ray

In the material is your bump strength settings at 30 (the default)? If it is - set it to 100. This is in the list of the different maps you can have diffuse, specular, etc. They all have a number next to them.

Also make sure you're using a normal bump node instead of just placing the bitmap straight into the bump slot.

[MightyPea]

I would seriously recommend you just don't render at all.
Normalmaps are a technique intended for realtime 3d, preview it that way. Not only will it more accurately reflect what it'll look like in the game-engine, it'll be faster and easier. You'll be able to rotate around the model, seeing the spec move around, which is much more helpful in determining whether the normals look good or not.

Xoliul's shader is free and works great.

As for normalmaps correcting the lowpoly shading, the short story is: yes, ideally it would. That's in the ideal case when what you use to bake and display the model use the exact same technique, which you'll often find is not the case.
Even in that rarely seen ideal situation, on something like this you'd still run into issues if you don't chamfer anything or use smoothing groups.

[martynball]

Computron, I will look into that, I did try to keep the edges smooth and not tight.

sprunghunt, I set it on 80. The normal maps are viewable, and yes I used the Bump material and didnt apply the normals straight to the slot.

MightyPea, I only rendered it to quick have a look if it looks okay. I usually use Xoliul's shader, but at the time I didn't feel like closing max down to switch from the Nitros rendering engine Autodesk have create, which sucks.

[sprunghunt]

sprunghunt, I set it on 80. The normal maps are viewable, and yes I used the Bump material and didnt apply the normals straight to the slot.


If its set to 80 it's 20% wrong. You need to set it to 100 to get the same effect as a real time shader, full normal replacement. You could still have smoothing errors at 100 but they'll never be completely fixed if you leave it at 80,

I also would not recommend mental ray. It's not designed for normal maps. Even the max scan line renderer is more normal map friendly. But you should probably use something like crysis or udk for most realistic results. And these engines are free.

[martynball]

I will use the Xoliu shader in the future as I like it. And I used mental ray because for some reason the scanline renderer was messing up, wouldnt render anything. Will see what I can do today though

[lefix]

so, as soon as you have a 90° or higher angle in your lowpoly -> add hard edges (smoothing group splits) and split the uv islands.
89° angles will still work fine however?

[sprunghunt]

so, as soon as you have a 90° or higher angle in your lowpoly -> add hard edges (smoothing group splits) and split the uv islands.
89° angles will still work fine however?

Sometimes you can get away without hard edges. Or just adding hard edges in some spots. Sometimes it's a better idea just to chamfer the edges. This is a judgement you learn to make from experience.

Usually I will model something completely then add 1-2 smoothing groups to fix the worst spots.

[AlecMoody]

so, as soon as you have a 90° or higher angle in your lowpoly -> add hard edges (smoothing group splits) and split the uv islands.
89° angles will still work fine however?

The angle of the edge is largely irrelevant. It is much more important to think in terms of separating flat surfaces and attempting to avoid smoothing across extreme topology. An even simpler rule of thumb is that if it shades badly without the normal map (and you are using an unsynced workflow) then it will shade badly with the normal applied.

[EarthQuake]

The angle of the edge is largely irrelevant. It is much more important to think in terms of separating flat surfaces and attempting to avoid smoothing across extreme topology. An even simpler rule of thumb is that if it shades badly without the normal map (and you are using an unsynced workflow) then it will shade badly with the normal applied.

Right, the "90 degree" thing is a one of these really annoying things people repeat over and over that has no real relevance, you just need to use your eyes and do testbakes, and then fix your smoothing errors either by adding bevels, or hard edges, or simplifying your topology. With experience you will be able to easily spot areas that will give you problems.

Simplifying your topology is a huge one, either after the fact or planing your high and low meshes so that you can avoid lots of little harsh angle changes. See: http://www.polycount.com/forum/showpost.php?p=1288654&postcount=9 for an example of how to change up your design to natually have more soft transitions.

[martynball]

I have added hard edges, but due to the lowpoly not aligning correctly, is there a way to avoid these nasty black lines on the edges and to have the smooth edge look smooth...

Problem:
http://dl.dropbox.com/u/63494571/Models/Help/bad_edges.jpg

UVW's:
http://dl.dropbox.com/u/63494571/Models/Help/bad_edges_uvws.jpg

Normal Maps:
http://dl.dropbox.com/u/63494571/Models/Help/scope_lowNormalsMap.jpg

Reason im just doing this one bit at the moment is to understand in the future why problems are there. So then I can make the rest lowpoly and model it in a way to avoid the errors i'm having here.

[sprunghunt]

Simplifying your topology is a huge one, either after the fact or planing your high and low meshes so that you can avoid lots of little harsh angle changes. See: http://www.polycount.com/forum/showpost.php?p=1288654&postcount=9 for an example of how to change up your design to natually have more soft transitions.

This is however a luxury that doesn't exist when working on a realistic game. So it's not such a good idea to rely on just changing the highpoly model all the time.

[EarthQuake]

Make sure you're baking with the cage and not that useless "use offset" thing. Also make sure the low is aligned correctly, so many little problems are fixed simply by making sure the low is as close to the high as possible.

[EarthQuake]

This is however a luxury that doesn't exist when working on a realistic game. So it's not such a good idea to rely on just changing the highpoly model all the time.

Bullshit dude, even in "realistic games" 99% of models aren't *that* accurate. Knowing how and when to simplify a model to smooth out your mesh normals is a hugely important skill, the link I posted is only one example on how to do it.

I think you'll find that at most studios being "realistically accurate" is far less important than making nice looking efficient models.