splitting smoothing groups for normal maps

Chase · Dec 2012

When separating Smoothing Groups for a normal map bake, I'm wondering if my process would work...

I watched Alec Moody's process at 3DMotive.com on how to render a normal map. He used the Text Tool script to automatically create the Smoothing Groups wherever he had made his UV splits. He doesn't worry about how the smoothing will look on the model until the very last step, which would drive me insane having shading errors everywhere. Is there any rhyme or reason to create UV splits first and then the smoothing groups?

I tried using the Text Tool operation that would create Smoothing Groups from UV Shells and all it does it make a mess of the Smoothing Groups I had already established. Not sure if you have to have everything in one smoothing group and then apply the operation. My process consists of selecting my entire model after I completing the unwrap, and Splitting the UV Shells by Smoothing Groups in the default 3ds Max Unwrap? Would that be the same thing? Would that be as effective in terms of creating less seems?

Psyk0 · Dec 2012

Here's a good read for you:
http://www.polycount.com/forum/showthread.php?t=107196

passerby · Dec 2012

the idea of it is that having your smoothing seams on your iv seams cost nothing extra, since those verts are already doubled for the uv's

Quack! · Dec 2012

I tried using the Text Tool operation that would create Smoothing Groups from UV Shells and all it does it make a mess of the Smoothing Groups I had already established. Not sure if you have to have everything in one smoothing group and then apply the operation. My process consists of selecting my entire model after I completing the unwrap, and Splitting the UV Shells by Smoothing Groups in the default 3ds Max Unwrap? Would that be the same thing? Would that be as effective in terms of creating less seems?

Yes, setting your smoothing groups by UV shells is the same as splitting your UV shells by smoothing groups. The order in which you do it is up to you, if you feel confident you can set your smoothing groups to where the uv shells should be, then go for it.

You want to split your UVs where seams would fall naturally on the high poly, in seams / crevices. This will hide the seam and make texture creation easier. After that you should do a few quick test bakes with an auto-layed out UV set and take the object and normal map into your engine of choice.

If there are shading errors in the engine, then you have 2 choices.

1. You can add more geo to the area with the bad shading to ease the smoothing errors and keep your original smoothing groups.

2. Or you can split the uv shells, making more of them, in that area and re run the Uv Shells to Smoothing groups script.

Chase · Jan 2013

So do you just use the Select Smoothing Groups by Faces button in the UV Unwrap and then just split them from there? Like you guys have suggested it wouldn't seem like the best resolution since it would double my vert count.

Here's what it looks like when I select faces by smoothing group 1

Not quite sure what the next step is since this selects the hard edges yet splitting them would give me a ton more seems...

Quack! · Jan 2013

If those islands are all hard edges then you need to split them off to their own island to avoid errors.

Is there a reason you don't want to use TexTools' script?

Chase · Jan 2013

That's one smoothing group and about 4 others so Id have to split those as well. Text tools has all of the same tools as the default unwrap tools. Should I be using text tools for any other reason?

Quack! · Jan 2013

Chase wrote: »

That's one smoothing group and about 4 others so Id have to split those as well. Text tools has all of the same tools as the default unwrap tools. Should I be using text tools for any other reason?

I have never seen the 'new' tools in the UV unwrap modifier work. But my experience is with Max 2012, so that may have changed with 13?

Anyway, I think we need to take a step back...

My general workflow goes:

High > Low > Unwrap > Smoothing Groups (TexTools Script) > Triangulate > Bake > Export

As I see it you are flipping the Smoothing Groups and Unwrap part, any reason for this?

I only set smoothing groups manually to achieve a certain effect, but that is relatively rare.

EarthQuake · Jan 2013

Quack! wrote: »

I have never seen the 'new' tools in the UV unwrap modifier work. But my experience is with Max 2012, so that may have changed with 13?

Anyway, I think we need to take a step back...

My general workflow goes:

High > Low > Unwrap > Smoothing Groups (TexTools Script) > Triangulate > Bake > Export

As I see it you are flipping the Smoothing Groups and Unwrap part, any reason for this?

I only set smoothing groups manually to achieve a certain effect, but that is relatively rare.

This is my basic workflow as well. The thing you really need to keep in mind though, is that while uving you're essentially laying out your hard edges/smoothing groups as well. With experience you'll sort of know where you want hard edges/SG breaks but it takes a little bit of effort to adjust to thinking like this.

Chase · Jan 2013

Here's what my unwrap/model looks like. You're right I do have my workflow flip flopped in terms of smoothing groups and UVing. For an example, the inside rim of the holes in my model are set to smoothing group 4 so they don't blend into the interior/exterior which are on smoothing group 2 and 3. I hadn't been thinking about unwrapping as I don't see how setting up your smoothing groups later is any different because the faces you assign would be assigned to the same group. When you say setting up smoothing groups manually you mean selecting faces and setting them to a smoothing group instead of Auto Smooth right?

By setting up my smoothing groups I'm able to have an shading error free model. That's why I thought I could just easily select my model's faces by smoothing group and Break away the edges in the unwrap.

Quack! · Jan 2013

Chase wrote: »

I hadn't been thinking about unwrapping as I don't see how setting up your smoothing groups later is any different because the faces you assign would be assigned to the same group.

Ok. So your workflow MIGHT work. Do a test bake. There are tons of variables that are going to happen in the next step. Depending on how you bake, vanilla max / handplane / xnormal, and depending on what 3D engine you will display it in, UDK, Unity, Max, etc, your errors that MAY appear from the 90 degree and other harsh angles you have on 1 UV island will be different.

Do not get hung up on what the smoothing of the low poly looks like, depending on the baking method your low can look like horse shit and the bake can correct it and make it look beeeeautiful. To a certain extent you can use the low polys smoothing as a basic guide.

It looks like you have UV islands that have very harsh angles within them, greater than 75°. Those types of islands are ones that will give you baking artifacts depending on your baking workflow. You may need to break those islands at the hard angle edge and create a new island and smoothing group, or add a chamfer, if you get triangle artifacts after you bake.

The reason I set my smoothing groups afterwards is exactly for the reason EQ stated. When creating your uv islands, you create them with these harsh angles in mind. You will learn to avoid / allow them as you do more and more bakes. After that setting smoothing groups becomes automatic with no manual labor involved aka FASTER!

When you say setting up smoothing groups manually you mean selecting faces and setting them to a smoothing group instead of Auto Smooth right?

Yes.

Chase · Jan 2013

Setting up the smoothing groups before hand makes a lot more sense to me though I do believe I may have finally understood your method. Does your process prevent any less seems? Now, working from what I have, and any model that I use this work flow with, what would you recommend I do to split the UV islands that have hard edges? I probably sound like a broken record right about now, but shall I do what I was originally going to do which meant selecting the faces by smoothing groups and separating them? I was watching this tutorial that put into practice selecting the faces that are 1 smoothing group and simply detaching them. Or I could use the Textools operation that created the UV islands based on my smoothing groups first and then continue to stitch things together which just seems to produce some fugly UV islands.

Thanks for baring with me guys!

Quack! · Jan 2013

Our process does what your process does with 1 less time consuming step, manually setting smoothing groups. If you create hundreds of objects for a game or even a handful of high triangle count objects our method will save you quite of bit time by removing that step. Again if you have the wherewithal to set your groups exactly where your UV islands need to be, go forth and do!

Autocon · Jan 2013

Quack! wrote: »

High > Low > Unwrap > Smoothing Groups (TexTools Script) > Triangulate > Bake > Export

I have TexTools installed but I see no option where it creates smoothing groups based off UV shells. Care to share how this is done? As I have always set my smoothing groups up before hand and then unwrapped, but if I can auto that shit after I UV it would be AMAZING!

Quack! · Jan 2013

Autocon wrote: »

I have TexTools installed but I see no option where it creates smoothing groups based off UV shells. Care to share how this is done? As I have always set my smoothing groups up before hand and then unwrapped, but if I can auto that shit after I UV it would be AMAZING!

Right above the "Edit UV" button is a small Tools▼ drop down. In there is "Smoothing groups from UV Shells"

Chase · Jan 2013

It makes sense now. Just to be clear you're making uv shells based on the smoothing groups you want to have. Then automatically assigning the smoothing based on the shells. Instead of hand assigning the SG your unwrap allows you to do it with one click. Yeah?

AlecMoody · Jan 2013

You can split your UVs based on your smoothing if you want, whatever makes more sense to you. I think working in that direction it is easier to stitch two shells together and then forget to smooth the edges that join them.
Also, in that briefcase video I was working as quickly as possible while also talking so the UVs / smoothing splits aren't exactly optimized. You should try and find the limit of how much stitching you can get away with for your baking software/engine combination.

Eric Chadwick · Jan 2013

Quack! wrote: »

Right above the "Edit UV" button is a small Tools▼ drop down. In there is "Smoothing groups from UV Shells"

It doesn't work properly though. See Joe's explanation here.
http://www.polycount.com/forum/showthread.php?p=1564703#post1564703

I think I found a solution that works. It's messy, but seems to work properly.
http://www.polycount.com/forum/showthread.php?p=1747860#post1747860

Quack! · Jan 2013

It makes sense now. Just to be clear you're making uv shells based on the smoothing groups you want to have. Then automatically assigning the smoothing based on the shells. Instead of hand assigning the SG your unwrap allows you to do it with one click. Yeah?

Yup.

Eric - yea I noticed that too. Will check out the fix.

Chase · Jan 2013

I put into practice what you guys have taught me so lets see if I got this down. The numbers represent the smoothing group. There's not a lot you can do to hide seems like these other then the one the arrow is pointing to right?

I went about this by setting the smoothing groups first, then creating UV shells based on the SGs using Textools with some edge refining. I know I could've just as easily forgone setting the SGs and went with creating the UV shells based on where the SGs would be and then using Textools to apply SGs based on the UV islands. If I were to be more vertex efficient I could stitch the hexagon shops together and the boxes that share the same SGs, thus lining up the smoothing splits and UV splits. Did I get it?

Snader · Jan 2013

You could make those 6-sided cylinders unwrapped into a 'flower' instead, that takes care of half or so of the seams. Or you could unwrap the sides of it like the side of the large shape. That would also take care of half the seams.

Then adjust your smoothing groups accordingly.

WarrenM · Jan 2013

Plus it would make texturing easier. Each side broken off of the cylinder like that will lead to hard to fix texture seams.

Optinium · Jan 2013

So what is the case when it comes to an organic model, for example a pelted island, when using Textools it gives the island one smoothing group which obviously causes this error

I've got this issue on some parts of a model I'm working on at the moment, I've tried to be smart with the unwrap but it can still be seen in places. Object space normals do the job but as far as I'm aware UDK isn't compatible with them

(Despite being an organic model it won't be animated if that makes a difference)

Chase · Jan 2013

Makes sense. I didn't even think of that. But otherwise does it look like I'm comprehending it?

Quack! · Jan 2013

You never know if what you are doing is correct until you bake....so bake it!

Autocon · Jan 2013

Eric Chadwick wrote: »

It doesn't work properly though. See Joe's explanation here.
http://www.polycount.com/forum/showthread.php?p=1564703#post1564703

I think I found a solution that works. It's messy, but seems to work properly.
http://www.polycount.com/forum/showthread.php?p=1747860#post1747860

Eric when you get to step 6. "Use EdgeSmooth to set the hard edges" Where is EdgeSmooth located? I dont see anything under the Edge Selection tab that would be EdgeSmooth. And adding the Smooth modifier gave me a lot of errors.

Also, thanks Nick for pointing out the drop down, this will save me SO much time!

joeriv · Jan 2013

He is probably talking about this in that step:
http://www.polycount.com/forum/showpost.php?p=1564713&postcount=18

And another thanks from me for finding that out and sharing

Eric Chadwick · Jan 2013

Yep, it's Pjanssen's tool. A hard/soft edge system like Maya's is so much better than smoothing groups, imho.

joeriv · Jan 2013

Links to the actual script seem to be down, and I cannot seem to build the script from the github page (VS errors, and googling doesn't really help).
Anyone perhaps have the actual script or a installer (for max2013)?

Or if I am missing it, a working link.

Eric Chadwick · Jan 2013

joeriv wrote: »

Links to the actual script seem to be down, and I cannot seem to build the script from the github page (VS errors, and googling doesn't really help).
Anyone perhaps have the actual script or a installer (for max2013)?

Or if I am missing it, a working link.

On GitHub, use the ZIP button (to the right of "Clone in Windows"). Open up the zip, and put what's in the Installer folder into a new zip file. Change the extension to MZP. Then start Max and drag-n-drop that onto the viewport.

Autocon · Jan 2013

Sounds awesome, will have to give that a try soon. After using Hard Edge Maya smoothing Max feels so inferior with smoothing groups. It is really nice begin able to select saved smoothing groups but trying to make a hard edge was always a nightmare.

Chase · Jan 2013

How does this script work uv in conjunction with textools since textools could create uvs based on your smoothing groups or make the smoothing groups based on your uvs. I assume you're unable to use that feature with this Maya script?

Eric Chadwick · Jan 2013

EdgeSmooth is a separate tool, it doesn't require TexTools at all. Both are Maxscripts for 3ds Max, not Mel scripts for Maya.

I want a tool that will change all my UV border edges into hard edges. TexTools has a quick way to select the border edges, but it does not send that selection down to the Editable Poly. Which is where EdgeSmooth does its work. So, I used TexTools "uv to 3d" tool to pass that selection down.

I hope that makes sense.

Chase · Jan 2013

I just meant during the unwrapping phase. So pretty much the "Smoothing Groups to UV Shells" and "UV Shells to Smoothing Groups" within Textools won't work with the EdgeSmooth script? When I say work with I just mean that it won't see the hard edges the EdgeSmooth tool makes as smoothing groups to make the UV shells in Textools. From what it sounds like you're saying it won't do that right?

Eric Chadwick · Jan 2013

EdgeSmooth doesn't do any UV work, it only changes the Smoothing Groups. It doesn't work with TexTools in any way, it is a totally separate tool.

TexTools can change the Smoothing Groups to match the UV shells, if you use the Split button. But it does not create hard edges for ALL shell edges... it only applies a single SG for each shell. For example, if you have a "flower" type shell, where you've split the model between the "petals", if you use Split those edges won't become hard edges because they all get the same SG.

You could use EdgeSmooth before TexTools, to set up smoothing groups the way you want. Then you could use TexTools to make UV shells out of those smoothing groups. Kind of a weird way to work, IMHO. But to each his own, it might work well for you.

Chase · Jan 2013

Yep, that's what I was wondering. Thanks Eric. I know, everyone else seems to worry about applying hard edges after they unwrap. Both makes sense to me, but do the smoothing groups first is just something I've been doing for awhile.

Quack! · Jan 2013

For now chase I would recommend sticking to the basics. Get a good bake going with just running the textools script. The fix that eric posted is a necessary step for better bakes, but just getting a base level bake done well has so many variables in it that adding more steps to it can only complicate. Start basic and work your way up to all the 'fixes'.

Enough talking, I want to see some normal maps! :]

Chase · Jan 2013

haha can do Quack! I wasn't doing a normal for this prop more so that I wanted to know what I would do if I were to do a bake. I'm working on a street scene that I'm sure will have some normal maps...fire hydrant, garbage cans, apartment buildings, etc. If it's ok with you might I keep you updated on what I have so you can tell me if I'm heading in the right direction? I also have a character I could do a bake for! I'll do that too! Oh man, so much to do and not enough time. I feel like the Alice in Wonderland rabbit. Off I go

Psyk0 · Jan 2013

Eric Chadwick wrote: »

On GitHub, use the ZIP button (to the right of "Clone in Windows"). Open up the zip, and put what's in the Installer folder into a new zip file. Change the extension to MZP. Then start Max and drag-n-drop that onto the viewport.

When i drop the MZP in the viewport i get "this 3d max version is not supported by the edgesmooth plugin in this installer"

Max 2010 x64 SP1

Any ideas?

Eric Chadwick · Jan 2013

Psyk0 wrote: »

When i drop the MZP in the viewport i get "this 3d max version is not supported by the edgesmooth plugin in this installer"

Max 2010 x64 SP1

Any ideas?

I had the same problem, haha. Solution here
http://www.polycount.com/forum/showpost.php?p=1750339&postcount=21

joeriv · Jan 2013

probably because if you do it that way it's always missing "something" untill you compile it.

see this post: (+ a new download link altough it seems to be for
2012-2013 only)
http://www.polycount.com/forum/showpost.php?p=1750339&postcount=21

Eric Chadwick · Jan 2013

Yeah his temp d/l link works for me, in 2010 32bit.
http://edgesmooth.pjanssen.nl/

joeriv · Jan 2013

Oh for some reason I read it as 2012 of 2010, woops

Psyk0 · Jan 2013

Sweet, thanks guys

Quack! · Jan 2013

Eric Chadwick wrote: »

EdgeSmooth is a separate tool, it doesn't require TexTools at all. Both are Maxscripts for 3ds Max, not Mel scripts for Maya.

I want a tool that will change all my UV border edges into hard edges. TexTools has a quick way to select the border edges, but it does not send that selection down to the Editable Poly. Which is where EdgeSmooth does its work. So, I used TexTools "uv to 3d" tool to pass that selection down.

I hope that makes sense.

Eric, I think this script is a fix to the problem? http://www.polycount.com/forum/showpost.php?p=1332118&postcount=20

Gonna try it out.

Quack! · Jan 2013

Ok I tested this out and it works!

If I run the script from here: http://www.polycount.com/forum/showpost.php?p=1332118&postcount=20

I get an error that asks me that it is about to collapse and break a bunch of shit, but if you turn off the error through the check box and choose Hold/Yes, the next time you run the script it will set only the uv edges to hard. Works a treat!

I have it 'fanned' on the bottom edge of the small face.

Eric Chadwick · Jan 2013

Cool! Curious how that works with a more complex mesh. EdgeSmooth seems to do some wonky things with the normals. Could be the 32-SG limit.

splitting smoothing groups for normal maps

Replies