Reply
Reply
 
Thread Tools Display Modes
MoP's Avatar
Old (#1)
OK, sensationalist headline. But it got your attention! This was a spinoff from another thread, I thought it deserved its own.

Here are my findings based on Max vs. Maya baking using the native tools and displaying the results in the viewport using the native tools.

Summary:
Unless your game engine is set up to use the same tangent basis as 3ds Max's renderer (not it's viewport), your normal-mapped models will never look 100% correct!

I have just done some simple tests tonight, and I can confirm that Max's "baking" tangent space is the same as it's "render" tangent space (no surprise, since the scanline renderer is used to bake normals in Max), however the "render" tangent space does not match the tangent space that it uses in the viewport. This strikes me as just plain insane. It means you'll never get the same result between rendering and viewport display.

If you use a "correct" normal-map for viewport display in Max, then it will look messed up in render, and vice versa... horrible.

I baked the same objects in Maya, and they are displayed perfectly in the Maya viewport. Interestingly enough, this map also looks perfect in Max's viewport! That means that the viewports in both apps derive tangent basis in the same way. However it also means that Maya's baking renderer correctly uses this tangent space! Which makes sense, of course.

It basically seems like since the viewports (and Maya's baker and Mental Ray, by extension) seem to all use the same tangent basis derivation, this is the most "correct" one, and therefore 3ds Max's renderer and baker are "wrong". This should be a standard that all applications adhere to, I see no reason why it shouldn't be.

Screenshots as promised...
Comparison of viewport shaders (just using both apps' native display of default shaders, nothing fancy or custom - grey Phong materials with high specular):


Max's scanline renderer normal map output:


Max's Mental Ray normal map output (ignore lack of anti-aliasing, that's not what's being tested here):


Maya's normal map output:


If you're feeling intrigued you could download those normal maps and flick back and forth between them in an image viewer to see the main differences. You should be able to see them clearly enough here, though.

So, out of interest, does anyone here work in a pipeline where they bake their normal-maps using 3ds Max, and if so, do you know what your game does with them? Have you ever had any problems?

EDIT: ADDENDUM!

Here are some direct links to some of the more useful posts from later in the thread:
EDIT 2! 3POINT SHADER RELEASED!
http://www.3pointstudios.com/3pointshader_about.shtml
Offline , MoP, 11,626 Posts, Join Date Oct 2004, Location London, UK  
   Reply With Quote

DarthNater's Avatar
Old (#2)
Did you flip the green channel? I take my bakes into Photoshop and invert the green...

Second question: Does the same thing occur when using Mental Ray in Max?
Offline , polycounter, 920 Posts, Join Date Jun 2009, Location PA Send a message via MSN to DarthNater Send a message via Skype™ to DarthNater  
   Reply With Quote

MoP's Avatar
Old (#3)
Quote:
Originally Posted by DarthNater View Post
Did you flip the green channel? I take my bakes into Photoshop and invert the green...
You shouldn't have to invert the green channel in Photoshop, there is a setting for that in the Render To Texture "Options" dialog. Just set it to the correct Y orientation before you bake.
Every application has a control for this (in Maya it's based on whether the mesh's "tangent space" attribute is set to Left-Handed or Right-Handed), if you're inverting the green channel in Photoshop then you're not baking it right

Quote:
Originally Posted by DarthNater View Post
Second question: Does the same thing occur when using Mental Ray in Max?
I have only tried the Scanline renderer so far. I will try Mental Ray on the same objects, hopefully it will give better results (although that also means more inconsistent results within Max...).

Edit: After trying Mental Ray inside Max (added image to original post), while it is slightly different to the scanline bake, it is still far from being correct, if we're treating the Maya bake as the "most correct" bake.
Offline , MoP, 11,626 Posts, Join Date Oct 2004, Location London, UK  
   Reply With Quote

Vrav's Avatar
Old (#4)
Gasp! Good thing I use Xnormal.
Offline , polycounter, 1,252 Posts, Join Date Nov 2008, Location Seattle, WA Send a message via AIM to Vrav  
   Reply With Quote

DarthNater's Avatar
Old (#5)
Quote:
Originally Posted by MoP View Post
You shouldn't have to invert the green channel in Photoshop, there is a setting for that in the Render To Texture "Options" dialog. Just set it to the correct Y orientation before you bake.
Every application has a control for this (in Maya it's based on whether the mesh's "tangent space" attribute is set to Left-Handed or Right-Handed), if you're inverting the green channel in Photoshop then you're not baking it right
Good to know, never saw that option Thanks!
Offline , polycounter, 920 Posts, Join Date Jun 2009, Location PA Send a message via MSN to DarthNater Send a message via Skype™ to DarthNater  
   Reply With Quote

Marine's Avatar
Old (#6)
so what you're saying is that maya is awesome and max sucks
Offline , dedicated polycounter, 1,628 Posts, Join Date Oct 2004,  
   Reply With Quote

Racer445's Avatar
Old (#7)
I pretty much bake all my normal maps with Max. I've thrown the straight Max normals into Source and Cryengine 2 and haven't once had any problems like this. Then again, I'm also mainly doing lots of hard surface and mechanical stuff.

It really does look like the green channel needs flipping, and you should try using Xoliul's max shader, since the max default doesn't display normal maps correctly.

Last edited by Racer445; 12-08-2009 at 03:06 PM..
freelance 3d artist and certified hood classic

http://www.racer445.com/
Offline , polycounter, 1,215 Posts, Join Date Jun 2008, Location the slums of shaolin  
   Reply With Quote

eld's Avatar
Old (#8)
This is definately most interesting, but I believe most people will not think this far, and even wont go as far as to properly set their hard edges even for baking
Offline , veteran polycounter, 4,220 Posts, Join Date Nov 2004,  
   Reply With Quote

aesir's Avatar
Old (#9)
I paint out annoyances in photoshop.
Offline , veteran polycounter, 4,060 Posts, Join Date Nov 2004, Send a message via AIM to aesir  
   Reply With Quote

MoP's Avatar
Old (#10)
I guess the questions we should ask here are:

1. Which application is using the "best" tangent basis derivation? As far as I can tell, Max's renderer tangent basis is the "odd one out" - more apps like Maya, XNormal, maybe others are consistent between each other, yet Max isn't even internally consistent!

2. Why is Max's renderer tangent basis different to it's viewport tangent basis?
This essentially means that if your normals look correct in the viewport, then they will look wrong in the render. Conversely, if they look correct in a render, they will be wrong in the viewport. This is ridiculous!

3. If you can write an FX shader that uses Max's "renderer tangents" in the viewport, why doesn't Max's default internal shader and supplied FX files use this method? Both Max's "Show Hardware Map in Viewport" option, and it's supplied external shader files like "StandardFX.fx" use the default viewport method, which means any bakes you do in Max will look crappy using these shaders! Insane!
Offline , MoP, 11,626 Posts, Join Date Oct 2004, Location London, UK  
   Reply With Quote

eld's Avatar
Old (#11)
aesir, I think you're missing the point of what mop is trying to explain though
Offline , veteran polycounter, 4,220 Posts, Join Date Nov 2004,  
   Reply With Quote

Ark's Avatar
Old (#12)
Interesting... the maya bake seems to capture more colour information than both of the max bakes.

Could we get the source files to test in other apps to see if its just max what does this?
Offline , dedicated polycounter, 1,657 Posts, Join Date Aug 2009, Location Lancashire, UK  
   Reply With Quote

eld's Avatar
Old (#13)
I'd love get to know exactly how a top of the line engine like unreal engine does this.
Offline , veteran polycounter, 4,220 Posts, Join Date Nov 2004,  
   Reply With Quote

Sage's Avatar
Old (#14)
Mop I find it odd that you are posting this now, I thought you used to be a Max user then had to go to Maya. I'm wondering if this happened in older versions of Max or is it an upgrade that came with 2010. I mean they did change how you could shift drag things to extrude edges, so maybe they thought it was a good idea to update the viewport display and not do so with the scanline renderer so now they don't match.

I'm also wondering if this is the case then how is it that you can make normal maps in Max and use them in unreal ed so they look correct or does unreal for example have code that can render this out properly because a coder took the time to do so.

It be nice if all normals maps were created equal
Alex Perez
Gallery
Offline , card carrying polycounter, 2,003 Posts, Join Date Oct 2004, Location Leominster, MA  
   Reply With Quote

MoP's Avatar
Old (#15)
Quote:
Originally Posted by Sage View Post
I'm also wondering if this is the case then how is it that you can make normal maps in Max and use them in unreal ed so they look correct or does unreal for example have code that can render this out properly because a coder took the time to do so.
(and eld's point, added the quote in an edit)

Well yeah, I was wondering that too, since I thought they baked stuff in Max?
That either means they've set up the Unreal Engine to use Max's renderer tangent derivation (meaning maps baked in any other app will look wrong!), or they're getting dodgy looking stuff in-game and dealing with it by adding geometry till the problem goes away... or... they just don't care that much

Quote:
Originally Posted by Sage View Post
Mop I find it odd that you are posting this now, I thought you used to be a Max user then had to go to Maya. I'm wondering if this happened in older versions of Max or is it an upgrade that came with 2010. I mean they did change how you could shift drag things to extrude edges, so maybe they thought it was a good idea to update the viewport display and not do so with the scanline renderer so now they don't match.
Nope, it's been like this as long as I've used Max. Definitely since max 8, and it was in max 2008 and still a problem in max 2010.

Quote:
Originally Posted by Sage View Post
It be nice if all normals maps were created equal
Yes, exactly - we should have some sort of standards for this (ie. how the normals/tangents/bi-normals/whatever are calculated).
Offline , MoP, 11,626 Posts, Join Date Oct 2004, Location London, UK  
   Reply With Quote

CrazyButcher's Avatar
Old (#16)
uhm I think even using custon FX in max doesn't save you from artifacts. Afaik the scanline baker bakes for the scanline renderer, which uses different kind of interpolation than what realtime rendering does (not just per-vertex tangent basis). At least that's what I read. One would have to modify the baker in max to get perfect matches. I also remember people here saying that some engines managed to load stuff artifact free in games, but had to do something "special" what that is, they couldn't tell due to NDA stuff.
how that is nowadays I don't know, but last time I checked it was still easy to create tangent maps that only look perfect in scanline.
If I recall correctly then ASE used in unreal import also doesn't encode tangent stuff, which means they indeed must do things the same way max does when importing.

Last edited by CrazyButcher; 12-08-2009 at 03:21 PM..
Offline , dedicated polycounter, 1,410 Posts, Join Date Nov 2004, Location Germany Send a message via ICQ to CrazyButcher  
   Reply With Quote

eld's Avatar
Old (#17)
Quote:
Originally Posted by MoP View Post
or... they just don't care that much
this
Offline , veteran polycounter, 4,220 Posts, Join Date Nov 2004,  
   Reply With Quote

EarthQuake's Avatar
Old (#18)
Quote:
Originally Posted by Racer445 View Post
I pretty much bake all my normal maps with Max. I've thrown the straight Max normals into Source and Cryengine 2 and haven't once had any problems like this. Then again, I'm also mainly doing lots of hard surface and mechanical stuff.

It really does look like the green channel needs flipping, and you should try using Xoliul's max shader, since the max default doesn't display normal maps correctly.
You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.
Offline , Moderator, 10,725 Posts, Join Date Oct 2004, Location Iowa City, IA  
   Reply With Quote

Sage's Avatar
Old (#19)
Mop how did you notice this? Did you start writing hlsl shaders or something and noticed the difference? I imagine since unreal is using hlsl for their shaders it's the same, so maybe they have a different baker. Maybe some of epic guys can say something about this.

Mop I think it's pretty obvious that Autodesk doesn't care that much. I mean look at the Max 2010, it's awesome..
Alex Perez
Gallery
Offline , card carrying polycounter, 2,003 Posts, Join Date Oct 2004, Location Leominster, MA  
   Reply With Quote

MoP's Avatar
Old (#20)
Sage: I noticed it first, a long time ago, when I was working purely in Max. I thought for a while that it was my lowpoly geometry causing the issues. Then after working with XNormal then Maya for a bit, I realised that they could bake the same stuff and display perfect results, yet Max's was always dodgy.

Thing is, it seems like Max's viewport calculates tangents almost identically to Maya's baker and viewport. So at least that seems like a more consistent calculation.
Is there any reason why the calculations should be different for max's renderer?!
Offline , MoP, 11,626 Posts, Join Date Oct 2004, Location London, UK  
   Reply With Quote

Sage's Avatar
Old (#21)
Quote:
Originally Posted by EarthQuake View Post
You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.
EQ you use Modo. How much easier is it to get proper results than in say Max.
Alex Perez
Gallery
Offline , card carrying polycounter, 2,003 Posts, Join Date Oct 2004, Location Leominster, MA  
   Reply With Quote

eld's Avatar
Old (#22)
Mop, are you implying that max should be and has always had perfect features?

tip of the iceberg.. ;)
Offline , veteran polycounter, 4,220 Posts, Join Date Nov 2004,  
   Reply With Quote

Sage's Avatar
Old (#23)
Mop the reason is most likely they forgot it was different and no one renders normal maps. Oh wait....
Alex Perez
Gallery
Offline , card carrying polycounter, 2,003 Posts, Join Date Oct 2004, Location Leominster, MA  
   Reply With Quote

Racer445's Avatar
Old (#24)
Quote:
Originally Posted by EarthQuake View Post
You're also adding 5000 hard edges to your models to get a correct result in max, you could take those same models with totally smooth normals in maya and bake them with results that are *near flawless*.
I highly doubt something with many many 90 degree angles would bake well in any app without adding edge bevels or adding in hard edges.
freelance 3d artist and certified hood classic

http://www.racer445.com/
Offline , polycounter, 1,215 Posts, Join Date Jun 2008, Location the slums of shaolin  
   Reply With Quote

Chai's Avatar
Old (#25)
Well I heard of this before, but very interesting to see an example cheers ...
Seems max can only overcome smoothing issues based on raytracer than realtime.

There's a way to get good maps in max (or any other engine) by setup your smoothing groups and uvs properly.
Below is a map I've done with max (realtime viewport) a while ago. (before I switched to xnormal)

Anyway thanks a lot MoP, I was using xNormal without even knowing I could get away with less smoothing errors.
(I have to revise my normalmap article now)

Offline , triangle, 450 Posts, Join Date Jul 2006, Location Brisbane, Australia  
   Reply With Quote

Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.

Copyright 1998-2014 Polycount