Home Unreal Engine

Correct way of baking normals for UE4

a3D
Feeling lost.

I've been having trouble trying to bake correct tangent space maps for UE4.
My first approach was the traditional one. I use Maya 2014.
I exploded my mesh, triangulated, managed smoothing groups according to UV splits, and baked a Tangent Space normal.

The results looked clean in every application I had experience with, including 3DO, Marmoset and obviously Maya itself.
(picture related)
35iq9p3.jpg

However, upon trying to view it in Unreal Engine 4, the results were less than satisfying.
wuo0nb.jpg

I am aware tangents are calculated differently in each application, and that for better results you are supposed to synch them.
Unfortunately I am not familiar with this kind of workflow. So I did some research.

I found this:
https://docs.unrealengine.com/latest/INT/Engine/Content/Types/Textures/NormalMaps/Creation/index.html

Apparently, there is this new workflow involving "tangents and binormals" which supposedly would remove the need for supporting geometry and produce clean results on the spot.
It would be great, IF I could figure out what I'm doing wrong...

This is what I did so far:

1) I exported an FBX with smoothing groups, without tangents & binormals (UE4 doc says T&B are important, but in the settings screen they are NOT checked)

2) I also exported an FBX with smoothing groups AND T&B checked, just to be extra sure

3) I imported the FBXs in xNormals 3.19.1, AND the hipoly

4) Baked separately the one with and the one without T&B. Got 2 maps.

5) Imported the lowpoly FBXs in UE4.
I DIDN'T assign the map in maya and re-export them. I imported the lowpolys I used for baking, and assigned the normalmaps in the UE4 material editor.
Yes, I changed the compression setting and texture group. The software recognized the files as Tangent Space Normals.
Here is where the firts problem appeared, however.
The documentation shows a checkbox, in the FBX importa settings, which says "import normals". This checkbox doesn't exist in my FBX import settings windows.
(see picture)
2yvpf91.jpg

6) Anyway, both models had horrible normals. They didn't look nearly as clean as I hoped. More, they looked bad in other applications as well, because I also tried to make a single smoothing group regardless of UV splits, as suggested in teh UE4 docs.
I also tried to keep my original smoothing groups, they still look bad in UE4.

PART 2: object to tangent

So I read xNormals can convert an object space map into a tangent space one, and this workflow is used to bake object space maps in your application of choice and then convert them in a tangent space map

This is what Handplane does, if what I read is correct. xNormals has this function as well, so I tried this too.

I baked my object space map in Maya. It looked super-clean as expected from an object space map.

Then I imported the lowpoly and the object space map in xNormals and converted it to a tangent space.

It suddenly looked terrible again, both in Maya and in UE4

conclusion

I feel like I've ran out of options and ideas.

I don't know what to do anymore.

Please help me.

Here are the lowpoly and the hipoly, if you want to examine them or attempt the bake yourself.
http://www.filedropper.com/barrellbake
If you can make this look good in UE4 and explain how you did it, I couldn't thank you enough. If I solve this, I promise I will help anyone I see struggling with similar issues, no matter how slow or lazy they look.


(baking considerations:

-since it's a cilinder, when baking in the traditional way I used to make a separate bake for the front, so that it was planar, and then blended things in Photoshop. However, after discovering NDO2 this is no longer needed... but the bullets were already modeled by the time I catched up with NDO2

-As you can see from the LOWPoly UVS, the object takes a very small part of the map. This happens because the barrel is just a small part of the whole weapon. I only shared the barrel as it's the part that looks the worst)

Replies

  • wirrexx
    Offline / Send Message
    wirrexx ngon master
    if you bake through Xnormal, than you have to set your greenchannel (y channel) to -Y.
  • a3D
    wirrexx wrote: »
    if you bake through Xnormal, than you have to set your greenchannel (y channel) to -Y.
    :poly102:
    Ok, this is a little embarassing. Thank you, this improved the normal map drastically.
    Really the devil is in the detail.
    Thank you very much. :)

    *next step*

    This worked, to a certain extent.
    This is the current result:
    2vno2a9.jpg

    These are the same (small) artifacts you can see in Maya as well, on a dark material with high reflections.

    It would be fine I guess, as with other maps it should become nearly unnoticeable. Or I could add supporting geoemtry.

    But since it's not perfect yet, and since handplane and UE4 say nowadays you can get clean results without supporting geometry... I have to try. I'm watching this series right now:
    https://www.youtube.com/playlist?list=PLJHtayLGTyCGKTQR1FZ6aKS34Vg4dsPRw

    I'll be making some more attempts in the next hours/days until I'm satisfied or completely devoid of energy (aka dead)

    Thank you again for the help so far
  • wirrexx
    Offline / Send Message
    wirrexx ngon master
    No problem, all the tips i know comes from Earthquake, Pedro and Perna (the main artist everyone should thank). You should take a look at Earthquakes Xnormal/ baking forums. They'll help you a bunch!
  • a3D
    I spent the whole afternoon trying various solutions, including

    -object space map baked in xnormals from FBX with TB checked, converted in Handplane to TS
    -object space map baked in xnormals from FBX without TB checked, converted in Handplane to TS
    -object space map baked in Maya, converted in Handplane to TS using both the FBXs as base meshes
    -object space maps baked in xnormals from both FBXs, converted in xnormal itself
    -object space map baked in Maya, converted with xNormals to TS
    +
    tried both straight green or inverted green (thanks, UE4 node system)

    and finally

    -just inverted the green channel on the whole old tangent space map baked in Maya

    results:

    -all maps that involved Handplane looked very bad. Maybe it's the lack of a specific support for UE4?
    -all maps baked in xnormals looked decent, but had visible shadows with high reflections
    -maps baked in Maya looked great, just the same as in Maya and Marmoset. Bothe the Object Space converted in xnormals and the Tangent Space baked directly in Maya produced good looking results.

    pic: the original TS map baked in Maya, as it looks in UE4 after inverting the G channel
    35ldu1f.jpg

    Conclusion:

    -Maybe it's me, maybe I don't understand what this "new" baking workflow is all about. I don't know. (and if so, please tell me) What I know is, in the end, the original map was the one that looked the best, once the green channel was inverted.

    * Should I stick with the traditional way of doing normals? *

    I am comfortable with baking in Maya and it seems like my techniques are still valid.
    However, I can't stop thinking about the fact that handplane, synched normals and the "new workflow" explained by Epic Games, all promise to produce even better results.
    (if only I could figure out what I'm doing wrong, that's it)
  • ZacD
    Offline / Send Message
    ZacD ngon master
    Handplane doesn't support Mikk TSpace which UE4 switch to in version 4.7. xNormal uses Mikk by default. Here's a good explanation of using synced normal map workflow with UE4.

    http://www.polycount.com/forum/showpost.php?p=2257792&postcount=1845

    Also you may need to change this setting in xNormal http://www.polycount.com/forum/showthread.php?p=2254880#post2254880
  • a3D
    ZacD wrote: »
    Handplane doesn't support Mikk TSpace which UE4 switch to in version 4.7. xNormal uses Mikk by default. Here's a good explanation of using synced normal map workflow with UE4.

    http://www.polycount.com/forum/showpost.php?p=2257792&postcount=1845

    Also you may need to change this setting in xNormal http://www.polycount.com/forum/showthread.php?p=2254880#post2254880
    Good information here, thank you.
    I'm gonna give it a shot.

    I'll be trying to bake this:
    20a5fdk.jpg


    EXPORT SETTINGS:

    23w8whk.jpg

    ACTIVATING PLUGIN FOR MIKK TANGENTS, and RAY DISTANCE CALCULATOR
    fnuwpc.jpg

    As I posted this image, the RDC was still running. Has been almost 3 minutes so far. I'm going to stop it now and press "copy results"

    SETTINGS USED AND RESULTING MAP:

    29vdzk6.jpg

    UE4 IMPORT SETTINGS (I'm using the original FBX I exported for xNormals baking)
    2gxr62x.jpg

    NORMAL MAP IN UE4:

    2m5yzj6.jpg


    INVERTING GREEN CHANNEL IN UE4:

    wl2aus.jpg

    RESULTS

    30cxfdz.jpg

    artifacts everywhere.

    - Setting the normal in xNormal to x+ y- z+ is the same as reversing the green channel in UE4? Did I skip an important step?

    - Did I miss any UE4 checkbox or settings related to tangent space or normals in general?

    - Should have I opened the FBX in Maya, assigned the normal and then exported with Embed Media, instead of importing the FBX and the normal separately in UE4? Does this even make any differences?

    - Should have I imported a custom-made cage instead of using the automated process?

    - Is it just the shape that's too weird to compute, or the UVs that are too bad? I thought synced workflow was supposed to produce clean results regardless of how messed up the geo and how steep the angles were, without needing supporting geometry or carefully planned UV splits... I guess this has limits as well?

    - The documentation even suggests to use a single smoothing group
    (should have I used a single smoothing group instead of making UV Splits hard?)
  • a3D
    SECOND ATTEMPT

    changes made:

    -Base FBX is a single smoothing group, as suggested in the UE4 documentation
    -Normal was created with x+ y- z+ in xnormals, all other settings unchanged
    -Opened in Maya, exported with Embed Media, imported in UE4, green channel not flipped

    changes not made:

    -No manual cage
    -UVs are the same as before

    RESULTS:
    9u3xu9.jpg

    This is getting worse

    (EDIT: NOTE THAT INVERTING THE GREEN CHANNEL DOES NOTHING TO IMPROVE THE LOOK IN THIS CASE. I TRIED)

    _________________________________________________________________________


    THIRD ATTEMPT:

    New UVs
    xqkgvr.jpg

    Hand-adjusted cage in Maya. Baking an Object Space
    2m2hzjc.jpg

    What I did here:

    -Still left the mesh with some difficult angles to better test the limits of the synced workflow.
    -Baked an object space map
    -Converted said map in xNormals

    Result:
    2141npz.jpg

    Same artifacts as the first attempt.



    CONSIDERATIONS:

    - THE OLD RULES STILL APPLY:
    Unless you can get a clean old tangent space bake in your application of choice, (i.e. Maya) you still have to mess with UV splits on steep angles, adding support geometry where needed, fighting incorrect shading and carefully plan each triangle orientation.
    Synced workflow doesn't make any of this easier. (but it should! the doumentation said it would!)

    - WHAT SYNCED WORKFLOW DOES:
    It solves ONLY the problems that can happen if your baking software isn't aligned with the destination engine.
    This has nothing to do with how you plan your normal map, your lowpoly, your hipoly etc.

    - what I think of this paragraph:
    2rpe4ic.jpg

    It just doesn't work. (???? but it should!)

    Did they have several UV islands, one per plane, to achieve clean results with that? Or something? What am I missing?

    How. The. Heck.

    Why can't I make it work.
  • Tzur_H
    Offline / Send Message
    Tzur_H polycounter lvl 9
    Did you convert the object space to tangent space with xnormal? Still getting errors?

    I'm working with modo and doing this workflow when baking using the rounded edge shader and it's working perfectly.

    Could you upload the model so we can have a look?

    Also, just so you know, you can flip the green channel via the textures editor:
    3ACXWcQ.jpg
  • a3D
    Tzur_H wrote: »
    Did you convert the object space to tangent space with xnormal? Still getting errors?

    I'm working with modo and doing this workflow when baking using the rounded edge shader and it's working perfectly.

    Could you upload the model so we can have a look?

    Also, just so you know, you can flip the green channel via the textures editor:

    Yes, the last attempt was all about baking an object space and converting it into xnormals.

    Here are the models.
    http://www.filedropper.com/archive_8

    Cubostrano_LowP = the first cube, edges hardened where UVSplits exist
    Cubostrano_LowP_OneSG = a single smoothing group
    Cubostrano_LowP_ObjToTang = the new UVs. Edges hardened where UVSplits exist. I used this for the Object to Tangent workflow
    Cubostrano_Hip = the Hi Poly

    Thank you for the information about flipping the green channel
  • ZacD
    Offline / Send Message
    ZacD ngon master
    There's too much pink and artifacts visible on the normal map itself, I'll try to do some of my own test bakes with that mesh. Either a setting is off in Xnormal or a Maya export setting.
  • Tzur_H
    Offline / Send Message
    Tzur_H polycounter lvl 9
    OK, had a look at it. Looks perfectly fine to me using xnormal to bake as tangent space normal.

    I use Modo as my 3d application.

    Here's what I modified in your low poly model:
    1. A lot of stretching occurred in one of the side pieces, I had to cut down the middle part of it and relax the UVs.
    2. Set all edges to soft, then set only UV borders to hard (there are scripts for this, 1 click process - I would have give you more details on where to find it but I don't use Maya, sorry)
    3. I duplicated the low poly, pushed all verts along their normals to create the cage, couple verts had to be modified manually since some intersecting occurred.

    - Imported into Xnormal, set files, cage and bake into tangent space normal with Y-

    - Test in unreal

    Here are some images:
    Modified UV: * Highlighted edges are set to hard.
    MO4rmXF.jpg

    UE4 Import settings:
    QC7UP1t.jpg

    Test screenshots: * I made a shiny metallic material, makes it easier to spot the errors and artifacts.
    yH2BmzN.jpg
    LxIdtpG.jpg
    fW4QE4J.jpg

    For production quality I would have bake in Xnormal as a 16-bit TIF, then convert in Photoshop to 8-bit, save and use that in unreal.

    Here are the low poly and the cage FBX files:
    Low Poly
    Cage

    I hope this helps you!
  • a3D
    Tzur_H wrote: »
    I hope this helps you!
    This will definitely help. I'll start trying right now.

    First, I see your cage has the same smoothing groups as the original model. After reading some information online, I was worried this could be a problem. Looks like it isn't.

    I'll try a bake right now in xnormals. If it comes out clean (and I really hope it does) I will try to export a cage from Maya and repeat the process.
    If it doesn't come out clean, then my problem lies within xNormals, and can be hopefully solved once identified.
    If the bake using your cage is clean, but the bake using my cage is not, then the problem is about cages.

    Experiments time, updates soon

    EDIT1:

    Baking worked with your cage! Wow! This is, like, super clean. It's amazing.
    I can't thank you enough!
    (tinypic captcha: Helping Hand :P )
    1418shi.jpg

    NOW

    The next step is about making a cage myself, and testing it. If it works, I will be independent from now on.


    EDIT2:

    Just to be extra sure, I unlocked the normals on the base lowpoly shape, and re set every edge by hand.

    I created a cage in Maya. I unparented it, and exported it as an FBX.

    I imported my own cage in xNormals. Baked.

    The results were perfect.

    My reaction:
    [ame]https://www.youtube.com/watch?v=QkjD3D5FgmE[/ame]

    There's really nothing I can say at this point. (I just had to use a custom made cage in xNormals and invert the green channel.)
    It wasn't a big deal but - man, it kept me blocked for oh so long.

    I am infinitely grateful to everyone who contributed to this thread.

    I'm looking forward to sharing this knowledge with other people struggling with the same issues.

    Thank you guys. You are the best.
  • Tzur_H
    Offline / Send Message
    Tzur_H polycounter lvl 9
    a3D wrote: »
    First, I see your cage has the same smoothing groups as the original model. After reading some information online, I was worried this could be a problem. Looks like it isn't.

    Where have you read that?
    As far as I know, the SGs of the cage doesn't affect anything, the cage is there only to affect the distance that the rays are being cast.

    I might be wrong though, I'm fairly new to this whole process myself.

    * I ran a test with 3 different cages, 1 with the same SGs as the low poly, 1 with all edges set to hard, 1 with all edges set to soft - all resulting bakes are the same.

    Glad you got it sorted out :]
  • a3D
    Tzur_H wrote: »
    Where have you read that?
    One guy in an old thread here on Polycount. Good to know it's not important.
    Glad you got it sorted out :]
    Yep. Thank you again.
    My map is clean now. I rebaked everything and it's looking just like I wanted.
    70ku0x.jpg
    Of course now it looks bad in 3DO but it's not like I really care or anything, as long as I can comfortably add detail and it looks fine in UE4.
Sign In or Register to comment.