Home Technical Talk

"Cavity" maps.

polycounter lvl 14
Offline / Send Message
fatihG_ polycounter lvl 14
Hey guys,

I was wanted to share my photoshop actions I use when texturing/concepting/image editing as a thank you for the game art community. I basically use my actions as someone would use their shelves in Maya. Basically my most used functions and what not are in button form.
Since it is specific to my workflow, not everyone might find it usefull. But hopefully they can take parts of the actions collection and use what works for them and maybe create their own customized "actions shelf" and share that with everyone.

However I am not too sure what to call a specific action.

nDo2 has a normals to "cavity" option. But it is not really a cavity map as it is a primarily grey map and has edge highlighting info in there as well. I was wondering what the correct term for a map like this would be.
No matter what I call it it will probably generate confusion.
I want to call it a "curvature" map, but that probably is not correct either. I think when someone thinks of a curvature map they think about a green/red map.
Right now I have called it "extract cavity cavity" and "extract cavity edges", which sounds kind of silly.

Help me name this action please? Basically what this actions allows me to do is separate the cavities and the highlights so I have more control on how to apply them to my textures. Instead of just using it as a single overlay layer.

Also since I haven't used a "proper" curvature map before, could someone provide a proper curvature map, so I can make a new action for that specific map type and have the same control as my current "highlight/cavity extract" action?

Cheers.

Replies

  • Blaisoid
    Options
    Offline / Send Message
    Blaisoid polycounter lvl 7
    i've never heard of any other names for it besides curvature and cavity.


    btw i had an idea of making a thread for sharing actions or just ideas for automating certain parts of the workflow (not necessarily restricted to just PS). unless there is something like that already.

    cos while i have many years of photoshop experience i never really took time to think about what could be automated and what are actual limitations of actions.
    Perhaps there's other folks out there who are like that.

    only recently i made an action that quickly previews the tiling of a texture, which made the process of making them noticably faster. the edge and cavity separation thing was also on my to do list for some time, though i think haven't recorded it yet.
  • passerby
    Options
    Offline / Send Message
    passerby polycounter lvl 12
    could just bake the curveature map from xnormal, it contains convex information in the green channel, and concave in the red channel.
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    Actually, I have played around with the curvature maps, convexity maps and cavity maps inside of xNormal and was not able to produce something that looks/works like the cavity map that nDo2 generates.

    The curvature map that xNormal bakes even though it bakes information in 2 separate channels, the green channel seems to simply be an inverted version of the red channel or vice versa.
    While the cavity map that nDo2 produces is more like a "find edges" effect for the highlights mixed with a cavity map.
    They are not so similar as I first thought they were.

    Anyway, I have to ask now, what exactly are they used for? I assume it will be handy to add dirt and what not into small crevices, by using it as a mask or something. But than you could do that with the AO bake as well. Perhaps it could be use as a base for an SSS map, or something. Other than that I cant think of any other uses of the top of my head.

    So how widely used are curvature maps?

    Blaisoid -
    I think having a general "improve your work-flow" topic might be too general and thus end up being too cluttered. Allot of people use different software anyway so it might not apply to simply everyone.
    So I think a more focused "improve your App X work-flow" topic would be better. That way people can just pick out what works best for them.
    If an "improve your work-flow" topic would be made though, I think it is best to just talk about the general work-flow instead of being too app specific.
  • passerby
    Options
    Offline / Send Message
    passerby polycounter lvl 12
    could just be your settings.

    but what ht found works very good is a script in maya called "r7_vertcurvature" that bakes the concave and convex information into the highpolys vertex colour and it gives me a really good result.

    if you use max i believe there is something similar called tension modifier.
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    I did a google search and ended up at a polycound topic.

    Looks like you have some experience with this passerby.
    I was wondering if you could make a quick screenshot(or just type it up) of the settings you use inside of xNormal.

    Either the default settings are wonky.. or the mesh I am trying to bake is all kinds of crap. I end up with a primarily green map, while I should be getting a black map with reds and greens, right?

    While writing this I made more test's and did some more google searches. Jogsy admitted he did not understand curvature maps well enough. So xNormal's outputs will never be correct.. so no use trying I guess.

    Ill give the Maya script a try though.
  • passerby
    Options
    Offline / Send Message
    passerby polycounter lvl 12
    ya the maya script works great for me, i just save it to vertex colour, than use xnormals vertex colour bake.

    the XN curvature maps are very hitting and miss and tried to use them in the past and usually had to spend like 30min messing with settings to get a good result.

    just remeber since your bakeing to verts the high poly needs enough verts to support it.
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    Tried playing with the Maya script..

    Having trouble getting it to work. I have put one of the files in the plug-ins folder and loaded the other through "load script" and tried running it. Doesnt seem to work.

    Checked my plug ins editor to see if it loading fine, it isn't. The "main.pyc" doesn't seem to want to activate. When I try to activate it I get an error saying it cant find "main.pyc.mll".
    Plus the fact that it mentions its using MEL while being a python script confuses me.

    Should these files actually go in "c:/programfiles/autodesk/maya/bin/plugins" or do these need to be in the scripts folder in "mydocs"? As the readme says it need to go in the "plug-ins" folder but isn't specific in what folder exactly it needs to be.

    I am using 2012 and have no idea what I am doing wrong.
  • passerby
    Options
    Offline / Send Message
    passerby polycounter lvl 12
    the main one wont activate, you want r7_vertCuratureCmd.py enabled in the plug-in manager.
    than you run the script with. the mel code.

    r7_vertcurvature;

    i just made a shelf button with that.

    they go in the plugins folder, since there implemented at a scripted plugin.
    i got both the "r7_vertCurvature_main.pyc" and "r7_vertCurvatureCmd.py" in that folder, and activated "r7_vertCurvatureCmd.py" in the plugin manager. and simply just run the mel code i got above to start the UI for the tool.


    All default settings in the tool work pretty great. you will just need to apply a material that can display the baked vertex colour so you can see the result in the viewport.
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    Awesome, got it working. Thanks for the help! =]

    As for my original question. I think I will make 2 different actions for the red/green curvature maps and the "cavity curvature" maps.
    And call em:
    Peaks/Pits (R/G).
    Peaks/Pits (B/W).
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    I've also used r7_vertCurvature successfully a few times before.

    I'm not sure how you guys are getting the vertexColors -> texture, but this is how I've done it before. You can use the xNormal SMB exporter to export your model with the vertexColors, then use xNormal which has a "Bake highpoly's vertex colors".

    A heads up. If you play with the scale value in the r7_vertCurvature plugin then you will get vertexColor values which go above 1.0 (and below 0.0?), and these would end up as black colors on my textures in the end. A workaround is to create the vertex color set in advance with the same name as the r7_vertCurvature calls it's vertex color set. Make sure you clamp the set you create in advance. Then when you run the plugin it won't have to create a set as it already exists. I've gotten this method to work.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    I'd love a better alternative than this though as it's limited to an older version of Maya (2011?) and could be done in a less time consuming way, but I've yet to find one which I think works well. If anyone has any good alternatives to this then please share.

    Doesn't ZBrush or Mudbox have anything similar to this?

    EDIT: Seems to work in 2012 even though zip lists plugins in a "2011" folder. Can't get it to run in 2013 though.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    I just invested some time in trying the xNormal vs r7_vertCurvature as I want to find an easier way of getting the results I get with r7_vertCurvature.

    From what I can tell xNormal does exactly what it claims to do in the manual. It does probe for differing surface vertex normal direction between the hp/lp. Similar angles = black, differing angles = white. I bet this can be useful for certain usages, but it's not commonly this I want.

    Here's an image to illustrate.

    xnormalconvexityvsr7ver.jpg

    The r7_vertCurvature is much more useful as a mask for usages such as "Where should I add scratches? What areas are more exposed? Where would dirt and grunge be? etc.". Whereas the xNormal one cannot be used as easily for these purposes.

    The xNormal one didn't have many options which I could find. Anyone know if these are any good alternatives to getting the r7 results?
  • cptSwing
    Options
    Offline / Send Message
    cptSwing polycounter lvl 11
    Shouldn't you be baking curvature maps in xnormal (instead of convexity)?
  • Eric Chadwick
    Options
    Offline / Send Message
    I've gotten good results with Xnormal's Curvature, by adjusting the scale of my models. The curvature range is very dependent on the scale of the model.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    Maybe I should :)

    *investigative goggles on*
  • cptSwing
    Options
    Offline / Send Message
    cptSwing polycounter lvl 11
    I've gotten good results with Xnormal's Curvature, by adjusting the scale of my models. The curvature range is very dependent on the scale of the model.

    Yeah that's been my problem mostly. I also get the impression that it's a lot harder to get good results with exploded meshes.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    So what scale are we talking about to get good results?

    I do get varying results based on scale. But these results are still not as useful for using as texture masks as the r7_vertCurvature is imo.
  • Eric Chadwick
    Options
    Offline / Send Message
    I can't remember what scale. I just fiddled with it until it looked good.
  • passerby
    Options
    Offline / Send Message
    passerby polycounter lvl 12
    I just find it easier to use r7 it still fits into my xnormal workflow anyways since xn can bake high poly vertex colour. it also dosnt touch blue channel so I can still have a custom painted mask baked out too.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    Might just be me not getting xNormal right. But r7 makes convex areas green, concave areas red and plain areas black. xNormal seems to be green pretty much all over and adds some red on angled areas?
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    Well, the thing with xNormals curvature is that you will be needing to do allot of testing per mesh/bake. Lots of rebaking until you have found the perfect settings for your assets. And even than you are eyeballing the "perfect settings".

    About xNormal and the green maps, like I said in post 6 I encountered this as well.
    I read however that if you match the search distance with the ray distance, you get better results. But there always seems to be slightly coloured red or green map, never a perfect black map. But that could probably be because I used a sphere as a test.
    On that test sphere however, it seemed like getting greens for sharp edges was pretty hard. I had no trouble getting the reds to bake properly though.

    Anyway setting up scripts and plugins to make something work, or using stand alone apps to make things kinda work is not really optimal imo. So having a more.. I guess "magic" solution would be nice. Having an easier method to bake this would be nice.
    Has anyone tried Meshlabs? I found that it has some form of "curvature colours" as well, but that also uses a different method of doing things. It bakes a "rainbow" ramp into the vertex colours.

    I am thinking though, couldn't we make a curvature map from a object space normal map, not tangent space? All the surface angles and what not should be in the object space normal map. So using that as a base to find the curvature of the surface could be done right? Kinda like nDo2 is already doing with its normal to cavity option.

    Maybe we should all just bug Jogsy and help him implement the curvature baking better? Though he is probably hard at work at making xNormal 4 though.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    Thing with creating a cavity map from a normal map vs using r7 on your highpoly is that the normal map method would only register differences between the low/high, whereas the r7 would give results on any convex/concave angles whether they are represented in the normal map or not. So while getting this from a normal map could be useful, it won't give the same results in all cases.
  • Cibo
    Options
    Offline / Send Message
    Cibo polycounter lvl 10
    r7 looks very nice.
    Make a high poly and use r7 as Mask for Decimation Master to save the outlines but decimate the holes.
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    kodde wrote: »
    Thing with creating a cavity map from a normal map vs using r7 on your highpoly is that the normal map method would only register differences between the low/high, whereas the r7 would give results on any convex/concave angles whether they are represented in the normal map or not. So while getting this from a normal map could be useful, it won't give the same results in all cases.

    Would that also be the case with object space normal maps? I mean an OSN would essentially "ignore" the low poly and only bake the normals of the high poly right?

    Since people are starting to use object space normal maps more now that Handplane is making them more popular. Using that hype wave and making tools for object space normalmaps might be a nice idea?

    But now that I think about it a bit more. Generating a curvature map from even an OSN might not work like you said, if uv borders would be in pits or peaks. If an uv border would be in a pit or peak, the magic a tool uses might think of it as a continuous surface, without changes in normal direction.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    Yeah you're right. Forgot that OSN aren't based on the vertex normals. The UV shells would be a problem as it would not know where a shell ends/begins and which edge goes with what other edge.

    Now that I think of it. There's a vertex color -> texture in Maya if I'm not mistaking. Minimizing the steps needed when using r7, keeping it all in the same app.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    It's under Paint Vertex Color Tool > Attributes Maps > Export. Kind of hidden away.
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    Well.. that means you would need to match your high poly uv's to your low polies right?

    I thought about it but didn't suggest it as you would need the HP's to be unwrapped as well.
  • passerby
    Options
    Offline / Send Message
    passerby polycounter lvl 12
    no need to unwrap the hp it stores the data in the vertex colour. all I do is use a little script in maya bonus tools to visualize it on the hp mesh, than bake the hp vertex colours, to the low poly mesh in xnormal.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    bb0x> Oh yeah you're right. I'm slow today. Yeah back to r7 -> xNormal workflow. Meh...
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    I guess we could use the Paint vertex tool> export method, by doing an automatic unwrap and use a big enough map, double/quadruple the size of the end result maybe. And use transfer maps?

    I'm sure that process can be automated as we stay inside of Maya.
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    Might be an idea. Then again, depending on how heavy your highpoly is those steps might be tough on Maya in the sense that it will take forever or seem to freeze. But might be worth a shot. Feeling up to the task? ;)
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    Yeah.. would probably be an issue. Needs to be tested though, if my crappy laptop can handle it I bet 99% of the users system can handle it as well.

    Maybe a good little project to start learning some MEL? =P
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    Yeah why not :)

    I'd even consider Python. But learning a bit of MEL first is always good.
  • fatihG_
    Options
    Offline / Send Message
    fatihG_ polycounter lvl 14
    Yeah, I'll dabble with it a bit in my free time.

    I'll have to prioritize my portfolio though, as in a few week "festival of games" will be held here in Amsterdam. And there will be some kind of Job Board.. I needz a job man. =D
    So it will probably take me a few weeks.. *A*
  • kodde
    Options
    Offline / Send Message
    kodde polycounter lvl 18
    Good luck mate!
  • combo_jerman
    Options
    Offline / Send Message
    Sorry bb0x,
    Im looking for a shader or PlugIn like this one.
    Du you have a new version of your r7_vertCurvature PlugIn for Maya 2014 AND Maya 2015?

    I will like to take a look on both Maya Versions.

    Thnx in advance.
    Cheers!
Sign In or Register to comment.