Home Technical Talk

UDK or Unity, manga-esque shader

Hello all! I'm working on a game concept that is ridiculously out of this world, and the art style is as well. Its art direction is heavily inspired by Viewtiful Joe, Mad World, and Soul Eater. Basically, I want it to look like a black and white manga, except in 3D. So, it's mostly cel-shaded, except with a twist that I haven't seen yet in 3D and especially in an interactive medium; masked screentone textures. I'm not sure if this is possible, but things like the toon shader, texture overlay, and sobel filters seem like they can be used to do the individual parts, and I'm not sure if they can be married. I'm an artist by skill and virtue, but I am pretty savvy at programming and the like when pointed in the right direction. Here's a quick sketch that I did to illustrate the look I want to achieve before I go into explaining.

cylinderShader.jpg

Basically, the biggest thing I want to know is possible is having the screentone texture masked off and behind all of the other elements with that specific material attached to it. There are a few examples of this in traditional animated cartoons, like Chowder. Pay attention to the texutres on Chowder's clothing, Schnitzel's skin, and Truffles' hair and skirt. The texture is flat and doesn't change, no matter how the characters contours animate on top of it.

[ame="http://www.youtube.com/watch?v=zalca7OkNWY&feature=player_embedded"]http://www.youtube.com/watch?v=zalca7OkNWY&feature=player_embedded[/ame]

I was wondering if this was possible, and if so, which engine to use to accomplish it; UDK or Unity. The art style is one of the biggest features, so it will take large part in what engine I use to create this project. So, if it can be done, could anyone point me to some good material. Thanks guys!!!

Replies

  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    I don't use Unity, but I know UDK, so here are some pointers on how it would work there.

    First, you will need to use CustomLighting materials, since what your described seems all like a custom solution as opposed to what UDK comes premade with.

    For your Specular/Shine/Gloss, you'll need to setup an Anistropic based Specular, so you get the sharp highlights straight like that. Although how good it will look, depends, I would suggest Lerping between a Blinn and Anistropic Specular to give it a softer look.

    For the overlay texture, that's really simple, you use the Screen Position, mask it out as an RG, multiply by the number you want tiled and connect it to your textures UV.

    So basically, Screen Position -> Mask RG -> x# -> UV of textures. I think you might also need to check "Align to Screen" in the Position node to get it work, but with this setup, basically whenever you look a certain way, or walk, you'll see the texture just like that of Chowder.

    For the cel-shaded outlines, there are several ways, one is to copy your mesh, push it/expand it and invert the normals. So basically an inverted copy of your mesh.

    Second way is to setup an outline post-process, there are plenty of tutorials on this, with the material network, you could take a look and see.

    Third way, is to setup a Fresnel, and have it Lerp to a black color, cheapest method I would think.

    To harden the shadow, I would suggest multiplying/power it by a number to harden the shadow on the model.

    For the ground texture...I would say setting up the same overlay texture, but connecting it to CustomLightingDiffuse instead, would work the answer.
  • krimzondestiny
    First of all, this has to be the most comprehensive response to any question posted on the internet of all time! For that sir, I thank you. Reading over your solutions, I believe I can pull this off in Unity. Unity is more efficient when it comes to rapid prototyping gameplay in my experience; not to mention my machine can't run UDK without killing my graphics card, lol. So, I will go ahead and try to translate your solutions in Unity. I'll still let you guys see my progress/ask questions, once I start cranking away at this. Thanks.
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    Ah, yeah, Unity is good for quick game creation, so it's totally understandable. So to go over certain things to make the translation easier.

    OK, so for the Specular, sharp highlights, it seems like someone already made the shader in the question exactly the way I mentioned: http://www.unifycommunity.com/wiki/index.php?title=Anisotropic_Highlight_Shader

    Overlay texture, Screen Position -> Mask RG -> x# -> UV of texture, make sure it's aligned to Screen Space/View.

    Same for the shadow texture, but in Unity I think you need to put it under the ambient?

    Cel-Shaded outlines for Fresnel: http://clocktock.com/vrav/resources/UDK-fresnel-outline.jpg

    And as a post-process: http://wdksoftware.nl/UDK/wp-content/uploads/2011/12/outline_shader021.jpg

    Hard Shadow, take the Lambert Term (NdotL) and Multiply it by a number to harden it.
  • krimzondestiny
    Wow! Just... WOW!!! I have already since added the antistrophic highlight page in my bookmarks. And I was looking into the different outline techniques you mentioned. Which brings up my next question. Could the fresnel technique allow me to have more control over the outlines rather than just a solid, constant width? I want to control different parameters like those seen in Maya and the image below. While sketching out the first image I posted, I was using my lame ass bamboo so I left the lines extending past the mesh. In doing so, I decided I liked that look.

    maya_toon_outline.jpg

    When the line is selected, it looks as though Maya creates about a 4 or 5-sided cylinder over each of the main edges of the mesh. It allows you to control such things as: line width, offset, line end thinning (tapered points), and line extend (how far the line extends). I'm pretty sure I won't be able to offset the outline with fresnel, but could I do the width, tapering, and extending? with it, or will it have to be more geometry?

    EDIT
    This is in no way me trying to "take a mile" here, because it seems like adding this feature would be incredibly more complex. It is a last minute thing that isn't even in my art bible yet, I was just wondering about it for mainly for learning experience.
  • LoTekK
    Offline / Send Message
    LoTekK polycounter lvl 17
    I'd need to spend a bit more time when I'm home tonight to look at the rest of the stuff, but a fresnel solution to creating black outlines shouldn't result in constant-width lines. What will end up happening is that the more tangential the surface is to the camera, the blacker it will get (and you can set up the falloff values to control the tightness of the black edge, and when the black actually kicks in in the first place).
  • krimzondestiny
    Oh okay. I think the constant-width just looks like that because the material is applied on a sphere. I have decided to see if the older versions of UDK can work, because I remember being able to use it in the past. The 2009 build seems to work fine, GPU at about 87 degrees C. I'm going to see how far I can move up update latter until it crashes my GPU, lol.
  • sprunghunt
    Offline / Send Message
    sprunghunt polycounter
    this might be a helpful image:

    5177430382_f929a65484_z.jpg
    halftoneShader_07112010 by sprunghunt, on Flickr
  • krimzondestiny
    Very interesting indeed. I can't believe I didn't come across it in all of my research. I've got a question, what are those "Param" nodes exactly, if you know? I believe I am using a different version of UDK than in the image; I am using the January 2010 build.

    The first (Param2d 'cartoonShade_pattern') node, I believe, is called (Param2D 'MaterialExpressionTextureSampleParameter2D_0') in my version and is a sample of the screentone texture I have. But, I don't know the node connected into the 'Dot' node, the 'uv_switch' node connected into the Lerp node, or the last 2 param nodes connected to the 'Append' node. What are they exactly?
  • sprunghunt
    Offline / Send Message
    sprunghunt polycounter
    Very interesting indeed. I can't believe I didn't come across it in all of my research. I've got a question, what are those "Param" nodes exactly, if you know? I believe I am using a different version of UDK than in the image; I am using the January 2010 build.

    The first (Param2d 'cartoonShade_pattern') node, I believe, is called (Param2D 'MaterialExpressionTextureSampleParameter2D_0') in my version and is a sample of the screentone texture I have. But, I don't know the node connected into the 'Dot' node, the 'uv_switch' node connected into the Lerp node, or the last 2 param nodes connected to the 'Append' node. What are they exactly?

    The parameter nodes have all been renamed. Thats why they don't use the long name you get. In this material there's only two kinds of parameters used. The tex2d parameter used for bitmaps like the normal map and a constant parameter like the one used for the node called uv_switch. Both of these can be made by either placing a tex2d or constant and right clicking on the node and selecting 'convert to parameter'.
  • krimzondestiny
    Here's a quick update of my progress thus far. There are things I like more about my shader and more about Sprunghunt's:

    mangaShader.jpg

    So far, I'm liking the way my shader's screentone works. There are a couple things that I don't like though. The cel shading doesn't affect crease edges, only those on the profile. I think the simplest fix for this though would be to paint the edges in the diffuse map. Also, I will probably just be using the inverted mesh technique to do the outlines because I came across some information stating the iOS and Android cannot use this method for cel-shading anyway. The project I am trying to develop this shader for is going to be a mobile game, btw.

    The thing that I like more about Sprunghunt's shader is the way it darkens when the camera gets further away. I believe this has something to do with the fact that it is a lighting shader whereas mine a diffuse one. The affect of mine is that when the camera gets further away, the texture 'visually' seems to increase. I would like the size of the dots to remain the constant, and multiply the dot density to create the illusion of darkness.

    I know there must be a way to track the camera position, but I'm not so sure about the texture. Increasing/decreasing the tiling doesn't work so much because the the dot size ratio should remain constant, unless I can change it 1:1 based on the screen position. Multiplying the final texture only makes the white part darker, of course. Any ideas?
  • arrangemonk
    Offline / Send Message
    arrangemonk polycounter lvl 15
    my suggestion us supposed to work only if you can use onscreen shaders
    first i would make 2 maps with decreasing pont density in each channel
    so you would have 8 shading types
    then render out zpass, lighting and some sort of mid thing betweeen ao and cavity map

    sobel on zpass gives outline
    then using the lighting * ao/cavity like a ramp/selector for the shading maps with blending
    just like a normal hatching shader but with using the textures in screen space
    which gifes you your sheet lighting

    lighting * outline = result (you could aso take the ao map into caluclation too, but that would give shades and stuff and is more grayscale thann bw then)

    edit:
    tried around in photshop with zpass, a0 and lighting stolen from some site

    stolen.png

    die result of the outlines isnt that sexy due to low quality zpass from jpeg

    anyways the 8 layer method with 4x4 pixel patterns seemed to work great
  • sprunghunt
    Offline / Send Message
    sprunghunt polycounter
    The thing that I like more about Sprunghunt's shader is the way it darkens when the camera gets further away. I believe this has something to do with the fact that it is a lighting shader whereas mine a diffuse one. The affect of mine is that when the camera gets further away, the texture 'visually' seems to increase. I would like the size of the dots to remain the constant, and multiply the dot density to create the illusion of darkness.

    If you look at the shader I made the top right hand part of the shader is actually setup so you can switch between using screen based coordinates and UV coordinates. If you set the parameter called "UV_switch" to zero it'll look more like the shader on the left.

    Also you might want to try doing the outlines as a post process instead of in the shader. I think it'd look better.
  • krimzondestiny
    my suggestion [is] supposed to work only if you can use onscreen shaders

    Would post processing count for onscreen shader? I've never done them before, but it seems like that is technically what they are. They are out of the question though because this particular shader is only going to be for a few elements: player, enemies, interactive objects. Everything else, the environment, will use a different shader entirely. The whole outline thing was utilized in animation to separate the characters from the environment in the first place. Besides, from what I've seen, the outline tends to be too consistent, making it look too flat and boring using this method.

    EDIT FOR RANTING
    This is where I think most games, especially amateur, that use the cel-shading technique fail. They just throw it everywhere and on everything and don't really understand why the technique works in the first place.

    AND... in my research on cel-shading shaders, I've found that a lot of people call any non-photorealistic art style with bold colors cel-shading. That's also annoying.
    END OF RANTING LINE----
  • arrangemonk
    Offline / Send Message
    arrangemonk polycounter lvl 15
    did it in forward shader ^^
    only thing i eeded thow would be screen size, otherwise the tiling is a bit wonky
    screenshot.png
    shadernetwork.png
  • DrunkShaman
    Offline / Send Message
    DrunkShaman polycounter lvl 14
    did it in forward shader ^^
    only thing i eeded thow would be screen size, otherwise the tiling is a bit wonky
    screenshot.png
    shadernetwork.png

    Nicely done sir.
  • krimzondestiny
    That's BRILLIANT! It's almost exactly what i'm shooting for. If the shadows and highlights are sharp, it'd be perfect. BTW, what program is this exactly?
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    ShaderFX for Max's viewport.
  • krimzondestiny
    Ah, well, what are the cone and UV nodes?
  • arrangemonk
    Offline / Send Message
    arrangemonk polycounter lvl 15
    vector compose and decompose
    (making float3 to 3 floats and vice versa)

    you also got to know the values of the muls and additions/substractions

    the minuses from top to bottom = 4 3 2 1, the mul on the right of the minuses is 5

    the muls on the bottom are 16 and 9 (screen aspect ratio) and the laswt mul is a multiplier for the dot structure, since the map im using is 4x4 pixels, on a fullhd screen this would be 120 -> (1920 x 1080)

    as you can see the map is 4x4 pixels : red and yellow diagonals on a blöack and white checker vhen channels used separately it gives the patterns for hatching

    the custom clamp note is just x > 1 ? 1 : x < 0 = 0 : x;

    edit: its an unregistred version of ShaderFX

    edit2: the uv node is the screen space uv coordinate

    depnding on what software you're using i could give you the shader
  • krimzondestiny
    depnding on what software you're using i could give you the shader

    Wow, Polycounters are by far some of the most helpful people on any forum I have ever seen. I'm actually using the UDK Material Editor. I'm learning about shaders and the like through this thread actually.
  • arrangemonk
    Offline / Send Message
    arrangemonk polycounter lvl 15
    Wow, Polycounters are by far some of the most helpful people on any forum I have ever seen. I'm actually using the UDK Material Editor. I'm learning about shaders and the like through this thread actually.
    nah, i am. XD

    boah... dunno if udk can import shaders from shaderfx
    .anyway, here you go
    http://dl.dropbox.com/u/1484680/polycount/hatching.rar
  • FicWill
    Offline / Send Message
    FicWill polycounter lvl 17
    There was a thread about this earlier, and I mocked this effect up in UDK:
    http://www.polycount.com/forum/showpost.php?p=1243848&postcount=15
  • krimzondestiny
    halftone.jpg

    This is really cool, and scary, but there's almost too much detail in there. At least, not the type of detail I'm looking for. I want something more like this image:

    bingeley_bingeley_by_drazebot-d30ntk6.jpg

    This is way flatter than yours, which is what I want, but there are lighting effects. This conveys space and adds depth, helping the players of the game to not throw up while they're running around in 3-dimensions...LOL.

    And for the simplified version of what I'm trying to get...

    mangaSphere.jpg

    I've got the outline and screentone, thanks to you guys, but now I want to add in the the highlight, shadow, and reflected light, in that order. Ace-Angel stated earlier,
    To harden the shadow, I would suggest multiplying/power it by a number to harden the shadow on the model.
    .
    Where would I do this exactly? Would it be in t he custom lighting slot, or do I have to somehow edit the way the default lighting works? I think I will have to incorporate a cubemap/bandmap because I know the Android and iOS supports them.

    ----EDIT----
    I've tried messing around with FicWill's shader to try to get it to do what I want, even named it after him, but I can't get it work properly. Straight out of the box it just goes looks like this:

    broken_FicWill_shader.jpg
  • arrangemonk
    Offline / Send Message
    arrangemonk polycounter lvl 15
    for hilight, you simply add the 5th or 6th square root to your effect (x^0.03125)-some clamping value and for the shadow you do te opposite

    some sort of, gotta try that myself to really say somnething useful
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    For the shadow, basically create a standard Lambert (Dot (Normal, Light) which is clamped setup, and multiply/power it from there.

    Although I need to also double check this, since I'm not sure if the clamp comes after or before the harden.
  • SirCalalot
    Offline / Send Message
    SirCalalot polycounter lvl 10
    Nothing really to add, but I just thought I'd say this thread is fascinating.
    Brilliant work guys, I'm learning a lot about custom lighting :)
  • krimzondestiny
    I was searching the Unity3D forums and came across the Strumpy Shader Editor (SSE). It's a visual shader editor akin to UDK's material editor. I decided to try to convert what I've got so far to Unity and SSE and it has been going well so far. I know you've been helping me in UDK, but I believe it's easily translatable to SSE. Here is the material in the game view of Unity.

    dot_screentone_shader_gameView.jpg


    Here is the Pixel Graph in SSE:

    dot_screentone_shader_pixelGraph.jpg

    and the Lighting Graph, which is where I believe I should alter the lighting, lol:

    dot_screentone_shader_lightGraph.jpg

    I've tried following the last two suggestions you guys posted to get the inner shadow. I've gotten a shadow, it's just white instead of black. It stands to reason that I should invert some value, I'm just not so sure which value that is.
  • BishyT
    I am sorry for bumping this thread, but just in case anyone is still interested in a manga-esque shader, i've been working on one in UDK for my masters degree.

    5Hphu.png

    Unfortunately I can't share the node setup yet as my masters degree isn't finished yet. Once it's handed in and marked though I do plan on sharing it. It's a surface shader, not a post-process shader, so different patterns can be given to different surfaces. It's also a pure black and white shader (no grayscale at all), so it's the most accurate screentone effect possible.
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    Very nice! I'm very interested in how you masked the shadows to have a different effect and cut the transition so harshly.
  • BishyT
    Ace-Angel wrote: »
    Very nice! I'm very interested in how you masked the shadows to have a different effect and cut the transition so harshly.

    The shadows are dynamic shadows from a moveable dominant light. These are naturally sharp. In fact, the shader has absolutely no lambert whatsoever, all shading is done by the light's shadows.

    Secondly, anything you plug into the "CustomLightingDiffuse" parameter of a shader will directly influence how the shader looks in shadow. As such, making the screentone appear required taking the base surface, multiplying a constant over it to make it darker, and then feeding that through the screentoning process again.

    However, after that, we're left with grey shadows that fade to black, because by default UDK produces realistic shadows. To fix that, we use a skylight, and boost the brightness of both shadows all the way to 5. This completely bleaches out the "paper" (white) area of the shader, while retaining the black screentone effect.

    You can do a similar thing on coloured surfaces (using a skylight to even out the shadows and boost them slightly) to get more of a nice cel-shaded effect. You can either set a single colour from the skylight to apply to the whole scene, or plug a colour/texture into CustomLightingDiffuse (preferably multiplied over the default surface) to get more control over the colour used.

    EDIT: Right now, the only issue i'm having is with the sobel outlines (as you can see in the image). I'm using the sobel shader provided by Epic as part of their UDK Gems. I've been able to reduce the appearance of these issues by fiddling with the settings (upping max distance and the resolution), but these seem to be unavoidable issues (the limits of realtime 3D).
  • Ace-Angel
    Offline / Send Message
    Ace-Angel polycounter lvl 12
    Thanks for the write up, much appreciated and really awesome.

    For the outline, did you guys try a Fresnel instead? Something with a really high Scale and Exponent value, with a Lerp to black out the edges?

    I found myself in your situation too, where the Sobel filter ends up being a hassle more times then not with overlapping geometry, and instead resort to said Fresnel, it take abit more planning with Geometry, but it's another viable solution.
  • BishyT
    A bit of fresnel is being used for areas like the nose and jawline. I kinda require the sobel for the effect though, as most manga (especially moe manga) is inked with even-width outlines.

    I've been able to reduce the appearance of the faults by fiddling with some of the options, so it looks a bit nicer now. It would also probably be less noticeable in a full scene.

    The main problem is the fact that the sobel effect doesn't actually use the z-depth/normal information. It simply uses a depth pass as an image and applies the effect to it, so it doesn't know which is the inside or the outside of a shape. Because we mask out the skydome from the effect (again, based on a depth pass), the outer lines are removed but the inner lines remain. If someone could alter the shader to remove all outer lines, the effect would be a lot more consistent (and would work a lot nicer with hair that uses transparency).
Sign In or Register to comment.