Render To Texture -Alpha Issues?

Tumerboy · May 2008

Hey guys, bear with me, there's some backstory here.

I'm working on a bush, which I have built fairly high poly in Max, and want to render out to be used as a Billboard. I also want to be able to render out the Normals of the polygons to be used as a Normal Map for the Billboard.

The Bush is constructed of many leaf cards, mapped with an Alpha leaf texture.

I tried rendering from an orthogonal view, and that's great for the diffuse, but I can't render out the normals from there.

I have Ben Cloward's Normal DX9 Shader for Max, which works great in viewport, but doesn't render . . .

So, I've ended up making a plane, behind my bush, which I'm projecting the bush onto, and then Rendering to texture. This does what I want, I can render out the diffuse, and the normal at the same time, from the same view, and it looks great.

Except. . . Render To Texture is not using the alpha map on the leaf cards. Or rather, it is and isn't.

Below is an image, on the left is the straight up render of a view of the bush. On the right, is the Render to Texture.

The regular render, draws my leaf cards with proper alpha, so that the edges are transparent.
On the render to texture, all of the areas on my leaf cards that would normally be transparent, are rendered black. So it's like it's not seeing the alpha map. However, the area outside my actual leaf area on my texture, is green, so the only way it would be rendering black, is if it DID see the alpha map.

Does anyone have any idea what's going on here? Is there some "use alpha maps properly" checkbox that I've somehow missed in the pages and pages of render settings? Am I an idiot and have my map in the wrong channel? Is there a better way to go about doing this in the first place?

Thanks in advance for any help you can provide.

MoP · May 2008

Render to texture doesn't work properly with alphas. You'll find the same in Maya's baking too. In fact most bakers I've seen won't do Alpha properly.

When I had to do some foliage like this, I ended up setting up a top-down camera and just rendering it out of the viewport, and using a special light setup to get a matching normal-map.

You won't be able to get it looking right with Render to Texture, I'm afraid.

The lighting setup I used for the normalmap, is described by Ben Cloward here:
http://www.bencloward.com/tutorials_normal_maps11.shtml

See the "Creating A Normal Map Right In Your 3D App" section.

I ended up using an instanced camera, and instanced geometry with a plain material (including alpha channel and bump map) applied to the normalmapped version, and the diffuse/alpha applied to the other one.

Tumerboy · May 2008

<

Thanks though.

Sage · May 2008

Hey I tried to recreate your problem but to my surprise it worked on my side so I'm a bit of wtf. I thought it might be an issue in which order you had the low and high poly model when you bake the texture and perhaps you might have to turn off the low like if you were going to bake an AO map with MR. The only thing you might be doing is you are using an alpha with in the the diffuse texture and I just made a mask with a gradient ramp. In my test I was making a grill with a bunch of holes in it. I guess I can try having an alpha in the diffuse to bake and see how that goes. I wasn't expecting to get good results but I did.

Alex

may sets

in rtt

1. have max output both alphamap and normalmap
2. set the baking option to raytrace instead of cage

I wonder if it's the raytrace thing that's making the difference here.

The edges are shitty on the leaf but I imagine it's the way the alpha was made. I did this to make sure it was not a fluke with the gradient map.

results with a leaf type alpha.

result

MoP · May 2008

Sage: The problem arises when having many layers of alpha-mapped geometry as a source texture. If you tried overlaying a plane with that texture on top of another plane with that texture, at an angle so you can see through to the "lower" leaf, the top leaf's alpha will mask out the lower leaf, which is wrong. It works alright for just single-layer alpha (although I find you can avoid that fuzzy edge thing if you just copy and paste the alpha channel from the preview window, rather than using the actual saved output - for some reason it displays better in the preview render.

Sage · May 2008

It sounds like the transparency look up at render time isn't set high enough then. If that is the problem then all you would have to do to fix it is go under render setup and look around for the options, I think by default it's set to three. I'll give it and see how that goes. The higher this amount is set the longer it take to render.

Alex

MoP · May 2008

Never heard of transparency look up... can't seem to find it in any of the Render to Texture, or scanline renderer options either. Where is it?

Mark Dygert · May 2008

I think he might be talking about the global Ray Depth Control, on the Raytracer Tab (Scanline)? I think it defaults to 9 and I thought it only worked on reflective surfaces, so that might not be it... Maybe its a MR setting?

Sage · May 2008

Nah it doesn't work. In Max you can set it under the raytrace material options. In other software this works fine. I thought Max had this but I was wrong. In other raytracers you can set how deep the renderer looks to calculate transparency, reflection and refraction. If the number is set too low it cut's off any object behind and just renders to black. So I guess Mops solution is the only one that works. In XSI for example you can set how deep this goes, I was hoping MR in Max worked the same way but it doesn't do it in RTT. Vig on the right track though, I thought Max scanline had this for some reason.

Alex

Eric Chadwick · May 2008

I've used MoP's way for impostors/billboards, works great. Sage, that's cool thinking though.

Tumerboy · May 2008

Thanks for trying guys. Ya I'm going to do the lighting trick. . . just more of a pain in the ass than I was expecting.

It seems like it should be very simple to just render normals instead of the regular image. . .

Oh well, thanks for all the help.

Sage · May 2008

An annoying work around could be to render out one plane at a time and then layer the results in photoshop or crazybump if setting up the lights is too annoying. If you export to png with alpha that export come into photoshop as an image with a transparent background. Somtimes I find it useful since it also does to normal maps as well. So instead of getting a black background where the uv islands end there is nothing. I found it a little useful when I was combining several different normal maps together.

Alex

Tumerboy · May 2008

nah, setting up the lights are easy:
Make a free cam pointing at your object, and set it to be "orthographic"

Then set up 5 directional Lights:

1 from camera angle - 127, 127, 255, intensity = 1, "overshoot"
1 from camera left- 255, 0, 0, intensity = 1, "overshoot"
1 from camera Right- 255, 0, 0, intensity = -1, "overshoot"
1 from camera Up- 0, 255, 0, intensity = 1, "overshoot"
1 from camera Down- 0, 255, 0, intensity = -1, "overshoot"

Then goto rendering>Environment, and set the background color to 127, 127, 255

Oh, and give you model a white texture (but in my case, still using the alpha of the original model)

Render To Texture -Alpha Issues?

Replies