How do you bake ao into vertex alpha channel in maya?

usersen · Mar 2013

So I have been banging my head against this for a couple days now. I am trying to bake ambient occlusion into the vertex alpha channel in maya. Its very simple to bake ao into vertex colors (rgb, or rgba) using the batch bake option but I cant for the life of me to get it to bake into the alpha channel only. I have played with the settings alot, created alpha only color set, selected the bake to alpha option in the batch bake settings, etc nothing seems to work, the alpha color set always appears completely black.
I have tried manually vertex painting the alpha channel too and while its possible its painstaking, long and does not give me the same quality result that baking it does, I also have a ton of assets I 'd like to do this to so its not very practical to try and hand paint evertything.

I need the ao in vertex alpha because I am using the ats shaders in unity that use vertex colors for wind modulation.
Im using maya on osx, I am not adverse to using any free or super cheap programs that could achieve help achieve this though.
If anyone know how this can be done directly or indirectly (transferring vertex data from rgb to alpha if thats possible for example) help would be much appreciated. Thanks

equil · Mar 2013

mel scripting makes this pretty easy. i have a shelf button containing this;

string $select[] = `ls -sl`; string $vertices[] = ls("-fl", `polyListComponentConversion -toVertex $select[0]`);
 for($vert in $vertices) {
     polyColorSet -currentColorSet -colorSet #SRC# $select[0];
     float $value[] = `polyColorPerVertex -q -r $vert`;
     polyColorSet -currentColorSet -colorSet #DST# $select[0];
     polyColorPerVertex -a $value[0] $vert;
 }

note the #SRC# and #DST# tags. replace those with the names of the colorsets you want to move the ao data from (src) and to (dst). you can probably change those to use the same colorset, i keep them separate as it's easier to rebake that way.

usersen · Mar 2013

Thank you so very much equil !!!
This is exactly what I needed, works great, you have taken taken a huge amount of stress off me.
Have a fantastic day

OmegaOne8 · Apr 2013

equil wrote: »
mel scripting makes this pretty easy. i have a shelf button containing this;
string $select[] = `ls -sl`; string $vertices[] = ls("-fl", `polyListComponentConversion -toVertex $select[0]`);
 for($vert in $vertices) {
     polyColorSet -currentColorSet -colorSet #SRC# $select[0];
     float $value[] = `polyColorPerVertex -q -r $vert`;
     polyColorSet -currentColorSet -colorSet #DST# $select[0];
     polyColorPerVertex -a $value[0] $vert;
 }
note the #SRC# and #DST# tags. replace those with the names of the colorsets you want to move the ao data from (src) and to (dst). you can probably change those to use the same colorset, i keep them separate as it's easier to rebake that way.

I have been looking for a solution to this as well. I have a black and white image i want to bake into geometry vertex alpha only. I am not that familiar with MEL, can you please explain how to use this script a little more. I have geometry with texture AO texture applied, Then I created a new vertex bake set (vertexBakeSet1). So I replace the #SRC# with vertexBakeSet1 right? and replace #DST# with vertexBakeSet1 also or new name like vertexBakeSet2? Then Do I run the script? and mental ray batch bake should work? Maybe a screen shot with specific settings Batch bakes uses. I am having trouble getting this to all work. Thanks for any help you can offer

How do you bake ao into vertex alpha channel in maya?

Replies