I've been mentally running over my entry to try and figure out Things for Consideration for future Dota2 items or sets I make. Generally list what went well and what failed in both the technical and design approaches and how I'd do them better next time.
There's a lot I don't like about the design and the execution, but it didn't seem worth writing it down. However, if Valve are trying to open up discussion over the art style and submissions, I thought it might be useful to put it out there (and for myself to have to condense it into actual words).
I'm not claiming any of these are correct, take everything with a pinch of salt.
This is gonna get wordy, sorry.
Design
Initial Ideas
The fantastic Ken Wong once mentioned in a workshop that you should carefully consider everything a brief doesn't explicitly forbid and then pull back from there. I tried to keep that in mind although I probably didn't follow it to the extremes he had intended.
The main inspiration came from re-reading American Gods over my girlfriend's shoulder at a part where the protagonist has a Native American influenced dream where he is climbing a tower of bones being circled by Thunderbirds and a character in there called Sam Black Crow put me on to the "Black Crow's Spirit" which was the original name for the set.
The bird-like masks of the French plague doctors also crept in here. I also decided that I wanted to stay away from the more traditional tribal masks as they felt too obvious.
I liked the idea of using the remains of one of these birds for the set, as if the Witch Doctor was attempting to embody it and, by proxy, it's power. Early sketches felt too close to the Native American origins of the mythology so I decided to push everything larger and go for an entire costume that represented the bird, with each item using recogniseable parts. This ended up working well as the details of the original designs would have gotten lost in translation.
I googled crow paintings rather than crow photographs to use as reference, so that I could get an idea of what people saw as being the bird's characteristics rather than correct anatomy. Namely; beady glossy eyes, a pretty specifically curved beak and splayed, often ragged looking wings.
I didn't realise that stormcrows were established fantasty fare until after I'd renamed the set. I'd simply merged the thunderbird and black crow's spirit ideas together. I'm not sure if this apparent "borrowing" of existing lore helped or hindered in the end.
In general, I think my changes to the Witch Doctor were cheeky in that they were kinda drastic, but I wanted to go further rather than be more reserved and that seemed to pay off.
What Went Right
Alter the silhouette.
Valve take character silhouettes very seriously. The Dota heros - like the TF2 team - need to be recogniseable and unique.
I knew that I wanted to fairly radically change the Witch Doctor's appearance rather than create more canonnical but less flashy items. I wanted the set to be unique both in design and silhouette.
Luckily, the Witch Doctor's silhouette is actually really hard to break in any meaningful way. He's highly asymmetrical to start with and his large left arm, stoop and tall staff mean that - so long as he remains mostly purple - you can do a lot to him and not break it.
That meant that I could add the large beak and the arm feathers and still have him remain recogniseable. I'm not sure other heroes would have survived the same changes. It also meant that I had to rely on colour more than I likely would have.
So, I'd say play with one or the other but not both.
A unified set.
One of the things I feel worked well was that the whole set feels cohesive and each item is clearly linked to the others and fills in a part of the whole. I think it's one of the reasons I got away with such big changes to the character.
Keeping the mouth clear.
This partly came down to simple polycount limits, partly due to the fact the mask would have looked too big with the lower beak included, but after reading that Valve were encouraging entrants not to entirely cover the face of their heroes, it worked out in my favour.
Contrast and gradients.
I was pretty happy with the final gradients and contrast. I ensured that the mask had the highest contrast so it could be the focal point, with the black and white feathers and bold paint design, and tried to fade both the contrast out and the gradient darker the lower down the hero I got.
Don't be afraid to push these pretty far. In general, I found that I was afraid to go bold enough with the gradients and colours; often what looked good in my 3D viewport ended up looking weak in game - it meant a lot of going back and changing textures.
What Went Wrong
Altered the silhouette too much. (or: Know what makes your hero)
I probably went too far with the size of the staff, it's bigger at the top than the original. As the cosmetics don't change the selection or targeting hitboxes of the character, I figured this wouldn't be much of an issue, and that the design was thin enough that it wouldn't significantly alter the "mass" of the staff too much.
However, one player mentioned to me that one of the strengths of the Witch Doctor is his ability to be fairly inconspicuous on the outskirts of a large fight - meaning that players are less likely to pick him as a target - and that a more noticeable staff could break that.
This is entirely unrelated to the under-the-hood mechanics of the game that I was taking into consideration and not something I had experience of, having not played the game. Unfortunately this was after submission so there wasn't a lot I could do about it.
Didn't go bold enough on some forms.
Parts of the design still feel flimsy and weak to me having seen them in Valve's screenshots as a full set. Namely the feathers on the arms. I had been worried that they would be too distracting if I'd made them bigger but the LOD1 versions ended up being a bit too flimsy due to the polycount restrictions. Other small features, like the wishbone buckle, also faded away a little more than I'd like and some of the ropes and ties I feel could have been bigger.
So, go big and always consider how big the details in your texture and model will be. If you want a detail to be seen, ensure it's at least 4px in your source texture if not bigger.
Technical
What Went Right
Tangent basis.
A bit of experimentation meant that I ended up discovering that xNormal's default MikkTSpace synced very well with Source's own tangent basis. This meant I could rely a little more on normal maps and save precious tris on supporting geometry.
Bolder forms in LOD1.
Simply by hitting the poly limit, I found that the detailed talons I had in LOD0 had to be reduced to chunky triangles stuck directly on the ends of the toes. This worked out nicely as it made the game-res forms far more obvious even though it wasn't a truly accurate representation of the LOD0 mesh.
Recreating the shader for Unity.
I had a stab at recreating the Dota shader inside of Unity which proved invaluable for testing towards the end. Unity also has the benefit of being able to easily clamp in-game texture sizes so I could get a good feel for the final resolution without the process of importing into the game over and over.
Unfortunately, I had used the wrong specular values for my testing and so there was a rushed editing period at the last minute to alter all the mask values to work with the correct values.
The shader is also not 100% correct so things still looked quite different in-game compared to my viewport.
What Went Wrong
Too much sculpting.
I got a bit carried away with the sculpting. I could have put a quarter of the polycount into the sculpts and no one would have noticed. This meant I spent too much time on small details that got wiped out at game res and made all my filesizes really big and awkward to work with. It also meant that I spent too much time focusing on details that would end up tiny and not enough on the medium-sized forms that would become small details in game.
It did help with the presentation shots, though, so maybe it wasn't all bad.
Not working to LOD1 (or LOD0).
A lot of the stuff I had planned ended up hitting the very solid brick wall of polycount limits. It's been a while since I've modelled anything that's had hard limits and, while I could just about squeeze things into LOD0, getting them into LOD1 was a real effort.
The cloak in particular was a big disappointment. Although I had originally thought of doing it using alphas (but found out the WD doesn't have alpha) I had to choose between the jagged silhouette at the edges or the silhouette at the back, with feather peaks sticking out. I went for the edges as it was easier on the polycount but in the end, I think that was probably the wrong choice. Sadly it was too late to go back and redo it by then.
In future, I plan to keep as much as I can as a solid mass - fill "gaps" with low-level detail, perhaps - and rely on the texture to carry the detail rather than the mesh.
More texture mirroring where possible, too. The Witch Doctor's asymmetry doesn't lend that well to mirroring and I wanted to keep him that way, but in the end I had to mirror some smaller parts to get the texture space back.
Reliance on normal maps.
Related to not viewing things at game res and not designing for LOD1, the normal maps fell apart a bit when LOD1 came into play and the UVs and normals get altered. Ultimately, it's not that noticeable once everything is at game resolution, but it's an annoyance.
Perhaps worth baking some things for LOD1 then up-resing that to LOD0? I'm not sure.
Losing volume in LOD1.
Generally when creating LOD1 I was just pulling polys out of LOD0 and leaving it there. I was thinking of it too much like a traditional LOD rather than it being an in-game representation of my portrait mesh. So it's a good idea to bulk back out your LOD1 mesh - just a little - if you've gone to town on optimising it, as chances are you've removed some volume from it. Remember the transition is never seen so noticeable changes in size or shape won't really matter. If a detail works better being larger at LOD1 then go for it.
Not working in-game enough.
Due to my entry mostly being created in half hour snatches at lunch time or the odd couple of hour or two at the weekend or after work, I didn't get a lot of chance to try things out in-game. This really hurt my submission as there were big discrepancies between the rendering methods used in game and in my viewport. This meant a lot of back and forth, tweaking gloss and specular values at the last minute and I'm still not particularly happy with them.
Also, the Witch Doctor's default animation pose made him much more hunched and his head looking further up than the base pose of the mesh, which meant I had to adjust some parts after completion to make them work nicely (the feathers in his mask being the most noticeable). If I'd spotted this earlier it'd have been an easier path.
Additionally, this led to things like the cloak being too noisy and too low res. Meaning I had to make some complex changes right at the end of development (re-UV the final LOD0 and LOD1 meshes, then bake all the textures across whenever I made changes to them). If I had been testing regularly in-game, it would have been clear that larger, bolder feathers was the way to go, but as it was I had to simply do the best I could with my textures to define them.
Not using point light maps.
As a stop-gap I was using the up vector (green channel) of an object space normal map multiplied with the baked AO to create my light maps. This was then used as a mask on a solid purple layer set to multiply over the top of my diffuse texture. Due to time constraints, I ended up using this method throughout. While it turned out pretty well and I think the results are similar, I'm almost certain that I could have gotten more "pop" out of my diffuse textures if I'd taken the time to create proper point light maps and used the blending methods that Valve suggested in their doc.
Keeping consistent values.
Ensuring the colour, spec, gloss and other masks were kept consistent across the textures was fiddly work. In future I may try and pre-atlas my UVs into one massive texture then split it up at the end, ensuring I can keep things consistent and easy to change.
Easier previewing would be helpful to get this stuff worked out "once and for all" at an early stage and keep the number of iterations down.
Keeping consistent internally.
This one's more for myself.
Using modo, I couldn't export my meshes directly to the Dota client (wouldn't recognise skinning or bones) or bake to a good tangent basis, so all the baking was done in xNormal (except the AO, which is far superior in modo).
Some of the sculpting was done in modo, but most in ZBrush and some of the ropes were using renderable splines that were only bakeable in modo. This meant keeping track of a lot of files, exports and "middle-men" files.
Next time, I'll try and stick to one app per task and less detailed sculpts, probably bake as much as I can within modo to object space maps and use handplane for converting to tangent space normal maps.
And, hopefully figure out why the FBX files from modo aren't working in Dota...
Things Valve Could Do
Allow multiple custom items to be viewed at once.
It was frustrating to only be able to see one item at a time in-game. Getting a feel for how the set worked together was hard.
Release sample shader code.
That would allow the community to implement the shader into our 3D programs, allowing much easier and faster feedback than re-importing everything. It's too much fuss to test things in-game at each step and often by then you find there's stuff that doesn't quite work.
Remove the cache in the import process.
Iterating on meshes and textures is a painful business, the cache appears to mean that you have to import as different files, each time appending things on the end of the item name.
Better file browser for the import process.
It's really awkward to browse through the default file browser, is it possible to make one closer to the native OS file browser?
Better debug information.
It's hard to know what's gone wrong and why. We're a lot of us technically minded folk, a bit more debug info and we could more easily figure out issues and workarounds on our end.
Congrats and a fantastic read! You provide very valuable information in this post mortem. something I think more people should do (myself included) to learn as much from their mistakes as possible. It teaches you to take a step back and observe what is created and critique it from that standpoint.
In future I may try and pre-atlas my UVs into one massive texture then split it up at the end, ensuring I can keep things consistent and easy to change.
I cannot stress how helpful combining the texture has made texturing. Easy to see overall colour changes and value changes, so you can make sure everything works together perfectly.
Its probably one of the reasons why I was able to finish so many sets. Just cut days off my workflow and let me tweak the final output to another level.
I cannot stress how helpful combining the texture has made texturing. Easy to see overall colour changes and value changes, so you can make sure everything works together perfectly.
Its probably one of the reasons why I was able to finish so many sets. Just cut days off my workflow and let me tweak the final output to another level.
It would be really cool if QuickSaveMaps would work with splitting up big atlas maps. Then you could have all maps (including normals and masks) for one set in one document... and save it out with one button.
Replies
Now I have to go grab a last hour of sleep before work...
nice seeing a phb face pop up
I've been mentally running over my entry to try and figure out Things for Consideration for future Dota2 items or sets I make. Generally list what went well and what failed in both the technical and design approaches and how I'd do them better next time.
There's a lot I don't like about the design and the execution, but it didn't seem worth writing it down. However, if Valve are trying to open up discussion over the art style and submissions, I thought it might be useful to put it out there (and for myself to have to condense it into actual words).
I'm not claiming any of these are correct, take everything with a pinch of salt.
This is gonna get wordy, sorry.
Design
Initial Ideas
The main inspiration came from re-reading American Gods over my girlfriend's shoulder at a part where the protagonist has a Native American influenced dream where he is climbing a tower of bones being circled by Thunderbirds and a character in there called Sam Black Crow put me on to the "Black Crow's Spirit" which was the original name for the set.
The bird-like masks of the French plague doctors also crept in here. I also decided that I wanted to stay away from the more traditional tribal masks as they felt too obvious.
I liked the idea of using the remains of one of these birds for the set, as if the Witch Doctor was attempting to embody it and, by proxy, it's power. Early sketches felt too close to the Native American origins of the mythology so I decided to push everything larger and go for an entire costume that represented the bird, with each item using recogniseable parts. This ended up working well as the details of the original designs would have gotten lost in translation.
I googled crow paintings rather than crow photographs to use as reference, so that I could get an idea of what people saw as being the bird's characteristics rather than correct anatomy. Namely; beady glossy eyes, a pretty specifically curved beak and splayed, often ragged looking wings.
I didn't realise that stormcrows were established fantasty fare until after I'd renamed the set. I'd simply merged the thunderbird and black crow's spirit ideas together. I'm not sure if this apparent "borrowing" of existing lore helped or hindered in the end.
In general, I think my changes to the Witch Doctor were cheeky in that they were kinda drastic, but I wanted to go further rather than be more reserved and that seemed to pay off.
What Went Right
Alter the silhouette.
I knew that I wanted to fairly radically change the Witch Doctor's appearance rather than create more canonnical but less flashy items. I wanted the set to be unique both in design and silhouette.
Luckily, the Witch Doctor's silhouette is actually really hard to break in any meaningful way. He's highly asymmetrical to start with and his large left arm, stoop and tall staff mean that - so long as he remains mostly purple - you can do a lot to him and not break it.
That meant that I could add the large beak and the arm feathers and still have him remain recogniseable. I'm not sure other heroes would have survived the same changes. It also meant that I had to rely on colour more than I likely would have.
So, I'd say play with one or the other but not both.
A unified set.
Keeping the mouth clear.
Contrast and gradients.
Don't be afraid to push these pretty far. In general, I found that I was afraid to go bold enough with the gradients and colours; often what looked good in my 3D viewport ended up looking weak in game - it meant a lot of going back and changing textures.
What Went Wrong
Altered the silhouette too much.
(or: Know what makes your hero)
However, one player mentioned to me that one of the strengths of the Witch Doctor is his ability to be fairly inconspicuous on the outskirts of a large fight - meaning that players are less likely to pick him as a target - and that a more noticeable staff could break that.
This is entirely unrelated to the under-the-hood mechanics of the game that I was taking into consideration and not something I had experience of, having not played the game. Unfortunately this was after submission so there wasn't a lot I could do about it.
Didn't go bold enough on some forms.
So, go big and always consider how big the details in your texture and model will be. If you want a detail to be seen, ensure it's at least 4px in your source texture if not bigger.
Technical
What Went Right
Tangent basis.
Bolder forms in LOD1.
Recreating the shader for Unity.
Unfortunately, I had used the wrong specular values for my testing and so there was a rushed editing period at the last minute to alter all the mask values to work with the correct values.
The shader is also not 100% correct so things still looked quite different in-game compared to my viewport.
What Went Wrong
Too much sculpting.
It did help with the presentation shots, though, so maybe it wasn't all bad.
Not working to LOD1 (or LOD0).
The cloak in particular was a big disappointment. Although I had originally thought of doing it using alphas (but found out the WD doesn't have alpha) I had to choose between the jagged silhouette at the edges or the silhouette at the back, with feather peaks sticking out. I went for the edges as it was easier on the polycount but in the end, I think that was probably the wrong choice. Sadly it was too late to go back and redo it by then.
In future, I plan to keep as much as I can as a solid mass - fill "gaps" with low-level detail, perhaps - and rely on the texture to carry the detail rather than the mesh.
More texture mirroring where possible, too. The Witch Doctor's asymmetry doesn't lend that well to mirroring and I wanted to keep him that way, but in the end I had to mirror some smaller parts to get the texture space back.
Reliance on normal maps.
Perhaps worth baking some things for LOD1 then up-resing that to LOD0? I'm not sure.
Losing volume in LOD1.
Not working in-game enough.
Also, the Witch Doctor's default animation pose made him much more hunched and his head looking further up than the base pose of the mesh, which meant I had to adjust some parts after completion to make them work nicely (the feathers in his mask being the most noticeable). If I'd spotted this earlier it'd have been an easier path.
Additionally, this led to things like the cloak being too noisy and too low res. Meaning I had to make some complex changes right at the end of development (re-UV the final LOD0 and LOD1 meshes, then bake all the textures across whenever I made changes to them). If I had been testing regularly in-game, it would have been clear that larger, bolder feathers was the way to go, but as it was I had to simply do the best I could with my textures to define them.
Not using point light maps.
Keeping consistent values.
Easier previewing would be helpful to get this stuff worked out "once and for all" at an early stage and keep the number of iterations down.
Keeping consistent internally.
Using modo, I couldn't export my meshes directly to the Dota client (wouldn't recognise skinning or bones) or bake to a good tangent basis, so all the baking was done in xNormal (except the AO, which is far superior in modo).
Some of the sculpting was done in modo, but most in ZBrush and some of the ropes were using renderable splines that were only bakeable in modo. This meant keeping track of a lot of files, exports and "middle-men" files.
Next time, I'll try and stick to one app per task and less detailed sculpts, probably bake as much as I can within modo to object space maps and use handplane for converting to tangent space normal maps.
And, hopefully figure out why the FBX files from modo aren't working in Dota...
Things Valve Could Do
Allow multiple custom items to be viewed at once.
Release sample shader code.
Remove the cache in the import process.
Better file browser for the import process.
Better debug information.
fantastic work!
I cannot stress how helpful combining the texture has made texturing. Easy to see overall colour changes and value changes, so you can make sure everything works together perfectly.
Its probably one of the reasons why I was able to finish so many sets. Just cut days off my workflow and let me tweak the final output to another level.
It would be really cool if QuickSaveMaps would work with splitting up big atlas maps. Then you could have all maps (including normals and masks) for one set in one document... and save it out with one button.