Facial motion capture hardware and software - what are you experiences?
I am looking into face mocap options and I would be glad to hear what other polycount users used or have knowledge on available systems :poly145:
I will add that I look on this systems from Maya user perspective, so Maya plugins are a plus for me, as I would prefer to stay on Maya an avoid adding Motionbuilder or any other software to pipeline if it is not necessary. Considering the fact that Maya have quite nice HIK retargeting that works nice for body mocap (I don't plan to use body mocap on my new project) and there are Maya plugins available for some of face mocap, I think it is viable option to stay on Maya and not complicated pipeline with extra software. Correct me if I am wrong.
Before I continue I want to mention some details about my project. I will need to provide a large number of face animation. Lipsync is the most important part, rest of the face is not that important, as animator can do it while animating body. Large quantity of lipsync equals a need for quick processing of data for me.
All the information I present below is what I found on the web, others people opinion, by testing test face mocap capture files provided by some of companies, by testing some of the equipment and by speaking/writing with representatives/support people of some of the companies. Please do not take everything as a fact.
That's the info I gathered so far:
Vicon Cara - One of the most expensive option(around £25k), if not the most expensive one.
The most well know company for mocap. There Head Mounted Camera (HMC) was released about 2 years ago, soon to be updated. It is a multi-view HMC, it has 4 cameras in total. System require markers on face. Great quality. Provides point cloud data as final product. Retargeting to your character need to be done by yourself (no extra tools or plugins provided). It seams to have one of the best quality, but the process is time consuming (applying markers and post processing data).
Faceware - Can get face motion capture from a standalone camera (ex. webcam), there is no need for HMC, however you can use HMC if you want. Maya plugin included. It was used on quite a few big game titles. Hard to get in touch with them
Faceshift - Cheap option (800$-1500$). Easy to use, easy retargeting, live preview on you character (maya plugin). Using cheap (300$) depth sensors (can work also on cheap used kinect). Inaccurate when it comes to closed mouth and because of that as well bad with mouth shapes like P B M. Can work as base for facial animation, but require a lot of manual clean up. Due to fact it acquires data from depth sensor final animation have less detail and is more smooth out.
Dynamixyz - Maya plugin included. Require HMC. I coud not find any big game/movie titles that used it. However it does look tempting. Would love to hear from someone who had experience with both Faceware and Dynamixyz.
Di4D - seams to be very detailed. I do not have any information about it yet except it is HMC (2 cameras) and no markers.
Optitrack expression - sit down only, no HMC. It looks like it is a bit out of date at the moment
Mixamo - Face Plus - web cam based mocap avaliable online from mixmo with 'All Access' subscription (some other plans or pay as go might be available). Looks quite simple, not sure how good it is.
ZIGN - seams to be out of date
Di-o-matic (Maskarad) - seams to be out of date
FaceFX Studio - It is not face mocap. It analyze audio and animate your rig using phoneme recoginsition as far as I understand it. Great idea, but results that I have seen are not that impressive. Seams to be out of date.
Thanks for going through that. I want to mention once again that those are only my opinions and I would really like to hear what other people have to say.
So if you have anything to share about facial motion capture, go for it and post
I will add that I look on this systems from Maya user perspective, so Maya plugins are a plus for me, as I would prefer to stay on Maya an avoid adding Motionbuilder or any other software to pipeline if it is not necessary. Considering the fact that Maya have quite nice HIK retargeting that works nice for body mocap (I don't plan to use body mocap on my new project) and there are Maya plugins available for some of face mocap, I think it is viable option to stay on Maya and not complicated pipeline with extra software. Correct me if I am wrong.
Before I continue I want to mention some details about my project. I will need to provide a large number of face animation. Lipsync is the most important part, rest of the face is not that important, as animator can do it while animating body. Large quantity of lipsync equals a need for quick processing of data for me.
All the information I present below is what I found on the web, others people opinion, by testing test face mocap capture files provided by some of companies, by testing some of the equipment and by speaking/writing with representatives/support people of some of the companies. Please do not take everything as a fact.
That's the info I gathered so far:
Vicon Cara - One of the most expensive option(around £25k), if not the most expensive one.
The most well know company for mocap. There Head Mounted Camera (HMC) was released about 2 years ago, soon to be updated. It is a multi-view HMC, it has 4 cameras in total. System require markers on face. Great quality. Provides point cloud data as final product. Retargeting to your character need to be done by yourself (no extra tools or plugins provided). It seams to have one of the best quality, but the process is time consuming (applying markers and post processing data).
Faceware - Can get face motion capture from a standalone camera (ex. webcam), there is no need for HMC, however you can use HMC if you want. Maya plugin included. It was used on quite a few big game titles. Hard to get in touch with them
Faceshift - Cheap option (800$-1500$). Easy to use, easy retargeting, live preview on you character (maya plugin). Using cheap (300$) depth sensors (can work also on cheap used kinect). Inaccurate when it comes to closed mouth and because of that as well bad with mouth shapes like P B M. Can work as base for facial animation, but require a lot of manual clean up. Due to fact it acquires data from depth sensor final animation have less detail and is more smooth out.
Dynamixyz - Maya plugin included. Require HMC. I coud not find any big game/movie titles that used it. However it does look tempting. Would love to hear from someone who had experience with both Faceware and Dynamixyz.
Di4D - seams to be very detailed. I do not have any information about it yet except it is HMC (2 cameras) and no markers.
Optitrack expression - sit down only, no HMC. It looks like it is a bit out of date at the moment
Mixamo - Face Plus - web cam based mocap avaliable online from mixmo with 'All Access' subscription (some other plans or pay as go might be available). Looks quite simple, not sure how good it is.
ZIGN - seams to be out of date
Di-o-matic (Maskarad) - seams to be out of date
FaceFX Studio - It is not face mocap. It analyze audio and animate your rig using phoneme recoginsition as far as I understand it. Great idea, but results that I have seen are not that impressive. Seams to be out of date.
Thanks for going through that. I want to mention once again that those are only my opinions and I would really like to hear what other people have to say.
So if you have anything to share about facial motion capture, go for it and post
Replies
Frame rate is also extremely important. You want to be capturing at a bare minimum of 60fps consistently. Any lower than that and your capture results will be tainted with motion blur. Next time I do face capture at home I am going to rent a pair of gopro's and use those to capture at 60 fps at 720p resolution(same as Cara minus the IR). I would stay far away from any system that relies on webcams or other sub 60fps cameras.
Of the systems you listed I only have direct experience with faceshift, and I did not find it's results compelling. Even with the carmine camera they recommend. It needed a lot of work to set up and over ride their default terrible shapes, and the tracking needed a lot of hand holding as it frequently had issues due to low resolution and low fps. Once depth sensors get better resolution and better fps faceshift might become more viable, however it is nowhere near being an integrated performance capture solution.
If I had the budget I would just go Vicon. I've used data from Vicon systems in the past and It's always been consistent, solid and flexible.
I've used faceshift with 3dsmax, I like it and was able to get some good reliable results from it. But being a max studio it isn't as nice as using it in Maya with the plug-in which is one of the main reasons I don't currently use it, there isn't a plug-in for max, you have to use the stand alone app to record and process, then export and import into max using FBX which had some annoying bugs, depending on what version of FBX you where using.
They've improved it a lot over the last year and it delivers better results now but the workflow is still too much of a headache to bother with, our animators do roughly the same amount of work in the same time and there isn't anything even slightly unpredictable about the output.
Things I didn't like:
Can't wear glasses. Well you can, but it builds depth data and depending on the glasses and the camera it can be a show stopper. Having your actors take off their glasses leads to squinty performances or bugging them to wear contacts is a pain.
If you're actor has light color eyebrows or thin brows, the eyebrow tracking can have trouble, using some eyebrow makeup will help but that's a pain and if you do multiple recording sessions you have to try to recreate the makeup. We experimented with fake eyebrows also, it was more consistent but a bigger pain than makeup.
The day to day tracking isn't always great, so re-training it to learn to use your face all over again is a pain, but they have improved this aspect in recent releases.
Faceware:
We looked into faceware, it has better results, but it wasn't in our price range at the time and the workflow was a lot more tedious.
https://www.youtube.com/watch?v=-cB8vS6rUbY
https://github.com/XantusStudio/FaceCap
Music: Canton Becker http://cantonbecker.com/sound/
unfortunately is not tested on windows
Hope this will helps your creative
youtube https://youtu.be/8d5YnCRyh5Y
Thanks