Motion capture technology 'SysMocap' that can reflect the movement of the face, hands and whole body in 3D avatar free & in real time
For VTuber (virtual YouTuber), AR, VR, motion capture technology ' SysMocap ' that reflects the movement of the face, hands and whole body taken with a web camera in real time to the 3D avatar has been released. It runs on Windows, macOS, and Linux, and can be easily installed in the local environment for free, so I actually tried using it.
GitHub - xianfei/SysMocap: A real-time motion capture system for 3D virtual character animating.
Node.js must be installed to install SysMocap. Node.js installer can be downloaded from the following page.
Download | Node.js
This time, click '64bit' of Windows Installer to download. The file size is 30.0MB.
Launch the downloaded installer.
You will be prompted to agree to the license, so check 'I accept the terms in the License Agreement' and click 'Next'.
Since the place to install is displayed, click 'Next'.
Since the installation contents are displayed, click 'Next'.
Click 'Install' to start the installation.
When the installation is finished, click 'Finish' to close the installer.
Type cd SysMocap to navigate to SysMocap's local repository.
Run it by typing ' npm i '.
Type npm start to run it.
VRM format , GLB format , and FBX format , and in addition to being able to load your own files, 9 types of avatars are included by default.
Then the SysMocap window will start. The 3D avatar file formats supported by SysMocap are
Select 'SETTING' from the top and select the camera in the 'Input' field.
Next, select 'MOCAP' from the top. Select the 3D avatar in '3D Model', select 'Camera' in 'Source', and click 'START'.
Then, loading is done and the image of the avatar and web camera is displayed as follows. My face and body in the webcam have bones that indicate recognition. Below the camera image are 'whole body', 'half body' and 'special shooting', and special shooting is a mode that recognizes bust up.
The avatar also moves in response to the direction of the face, the movement of the mouth and hands. Although the hand recognizes each finger, it was not perfectly reflected in the movement of the avatar.
'Half body' recognizes the movement of the upper body. Bones are also visible below the shoulder.
'Whole body' reads the movement of the whole body. The accuracy with which webcam recognition is reflected in the avatar is still not that high, but the rough movement of ``bending the left leg and extending the right leg'' was firmly reflected in the avatar.