AI puppetry | #MagPiMonday

This #MagPiMonday, Rob Zwetsloot explores interesting uses for the new Raspberry Pi AI Kit.

I’ve probably mentioned here or in past tutorials that I am a streamer when I’m not working on the magazine. I play games and build robot model kits on camera, which probably does not surprise anyone who knows me.

Specifically, I stream as a VTuber. What this means is that I have an animated avatar which I puppet with my face; it uses a mixture of technologies such as face tracking, which is mapped to specific image manipulation software to simulate a living, breathing cartoon character. You also get VTubers who use a 3D model that works similarly to mocap done for movies and video games.

I mainly stream with a 2D cartoon model – one I rigged myself, of course, defining how the different image layers move and warp as I do – however I do have a 3D model which can be controlled with my entire body.

Pose detection

Normally I’m sitting at my PC, so the full-body tracking is not something I use often. I’m able to load it into the popular VRChat software, and at least give the illusion that I am running around with full-body tracking. However, I recently did a very silly stream where I played a Sonic the Hedgehog hoverboard game on the Xbox 360 using the Kinect. It was horrendous, but very funny as I was using a web app full-body tracker with my 3D model.

Usually, full-body tracking requires motion sensors on specific body parts, however like the Kinect before it, modern software is able to make out your body using machine learning. Just like the new Raspberry Pi AI Kit.

This has got me thinking – can a Raspberry Pi power VTuber tracking? Can it even do the detailed face tracking which you usually need an iPhone to do? I need to start experimenting and hacking I think.

Mobile pose detection?

I’ve also been dreaming up a mobile puppeteering system. I mean, it’s not often that I would need to walk around with a cartoon version of myself on my chest but it could be very funny! And maybe it could be used for a silly costume where you put a screen on your stomach and pretend you’re being controlled by Krang from the old TMNT cartoon?

I love it when Raspberry Pi releases new technology that I can get to mess around with and possibly make with stuff with. I really should learn to polish up my projects though so they’re a little more presentable. Looking at you, NES controllers with a Raspberry Pi Zero inside that I hacked apart with a Dremel.

The MagPi #144 out NOW!

You can grab the new issue right now from Tesco, Sainsbury’s, Asda, WHSmith, and other newsagents, including the Raspberry Pi Store in Cambridge. It’s also available at our online store which ships around the world. You can also get it via our app on Android or iOS.

You can also subscribe to the print version of The MagPi. Not only do we deliver it globally, but people who sign up to the six- or twelve-month print subscription get a FREE Raspberry Pi Pico W!

The post AI puppetry | #MagPiMonday appeared first on Raspberry Pi.

from News - Raspberry Pi https://ift.tt/RyvYwkb