Puppet Land

How it's built

Everything runs in the browser as a static site on Cloudflare Pages, with no server code. Google's MediaPipe Pose Landmarker finds up to several people per frame and returns 33 body points for each, using the GPU when it can and the CPU otherwise. A tracker matches people between frames so each kid keeps their puppet, drops duplicate detections and smooths every point to cut jitter. A rig module turns those points into puppet joints (head, shoulders, elbows, wrists, hips, knees, ankles) in screen space and then places the figure in the world. A spring simulation on each bone makes hands and feet swing a little loose. Each puppet is a module that draws itself on a Canvas 2D element from the joint positions, and the props test circles around hands, head and feet to see what they touch. The same pipeline can be fed synthetic bodies, which drives the demo mode and repeatable tests.

Built with

• JavaScript (ES modules)
• Canvas 2D
• MediaPipe Pose Landmarker
• Web Audio
• Cloudflare Pages

The hard parts

1. An arm held up kept firing

   Holding an arm up changed puppets over and over. A hand above the top of the camera frame comes back with very low confidence, and the gesture read low confidence as the arm being down, so it re-armed and fired again. Down is now judged by wrist position only, and in simulation one 3.5 second raise went from 5 changes to 1.

2. Distance without a depth camera

   We use how big someone looks, shoulder width as a fraction of the frame height, as a stand-in for how close they are. Body size also takes torso length into account, so turning sideways does not shrink it. A person has to stand in the change zone for half a second before a raise counts, which stops people who are just walking past.

3. Legs the camera cannot see

   A kid standing close has no legs in the frame. Every joint blends between a natural hanging pose and the tracked point according to how visible the point is, so a close-up kid still has puppet legs. A person who leaves the frame fades toward limp, and their puppet stays on stage for 1.2 seconds in case they come back.

4. Puppets that shrank when you stepped back

   A puppet dropped to about a quarter of the screen height when its person walked back, which made it hard to see. Size now barely changes with distance, from 47% of the screen height far back to 51% up close. Bigger puppets could stand under the camera guide, so when the guide shows the right-most spot stops short of it.