Show HN: Controlling 3D models with voice and hand gestures

98 pointsposted 8 months ago
by getToTheChopin

28 Comments

corysama

8 months ago

I understand you need your face in the videos for the demos. But, want to mention that you should make sure your system works with your hands in your lap. As shown, the user is going to experience "gorilla arm" fatigue very quickly.

getToTheChopin

8 months ago

Good points, maybe a second camera (phone?) pointed downwards at the tabletop would be good for that. Then the user can rest their hands in a "normal" position.

Thank you for the feedback!

doakes

8 months ago

Very cool! I like the different modes. I've always been fascinated with this space and products like Leap Motion: https://www.youtube.com/watch?v=zXghYjh6Gro

getToTheChopin

8 months ago

It's a shame that computer vision tech like Leap / Eyetoy / Kinect didn't have lasting power. So much fun to build with

ugh123

8 months ago

Amazing! Maybe use specific finger positions/gestures to trigger a rotation and scale functions (index finger up and within a bounding box of the model perhaps for rotation, similar for pinch to fingers to scale).

getToTheChopin

8 months ago

I'll try it, thank you! I separated them into completely different interaction modes to avoid misfires, but there's definitely room for efficiencies

Ey7NFZ3P0nzAe

8 months ago

I would be surprised if sign language didn't have an efficient way to convey digits.

danvoell

8 months ago

Awesome, nice work! This type of tech opens up a world of physical games.

chimpansteve

8 months ago

Slightly on topic - anyone remember LeapMotion and is anyone aware of any current support for that? Found an original one in a drawer when I was having a clearout the other day

one_more_q

8 months ago

I have one as well! Sorry no idea what it can be used for nowadays

xnx

8 months ago

Sounds very cool, but I could not make sense of the on-screen instructions. Some images or animations would go a long way to explain the controls.

getToTheChopin

8 months ago

Sorry about that, the instructions need to be improved.

Does this video demo help?

https://x.com/measure_plan/status/1929900748235550912

If it makes it clearer, I'll upload it to the github repo directly

xnx

8 months ago

That video did help. I think I was thrown off by two things: 1) I was expecting 3D controls with more direct mapping (e.g. rotating my hand rotates the model). This is more like gesture mouse controls. 2) Some of the controls were too subtle. The scaling between my gesture size and effect on screen was smaller than I expected.

Great area to develop though. There's so much untapped potential in applying Mediapipe.

getToTheChopin

8 months ago

Thank you for the feedback. I'll continue to work on it!

ednite

8 months ago

Great job! Looks very useful for interactive content creations and product showcasing. Definitely will testing it more. Thanks for sharing.

getToTheChopin

8 months ago

yes I'd love to go further with this concept so that 3D / CAD designers could easily present their models during video calls.

thank you!