r/computervision • u/RoofProper328 • 1d ago

Discussion How are you handling occlusion in hand pose estimation during grasping?

Hand pose is fine on an open hand in free space. Soon as the hand actually grabs something, half the keypoints disappear behind the object and predictions start drifting.

What do people actually do here? Temporal smoothing across frames to fill the occluded joints, or train on data with occluded joints properly labelled? Second one seems more correct but way harder to get, since annotating a joint nobody can see means either guessing or multi view capture.

Also not sure how much the object should be in the model. Predicting hand and object together should help, contact points constrain the pose. But I've seen people say it just overfits to whatever objects were in training.

Anyone got a setup that holds up on real grasping footage?

• Upvotes

100% Upvoted

1 comment sorted by

•

u/bfyvfftujijg 20h ago

End-to-end grasping models?

Seems like with enough data a model can learn to infer the position of the hidden parts. Probably should incorporate temporal “memory” too. Give it the prior key point trajectory or something.