.
Project
Gestura
↓
Project origins
Detecting hand gestures using magnets
Gestura is my first-year solo project as an MSc Innovation Design Engineering student. The journey began by looking for natural ways to interact with the smart devices around us using natural body language.
The result is a universal control system for smart home devices that recognizes hand gestures using a simple magnetic ring and a smartwatch.
Entering spatial interactions
Our perception of space plays a big role in the way we naturally communicate with other people. We are more likely to ask someone "could you pass me that [object]" while pointing at the right object, than to provide a lengthy description of it. However, this efficient communication method is lost in the way we traditionally interact with technology.
What if...
instead of saying “Hey Siri, dim ‘Living Room Light #2’ to 50%”, we could simply point at the light and fine-tune it with our hand?
Enabled by devices that are always with us, Gestura can be quickly activated by closing our hand in a fist twice. Then, we can operate whatever device we are pointing at using gestures.
Design challenge
Passive hand pose detection
Not relying on electronic devices outside those currently in our lives was an essential requirement of Project Gestura, as it was determined that making users own yet another electronic product to charge at night would be an important barrier in the universalization of this control system.
Our hands convey information that can be of incredible value to control smart devices. Nonetheless, tracking their motion can be challenging when trying to avoid the use of additional electronic equipment and active sensors.
The solution was found in magnets: most smartwatches and bands around us have magnetic sensors to allow for geolocation capabilities (i.e., detecting where the North is). However, these can also be used to track the position of nearby magnets.
If such magnets ar attached to our fingers (e.g. as a ring or as magnetic nails), a smartwatch can track our overall hand motion — something that can be combined with the wrist motion data of the smartwatch itself to determine the exact interactions we intend to perform with a device.
Technical challenge
Spatially aware control recognition
Gestura's product vision was to allow for a seamless switch between the devices we intend to control. Pointing towards the target smart device was seen as the most intuitive way to achieve this, but this required the system to be spatially aware of our position and the layout of our surroundings:
GPS is not accurate enough for indoor settings, so the challenge led to exploring the development of an indoor positioning system that, once again, would not require the user to acquire additional electronic equipment.
An efficient solution was found in using the Bluetooth signal from the smart devices themselves (i.e., from the speakers, TVs & lights the user is trying to control). From the strength of such signals, one can roughly triangulate the position of the user's smartwatch. This is added to the orientation data of the watch itself to determine which particular device they are pointing towards.
Pointing towards each device allows users to switch between them. The system knows to which device we are pointing by triangulating based on the Bluetooth signal strength of the smart devices themselves.
Smart indoor positioning
The RSSI —or signal strength— of a given set of fixed Bluetooth beacons can offer an estimation of the distance of a Bluetooth terminal to each of said beacons. Each individual estimation can be vague at first, since the signal strength varies even when the terminal is not moving. Nonetheless, combining the several readings from different devices and using noise filtering, we can determine the position of the Bluetooth terminal with sufficient accuracy [1][2].
Gestura uses this Indoor Positioning System resourcefully by using the smart home devices themselves as the Bluetooth beacons, and the smartwatch or band as the terminal, so no additional equipment is needed.