How to control the Apple Vision Pro: hands, eyes, voice

Now that Apple’s “spatial computer” is here, how do you control the Vision Pro? Pretty much with your hands, eyes, and voice.

For hand gestures, your hands have to be visible to the Vision Pro’s external cameras. However, you don’t have to keep your hands in the air in front of you. As long as the external camera can see them, you can place ‘em pretty much anywhere you want — even in your lap.

The Vision Pro sports multiple cameras on the outside, a lidar sensor and TrueDepth camera, infrared sensors and more cameras underneath, and extra eye-tracking cameras and sensors on the inside. You interact with you hands or eyes. Or you can use talk to Siri via the built-in microphones. Hand and eye tracking even works in the dark (though you’ll need light for room tracking).

Controlling by gestures takes a bit of adjusting. But once you do, navigation is easy. You can scroll from left to right and up to down by pinching your fingers together and drag the direction you want to move. You can zoom in on a photo, image, or web page by pinching your fingers together with both hands and spreading them apart.

You have to eyeball the objects and apps you wish to manipulate. The Vision Pro’s sensors track your eyes with amazing accuracy. You orbs can even be used to verify online purchases via a feature Apple calls Optic ID.

Of course, everyone’s eyes are different. For example, eyes on my noggin may be spread apart at a slightly different distance than on you noggin. However, Apple has a quick setup process that aligns the headset to your eyes and then has you look at a series of dots, pinching your fingers as you go so you can calibrate. And if you wear glasses, Apple also sells inserts that magnetically attach to the Vision Pro.

As you stare around the home screen view and move your eyes from icon to icon, they light up ready to be selected. Once everything is calibrated, you can look where you want to go and then tap your thumb and index finger to select a button or app. There’s a white bar at the bottom of every app that you can grab to pull and push it around. You can adjust the size of any app by looking at the corner and then dragging it out or in at a diagonal angle. And you can swipe through photos or scroll websites by holding your index finger and thumb together while pulling up or down.

As for voice navigation, you can use Siri the same way you’d use it on a Mac or iPad. The Vision Pro also offers a floating virtual keyboard. But using it is slooooow as you have to look at each letter on a digital keyboard and select it, or reach out and tap the digital keyboard. If you need to use a keyboard with the Vision Pro (and I do), get a physical keyboard. If you have a Mac or an iPad with a keyboard, their keyboards will work just fine with the headset.

By the way, watchOS 10.4 and iOS 17.4 betas introduce a new toggle on Apple Watch Series 9 and Ultra 2. This is designed to avoid a conflict between the Apple Watch and Apple Vision Pro when gesturing. The toggle lets you choose to ignore the double tap gesture on the smartwatch while you’re wearing the spatial computer. 

The floating keyboard is useful for search or typing quick messages, but you won’t be able to type very fast at first. I got faster during my time with the Vision Pro, but nowhere near as quick as I am on my iPhone or a real keyboard. You can just use Siri voice-to-text to respond to iMessages or enter URLs in the browser (and launch apps). Still, you’re going to want to use a keyboard if you have to do a lot of typing.