Apple has internally tested a new Apple Pencil with visionOS support, MacRumors reported in March, quoting an unnamed “source familiar with the matter.”
This would allow the Apple Pencil to be used with drawing apps on the Vision Pro, such as Freeform and Pixelmator, the article says. MacRumors adds that no specific details are known about visionOS support, but one possibility is that users would be able to draw with the Apple Pencil on a desk or another flat surface, and the drawing would appear in visionOS. This would essentially turn a person’s surroundings into a giant canvas, complete with pressure and tilt sensitivity.
This isn’t the first hint of an Apple Pencil/Vision Pro collaboration. In January Apple was granted a patent for a “Stylus-based Input System for a Head-mounted Device.” Input to the “spatial computer” would be accompanied by writing on an iPad.
In the patent Apple noted that a head-mounted device (the Vision Pro, of course) can be worn by a user to display visual information within the field of view of the user. The head-mounted device can be used as a virtual reality (VR) system, an augmented reality (AR) system, and/or a mixed reality (MR) system. A user may observe outputs provided by the head-mounted device, such as visual information provided on a display. The display can optionally allow a user to observe an environment outside of the head-mounted device. Other outputs provided by the head-mounted device can include speaker output and/or haptic feedback.
A user may further interact with the head-mounted device by providing inputs for processing by one or more components of the head-mounted device. For example, the user can provide tactile inputs, voice commands, and other inputs while the device is mounted to the user’s head. And Apple wants one of those tactile inputs to involve the Apple Pencil.
Summary of the patent
Here’s Apple’s abstract of the patent: “A tracking system includes a stylus comprising a light emitter, a head-mounted device comprising an optical sensor configured to detect the light emitter, and a processor communicatively coupled to the optical sensor and configured to determine a position of the stylus based on detection of the light emitter.”
And last month Apple filed for a patent for detachable handheld controllers such as the Apple Pencil for use with the Vision Pro.
In the patent filing, Apple notes that input devices such as mice and touch screen displays aren’t ideal for spatial computers such as the Vision Pro. However, the company thinks an Apple Pencil would work just fine.
Apple said “a “A handheld controller [which would apparently be an Apple Pencil] may, as an example, include an inertial measurement unit with an accelerometer for gathering information on controller motions such as swiping motions, waving motions, writing movements, drawing movements, shaking motions, rotations, etc.”
The Apple Pencil could also provide haptic feedback to the hands of a Vision Pro user. And Apple thinks it would be useful to provide a Vision Pro strap to which the Pencil could be magnetically attached — and perhaps even charge it.
Summary of the patent
Here’s Apple’s abstract of the patent filing: “A system may include an electronic device such as a head-mounted device and a handheld controller for controlling the electronic device. The handheld controller may have a housing with an elongated shaft extending between first and second tip portions. The handheld controller may have power receiving circuitry configured to receive power from a power source.
“The power source may be incorporated into an electronic device such as a wireless charging dock or stick, a battery case, or a head-mounted device. The power source may supply power through terminals that form ohmic contacts with mating terminals in the finger device or may transmit power wirelessly using capacitive coupling or inductive charging arrangements. Magnets may be used to hold and align the elongated shaft of the handheld controller on the power source.”
