Why AR Alignment Goes Awry on Boats: Insights from an AR Cameraman

This article is a translated version of my original post on Qiita. Original (Japanese): https://qiita.com/segur/items/0519088eec7074230bc4

image.jpeg

Why AR Alignment Goes Awry on Boats: Insights from an AR Cameraman

I love taking boat trips, and there's something I always do whenever I'm on a boat: taking photos with smartphone AR!

Recently, I tried AR photography during a boat trip to Himakajima, Aichi Prefecture. Here's what it looked like:

https://x.com/segur_vita/status/1923650697070891452

As you can see from the video, the CG is being pulled towards the pier or floating in mid-air—the alignment is completely off.

This video was shot on the "Kashikojima España Cruise" in Shima City, Mie Prefecture. While not as extreme, the alignment is still off.

https://x.com/segur_vita/status/1863790075383091263

Why Does This Happen?

The reason is that smartphone AR uses two types of sensors to estimate position:

The IMU combines accelerometers and gyroscopes to capture the phone's rotation and acceleration at high frequency.

However, when the boat accelerates or turns, the IMU perceives that "the smartphone is moving!" Meanwhile, the camera, seeing that the visual features like the deck remain unchanged, estimates that the phone is "almost stationary."

This contradiction triggers confusion in the AR system, causing the CG elements to misbehave.

Sensor Main Function Reaction to Boat Movement Impact on AR on Boats
IMU Detects smartphone's acceleration and rotation Detects boat's acceleration as "smartphone movement" Causes CG misalignment
Camera Estimates position from visual features the smartphone sees Assumes "stationary" as visual features don't move Attempts to fix CG in place

Thus, the differing "perceptions" of the world by the IMU and camera lead to chaos in AR coordinate systems.

It's Not Just a Boat Problem

This phenomenon is not limited to boats.

Such issues can occur in any "vehicle-based AR experience."

So, if you're planning any vehicle-based AR events or services, be aware of this challenge. Implementing it will require considerable ingenuity.

"But I've Done AR on a Plane Successfully"—If This Is You

When I share this story, some respond with:

"I've used AR on a plane without issues."

"It worked fine for me on a boat, too."

Perhaps that's because the vehicle was moving in uniform straight motion, making sensor contradictions less likely.

For example, in the video below, the boat was moving steadily straight.

https://x.com/segur_vita/status/1901954736032714970

The CG doesn't shift even when moving the phone.

This is because the IMU, while sensitive to acceleration and rotation, is less reactive to uniform straight motion. While moving at a constant speed, the IMU perceives it as "not moving," minimizing contradictions with the camera. Thus, a straight-moving boat is effectively akin to being 'on the ground.' (Though it doesn't completely align with GPS, pressure, or rocking.)

Additionally, shooting in an area rich in visual features, like posts and rails inside the boat, likely helped stabilize the camera's estimation.

Conclusion: Smartphone AR Events Are Challenging on Boats!

In summary:

Thus, while a skilled AR cameraman might cope by "only shooting during straight motion," it's not feasible for events aimed at the general public.

Operationally, while it might be possible to say "AR experience only during the 10 minutes of straight sailing," such an approach would not be practical for real-world event management.

In Conclusion

I will continue to tackle AR challenges on boats. I aim to communicate tips and the challenges of the "vehicle x AR" combination.

If you're facing similar challenges, let's connect! I'm also sharing insights on X (formerly Twitter), and I'd love to hear your comments.

https://x.com/segur_vita