There are now over 1 billion augmented reality enabled iPhones and iPads which gives an incredible reach for your experiences. Here is an overview of what we know so far from this year’s WWDC when it comes to augmented reality, with more details to unravel this week after the engineering sessions. You can find a list of recommended AR sessions and challenges to check out from this year’s WWDC at the end of the article.

RealityKit 2

Introduced in 2019, RealityKit is Apple’s rendering, animation, physics, and audio Swift framework built from the ground up with augmented reality in mind. This year, RealityKit gets some great improvements:

Object Capture API

Image source: Apple

Object Capture is a new API on macOS Monterey that enables anybody to create high-quality, photo-realistic 3D models of real-world objects in minutes. The biggest step-back I have seen for adding augmented reality capabilities to businesses is the cost of 3D content creation. So this new feature is massive for the AR world, especially for the E-commerce industry. You can use an iPhone, iPad or even a DSLR to take pictures and transform them into 3D models optimized for AR. You will be able to generate USDZ, USD and OBJ files with the level of detail you wish.

AR Quick Look is a great way to showcase your models created with Object Capture. Apple has even updated their AR Quick Look Gallery with new 3D model examples created with Object Capture, and I must say, I am very impressed!

Unity has been working closely with Apple to bring this new feature into their AR Companion app (Unity MARS). You can read more about Apple’s collaboration with Unity in this article.

Custom Shaders

Custom shaders give developers more control over the rendering with custom render targets and materials. This update will make Apple AR experiences even more realistic.

Video source: Apple

Dynamic assets

The loading of assets is now more flexible, giving the option for example to programmatically change an image with every frame.

Improved Entity Component System

A new System object is available which will affect multiple entities in every frame of a RealityKit scene. Developers will be able to organize their assets better and build more complex apps.

Character Controller

Great addition for AR games, developers will now be able to control the characters and make them jump, scale and explore the AR worlds.

ARKit 5

ARKit helps developers build powerful augmented reality experiences for millions of users worldwide. Here are the main updates announced this year:

Location Anchors (in more cities)

Location anchors were introduced in 2020 but with a limited availability in major United States cities. The update we are getting this year is additional cities including London. AR Coaching UI will also now support location anchors with a new geoTracking goal, meaning that you can easily guide your users through location based experiences.

Expanded Face Tracking support

Face Tracking support has been extended on the Ultra Wide camera on iPad Pro (5th generation). If you want to learn more about how to create AR Face filters, you can check out my Borderless Engineering Conference talk on the topic Youtube video.

Improvements to Motion Tracking

With ARKit, you can capture the motion of a person in real time with a single camera, previously possible only with special equipment. Great potential with this ARKit feature, so it is exciting to see what improvements they are bringing to it.

App Clip Code Anchors

Great addition to the list of supported ARKit anchors, you will be able to place your virtual content to a printed or digital App Clip Code.

Video source: Apple

Apple Smart Glasses

Still no Apple Smart Glasses announced, but that was expected. However, if you would like to catch up with the latest rumors, I can recommend this in-depth roundup article from MacRumors.

WWDC21 AR sessions

WWDC21 AR challenges

WWDC21 Challenges are a fun way to dive right into the latest technologies and frameworks. Each day brings new challenges you can share on the Apple Developer Forums. This year, we have 2 augmented reality related challenges:

  1. Create your first 3D model with Object Capture (Tuesday)
  2. Framework Freestyle (Thursday) - build and ARKit sample app in 100 lines of code or less


Article Photo by Apple


Roxana Jula

Senior Mobile Developer

👩🏼‍💻 Augmented Reality and Future Tech Enthusiast

You may also like

How to setup CI/CD for iOS App Development with Fastlane, CircleCI and Firebase App Distribution

App developers always strive to improve the product they are working on. So it is definite that either(mostly) you will be adding new features to existing products or (luckily) you will be setting up a whole new project. Either way, onboarding new developers or bootstrapping an entirely new project is...

RubyKaigi Takeout 2021: A Look Back

Can you believe it’s already been an entire year since I had the opportunity to join my first ever developers conference, the RubyKaigi? Over the past three days, I was blessed with the opportunity to also enjoy this year’s talks and sessions, and get once again the opportunity to immerse...