23 Mar Developing Augmented Reality for Xamarin
Recently, I was tasked to develop an augmented reality (AR) project using Xamarin.Forms. For the project, we experimented with image tracking, object tracking, model placement, occlusion culling, and depth sensing using Xamarin. All 3D models and images were saved to the device and load at runtime. I wanted to write down a small article about my thoughts on what was going on.
Just to note, this will mainly be focusing on the AR side of development. There wasn’t really much to be said about the UI (user interface) layer of the AR implementation. Xamarin.Forms worked well in terms of providing similar experiences for both with minor idiosyncrasies.
iOS (ARKit and SceneKit)
Developing AR for Xamarin.iOS, while having its own challenges and hurdles, wasn’t a particularly bad or off experience. Most of the issues I encountered were getting used to the system and getting used to the Xamarin environment. Luckily, there were several tutorials and examples discussing how to implement several of the functions of ARKit and SceneKit.
Unlike in Android, SceneKit and ARKit aren’t separate plugins that you download but are already part of the package when you begin. Whenever you update Xamarin.iOS, you receive the latest functions and calls of the operating system, including AR features. This was very convenient because that meant I didn’t have to keep an eye on the plugin version in the same way that I did on Android. If there was a flaw or a bug on the AR side, it was usually resolved in the next update, and it was easy to find patch notes and details on what happened between versions.
Loading 3D models in Xamarin.iOS was trivial once you figured out the method was created. I didn’t need to build any additional factory scripts or the like; just one method call, and the model would be loaded. Models for SceneKit are required to be data files. While this may be a little confusing at first if you are coming from Unity like me, it’s extremely easy to batch convert your FBX (3D data interchange format) to DAE.
Once loaded, models are saved to files called SCNNode. These SCNNode objects are functionally similar to Unity’s GameObject class, and it’s pretty easy to move between the two once you understand the difference. The lighting, however, is a bit off at the start. Many of the colors when we initially built were either too saturated or too dark. While this was corrected as time went on, it was a large initial hurdle.
Image tracking and object tracking were also easy to get set up. Apple provided a very easy-to-use tool to scan 3D objects into a format that ARKit can handle. In combination with the ease and stability of model placement, we could get robust and creative scannable for the projects.
AR development for iOS was a comparatively easy experience. While lacking some of the features and ease of use of Unity, it was comparable to using Xcode for development with minimal problems.
Android (ARCore and Sceneform)
One of the most striking things about developing AR for Xamarin.Android is the sheer lack of knowledge that exists regarding it. Even the Xamarin provided sample projects are lacking in terms of details or just don’t work out of the box. There are hardly any tutorials online and not many Stack Overflow, GitHub, or articles in general regarding programming for Xamarin ARCore. To figure out many of my issues with developing AR for Android, I had to reference a huge chunk of details from Android Studio tutorials and translate them to Xamarin. One reason why this occurs may be that Android wasn’t as focused on AR as iOS was. In the future, I would like to write a tutorial series explaining how I implemented AR in Xamarin.Android.
There are several limitations to ARCore/Sceneform compared to ARKit, and these are limitations inherent to the plugins and not just Xamarin’s implementation of it.
The most critical issue came regarding 3D models. Like iOS, Android requires that models are in a particular format. The difficulty lies in that the format (SFB) is proprietary. The only way to produce an SFB (Sceneform Binary) model is to go through Android Studio and follow Google’s instructions. However, this belies a small but important detail. Older versions of Sceneform can’t use models created in newer versions, nor any versions made before it. If you are creating a project in Android Studios or JetBrains, this isn’t an issue. With a few button clicks, you can update your models.
This fact, however, becomes more relevant when you learn that Xamarin.Android.Sceneform 1.11 isn’t equal to Google Sceneform 1.11. Instead, it’s closer to 1.11.1, meaning I had to find the specific plugin. No other one would do. Use 1.12, and the model won’t load. Use 1.11.0, and the model won’t load. It has to be just right. This becomes increasingly difficult when Android Studio updates discourage the use of older plugins, pushing you to go forward. The release schedule of Xamarin exacerbates this problem for Android.Sceneform plugin updates. By 2021, we were forced to keep one machine in the office running an earlier version of Android Studio with all the correct settings just to build a few new models.
Xamarin’s updates to their ARCore and Sceneform plugins are incredibly sparse. When I initially started using Xamarin’s Sceneform plugin, there was no implementation to build out basic primitives. I had to wait several months before I was given the ability to do so. The time between the Xamarin.Android.Sceneform from the 1.11 release and the 1.14 release was about six months with no updates or revisions in between. The time between 1.15 and 1.16 was almost a full year. That means that any improvements or innovations created in Sceneform in-between versions were unavailable unless a user found a way to pass the middleman and directly bind the Google Sceneform and ARCore plugins, which I, unfortunately, didn’t have the time to experiment with. This problem was made worse because the Xamarin.Sceneform 1.14 and on have bugs in them that make loading models impossible. Which meant we couldn’t update the plugins, meaning we were becoming increasingly outdated. As of this writing, Xamarin.Android.Sceneform from 1.11 was the latest version that I could find and still work.
Returning to models, however, once you get past the initial hurdle of just creating an SFB, there were still issues that had to be addressed. To load your hard-won SFB, you had to create several Model Creator and factory scripts to help build it out. While not hard to develop, the lack of information online meant that I had to guess and experiment to figure out how to load anything. All renderables have this problem, models, materials, and even AR text to go through a factory script initially.
Another issue with Sceneform is that it has a different notion of submeshes than SceneKit does. In platforms like Unity or other plugins like SceneKit, submeshes are treated as individual objects that can easily interact with and transform data modified. Sceneform does not work like that. Submeshes are parts of the whole node object, and while their material can be changed, they don’t act nor react like individual objects of their own. It isn’t easy to interact with or touch submeshes in the object without interacting with the whole. While this may not necessarily be a problem if you don’t need to interact with submeshes or component pieces, it is frustrating when these submeshes are to provide the user with additional data. Once placed, the models are well lit and look more compelling with fewer user actions for all those problems. Light measurements are provided and are easy to modify and change to get the correct settings. This led to AR models looking better on Android than on iOS.
Regarding image tracking, the iOS implementation was simply superior. When testing images on comparable devices in terms of specs, the iOS version was regularly able to scan and recognize images that the Android version could not. Google provides a free tool that will allow users to input their images and have the program rate them in terms of ability to be scanned. This tool gives an accurate representation of how well it will recognize the image. While this was useful, most of the images we needed to scan were simply unreadable by ARCore but could be found by ARKit easily. If you are using images that can’t be changed or modified, it may be more useful to use a computer vision solution instead.
Due to project-mandated demands, we were unable to use cloud storage to hold models and images. While this wasn’t a particularly big problem on iOS (albeit the .ipa being huge in terms of size), the Google Play store will not allow any APK file larger than 100 MB. Being unaware of this at the start, this created a huge scramble to compress the assets and develop a framework that will allow the system to reference the APK expansion files.
With all that said, I would lean away from developing AR in Xamarin. There is little support for ARCore and Sceneform in general, and the release schedule for the plugin from Xamarin is abysmal. It takes twice as many lines of code to do something in Android vs. iOS. When developing for AR for Xamarin.Android, I often looked at examples from Unity ARCore and Android Studio to figure things out. It got to the point where I had to ask, why not just use those tools instead.
Future Considerations
One thing that I have been working on is Unity as a Library with Xamarin. Unity has created the notion to take the engine and use it as a plugin to handle 3D models and interactions. Many companies have used Unity as a library in conjunction with a native development environment (Xcode and Android Studios). While my work with UaaL is still in its infancy, the result is interesting. Naturally, there aren’t a lot of instructions on how to actually implement the system. What is interesting is the potential. I was able to figure out how to get the system working for Android, and I’m working on the iOS version now.
