Andreas Atteneder's website

Breaking radio silence

2025-12-07T00:00:00Z

I noticed I didn't write a blog post in over 3 years and I reflected on why I think that is.

TL;DR: Founding a family, focusing on work instead of public relations and imposter syndrome.

A blog is born

I started this blog in 2019. At the time I wasn't finding too much joy with the direction work related projects took, so I had started hobby projects. Those cured the itch for technological excitement and gave me a sense of freedom I haven't felt in my job at the time. Naturally I wanted to write about what excited me and, inspired by other great developers who were blogging and sharing gems of wisdom, I followed their path.

It didn't take long and my work got noticed. I was contracted to work on Khronos Group's KTX-Software, to implement features for glTFast (my very own open source project) and got solid job offers. One of which was from Unity. They asked me to continue working on my hobby projects on their payroll, a dream come true offer that I gladly took. I'm not sure if my blog played a significant part in these things happening. While I think my work on open source projects spoke the loudest, I'd like to believe that the blog helped ever so slightly.

Activity going down

Before I joined Unity I dished out 11 blog posts over the course of 1.5 years. The next 1.5 years only three more posts and then I stopped completely. What happened?

Pack up all you belongings

Starting at Unity forced me to relocate to a country where they had a legal entity to be employed by. While I interviewed with Unity my girlfriend got pregnant. We decided to move to Hamburg, Germany, as my girlfriend is German and we would live close by my in-laws. Our first child was born 4 months into my new role. We married later that year, another two summers later our second child was born and we moved yet again to a place that was big enough for the four of us.

There's no guarantee for health

During her second pregnancy my wife developed gestational diabetes. Definitely something to be taken seriously, but usually not too concerning long-term. When she continued to check her blood sugar levels after childbirth (out of pure curiosity) it was still way too high. She then got diagnosed with diabetes type 1, a permanent condition. It turned her's and our life around, as you can imagine.

Before becoming a parent I used to stay up late and sleep until shortly before I got to work. After working hours I had plenty of time and energy for hobbies and even more so on the weekends. It was in those hours where I worked on blog posts. Those precious hours and my energy got directed towards a much more important thing now, supporting my family.

Doing vs. writing

Most people have heard this public relations quote at some point:

Do something good and talk about it.

Though promotion might help getting more support for my projects and possibly advance my career as well, I enjoy hacking on projects much more than rambling on about it. It's a bit different when it's purely about sharing knowledge, but even that takes time away from doing productive work.

Never good enough

I rarely feel that my work is good enough to hold up in the spotlight and tend to focus on the remaining flaws. That relates to both my software as well as my blog posts.

By now I started many blog posts which ended up unpublished and abandoned in half-finished state as I didn't feel they were of substantial value.

A little reminder for me and my fellow impostor syndrome victims: Perfect is the enemy of done.

A no-reason to highlight

One would have to think the fear of being exposed as fraud must stem from experiencing malevolent, adverse criticism.

Quite the opposite is true. In all those years putting out my work all I've gotten has been acknowledgements, words of encouragement and praise. Criticism has only been formulated in positive and constructive manner. Even when I haven't answered issue reports or pull requests on GitHub in weeks, I was never met with hostility whatsoever.

Given that there's so many stories about open source maintainers who have to endure toxic behaviour I'm super grateful that this never happened to me, not even once.

Thank you, people of the internet ❤️ for keeping my hope in humanity intact.

So, where does that lead us

I don't plan on becoming a volume writer anytime soon, but I'd like to slowly resurrect the habit of writing posts to share my learnings.

The kids grow and become more independent by the day. My wife has gotten somewhat comfortable with her condition and we are steadily learning to better balance our individual needs within the family. All that leads me to believe I'll able to spend more time on things like sports and hobbies, including this blog.

I might broaden the topics I write about, starting with this blog post. Coding, personal stories, non-work-related hobbies, you name it.

At the very least some AI bots will enjoy crawling it. Maybe my kids dig it out, once they're older. Who knows.

Thanks for sticking around.

glTF in Unity optimization - 8. Asynchronous Scene Instantiation

2022-09-16T00:00:00Z

This is part 8 of a mini-series.

TL;DR: Asynchronous instantiation reduces frame rate drops on big scenes (i.e. scenes with many nodes and lots of content), but it required breaking more of the API
than I presumed.

Introduction

glTFast currently loads a glTF file in two phases.

Reading the source glTF and converting it to Unity resources (meshes, materials, textures)
Instantiation: Actually creating GameObjects/Entities of a glTF scene

The whole process might exceed the computational time budget available for a single frame and thus causing stutter. That's why async programming was used to spread work across frames, but unfortunately this was only done for the first phase. This post is about the adventure of making phase 2 async as well.

Preparation

First, we need proper test scenes. From previous day-to-day observations I know that scenes with many nodes (and lots of content in general) perform worst and make the frame rate stutter apparent. To create such a test glTF I used Blender and a small python script, that creates thousands of cubes.

import bpy

# count - final count will be 15 to the power of 3 = 3375
c = 15
size = .95/c
# Pick a mesh from a previously created cube
m = bpy.data.meshes['Cube']
for x in range(c):
    for y in range(c):
        for z in range(c):
            bpy.ops.object.add(type='MESH',location=(x/c, y/c, z/c))
            obj = bpy.context.active_object
            obj.scale=(size, size, size)
            obj.data = m

To my negative surprise this script took quite some time to execute. It could probably be sped up by creating the objects in bpy.data and linking it to the scene instead of using the bpy.ops.object.add operator, but I'm here to optimize glTF loading, not this helper script. So I decided to refill my glass of water in the meantime and manually duplicate the resulting cubes a couple more times to have an even bigger scene. It ended up with 13,500 nicely arranged cubes.

Baseline

Overall it takes between 550 and 580 ms to load the model. As expected, frame rate is smooth until instantiation. Here's that nasty last frame that takes ages:

The instantiation blocks execution for 353 ms and the frame overall takes almost half a second. Key observation breakdown:

54 ms creating GameObjects
250 ms populating the hierarchy (e.g. adding Renderer components)
33 ms adding the scene (22 ms of that was parenting root nodes to the scene GameObject; more on that later)

Baseline DOTS

I also tested the experimental DOTS support. Loading time in the Editor is really bad (almost one second), but quite fast in builds:

250 ms to 300 ms overall
133 ms for the last (instantiation) frame

Let's see if we can do something about it.

First Async Attempt

The first approach was to make the instantiation methods async and add breakpoint in between creating or altering GameObjects.

A problem that surfaced right away was that now also half-way loaded scenes are already rendered. A quick solution was to de-activate all GameObjects right after they are created and activate them at the end, after all of the scene is ready. The IInstantiator API does not offer a final callback, so I crammed that into AddScene, which is not very clean, but does the job for now.

Here's the first result:

Although the frame rate drops from 60 to 30 (which could be tweaked), most of the work is spread evenly now. First highlight is GameObject creation and following portion is adding components.

What's problematic is that there is still a sharp spike in the end. Let's look at this frame in detail.

Breakdown of causes of this 200 ms frame time:

Scripting
- 56 ms AddScene
  - 20 ms Re-parenting root nodes onto scene GameObject
  - 22 ms activating all nodes so that they are rendered correctly
Engine
- 122 ms PostLateUpdate.FinishFrameRendering

It's a bit scary that the engine takes 122 ms, but I assume it's a consequence of the way the scene is assembled. Instead let's focus on the scripting part and AddScene, since we're in direct control of that.

Re-parenting thousands of nodes takes some time. We could go ahead and make AddScene itself async to split it up, but I'd like to address a deeper conceptual problem here. Re-parenting a GameObject has a certain computational overhead (that becomes even bigger if it has a deep hierarchy of children). To avoid this overhead, it's best to create hierarchies from the root upwards without any unnecessary re-parenting. glTFast does not do this currently.

IIRC glTFast instantiation historically worked like this:

Create all nodes upfront (i.e. nodes of all scenes; IInstantiator.CreateNode)
Parent non-root nodes to their respective parent node (in IInstantiator.SetParent)
Add components
Add the scene, which parents all root-level nodes/GameObject to the scene GameObject (IInstantiator.AddScene)

This was nice (I thought) since it did not require the overhead of iterating the hierarchy tree in a particular order. At some point support for scenes was added, and then iterating a scene's node tree had to be done anyways (see GltfImport.InstantiateSceneInternal.IterateNodes).

Refactor API

The obvious thing to do now is to create the scene hierarchy starting from the root, so that a parent is always created before its children and parented right away. GltfImport.InstantiateSceneInternal already iterates nodes in that order, but we have to refactor IInstantiator to pull this off. The new approach is:

Create the scene (which certifies and prepares the root level GameObject)
Create nodes (in correct order and parented right away)

Also, previously the scene's node tree was iterated twice. Once for creating the raw hierarchy and a second time to add components (like Renderer, etc.). We can now do it in one go.

While the overall loading time did not suffer much at all the worrying last frame improved a bit.

EndScene, the logical successor of AddScene now takes less than 16 ms instead of 56 ms. I book that as a success, even though the engine itself still spends a lot of time preparing the rendering. Still the last frame is now at around 100 ms instead of 450 ms for this corner-case benchmark file.

DOTS

Besides adopting the EntityInstantiator to the new IInstantiator API I had to fix the visibility problem of half-way loaded scenes as well.

In the Editor this still perform really bad. Up to 4 full seconds to load the test file. I haven't certified, but I blame the DOTS Editor UI for the overhead.

Briefly I thought about optimizing this API for DOTS right away. At the moment I create each node one at a piece via EntityManager.CreateEntity, which the documentation says this about:

Creating entities with the EntityManager is typically the least efficient method

Spawning them all at once in a job seems to be a better solution, but it is a bit more complicated to calculate the number of required entities (which is not necessarily the number of nodes) upfront.

Taking a look at an actual build (which you should always do), the test file loads in ~400 ms overall with maximum frame times below 33 ms.

Even if it's slower overall, the frame rate is quite solid and it's still better than the GameObjects workflow.

For now I decided to not optimize further, but once DOTS nears maturity, I'll definitely have to revise this part.

Side Quest - Materials Count

While benchmarking the test scene I noticed that there was an absurd high count of materials, one per cube to be precise.

The glTF does not contain any materials, so they fallback to the default material. What's going on?

Turns out the fallback material was not cached and re-generated on every occasion. Another small fix along the way.

Conclusion

Reducing maximum frame time from 450 ms to 100 ms (GameObject) and 133 ms to 33 ms (DOTS) is quite the result I was aiming for. There's still potential for improvements (especially for the DOTS implementation), but that has to wait for later.

This work landed in glTFast's main branch and will ship in the upcoming 5.x release.

Next up

I have a couple of half-finishes ideas/posts lying around and haven't decided which one and when to finish, but I strive to put out posts more regularly in the future.

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Overview of this mini-series

glTF in Unity optimization - 7. Performance Tests

2021-02-08T00:00:00Z

This is part 7 of a mini-series.

TL;DR: Being able to check and see how changes to your code affect the performance is super helpful and fun.

I have to admit that lots of my work is not test-driven, but whenever I tests save my ass (which happened numerous times) I promise to improve.

Status Quo

Right now glTFast has a couple of Editor and Play Mode tests. The most important ones are the "(Quick)Load" play mode tests. Given a set of sample glTF files, one test per file is created. The only set that is tested currently is Khronos' official glTF-Sample-Models (this could and should be extended to further sample files in the future).

In Unity's Test Runner window it (hopefully) looks like this:

The tests themselves only trigger loading the file, wait until it finished and check if no exceptions or error logs happened. What they do not consider is:

The speed or duration of the loading process
The resulting data
The visual result

In the last post I already mentioned I need a way to automatically and systematically measure performance changes to avoid regressions and this is what the rest of this post will be about.

Checking the resulting data and the visual results will be a topic for follow-up posts.

Measuring Performance

At first I made the mistake of trying to create my own time measuring tests, only to realize that this is a lot of work (if done well). I started searching for existing solutions and quickly found the Performance Testing Extension for Unity Test Runner, a Unity package in preview.

To get started, you decorate your tests with the Performance attributes:

[Test, Performance]
public void MyTest() {...}

There's a couple of things you can measure. I for once wanted:

Repeat test procedure ~ 10 times to even out deviations
Warmup (run the test at least once first without capturing)
Capture total duration
Capture frame times

The performance tests framework offers all of these out of the box, but I couldn't find a straight-forward way to combine them all. So took care about warmup and repetition manually. Here's one of the results:

[UnityTest]
[UseGltfSampleSetTestCase(glTFSampleSetJsonPath)]
[Performance]
public IEnumerator SmoothLoading(SampleSetItem testCase)
{
    Debug.Log($"Testing {testCase.path}");
    var go = new GameObject();
    var deferAgent = go.AddComponent<TimeBudgetPerFrameDeferAgent>();

    // Set up measuring total duration
    SampleGroup loadTime = new SampleGroup("LoadTime", SampleUnit.Millisecond);

    // First time without measuring
    var task = LoadGltfSampleSetItem(testCase, go, deferAgent);
    yield return WaitForTask(task);

    using (Measure.Frames().Scope()) {
        // Repeat test `k_Repetitions` times
        for (int i = 0; i < k_Repetitions; i++) {
            task = LoadGltfSampleSetItem(testCase, go, deferAgent, loadTime);
            yield return WaitForTask(task);
            // Wait one more frame. Usually some more action happens in this one.
            yield return null;
        }
    }
    Object.Destroy(go);
}

Inspecting Results

The package comes with a brilliant, dedicated window for test inspection.

The top bar chart shows the frame times. This particular test has extreme outliers which should be investigated.

The second chart shows the overall loading times with 10 repetitions.

Comparing Results

So far this package is already very useful, but the next step is to re-run the tests after changes were made to see how the results compare. Or run the tests on different hardware or build configurations.

Performance Benchmark Reporter

The documentation refer to a tool called Performance Benchmark Reporter. It's a .NET based CLI tool that takes the output files that performance tests creates and produces an HTML page with comparison bar charts for each test. Here's how it looks like:

Initially I was keen on it, since it provided a solution out of the box. The more I played with it, the more flaws I found:

It does not really give you a quick overview across all test. You have to disable "filter by failed" and scroll through results a lot to get a feel.
It is very slow to view and react as the number of tests goes into the hundreds. Resizing the window took seconds. It uses HTML canvases to draw the charts for each test on a single (non lazy-loaded page)
No way of filtering or sorting for total load times or frame times (those are called sample groups, in performance test speak)
Result files have to be copied and building the page has to be triggered manually.

Custom Charts

Disappointed with the fact that I couldn't quickly make sense of all this cool results that I had gathered I thought that I could probably make some charts myself if only I had the data in a spread sheet.

Since the test data is written to disk in a JSON file I tried to quickly hack a script that does exactly that: Output the data I wanted in a CSV file that can be copy pasted into my spread sheet app of choice.

The output of the Python script allowed me to create my own charts.

I still have to trigger the creation of CSV files and copy-paste them manually, but making and updating meaningful charts is doable now.

Putting it to use

Before I created this setup I used to compare a handful of manually selected models and watched at the total time all tests would take, which was tedious and prone to overseeing something.

Creating this kind of performance benchmark set me back a bit, but I cannot stress enough how much more fun it is to tweak performance now and seeing actual, exact results across all test files in a matter of minutes.

It helped me spot regressions already and I'm sure it will in the future.

Follow-Up

There's still a lot room to improve:

Add more real-world and corner case glTF models to tests (e.g. high polygon models)
Measure memory usage
Run benchmarks on mobile and WebGL platforms (at the moment it's only Editor and standalone)
Automate creating comparison charts (further)
Automate running the tests

When it comes to quality assurance in general, there's even more missing pieces

Code Coverage
Render Tests
Integration of all test in Continuous Integration

I hope I'll be able to push in this direction as we go. It certainly won't get boring soon.

In the next post I'll try to show how the performance benchmarks helped me improve frame rate and loading speed alike.

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: Asynchronous Scene Instantiation

Overview of this mini-series

glTF in Unity optimization - 6. Asynchronous Programming

2021-01-27T00:00:00Z

This is part 6 of a mini-series.

TL;DR: C# Asynchronous Programming opens opportunities to make your code clean and fast

Disclaimer

Before we even start, two things:

This is not an extensive comparison between Coroutines and asynchronous programming. I highly recommend watching the presentation Best practices: Async vs. coroutines - Unite Copenhagen.
Async is not generally a replacement for Coroutines in Unity. The two have their respective pros and cons. Always choose the right tool for the task!

Cool? Then lets get started!

Why do we need coroutines

When a glTF file exceeds a certain size, loading it might take longer than there is time available within a single render cycle (frame). If that happens, certain frames take very long, your framerate starts to drop and you will see visual stuttering (which is especially bad in immersive XR). A way to prevent this is to split up the work into smaller portions and spread those evenly across frames.

Another use case are I/O bound tasks, like loading a file into memory or performing a download. While the data loads, rendering should continue without hiccups.

Unity Coroutines

One way to achieve this is to use Unity Coroutines. They creatively use an enumerator pattern to defer the execution of parts of a method to a later point in time. Whenever you call yield return null; in your coroutine, execution would stop at this point and continue in the next frame.

Some pseudo code example demonstrating how coroutines were used in glTFast


// The lack of return values is compensated by storing them in class variables
int globalVar;

void Start() {
  // Notice: you need a reference to a MonoBehaviour (this)
  // to start the coroutine
  this.StartCoroutine( SplitTheWorkCoroutine() );
}

IEnumerator SplitTheWorkCoroutine() {
  Debug.Log("I'm shown right away");
  yield return null; // Pause here and return a frame later
  Debug.Log("I'm shown on the next frame");
  yield return new WaitForSeconds(1); // Pause here and return a second later
  Debug.Log("I'm shown a second later");
  // storing some result state
  globalVar = 42;
}

Coroutines are lean and effective. They served me well a lot of times. With regards towards the future of glTFast optimizations, they raise some concerns though:

You need a MonoBehaviour to run the coroutine
Coroutines must be run exclusively on the main thread
No support for return values, so you need a workaround (storing results in class variables mostly)

The future of high performance computing in Unity is called DOTS. Part of this philosophy is about replacing GameObjects and MonoBehaviours with Entities and (data only) Components. At the time of writing DOTS is still under development and I haven't found the time to experiment with it. What seems clear though is, that depending on MonoBehaviour and coroutines is not future proof.

glTFast already uses the C# Job System to offload work to other threads. Code that operates on managed data cannot be run with C# Job system though.

Async to the rescue

The more I learned about asynchronous programming, the more it appeared to be a viable option for glTF. Replacing coroutines would make changes to the API necessary, but turned out to not be too much of work.

Example above converted:


async void Start() {
  // This method runs non-blocking, but is "awaited". Its return value can be used afterwards.
  var result = await SplitTheWork();
}

async Task<int> SplitTheWork() {
  Debug.Log("I'm shown right away");
  await Task.Yield(); // Pause here and return a frame later
  Debug.Log("I'm shown on the next frame");
  await Task.Delay(1000); // Pause here and return a second later
  Debug.Log("I'm shown a second later");
  return 42; // You can return values like any regular method
}

Benefits

To sum up some of the benefits of async:

No more MonoBehaviour
Pushing work on other threads should become easier (more on that in a next article)
Async method can have return values
- This resulted in less boilerplate code and removal of class variables for result state
More meaningful stack traces (also in profiler)

Benchmarking the results

I did not expect any significant change in performance between the two approaches. In order not to introduce performance regressions, I did some random benchmark.

For the tests I used a couple of different glTFs with unique characteristics, loaded them repeatedly and picked average values. The differences were similar across models.

Example model: Buggy. This glTF has a somewhat complex scene graph (and thus bigger JSON part), but no textures.

glTFast 2.5.1: 80 ms (17.6 ms max frame time)
glTFast async: 44 ms (21 ms max frame time)

Wow, that's a LOT of unexpected difference. Looking at the profiling data it seems that the coroutine version renders another two frames in between starting loading and instancing the glTF. The CPU cycles of those two frames should be put to use. It turns out, I was a little too aggressive in the past when it comes to spreading work across frames. The code waited for no reason for the next frame twice.

On the other hand, notice the increased max frame time, which botches the desired frame rate. The async version does not defer work so aggressively and thus executes too much work within one cycle. This was especially bad on models with many Jpeg/PNG textures.

The conclusion here is that the decision at what point execution should pause until the next frame is subject to optimization.

Conclusion

I am not happy with the fact, that I discovered these unexpected changes by coincidence and not systematically. The regressions of the async branch need to be eliminated/reduced.

But even more important, I'm gonna need some automated performance testing, so that regressions don't go (un)detected by coincidence in the future.

Next up

I've already started working on both performance test automation and optimizing async processes. Very likely I'll write about it.

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: Performance Tests

Overview of this mini-series

Matrix Decomposition in Unity

2020-11-16T00:00:00Z

TL;DR: Unity's Matrix4x4.rotation and Matrix4x4.lossyScale deliver incorrect results on some corner-cases. Here's an alternative solution.

In this article I'll tell you about the journey from detecting the problem towards solving it but won't go into math details. Feel free to jump ahead to the problem definition and the solution.

The Issue

A while ago an issue was raised which observed strange rotation errors when loading a glTF 3D assets in Unity with glTFast and I could reproduce it:

Some objects like tires, doors and head lights are rotated incorrectly ☹️. In order to draw comparison (and rule out content errors) I viewed the file in the glTF Viewer and imported it into Blender. In both cases it seemed to work fine:

What's cool in Blender is that you can inspect and compare the transform values. I picked one of the tires to inspect closer. If you look at the transform panel (on the right side in the image), you notice that the scale is a uniform negative 1. While that's not good practice, it certainly shouldn't break positioning of objects. The rotation is shown as quaternion. I switched it to Euler, so it's easier to compare it with Unity inspector's transform component.

Let's compare this to the transform values in Unity:

They differ! Even when considering that Unity and Blender have different coordinate systems (Y-up versus Z-up), in Unity…

…the scale is negative in X only
…rotation values are not equal

Let's found out why!

Lucky guess

In the past I rarely had problems with glTF files exported from Blender, so I re-exported the asset and imported this second version in Unity again. This time it worked, so I started comparing the original and the Blender export. glTF's scene definition is JSON based and thus readable.

In glTF, a node's transformation can be defined by either a tuple of translation, rotation, and scale or a (4-by-4) transformation matrix. It turned out the original was using matrices only and the re-export separate transformations 💡.

Enter the Matrix

I decided to dig down into the matrix import code, which consists of two steps:

Convert the matrix from glTF's coordinate space into Unity's (by flipping signs on certain values)
Decomposing the matrix into separate translation, rotation, and scale (since you cannot assign a full matrix to a Unity Transforms).

I tinkered with the space conversion without any luck. I tried a different approach (via conversion matrix multiplication), which yielded the same result as before but didn't solve the issue.

I then wanted to see if omitting the space conversion yielded in the desired uniform scale of -1, which it didn't. I started to suspect the error lies in the matrix decomposition, which looked something like this:

// Given is a matrix (already converted to Unity space):
Matrix4x4 m;

if(m.ValidTRS()) {
  // Translation is the first three values in the last column
  position = new Vector3( m.m03, m.m13, m.m23 );
  rotation = m.rotation;
  scale = m.lossyScale;
}

Both Matrix4x4.rotation and Matrix4x4.lossyScale are undisclosed, so I couldn't investigate how they work.

Before going on I had to freshen it up my linear algebra theory and picked up Foundations of Game Engine Development, Volume 1: Mathematics by Eric Lengyel (great book!) and found some useful inputs. It was immensely helpful when trying to understand other people's code.

There are many different ways to decompose a matrix. I found some algorithms and tried one of them out. Still the same error, but identical (which is also a great observation). I now had a starting point to compare to. Unfortunately I discarded this interim result and cannot find the source anymore.

The next thought was "What does Blender do different than this Unity script?". Since Blender is open source it only seemed natural to me to look it up. Turns out Blender does an additional negativity check on the rotation matrix and flips both scale and rotation in some cases. Sounds exactly like what it's missing, so I decided to port this code and use the Unity.Mathematics package for it. It already contains types and methods I'd need (like float3x3, a 3-by-3 matrix), which saved a lot of time. It's also said to be well optimized, so yay.

Behold the results:

🎉🚙😎

Performance

Whilst at it, I ran the conversions in a loop to see how they perform:

So the downside of the correct solution (Matrix4x4.DecomposeCustom) is that it's ~2.3 times slower. This is done once per node. The typical scene won't have a large number of nodes, so it's safe to neglect the minor performance loss.

There's also a tiny raise of memory allocations. I made another, pure Unity.Mathematics types based variant (float4x4.Decompose), which does eliminate this flaw. Another good reason to switch to Unity.Mathematics types overall.

Problem Definition

The original problem turned out to be a corner-case matrix with…

Negative scale
Rotations in multiple axis, one of them being 45° (so perfectly in-between quarter rotations)

Separating rotation and scale is a non-trivial problem that only gets harder if you cannot assume that the scale is positive. I assume Unity's Matrix4x4.rotation and Matrix4x4.lossyScale was written with only positive scales in mind. It's also worth mentioning that they are two separate, non-coherent calculations (hidden behind properties). It may be, that they're not consistently aligned for corner-cases.

Solution

The solution was to port the matrix decomposition algorithm of Blender to C#:

Here's the Solution Source Code in C# ✅

It can be used like so:

// Given this matrix
Matrix4x4 m;
// But could also be this Unity.Mathematics type
float4x4 m;

m.Decompose(out var t, out var r, out var s);
position = t;
rotation = r;
scale = s;

I hope that was helpful.

glTF in Unity optimization - 5. New Mesh API - The Refactor

2020-04-13T00:00:00Z

This is part 5 of a mini-series.

TL;DR: The speed improvements from the advanced Mesh API for glTFast are marginal and needed to be earned via a time-consuming refactor.

Goal

In the last post I concluded that in order to use the advanced Mesh API to its fullest, I'd have to refactor a lot of things. In this post I want to show you what I did and compare the results.

Problem

Meshes' vertices come with a couple of attributes. At least they have positions (in 3D space), but most often also normals, texture coordinates, tangents and sometimes colors, weights and even more texture coordinates. The advanced Mesh API only supports up to 4 vertex streams.

Solution

Once you have more than 4 vertex attributes, you have to combine some of them into one interleaved vertex stream. From a memory layout perspective this means instead of having one array per vertex attribute you now have to create an array of structs (AoS), where the struct contains multiple vertex attributes.

So instead of this…

vertexCount = 100;
var positions = new Vector3[vertexCount];
var normals = new Vector3[vertexCount];
var tangents = new Vector4[vertexCount];

…you create something like this:

[StructLayout(LayoutKind.Sequential)]
struct Vertex {
    public Vector3 position;
    public Vector3 normal;
    public Vector4 tangent;
}

vertexCount = 100;
var vertexData = new NativeArray<Vertex>(vertexCount,Allocator.TempJob);

The next step is to retrieve the data from the glTF buffers into the NativeArray. This was done via C# jobs before and has to be changed, so that the interleaved nature of the output array is respected. I introduced an outputByteStride parameter for that reason. A new C# Job looks something like this:

public unsafe struct GetVector3sInterleavedJob : IJobParallelFor {

    public int inputByteStride;
    public byte* input;
    public int outputByteStride;
    public Vector3* result;

    public void Execute(int i) {
        float* resultV = (float*) (((byte*)result) + (i*outputByteStride));
        byte* off = input + i*inputByteStride;
        *((Vector2*)resultV) = *((Vector2*)off);
        *(resultV+2) = -*(((float*)off)+2);
    }
}

I had to change all existing >50 Jobs to support output byte strides. On top of that, I had to create new data structures/classes and determine the way vertex data will be clustered retrieved before being able to schedule these new Jobs.

This better be worth it 😃

Benchmarks

High polygon scene

First test scene is a high resolution mesh (4 million triangles) with normals tangents and texture coordinates. These are the most important timings when loading it with glTFast 1.0.0

Phase	Duration
Prepare	12 ms
Retrieve Data	19 ms
Set Positions	21 ms
Set Indices	65 ms
Set UVs	34 ms
Set Normals	59 ms
Set Tangents	101 ms

Initial test were not very promising. Overall loading times were identical or worse, not better. A little investigation showed, that setting the data was taking too long. This led to a first learning: the advanced Mesh API is faster, because it makes certain sanity checks (like index out of bounds) optional. But to benefit from that, you'd have to actually disable those checks via MeshUpdateFlags:

MeshUpdateFlags flags =
  MeshUpdateFlags.DontNotifyMeshUsers
  | MeshUpdateFlags.DontRecalculateBounds
  | MeshUpdateFlags.DontResetBoneBounds
  | MeshUpdateFlags.DontValidateIndices;

msh.SetVertexBufferParams(…);
msh.SetVertexBufferData(vertexData,0,0,vertexData.Length,stream,flags);
msh.SetIndexBufferData(indices,0,indexCount,indices.Length,flags);
msh.SetSubMesh(0,new SubMeshDescriptor(indexCount,indices.Length,topology),flags);

After I did this, I got improved timings.

Phase	Duration
Alloc VB array	55 ms
Alloc UV array	8 ms
Alloc Index array	11 ms
Retrieve Data	30 ms
Set VB Params	47 ms
Set VB Data	14 ms
Apply UV	3 ms
Set Index Data	31 ms
Set SubMesh	0.004 ms
Recalculate Bounds	38 ms

Since so many things changed at once here (the structure of the code base and the procedure of loading in general), we have totally different load phases and to some extent this is like comparing apples and oranges. I'll try to interpret this as fair as possible and figure out the actual factors of causation.

The former Prepare phase consistent mostly of allocating C# arrays for mesh data (like Vector3[] for positions) and was surprisingly fast compared to the newer allocations of NativeArrays (Alloc VB=VertexBuffer, UV=Texture Coordinates and Index arrays). Since in the past I got errors when holding NativeArrays for more than 4 frames (which occasionally happens when bulk-loading many scenes), I used the slower persistent allocator. But even changing the Allocator type to TempJob didn't bring much relieve.

Retrieving data (via C# Jobs) became up to 50% slower. I assume the addition of the output byte stride is the reason. The positive thing to mention is that the vertex buffer data jobs start in parallel before the Index buffers are allocated now (5 ms - 10 ms overlap). In some tests runs data retrieving was not slower overall because of that. Makes me wonder how much potential lies in restructuring the Coroutines.

Let's compare setting the data. The new Set VB params and Set VB Data (61 ms combined) are replacing the former Set Positions, Set Normals and Set Tangents (181 ms combined). Hard to tell why it's so much faster. Maybe the promised benefits of the advanced Mesh API, maybe the old interface triggers additional, dispensable calculations.

Set UVs was replaced by Apply UV and is 11 times faster!

Set Indices (65 ms) was replaced by Set Index Data and Set Submesh (combined 31 ms). I assume this is because the old interface calculates the mesh bounds automatically. I added it (after getting errors about invalid bounds), which takes 38 ms. So overall this is comparable to slightly slower.

Most important is the overall load times, so I made 10 runs reach and it got ~14% faster.

	1.0.0	new Mesh API
High Polygon Scene	716 ms	618 ms

Test sample set

Even with a specific test scene it is hard to lay the finger on where exactly things got better or worse without knowing the inner workings. So I tried to get a feeling for the overall impact on a wide variety of lower complex scenes. I took the glTF Sample Model set, removed irrelevant scenes (embed buffers, draco compression) and loaded the 114 files all at once.

	1.0.0	new Mesh API
glTF Sample Models	10.6 sec	10.1 sec

That's an ~4% - 5% improvement.

Conclusion

The way I implemented usage of the advanced Mesh API is an ever so slight improvement, no doubt. It is not the silver bullet I hoped it would be though.

I still think it is a good foundation to build upon and try the next couple of things. Maybe I missed something and use it in an non-optimal way. I'd be delighted to be corrected, in that case.

This work is not in a release yet, as I think it needs some more work and polishing before going live.

Where to go from here

With this refactor work being out of the way, I can move on to try other ideas for improvement. Some candidates that popped up as a result of this post:

Maybe it is better to set a mesh's vertex parameters before/in parallel to data retrieval.

One thing is to take a look at are the now slower Jobs:

Schedule them as soon as possible for better parallelization
Burst and Mathematics
Optimized special Jobs for most used cases

Draco meshes were not involved thus far, so those should be using the advanced Mesh API as well.

Looking at the profiling data, more specifically the worker threads, it seems that the C# Jobs are not the bottleneck. So looking into smarter flow of the async loading process seems to be the most promising endeavour.

Next up

I haven't decided yet. It'll be a surprise 😃

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: Asynchronous Programming

Overview of this mini-series

glTF in Unity optimization - 4. New Mesh API - The Failed Attempt

2020-04-06T00:00:00Z

This is part 4 of a mini-series.

Goal

With version 2019.3 Unity introduced a new Advanced Mesh API for creating meshes and announced the following advantages

Faster (yay!)
Flexible vertex attribute data layout

It's faster, since it omits all validation checks the simple API does. In this post we will see if that's true for my cases.

The plan I have in mind is start from the rear end (mesh data submission) and gradually improve the workflow towards data retrieval from glTF buffers.

Replace simple by advanced API calls
Replace existing data structures (C# arrays) by Unity NativeArrays
Experiment with data types (instead of using floats for everything, use smaller types; esp. if the original glTF type is not a float)

Step 1: Advanced API calls

First thing I did was replacing the simple API calls by the advanced ones (see commit).

This is an approach from the rear end, where all vertex data is already retrieved from buffers in the form of arrays (e.g. Vector3[]) and ready to be pushed. I created one vertex stream per attribute.

Test 1: Full high resolution mesh

First comparison loading a high resolution mesh with UVs, normals and tangents (repeated 10 times)

	glTFast 0.11.0	glTFast dev	speedup
SetVertices	20.51 ms	3.17 ms	6.5 x
SetIndices	66.42 ms	29.34 ms	2.3 x
SetUVs	31.99 ms	2.07 ms	15.45 x
SetNormals	54.14 ms	3.16 ms	17.1 x
SetTangents	93.80 ms	4.42 ms	21.2 x
RecalculateBounds	34.63 ms	30.02 ms	1.2 x
UploadMeshData	117.70 ms	121.69 ms	1.0 x

That's some great improvements 😀! Vertex data is 6x to 21x times faster and setting indices is twice as fast. As a result the total loading time for 10 huge meshes went down from 8.0 sec to 5.5 sec ( 45% faster ).

Test 2: High resolution mesh without normals and tangents

The second test is the same mesh, but without normals and tangents

Note: In previous posts of this miniseries it became clear that normal/tangent calculations are a bottleneck. Still I'd like to see if the new mesh API improves the situation.

	glTFast 0.11.0	glTFast dev	speedup
SetVertices	32.64 ms	3.09 ms	10.6 x
SetIndices	69.80 ms	27.38 ms	2.6 x
SetUVs	35.42 ms	2.27 ms	15.6 x
RecalculateNormals	130.61 ms	75.62 ms	1.7 x
RecalculateTangents	960.36 ms	892.66 ms	1.1 x
RecalculateBounds	35.81 ms	26.77 ms	1.3 x
UploadMeshData	134.27 ms	132.60 ms	1.0 x

The normal and especially the tangent calculations are still devastating, but they got a bit faster. Other than that we see similar results. Setting positions is even 10 times faster. The total time went down from 16.4 sec to 14.5 sec ( 13% faster ).

Test 3: Generic sample sets

Moving on to sample set with more variation and practical real-world features.

	glTFast 0.11.0	glTFast dev	speedup
glTF sample models	9.48 sec	8.82 sec	+7.5%
furniture set	9.08 sec	8.32 sec	+10%

This looked promising at first sight, but I saw that I introduced regressions (like not supporting a second UV set or vertex colors). When I tried to fix those to re-run the tests, suddenly Unity crashed 😱 .

I tracked down the crash at Mesh.SetVertexBufferParams. Turns out the troubling mesh primitive uses positions, normals, tangents and two sets of texture coordinates. Reading the docs carefully I found that Unity supports up to four vertex streams maximum and my "one stream per attribute" approach exceeds this limit and causes Unity to crash.

I filed a bug report and Unity fixed it in 2020.2. It still won't work, but at least fellow developers don't have to wonder why it crashes anymore 💯.

I investigated a bit in the code base and came to the conclusion, that for the moment this is a dead end. The experimental Mesh API support stops here 🛑.

Next up

In order to use the advanced Mesh API I have to group vertex attributes in a way that results in 4 vertex streams or less, no matter how many attributes there are.

This totally screwed up my plan of starting at the "rear end" and improve from there in tiny steps. I'll have to refactor the data retrieval from start to end in order to support this. The positive initial results motivate me to do exactly that, so I decided to draw a line under the current version of glTFast and call it version 1.0 before I proceed doing this major refactor.

On the plus side, I'll build the refactored version based on NativeArrays from the start, so that's two things at one sweep.

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: New Mesh API - The Refactor

Overview of this mini-series

glTFast Next Steps

2020-03-13T00:00:00Z

State of glTFast

A while back I started a mini-series, in which I document my aspiration to get my Unity package for loading glTF (glTFast) more efficient, modern and elegant.

I'm far from running out of ideas yet, but I have to balance backwards compatibility vs. innovation speed.

First of all, due to excessive profiling I figured out where the biggest potential for improvements is and followed the trace. I started using Unity's new advanced Mesh API and was making first progress (up to 45% faster loading of big meshes). I also figured that in order to get ideal results and to not introduce regressions, I have to do a significant refactor.

I wasn't in the mood for that yet, so I did a quick stab at Unity's new Burst compiler. Again I could strip off some milliseconds and retrieving data from buffers got ~50% faster in a test case. But again, this is not backwards compatible to the currently supported 2018.2.

Next I tried to tackle some last, missing features (outside of animation; like support for multiple UV sets), but the thought that this is not important (to me) and possibly obsolete in upcoming tech stacks did not appeal to me at all.

So what to do then

Today (on Friday the 13th) I freeze glTFast's feature set and release version 1.0.0! 🎉🎉🎉

I think the project is at a point where it can be used in production. Possible fixes will be released in 1.0.x cycles.

Future development will be based on Unity 2019.3/4 and break backwards compatibility with older versions. Dropping the baggage of legacy support allows me to innovate faster. Focus will be speed and efficiency foremost. I think about:

Advanced Mesh API
Burst
DOTS
Loading improvements (maybe async)
Streaming

Secondary features/goals that might come in are

Newer glTF next gen material features, as they evolve
Universal Render Pipeline support
High Definition Render Pipeline support
Better interface and more developer convenience

I'll release those innovations in a 1.x cycles until it is ready for production in a final 2.0 release.

I think this project has some exciting times ahead. I'd be glad if you follow and star it on GitHub.

Cheers!

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

glTF in Unity optimization - 3. Parallel Jobs

2020-02-28T00:00:00Z

This is part 3 of a mini-series.

Goal

In this episode I wanted to investigate, if the task of converting index/vertex data from binary buffers into Unity structures could be sped up by using parallel jobs.

The abstract steps of converting data in glTFast look like this

Prepare Jobs
- Allocate memory for the destination Unity data structures
- Schedule Jobs
Execute Jobs
Create Meshes from retrieved data structures
Instantiate GameObjects

The job execution is already threaded via the C# Job system, but one glTF Accessor is always retrieved in one thread and not split up further. For smaller meshes this is probably fine, but I can imagine large meshes may benefit if the work is spread across CPU cores more evenly.

Analysis

Lets do some profiling to get a measure of the status quo. I used a 4 million triangle mesh with indices, positions and texture coordinates only.

The interesting bit is in the lower half. After scheduling the jobs, the worker threads start the execution roughly at the same time. 3 Workers have nothing to do, the UV job is done after 12.56 ms while the indices job takes the longest ( 24.55 ms ). Since we can only continue to build the mesh after all jobs are finished, 24.55 ms is the starting measure.

What I try to achieve is to spread the work more evenly across the worker threads, so the total computation is finished earlier. If I add up the Job's execution times and divide it evenly by the number of workers (6), I get 9.21 ms, which would be the absolute optimum if there was no overhead for spreading the work. Let's see how close we can get.

Status Quo: Job implementation

There are over 30 different C# Jobs in glTFast, all of which implement the IJob interface and most of the have a main for-loop at their heart.

To make them parallel, they have to implement the IJobParallelFor interface. Let me demonstrate this on one simplified example. This Job retrieves the mesh indices in one loop:

public unsafe struct GetIndicesUInt32Job : IJob  {

    public int count;
    public System.UInt32* input;
    public int* result;

    public void Execute()
    {
        int triCount = count/3;
        // This is the main loop we want to spread
        for (var i = 0; i < triCount; i++) {
            result[i*3] = (int)input[i*3];
            result[i*3+2] = (int)input[i*3+1];
            result[i*3+1] = (int)input[i*3+2];
        }
    }
}

Let's go parallel

Instead of this, we want the Job Scheduler to split this up onto the threads.

public unsafe struct GetIndicesUInt32Job : IJobParallelFor  {

    public System.UInt32* input;
    public int* result;

    public void Execute(int index)
    {
        result[index] = (int)input[index];
    }
}

Scheduling this job changes a bit:

// old way for IJob
var jobHandle = job.Schedule();
// new way for IJobParallelFor
var jobHandle = job.Schedule(indexCount,1);

We now have to provide the number of iterations (index count in the example) to the Schedule method. Before it was a member of the Job struct (public int count;);

The second parameter is called batch count. We get to that soon and leave it at a default 1

First test

Now the work is spread across workers evenly, but the total execution time went up to over 32 ms! 😱

Reading the documentation explains why. Our new job has little computational complexity but is repeated millions of times. Spreading this iterations across threads does have an overhead and if this is done on a very small/granular level, the overhead is bigger than the gain.

Batch count counts

This is where the batch count parameter comes in. It tells the Job System to not make batches with less iterations than this number. So next thing I did was increase the batch count in test iterations and tried to find a sweet spot. I settled for 50000 for this Job, but results may vary on different (more complex) jobs or different hardware.

Let's see how it performs now:

Down to 14.6 ms. That's not as low as the targeted 9.21 ms, but 40% less than the original 24.55 ms. And look how neatly aligned the blue bars on the worker threads are 😍. Good job 😎!

I think the result may be improved further if the batch count is tweaked more carefully on the actual target hardware (which will be mainly mobile in my case). That I still have to do.

Hot loops

Some jobs had an option for normalization, like this one:

public void Execute()
{
    if(normalize) {
        for (var i = 0; i < count; i++)
        {
            result[i].x = input[i*2] / 255f;
            result[i].y = 1 - input[i*2+1] / 255f;
        }
    } else {
        for (var i = 0; i < count; i++)
        {
            result[i].x = input[i*2];
            result[i].y = 1 - input[i*2+1];
        }
    }
}

There were two design decisions I made when writing this:

One Job for both normalized and not normalized variants for more convenient Job usage.
The normalize condition is outside of the hot loop. It's a bit more redundant because of this, but it's faster.

When making them parallel, I have to either make two Job types or move the if condition into the loop. I chose the first option for performance reasons:

// "regular" variant:
public unsafe struct GetUVsUInt8Job : IJobParallelFor  {

    public byte* input;
    public Vector2* result;

    public void Execute(int i)
    {
        result[i].x = input[i*2];
        result[i].y = 1 - input[i*2+1];
    }
}

// And the second, normalized variant:
public unsafe struct GetUVsUInt8NormalizedJob : IJobParallelFor  {

    public byte* input;
    public Vector2* result;

    public void Execute(int i)
    {
        result[i].x = input[i*2] / 255f;
        result[i].y = 1 - input[i*2+1] / 255f;
    }
}

I applied this everywhere and ended up with 43 parallel jobs now (See the commit for details). This was released in glTFast 0.11.0.

Putting it in perspective

Shaving off a couple of milliseconds sure feels nice, but looking at the overall loading time (which is still hundreds of milliseconds), frankly it doesn't really matter that much 🙁

The demo scene profiled here also is an extreme case. Yes, high triangle counts will benefit more, but I guess most content out there is not.

Originally I wanted to go on and see how much Unity.Mathematics and the Burst compiler can speed up the Jobs, but it seems that the energy is spent wiser elsewhere first and get back to that later.

Next up

I'll probably tackle the new Mesh API in an attempt to get these crazy mesh instantiation times down. Stay tuned!

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: New Mesh API - The Failed Attempt

Overview of this mini-series

glTF in Unity optimization - 2. Avoid Tangents and Normals Calculation

2020-02-26T00:00:00Z

This is part 2 of a mini-series.

Instant side quest

Originally I wanted to investigate the benefit of parallel jobs in this post. I started by creating test cases and profiling the status quo. That's where I figured that there's higher potential in other optimizations.

Analysis

I created two test scenes

The "HighRes" scene with one 4 million triangle mesh
The "Sixfold" scene with six meshes with ~252k triangles each (1.5 million triangles total)

This is the HighRes scene in Unity's profiler. More specifically, the first frame:

Most time is spent/lost allocating Unity's data structures (a hint to switch to the new Mesh API, maybe). Let's look at the second part in the following frame:

Here the Unity Mesh is created and it takes a devastating 1.279 seconds !!! Most of that time (1.06 seconds) is spent calculating tangents. Overall this scene loads in ~1.6 seconds.

For the Sixfold scene the result is similar, except that the six Mesh creations are spread amongst six frames, so at least the frame stall is not as bad.

Optimize

So tangent recalculation is expensive. The best and first way to make code faster is to not execute it, so why do we need tangents in first place? Some materials/shaders rely on correct them. I know they are needed for consistent normal mapping, but I'm not sure if for anything else (maybe anisotropic shading?). For now I assume we only need them for normal mapped materials.

Calculating normals is less expensive than tangents, but while we're at it, let's think about them as well. They are necessary for all shaded materials, but not for unlit ones.

So in glTFast 0.10.2 I changed the importer to only calculate normals/tangents, if the material requires them. Let's have a look at the final frame when loading the previous scene again:

Boom, down a full second. The whole scene now loads in ~700 ms. It's even faster when using unlit materials:

Loads consistently in under 400 ms. Nice achievement for a couple of lines of code.

This was released in glTFast 0.10.2.

Next up

What about all the cases where having normal/tangents is absolutely necessary? The Unity mesh API does not provide us with a threaded version of RecalculateNormals or RecalculateTangents, so the only way around this would be to make one's own C# job that does the normal/tangent calculations on a thread.

I definitely plan to do this, but only after glTFast switched to the new Mesh API (based on NativeArrays) and other improvements, so stay tuned!

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: 3 Parallel Jobs

Overview of this mini-series

glTF in Unity optimization - 1. Buffers, Accessors and Primitives

2020-02-18T00:00:00Z

This is part 1 of a mini-series.

TL;DR:

Reading the glTF specification carefully right away would have spared me performance issues in corner cases later on.
How to get from 8 GB down to 0.034 GB of mesh memory usage. glTF may be a bit too generous and allows exporters to create valid, but troublesome data.

Observations

Though glTFast did a decent job on all of the assets in the official sample model repository, some users reported poor performance with certain scenes of theirs.

The investigation showed, that I was approaching loading (mesh) primitive data from the wrong side. Let's look at how a glTF scene is structured.

I'll give a brief overview of the glTF 2.0 specification. If you already know the intrinsic structure of glTFs, you might want to skip ahead to the solution.

In this article I'll use words that can stand for a general concept and a specific glTF schema entity as well (like a mesh). I'll use capitalized words when I talk about the glTF schema entity in particular and lower-case when it's about the general idea/concept.

glTF scene structure

A glTF scene consists of a hierarchy of Nodes. Nodes have a transform (position, rotation and scale) and can have any number of child Nodes. They can also have a reference to a Mesh, which indicates that the Mesh's geometry is supposed to be rendered with the Nodes transformation. A Mesh can be references by multiple Nodes in a Scene.

glTF data structure

In glTF all geometry data is inside of Buffers, which are plain binary data. Several entities in the glTF's JSON describe how this data is structured and how it can be accessed.

The smallest unit of visible geometry is a Primitive. It consists of triangle indices, references to vertex attribute buffers via Accessors ( positions, normals, texture coordinates, colors and tangents) that those indices reference into and a material. A Primitive belongs to a Mesh, which can contain multiple Primitives.
The Primitive's references to vertex attribute buffers are actually references to Accessors. Accessors define the attribute's type (scalars, vectors of 2,3 or 4 dimensions, etc) and its components data type (byte, short, float). They also reference which BufferView contains the described vertex attributes. The BufferView is the final connection to the actual data inside the Buffers. BufferViews define in which Buffer, at what position (and in case of interleaved attributes, the byte-stride) the actual data is.

This structure gives DCC (digital content creation) tool architects a lot of possibilities and freedom to put 3D scenes into glTFs. It allows efficient re-use of Accessors and Meshes, so there's no need to make data redundant.

glTFast's behavior status quo

When I sketched out glTFast, I apparently was approaching things from the final scene structure's perspective and not from a data perspective. I can hear DOTS folks shaking their heads now 😉

glTFast iterates over all Primitives and then retrieves their data through Accessors and BufferViews from the Buffer. Retrieving means the raw byte values are transferred into C# data types / Unity data structures (like Vector3 arrays for positions) and converted into Unity's coordinate system's space.

This is simple and in many cases fast enough (or not slower), but as soon as multiple Primitives reference the same attributes or indices, those will be retrieved from the Buffer multiple times redundantly. This is bad for two reasons:

It takes longer
It wastes memory since more copies of the same data are created

I can't remember why I designed it this way (other than not being aware of this problem), but I probably thought unused Accessors will never get imported this way. A fair point, but a common/sane glTF files don't contain unused data in first place 😃

Analysis of the problem

I didn't want to repeat the mistake of jumping into coding before I understand the problem, so I started researching the glTF structure as well as Unity's Mesh API (the legacy and the new 2019.3 version), because it all comes down to bringing those two together efficiently.

I've had two bad-performing scenes on my hand which illustrate two different problems.

Case 1: Smart re-usage

The first case had Primitives of one Mesh reference the same vertex attributes, but with different materials (and indices). This is actually very common with assets. Triangles that share the Node's transform and even vertices, but have different materials. It's glTF's counterpart to Unity subMeshes and that's what they should end up being imported to.

Case 2: Let's go bonkers

The second case took the concept of re-using Accessors even further. The scene had ~400 Primitives, but only two very large vertex attribute arrays (with Accessors for positions, normals, UVs for each). One buffer counted ~234.000 vertices and the other one ~138.000. This is pretty much the worst-case scenario for glTFast and thus an excellent use-case.

When turned into Unity meshes, Primitives would occupy ~18 MB or ~13 MB each, but most of them consisted of only of a dozen or up to some hundred triangles. The hundreds of vertex data copies took their toll. Overall this 30 MB glTF binary file ate up over 7 GB of memory after import 😱

The official glTF Validator thinks this glTF is perfectly valid. Don McCurdy's glTF Viewer reports 97 million vertices!

That's roughly the number of Primitives multiplied by the vertex count. So either this viewer does duplicate all vertex attributes as well, or the number in the report is wrong. It loads it very fast though ( ~3 seconds on my laptop ).

My gut feeling tells me, this scene is not using glTFs ability to re-use accessors to the its benefit.

Finding a solution

The rationale of the analysis is that the ideal glTF importer retrieves the data once per Accessor instead of once per Primitive.

Proper sub meshes

I refactored the loading routine to the following, simplified steps

Iterate over all Primitives
- Assign the Accessors that they reference their usage type (used for positions or normals or …)
Iterate over all Accessors
- Now that we know how an Accessor is used, we can properly retrieve and convert its data
- This step is still fast, because threaded via C# Job System
Wait for all Accessor data to be ready
Iterate over all Meshes
- Iterate over a Mesh's Primitives and cluster them by identical vertex attribute usage
- Create one Unity mesh per Primitive cluster
  - In case of multiple Primitives per cluster, create subMeshes for them
Iterate over Scene's Nodes
- Create a GameObject/MeshFilter that references the corresponding Mesh.

As a consequence, Accessor data is retrieved only once in any case 🎉🎉🎉

This solution has a bit more pre-computation overhead compared to the old, per Primitive approach, benchmarks showed, that from a performance perspective this is negligible. Scenes that don't re-use vertex attributes might load a couple of milliseconds slower.

Scenes with similar Primitives per Mesh are faster and more memory efficient now. For example, this cube consists of one Mesh with six Primitives (one per side). Instead of importing the vertex data six times and creating six Unity meshes/GameObjects it now does everything only once.

These changes will soon be released in glTFast version 0.10.0.

Arbitrary accessors dilemma

Unfortunately, the problem case #2 cannot be resolved 😦

In Unity you render geometry via a Unity Mesh, which is a data structure that tightly bundles vertex attribute buffers and (sub) mesh indices. You cannot keep them separate, even with the improved new Mesh API from 2019.3 (Aras verified that). If Accessors are re-used across multiple Meshes their imported Unity data still gets duplicated.

If your rendering API or game engine gives you access to low level functionality like binding vertex attribute buffers manually, then there's no problem in loading glTF data "as-is".

Solution idea 1

Detect non-overlapping fragments of large vertex buffers and split them up at run-time.

Even if this can be done fairly efficient (which I doubt), it will worsen the computation time of the import. It also means more code to write, stabilize and maintain.

Solution idea 2

Create one Mesh per vertex attributes combination and use Graphics.DrawMesh instead of MeshRenderers for rendering.

In problem case #2 that would mean two giant Unity meshes with hundreds of sub-meshes and a scene hierarchy with custom/non-standard MonoBehaviours for rendering.

I presume this approach is faster than solution 1 at run-time and maybe even more efficient in terms of memory. Still, the non-standard components is what I dislike.

Finally: the "no" solution

This is a classic example of "garbage in, garbage out".

The vast majority of glTF files I came across (including the official sample models), have a sane structure that doesn't yield inefficiencies. As a matter of fact, when trying to produce worst-case example glTFs with the Blender glTF addon, I couldn't achieve it, since it created efficient files with separate vertex attribute buffers all the time.

So I came to the conclusion it's a content problem created by the glTF generator.

Workaround

Fortunately there are tools that optimize glTFs. I used the excellent gltfpack to optimize the named scene and look at the results.

glTFast 0.9	time	memory
original	16.3 sec	8.000 GB
gltfpack	2.49 sec	0.034 GB

glTFast 0.10	time	memory
original	9.20 sec	6.010 GB
gltfpack	2.53 sec	0.034 GB

Optimized glTFs make all the difference! Apart from being ~40 milliseconds (1.6%) slower on optimized content (I assume that's the additional overhead), overall glTFast 0.10 improved quite a bit.

Where to go with these problems

The next steps I eventually intend to make

Figure out if other glTF importers have problems with this as well
Certify that this is not an implementation problem of glTFast
Let affected glTF generators' developers know that their output is troublesome

If this proves to be a generic problem there's gotta be a discussion about possible solutions

Make the next iteration of the glTF specification more rigorous
Make the glTF validators throw warnings (or even errors) that inform exporter developers about potential performance problems

Next up

This optimization step, although good and necessary for some scenes, made simple scenes a tiny bit slower. I'm tempted to optimize the nested structure of Coroutines to make up for this right now, but I know that there are lower hanging fruits in other topics first. This has to wait, maybe until after a switch to .NET 4 only and async/await.

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: 2. Avoid Tangents and Normals Calculation

Overview of this mini-series

glTF in Unity optimization - 0. Introduction

2020-02-17T00:00:00Z

I see some optimization potentials for my glTF loading library glTFast for Unity. I decided to document the process and results in a mini-series:

History of glTFast

glTF, in a nutshell, is an open 3D asset format that is striving to become a standard, the "Jpeg of 3D".

Early 2018 I was investigating in possibilities to load glTF files at run-time in a Unity project. Back then the only option was UnityGLTF, the official importer/exporter from the Khronos Group.

While on the surface it seems to work nicely pretty much out of the box I noticed that it's not very fast 😦
First of all, the WebGL build's JS/WebAssembly file was big, so starting the project took long. Loading glTF files also wasn't blazing fast.

During investigation the thing that stood out to me was UnityGLTF is using Newtonsoft's Json.NET library for parsing. From previous projects I knew that feature-wise it's a great, flexible library, but it takes its toll by pulling in a lot of dependencies which eventually bloat the build by some mega bytes.

I also knew Unity has its own JSON parser, which is less powerful but:

It's fast
It does not use a lot of memory
It's already built-in (or nowadays: a package)

So in ~May 2018 I started tinkering with my own solution based on that parser. Initial results were promising. This is a screenshot (made in May 2018) of WebGL builds of UnityGLTF and glTFast loading the same asset.

Admitted, this is an old screenshot and not a meaningful, significantly measured test result. The test wasn't repeated/averaged and I cannot reconstruct the exact setting. But still, loading 56% quicker seems pretty good to me.

Over time I kept on improving:

Speed up due to threading via C# Job system
Using custom shaders that can consume glTF textures without re-processing
Improved standard feature compliance with official sample files
Added support for official extensions like Draco or mesh quantization

Future

Topics I want to tackle in the future that may be worth a post

Treating Buffers and Accessors correctly
Parallel jobs
Unity Math and Burst
New Mesh API
Speed up tangent calculations
DOTS
Coroutines vs. async
Custom DownloadHandler
Streaming support
Draw call reduction due to Material batching (Property blocks)
Comparisons with recent alternatives and benchmarks.

Stay tuned!

Follow me on Mastodon or Bluesky or subscribe the feed to not miss updates.

If you liked this read, feel free to

Next: 1 Buffers, Accessors and Primitives

Benchmarking Basis Universal Transcoding in Unity

2019-11-10T00:00:00Z

Basis Universal super-compressed textures are awesome (read the details). Let me show you some benchmark results of it being integrated into Unity.

In these test an 1024 by 1024 pixel images is loaded. One is color only, one has an alpha channel. At first it's loaded from Jpeg/PNG format and then via a BasisU super-compressed KTX 2.0 file (both pre-loaded into memory).

Feel free to run the benchmark project yourself. It uses the KtxUnity package for loading the KTX files. Let me know if your results differ.

macOS standalone build

Let's load the image 50 times.

Tests are run on a MacBook Pro (2017) with an Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz ( 4 cores, 8 threads ).

Jpeg / PNG

For the color only trout Jpeg image

Decoded to RGB 8, which occupies 4 MB per image
Blocks the main thread for ~564 ms

For the PNG image with alpha channel it's even worse

Decoded to ARGB 8, occupying 5.3 MB each
1064 ms (yikes)

So there are two takeaways.

First, we can see that all the work is happening on the main thread (done via Texture2D.LoadImage), which freezes the demo for up to a second. Maybe Unity makes an async version of LoadImage some day that'll run on a worker thread, but right now it's a clog.

Secondly, the amount of RAM needed for each texture is huge!

Super-compressed KTX files

Let's load the same content from KTX 2.0 files with Basis Universal compression.

This is the first frame of action. You can immediately see that the transcoding part is off-loaded to the worker threads (via the C# job system). Only the actual GPU upload has to be done on the main thread.

In one of the following frames there's a lot of transcoding going on on the worker threads and while the frame time is too high, it's far from being as bad as with Jpeg/PNG.

Results

Color only trout Jpeg image

Transcoded to BC1 (DXT1) occupying 0.5 MB per image
Takes ~95 ms spread across 4-5 frames

Star image KTX with alpha channel

Transcoded to BC7 RGBA occupying 1 MB per image
Takes ~85 ms spread across 4-5 frames

It's mind-blowing. The transcoded images take up up to 8 times less memory and load an order of magnitudes faster.

iOS

Let's look at the memory consumption on an iPhone 8. The benchmark loads one image per frame until the app runs out of memory and is stopped.

https://www.youtube.com/watch?v=sYzkwQvaFSc

I think the video speaks for itself. The sheer amount of additional image data you can consume with this technology is amazing.

Recap

If you have to load image data on WebGL or more than one platform and you can encode it at build-time or later on server-side, super-compression like Basis Universal is a fantastic solution!

At the moment the tools are rather raw. For example you have to compile the CLI tools that encode those files yourself. I hope and assume that this will get easier/more accessible over time and more technology providers will offer those benefits to developers and users.

If you liked this read, feel free to

Cross-platform C/C++ Plugins in Unity

2019-06-24T00:00:00Z

Have you ever wanted to use a native library in Unity or write parts of your game in super portable, efficient C/C++?

Recently I stumbled over an interesting C++ library and integrated it in a Unity project. Let me show you what I did.

I'll focus more on the workflow and tools to create comprehensive multi-platform support than on the actual interface source code. There's already quite good documentation about that^[1].

The target

The library in focus is Basis Universal (see this other post for more info). It consists of:

an encoder that converts images into the basis file format
a transcoder that converts the basis file into GPU-friendly format at runtime

The transcoder is what I was interested in. It's also a great example, since it has a straightforward API, consist of a single file (header files not counting) and has no 3rd party dependencies.

Let's get started

In this post I'll only give code excerpts and examples. To see the complete code look at BasisUniversalUnityBuild (for building the native libraries) and BasisUniversalUnity (the matching C# interface).

Inspect the library

First, let's look how it's supposed to be used^[2]. In a nutshell like this (warning: pseudo code):


#include "transcoder/basisu_transcoder.h"
// one-time initialization at startup
basist::basisu_transcoder_init();
basist::etc1_global_selector_codebook sel_codebook(basist::g_global_selector_cb_size, basist::g_global_selector_cb);

// load the basis file content into a buffer
uint8_t * data = ...
size_t length = ...

// creating a basis_file instance
basis_file* new_basis = new basis_file(data,length);

// and now you can do things like retrieving the image size
uint32_t image_index = 0; // first image
uint32_t level_index = 0; // highest mipmap level
uint32_t width = new_basis->getImageWidth(image_index,level_index);

// and finally transcoding
new_basis->transcodeImage(destination_buffer, dst_size, image_index ,level_index , ...);

Wrapping

You can call native C functions from C# by using the DllImport method attribute^[1:1]:

[DllImport ("PluginName")]
private static extern void some_function ();

No, I did not forget the ++. You can only call pure C functions and use pure C data types (no C++ classes) from C#. Luckily C and C++ are compatible, so we need to wrap the logic within C functions like this:

// the extern "C" block tells the compiler to use C name mangling instead of C++
extern "C" {
    void aa_basis_init()
    {
        basisu_transcoder_init();
    }
}

Build via CMake

After writing all C wrapper code we need to build the libraries. In the end, I want this to work on multiple platforms:

WebGL
macOS 64-bit
Windows
- 32-bit
- 64-bit
iOS
- armv7
- armv7s
- arm64
Android
- armeabi-v7a
- arm64-v8a
Linux 64-bit

The library needs to be built for each one of these, so it makes sense to use a build system. Since I'm familiar with it, I chose CMake.

CMake rules

All you have to do is describe to CMake what you want to build/compile and it takes care of creating projects and invoking the right compiler/linker commands. This description has a certain syntax and needs to be saved to a CMakeLists.txt file. This is how a minimal version looks like:

# define a CMake version
cmake_minimum_required(VERSION 3.0)

# give the project a name
project(basisu_transcoder)

# create a list of files to compile
set(BASISU_SRC_LIST
    # this is the transcoder itself
    basis_universal/transcoder/basisu_transcoder.cpp
    # this is my wrapper code
    basisu_wrapper/basisu_wrapper.cpp
    )

# add a library target
add_library(basisu SHARED ${BASISU_SRC_LIST})

# In order to find header files, we have to tell the compiler in which 
# folders he's got to search: include directories
target_include_directories(
    basisu
    PUBLIC
    basis_universal/transcoder
)

In theory you can now invoke CMake and build the library. Unfortunately in order to get all platform libraries in a way they work in Unity I had fix some more things:

macOS Unity requires a .bundle file library without the lib prefix.
iOS
- lib has to be static
- A special toolchain is needed
Android
- special toolchain
Windows
- Had to setup everything on another machine
- Enable __declspec via compile definition (pre-processor) when using MSVC compiler
Installing
- Install lib to correct destination folders in Unity project
- Copy the sources for WebGL builds

I won't go into all details, but I want to point out the Installing part.

Install to correct destination

Within the consuming Unity project's Assets folder, we need our libraries at certain destinations. Here's an (incomplete) overview how it should look like:

Plugins
├── Android
│   └── libs
│       └── arm64-v8a
│           └── libbasisu.so
├── WebGL
│   ├── basisu_wrapper.cpp
│   ├── basisu_transcoder.cpp
│   └── ...some more header files
├── iOS
│   └── libbasisu.a
├── x86
│   └── basisu.dll
└── x86_64
    ├── basisu.dll
    ├── basisu.bundle
    └── basisu.so

In order not to have to manually copy every library file into the Unity project, I tweaked the CMake install target.

...
# first, add a path parameter/option where the user can provide the path
set( BASIS_UNIVERSAL_UNITY_PATH "" CACHE PATH "Path locating the BasisUniversalUnity package source. When installing, native libraries will get injected there" )

# Set the output sub-path within the package project
# In this case for standalone x64 targets
set( DEST_PLUGIN_PATH "${BASIS_UNIVERSAL_UNITY_PATH}/Runtime/Plugins/x86_64")

# We can check if it's correct by logging it
message(STATUS "Will install native libs to ${DEST_PLUGIN_PATH}")

# Check if the path actually exists
if( NOT EXISTS ${DEST_PLUGIN_PATH})
    message(SEND_ERROR "Invalid path!")
endif()

# Finally tell CMake to install to our path
install(TARGETS basisu DESTINATION ${DEST_PLUGIN_PATH})

This was a huge time saver!

WebGL

WebGL is a special case when it comes to native libraries. You cannot build/provide a pre-compiled library for WebGL, but you place all C++ source files within the Assets/Plugins/WebGL folder. Unity will detect and include them when compiling the rest of the game code (which is also C++ generated from C# via IL2CPP).

All we need to do is copy the right source files over to this destination, which I also did via CMake:

# Set the right sub-folder
set( UNITY_PLUGIN_DIR_WEBGL "${BASIS_UNIVERSAL_UNITY_PATH}/Runtime/Plugins/WebGL")

# Let the install target copy the listed source files
install(
FILES
    # The two source files
    basisu_wrapper/basisu_wrapper.cpp
    basis_universal/transcoder/basisu_transcoder.cpp

    # Plus all the header files they reference
    basis_universal/transcoder/basisu_transcoder.h
    basis_universal/transcoder/basisu_transcoder_internal.h
    basis_universal/transcoder/basisu_global_selector_cb.h
    basis_universal/transcoder/basisu_transcoder_tables_bc7_m6.inc
    basis_universal/transcoder/basisu_global_selector_palette.h
    basis_universal/transcoder/basisu.h
    basis_universal/transcoder/basisu_transcoder_tables_dxt1_6.inc
    basis_universal/transcoder/basisu_file_headers.h
    basis_universal/transcoder/basisu_transcoder_tables_dxt1_5.inc
DESTINATION
    ${UNITY_PLUGIN_DIR_WEBGL}
)

The only down-side is that this is always done as part of another platform's build (in this case redundantly every other platform), but that's not really a problem.

Configure build and generate

Now we have a final CMakeLists.txt and we are ready to generate all Xcode / Visual Studio projects or Makefiles and fire up the compilers.

The extensive documentation how this is done can be found at the BasisUniversalUnityBuild project site.

Future work

Linux

I haven't built the Linux libraries yet. Definitely will, especially since the Unity Linux Editor is coming out of preview later this year.

Continuous Integration

Although I made some small improvements, building a library for so many platforms requires a lot of small steps. It would be nice to further automate the building process, so updating the library is fast and less error prone.

This means either having multiple build workers with different platforms or doing more cross-compiling. I already cross-compiled for iOS and Android from my machine (macOS).Adding Windows and Linux as target platform would be nice for development. On the other hand I'd love to be able to cross-compile all variants from one server platform (preferably Linux). Maybe I'll look into that some day.

Unity Package

To be able to re-use the library (all native libs plus the C# interface code) amongst multiple Unity projects easily, the proper way is to create a custom Unity package. I did this in this particular case ( see BasisUniversalUnity on GitHub ) and plan to write about it in the future.

If you liked this read, feel free to

Basis Universal Texture Format Introduction

2019-06-19T00:00:00Z

TL;DR: Basis Universal is the image format for cross platform/API usage in graphics/GPU contexts.

It is also super useful for creating a platform/API/framework agnostic 3D content format. For textures, it is the missing piece in that puzzle.

Intro

It's pretty straightforward to put an image on a website. Save it as a Jpeg or PNG, add an <img> tag to your HTML and it'll work on all browsers across all platforms.

Some people (me included) also want to do this in graphics applications (e.g. 2D/3D game) that run on GPUs. To do this, you need the texture in a special format, the graphics API and GPU can work with.

The objectives are:

Images/Textures to load as fast as possible
- Small file/download/storage size
- Small runtime load overhead
Straightforward content creation pipeline

GPU texture formats

GPUs can handle only certain types of texture data.
Apart from uncompressed formats (bitmaps; huge in size) there are a couple of compressed formats, also called GPU-friendly formats. They have a lossy compression and thus are small in size (~8 times smaller than bitmaps), yet GPU texture units are able to access them very efficiently (random access). Examples are DXT1/BC1^[1], PVRTC^[2], ETC2^[3] or ASTC^[4].

Unfortunately there isn't a single compressed GPU texture format that is supported throughout all GPUs and graphic APIs.

If you want to provide a broad range of target devices/platforms with their respective ideal format choice, you have to encode every texture into a bunch of different formats.

To make matters worse, I'm not aware of a single encoding tool that supports all or most formats, so you'll also need to install a handful of encoders.

Note: If you're using a game engine or graphics framework (like Unity3D), chances are it assists you with automatic conversions by abstracting them away...maybe not always in the most efficient way.

Decoding / Encoding at runtime

If your GPU does not support the format you have directly, you have the option to convert it at runtime. For example decode a PNG file into an uncompressed texture. The smaller PNG gives you a decent download size, but it has two downsides:

Decoding is slow
The decompressed bitmap will use a lot of video RAM at the end

The second point could be fixed if you encode the bitmap into a supported GPU-friendly format after decoding, but that would make the loading process even slower.

Enter Basis Universal and Transcoding

Basis Universal is a supercompressed intermediate texture format. BasisU files can be transcoded into GPU friendly formats at runtime.

The transcoding is quite fast and it does not create an interim bitmap.

For technical details, jump right to the repository/code^[5] or read this related paper^[6].

So BasisU fulfills all requirements

Small file sizes for storage/download (compression)
Fast transcoding
Low video RAM usage (due to GPU-friendly formats)

Seriously, these benchmarks speak for themselves.

And on top of that:

It's open source! (Apache License 2.0)
The creators of it (Binomial^[7]) are working with the Khronos group on the specification to make it an industry standard.
It is supported by Google^[8]

Basis Universal for Unity

When BasisU was released, I got excited and created a small wrapper/library so you can load it in the Unity3D game engine. You can find it on my GitHub:

https://github.com/atteneder/BasisUniversalUnity

Let me know what you think. I'll try to prevent it from going to the graveyard of side projects before it is usable.

I think that if BasisU gains the traction it has the potential to, eventually Unity will support it by default and maybe even use it for texture/asset transmission internally.

Generic 3D assets: glTF

As mentioned above, efficient texture transmission is key for having a universal 3D asset format.

We're talking about a "last mile" format, with the intention to deliver final content to the end user. Not a format to deliver assets between digital content creation applications.

The Khronos group made an effort to give us a format like this called glTF^[9].

It is getting some attention lately. For example you can load it in Google's model-viewer web component and view it directly in 3D or augmented reality. Also facebook does support posting 3D content via glTF-binary^[10].

At the moment textures in glTF are encoded in either PNG or Jpeg, which suck, as I explained before.

But Khronos is working on including BasisU in glTF^[11], so the future for this format is bright, once the industry starts to adopt to it.

Oh yeah, I've also created this Unity glTF loading library called glTFast (because it's really fast). I'll try to get BasisU support in there as well. Also on my GitHub:

https://github.com/atteneder/glTFast

Final notes

To stay up to date on this topic I recommend to follow the following sources:

The glTF draft pull request regarding compressed texture formats.

Stephanie Hurlburt and Richard Geldreich, the people behind Binomial.

Richard also has a blog, where (amongst other things) he gives lots of insight into texture encoding details.

Star/follow the repositories of Basis Universal and glTF on GitHub.

If you liked this read, feel free to