Faster GPU depth map inference, new controller mode, pairing fixes
Hey guys, another hot release coming regarding the AI 3d Conversion - This release streamlines the feature a bit and introduces GPU inference, the main goal being to improve the depth map latency to 20hz. I still recommend auto mode, as this will automatically cap the resolution and depth model according to what your currently selected resolution and passthrough can actually handle. But you can override it and choose an uncapped (standard) mode or a CPU model. I yeeted YOLO26 because it's performance wasn't great vs MiDaS.
To get the performance up to standard to actually handle GPU inference at around 50ms required switching to GLES rendering, and removing all projection based rendering, causing quite a few elements to be replaced, including removing 1 background animation and our controller models - they have been replaced with indicators. At the end of the day, stream performance matters more than being able to look at the controllers, but I will look into bringing it back some how.
This is specifically tailored to the Quest 3/3s - I don't know what would happen on another headset like the Quest 2, but I find that inference with passthrough off at HD to be extremely performant, so that is what I would try on a weaker headset.
The AUTO resolution for AI conversion is capped at 1440p, and the model strength and quality increases if you drop res or turn off passthrough, but I try to hit that 50ms target.
I've left (CPU) Depth Anything in, for experimentation and the future, but it's just not fast enough to be usable, stick to the MiDaS models.
GLES rendering conversion (Quest)
Nightfall now renders on Android via gl_compatibility (GLES) instead of the previous Vulkan/mesh-projection pipeline. All the pieces that used to be real 3D meshes in the scene - controller rays, the grab-bar, corner-handle resize indicators, cursors, the status bar, and the environment background - are now composition-layer equivalents instead, since projectionless mode never renders ordinary scene geometry.
- Fixed a composition-layer crash on grab/resize, an invisible laser/cursor, an invisible status bar, blurry video (a composite-viewport resize regression), and a stuck 0ms latency readout - all specific to the GLES path
AI 3D depth: GPU inference + reworked Auto mode
- Added a GPU-accelerated depth-inference path (GLES) and generalized the inference machinery to support it, alongside the existing CPU path
- Redesigned the AI 3D "Auto" setting as a benchmarked model+tier lookup table, replacing a simple threshold - Auto now picks a tuned model/tier/resolution-cap combination per resolution and passthrough state, based on real on-device benchmark data
New controller mode
- Added PAD-ABXY, a new controller layout, alongside the existing mode (renamed PAD-HAND)
- Added position markers for controllers under the new GLES/projectionless rendering path
Pairing & connection fixes
A batch of real-world connection issues reported by users, all now fixed:
- Pairing could time out before you finished entering the PIN on the host - Apollo/Sunshine intentionally hold the connection open until you do, so Nightfall now waits for that instead of giving up early
- Added optional custom-port support (
ip:port) to server entry, for hosts running on a non-default port - Fixed a rare race where streaming failed to start immediately after a fresh pairing
- Fixed an issue where, if two of your servers were reachable via the same IP (e.g. a VM port-forwarded through its host machine), Nightfall could connect to the wrong one - saved servers are now identified correctly regardless of shared IPs
- Fixed reconnecting to an already-paired server sometimes hanging indefinitely instead of connecting
- The saved-servers list now shows each server's IP so entries with similar names are easier to tell apart
Sorry, no Linux build this time
The GLES conversion and GPU depth-inference work touched a lot of shared rendering/inference code, and I haven't had time to fully test the Linux client against these changes yet. Rather than ship something I haven't verified, I'm holding back the Linux (AppImage) build for this release - it'll be back once it's had a proper pass.