August 02, 2019
As part of its unwavering commitment to open source and open standards, Collabora is proud to be part of bringing the recently-released OpenXR 1.0 to life. We are pioneering the Monado open source runtime for OpenXR to ensure the future of XR is truly open and accessible to all hardware vendors. As the OpenXR specification editor, I am grateful for the diligent efforts of the working group, as well as the community feedback that shaped this release.
There have been a lot of changes since the last post about OpenXR and Monado. On the working group, we've brought the concerns of the open source and Linux communities to the working group. We have worked to improve the loader and provided API layers in both cross-platform and Linux-specific ways, together with the Monado community. As specification editor, I developed or enhanced a variety of specification-related tooling to ensure a continuous standard for consistency and high-quality in the specification text and registry.
For example, xml_consistency uses specification-specific "business logic" to check the internal consistency of the XML registry. Among other things, it compares the return codes listed for a function with those inferred from parameter types, and raises an error if an expected code is missing or an existing code seems unnecessary. The comprehensive check_spec_links tool processes the AsciiDoctor source of the specification, ensuring that the spec-specific markup macros are used correctly, that all members and parameters are documented, that all entities referred to actually exist and are spelled correctly, and more.
Over the past several months, I have worked with participants in other Khronos working groups that share a related spec toolchain, such as Vulkan, to harmonize our forks of the tooling scripts and bring these additional quality checks to their specifications as well. The first such changes were published in Vulkan 1.1.96, and have been rolling in to later releases as the tools were improved across the projects. In addition, I introduced the use of Jinja2 templating in the code generation process for Khronos specs and code, in place of pure Python string interpolation. Uptake in the OpenXR group has been promising, and it is now being used in the new-for-1.0
openxr_reflection.h header as well as in-progress conformance tests.
Once the OpenXR 0.90 specification was released, we were able to begin releasing some of our previously-internal utilities and tools. These are currently using 0.90, but are in the process of being updated. [Edit 14-August-2019: xrtraits and xr-system-tests have been updated to 1.0.] These standalone projects, maintained under the broader Monado umbrella but not tied to the Monado runtime specifically, include:
With the release of OpenXR 1.0, we were able to publish the few changes required to adapt Monado to the 1.0 API, which will be our target ABI from here on.
To answer the simple question "Are we there yet?", no. There are many pieces missing to make Monado a complete OpenXR runtime. We are working hard to soon get into a state where people can try it out and have a good experience. Already, people who are very adventurous can join in and hack if they want to; please feel free to join us on IRC or Discord.
As mentioned above, as a product of running test suites on Monado, we have made lots of improvements to the OpenXR state-tracker to make it more complete and bring it closer to conformance. A fairly large chunk of work was needed to bring in the action system, and code to handle rebindings of actions to various hardware still needs to be written.
On the driver side we have merged four new drivers, plus the V4L frameserver driver. A device detection subsystem has been merged as well, for use by drivers. The new drivers are:
One of the biggest chunk of work ahead of us is tracking. Broadly speaking, there are 2 main types of tracking: outside-in where sensors in the world track an object, and inside-out where sensors in the object track landmarks in the world (this is a gross simplification). In many cases, these sensors are cameras. These can then be based on either: stereo color cameras as used by PSVR; IR camera(s) in the case of Constellation; or in the case of Lighthouse IR beam and sensors. The term "inside-out tracking" is usually used to refer to markerless SLAM ("Simultaneous localization and mapping") tracking such as used on the Windows MR headsets. Note that in the case of Lighthouse, despite the light sensors being on the tracked object, it could be logically considered an outside-in tracking system because the scanning base station beam is somewhat like the scan of a camera readout: the math is more similar to camera-based outside-in systems than to SLAM.
To help with our goal of being maker/hacking friendly, we chose to first focus on a outside-in color based tracking system, more specifically the PSVR. As there are a lot of PSVR headsets out there, hacking on something vision-based requires just a webcam and some printed markers.
Right now, we have merged infrastructure for passing frames of data around into master. We are in the middle of expanding that to be able to hold various metadata needed to make sense of the images. On a branch we have experimental code that is able to track a PS Move controller in 3D space, can identify the PSVR LEDs. We also have a branch with some in-progress improved colour filters to improve tracking speed.
As mentioned above we have IRC and Discord channels. We will also be at SIGGRAPH this week, so please feel free to send a message to jakob at collabora.com if you want to meet up or come to the Khronos BoF sessions on Wendsday. We will also be attending various different open source conferences to talk about our work, including GUADEC, Akademy, and XDC. Finally, we will be hosting a new conference specifically for Open-Source XR in Amsterdam on the 26th of October.
While the general shape of OpenXR remains the same between the provisional 0.90 release and the full 1.0, changes have been made for improved usability and consistency, as well as to resolve issues discovered during use of the provisional release. Most changes are fairly small (specifying more corner case or validity behavior, renaming, resolving contradictions, etc.) The actions system underwent larger changes, in part to make it easier to use correctly and harder to use incorrectly.
As an obligatory note, this post isn't an official product of the working group. Anything that can't be derived from a comparison of the released specs should be considered my personal opinion. While I have tried to be thorough, it is not necessarily comprehensive, and it is definitely not normative: just informative for those who have been with OpenXR longer.
Jump to a section: New helper header: openxr_reflection.h | next Chains | Return codes and valid usage | Undefined behavior | Two-call idiom | Versioning | Space Relations | Spaces | Session | Rendering and Swapchains | Input and Actions | Extensions | Misc | Public spec issues closed
This header provides expansion macros ("X Macros") for generating data about OpenXR functions, types, and enums. The way these work is that you define a macro taking a particular number of arguments (depending on the usage), and pass the name of that macro to one of the macros defined in
openxr_reflection.h. It calls your macro repeatedly with the data you want.
One of its goals is to make
xrStructureTypeToString obsolete, though we weren't able to get to full agreement and code revision on that in time. The header does include sample code showing how to make an "enum to string" function.
While use of
openxr_reflection.h is not required for OpenXR usage, it provides useful utilities, especially to those working at a level above end-user application software, such as game engines, layers, and runtimes. I am certainly excited to have it available!
The file isn't currently all-encompassing, but I anticipate its gaps will be filled in over time.
For each enum type, there is an X Macro
#define XR_LIST_ENUM_enumname(_) that calls its argument with the name and numeric value of every enumerant of that type.
For example, the first few lines of
#define XR_LIST_ENUM_XrResult(_) \ _(XR_SUCCESS, 0) \ _(XR_TIMEOUT_EXPIRED, 1) \ _(XR_SESSION_LOSS_PENDING, 3) \ _(XR_EVENT_UNAVAILABLE, 4) \
Note that all extensions-defined enumerants, as well as the dummy
MAX_ENUM value (which forces enum type size), are provided in addition to the core values.
For each bitmask
Flags type, there is an X Macro
#define XR_LIST_BITS_flagsname(_) that calls its argument with the name and numeric value of every bit for that type (from the corresponding
XR_LIST_BITS_XrSpaceVelocityFlags is defined as:
#define XR_LIST_BITS_XrSpaceVelocityFlags(_) \ _(XR_SPACE_VELOCITY_LINEAR_VALID_BIT, 0x00000001) \ _(XR_SPACE_VELOCITY_ANGULAR_VALID_BIT, 0x00000002)
Note that all extensions-defined bits are provided in addition to the core values.
For each structure type, there is an X Macro
#define XR_LIST_STRUCT_typename(_) that calls its argument with the name of each member of that type in order.
#define XR_LIST_STRUCT_XrInstanceProperties(_) \ _(type) \ _(next) \ _(runtimeVersion) \ _(runtimeName)
nextchain isn't solely for extensions.
nextchain, and that apps should avoid putting duplicates in. (but not "must not")
XrCreateInstanceInfo-chained structs no longer break the convention by requiring blacklisting by runtimes.
XR_ERROR_..._UNSUPPORTEDreturn codes: For when something is valid but not acceptable in a given usage. For example, XR_ERROR_PATH_INVALID is if you pass in uninitialized memory containing garbage as XrPath. XR_ERROR_PATH_UNSUPPORTED is if you pass in
/user/hand/leftas an interaction profile path.
nextchain text updated to reflect the prose in Fundamentals.
xrStructureTypeToStringno longer use a loader-supplied implementation for null instances.
openxr_reflection.hheader to generate your own version.
There was some existing text in the spec that was very Vulkan-like about the risks of "invalid usage". However, Vulkan is much more "dangerous" than OpenXR, by design. Vulkan tries to avoid overhead at nearly any cost, because it's a high-frequency API. Therefore, incorrect usage in Vulkan without the validation layer may cause terrible things ("Undefined Behavior" in the nasal demons sense) to happen.
OpenXR is comparatively very low frequency, so most error cases are required for the runtime to catch. The various references to return codes have been clarified, most by adding/changing to "the runtime must return". The actual places where Undefined Behavior may occur are very few in OpenXR. Notably,
XrPath usage is not intended to cause undefined behavior since
XrPath is just an easier way to refer to a path name string. They may vary between instances, so mixing them or using random garbage might cause unexpected behavior depending on what the value maps to, but the behavior is always defined: an error code or a function working "properly" on possibly-wrong data. No nasal demons.
The "count output" parameter is now always required - it cannot be NULL. Previously, this was self-contradictory. Additionally, for the sake of one current usage, a "struct"-style two-call idiom is described, along with behavior when two things need to be enumerated in the same call. The "visibility mask" extension is the troublemaker here: it enumerates vertices and indices at once - two lists of different size.
Most two-call idiom calls have had their spec text updated to indicate if or when their results may change between calls.
XrVersion, that is used whenever such a packed version is expected.
XR_MAKE_VERSIONnow masks its arguments, so you can specify the max value for a component with e.g.
XR_HEADER_VERSIONwas removed because it was unneeded.
XrSpaceRelation structure was replaced with two structures:
XrSpaceLocation (which has pose, usually all you need) and
XrSpaceVelocity (which adds linear and angular velocity). For those cases when you really need velocity (such as when initializing the velocity of a thrown object) add an instance of
XrSpaceVelocity to the
next chain of
XrSpaceLocation. The time associated with a relation is no longer in either of those structures. The ability to get acceleration was removed, since no compelling application use case was identified that warranted the cost in complexity. (If this changes, there are always extensions.).
Note: You should not attempt to use
XrSpaceVelocity to do any kind of tracking prediction: leave that to the runtime by just asking for a pose in the future.
Runtimes now must support local and view space. Contradictory information about space change events was fixed.
Action spaces are now unlocatable any time their action's action set was not in the most recent sync call (making them inactive), rather than unlocatable only until the first sync call.
Because of the hierarchy change, action spaces are now children of sessions, rather than of actions.
The session lifecycle was further refined and clarified. The "RUNNING" state is now called "SYNCHRONIZED", for clarity, and what it means for a session to be "running" was more clearly defined: it means there has been a successful
xrBeginSession but no call to
xrEndSession. There is now a function to request a session exit:
xrRequestExitSession, which fills in a missing transition on the state diagram.
A lot of spec text in this section has been replaced/revised, so see the text for the nitty-gritty details.
Clarification was added that if there's a "user engagement sensor" (proximity detector or other presence detect), it should be used to drive the STOPPING state.
A number of events were missing a session handle to provide them with necessary context: this has been fixed.
If a swapchain create flag (static image or protected content) in
XrSwapchainCreateInfo is specified but not implemented by the runtime, the runtime now must indicate this with
XR_ERROR_FEATURE_UNSUPPORTED. The other fields of
XrSwapchainCreateInfo now have more explicit descriptions of their valid values.
It's now explicit that trying to acquire a "static image" swapchain more than once results in an error.
Some structure parameters related to swapchains are "empty" (only type and next), included only to provide a
next chain for future extension. These are now consistently noted as optional.
XrViewLocateInfo now additionally takes (as input to xrLocateViews) a
Some editorial, non-normative content related to IPD was removed. Other non-normative content that was still important to understanding the spec was moved to informative "Note" boxes.
Requirements on the runtime for frame timing and waiting have been made more consistent.
XrFrameState now has a
shouldRender field indicating if the application should do its heavy GPU work of rendering. This is to ensure that the frame loop is kept synchronized (begin/end frame) even if no layers should be submitted. This replaces the removed
XR_SESSION_VISIBILITY_UNAVAILABLE "success code".
The composition layer flag bit
XR_COMPOSITION_LAYER_UNPREMULTIPLIED_ALPHA_BIT was added to indicate if you are submitting a layer whose color has not been pre-multiplied by the alpha component. Related cleanups to blending specification were also made.
View configuration type is now consistently used in conjunction with a system ID: the system ID doesn't necessarily imply a single view configuration type. (Some systems might support multiple view config types: an HMD can support mono viewing, etc.)
This where the most changes took place.
XrActionSetis now created directly under
xrSuggestInteractionProfileBindings(moved earlier in the process and renamed from
XrActionsSyncInfoinstead of an array of
XrActionStateGetInfoinstead of lots of parameters, for extensibility.
Relatedly, it is clarified that destroying an action set handle just invalidates your access to it: the action set continues to exist (with no way for you to reference it) until it is no longer used by any handles.
The same applies when you destroy an action handle.
This solves the question of what happens when you destroy an action set during a session, etc.
Path related changes:
Interaction profiles were updated accordingly.
Other minor action changes:
localizedNamefor action/action set must also be unique.
XrActionTypeenum values were renamed to be more consistent.
xrGetBoundSourcesForActionhas been renamed
xrEnumerateBoundSourcesForActionfor consistency, and now takes a session as well as an info struct with the action handle in it.
xrCreateActionSpacehas moved into the info struct.
xrGetInputSourceLocalizedNamehave moved into an info struct.
/<component>part is optional.
Interaction with the session lifecycle and states is specified.
Removed as obsolete.
Pulled from release due to some issues related to the session lifecycle. Collabora is working on a temporary vendor extension (edit: XR_MND_headless now published in 1.0.1), and hopes to resolve the underlying issues soon to be able to release a KHR headless extension.
XrVector3f offset was removed, along with a large amount of explanatory and editorial text about it. Its behavior can be provided by supplying a space created with a transform.
Clarified that only the interior of the sphere is visible, and that it's more similar to
XrVector3f offset was removed. Its behavior can be provided by supplying a space created with a transform.
float radius was added.
XR_ERROR_TIME_INVALID is now returned if a conversion cannot be performed.
Clarified that an implementation is only required to support the structs associated with the platform it is running on. (Windows runtimes don't have to implement xlib structures, etc.)
Clarifies creation: app must create context, runtime creates textures for swapchain.
Clarifies multi-GPU error case on Windows if there is a mismatch:
uint32_t screen_number was renamed
screenNumber for consistency.
A copy-paste typo in the Wayland section was fixed.
Clarifies creation: app must create context.
XrVisibilityMaskKHR struct was revised to use the new "struct-style" two-call idiom, with separate capacityInput and countOutput members for each array. Its associated function,
xrGetVisibilityMaskKHR, was updated to link to that part of the fundamentals.
XrEventDataVisibilityMaskChangedKHR event now has a session field.
xrGetVulkanDeviceExtensionsKHR had their two-call idiom parameters renamed for consistency:
XR_ERROR_TIME_INVALID is now returned if a conversion cannot be performed.
Reference page markup was added.
A new message type was added, for future use in conformance utilities:
XrResult value previously added by this extension,
XR_ERROR_DEBUG_UTILS_MESSENGER_INVALID_EXT, was removed and replaced with
XR_ERROR_HANDLE_INVALID to match the rest of the spec.
Thread safety/external synchronization concerns were added to the XML and generated portions of the spec for
The sample code snippets were updated and fixed to actually compile (as tested by CI with
make -C specification build-examples).
Reference page markup was added.
Reference page markup was added.
XR_VARJO_quad_views was added.
XR_MAY_ALIASno longer appears all over the spec in source code snippets, for clarity, but it is still put in place and used in the header.
*BaseHeadertypes and not add an impossible requirement.
Earlier this year, I joined Collabora as an intern to work on improving testing in libcamera and automating it through KernelCI. Having…
With the LLVM toolchain seeing increasing development and adoption alongside the older, more established GNU toolchain, projects needing…
This summer, Christoph Haag and I had the pleasure of taking part in Google Summer of Code as mentors for xrdesktop, the Open Source project…
Earlier this year, from January to April 2021, I worked on adding support for stateless decoders for GStreamer as part of a multimedia internship…
In our previous post, we presented a project backed by INVEST-AI which introduces a multi-stage neural network-based solution. Now let's…
Initiated as a joint effort by the Google Chrome OS team and Collabora, the recent KernelCI hackfest brought the addition of new tests including…