Open Standards Powering the Future of Embedded Vision

May 02, 2023

Blog

Latest updates on industry open standards for portable accelerated vision processing and inferencing, and insights into the goals and design directions of the new Kamaros embedded camera API Standard.

Open standards streamline interoperability between critical technologies, reducing product development costs and time to market while speeding industry innovation. As a non-profit standards body with close to 200 member companies, The Khronos Group is committed to developing and evolving open, royalty-free interoperability standards for the good of the industry.

Our tagline, "Connect Software to Silicon," reflects our mission to provide key markets with timely and effective standard APIs that allow software applications, libraries, and engines, to harness the power of silicon acceleration for demanding use cases such as 3D graphics, parallel computation, augmented and virtual reality, vision processing and inferencing. We are a member-driven organization that believes that open standards that are not controlled by, or dependent on, any single company can often be the thread of continuity for industry progress as technologies, platforms, and market positions swirl and evolve.

Figure 1. Khronos Member Companies

The Family of Khronos Standards

The family of open standards being actively developed by Khronos include 3D acceleration APIs such as OpenGL and the new generation, Vulkan GPU API; initiatives around 3D format standards including glTF for assets and KTX for textures; the OpenXR API standard for portable augmented and virtual reality; and a family of APIs and languages for parallel computation, vision acceleration, and inferencing – including OpenCL, SYCL, SPIR-V and OpenVX— and now the new Kamaros embedded camera API currently in development.

Figure 2. Khronos Active Standardization Working Groups

Khronos also has a history of adapting mainstream acceleration APIs for safety-critical markets. This commitment continues as demand for advanced GPU-accelerated graphics and compute builds in a growing number of industries where safety is paramount. Vulkan SC is derived from Vulkan API, to streamline safety certification of systems that employ GPU acceleration, by enabling system implementers to provide certification evidence packages with reduced cost and effort. OpenVX has a safety-critical profile that enables the rapid deployment of trained neural network models.

Finally, the newly created Khronos SYCL Safety-Critical Working Group is investigating industry requirements for a general parallel programming API for accelerated compute using SYCL’s standard C++ single source programming model in safety-critical markets. SYCL SC will bridge the gap between low-level APIs such as Vulkan® SC and the C++ high-level language to streamline the development and safety certification of systems incorporating sophisticated parallel processing acceleration, including Artificial Intelligence and Machine Learning pipelines. The SYCL SC Working Group has already gained support from industry leaders including AMD, Arm, Barcelona Supercomputer Center, Codeplay, CoreAVI, Intel, Intellias, Mercedes-Benz, and Qualcomm Technologies Inc.

Khronos Compute Acceleration APIs

Khronos compute standards are often used to harness the hardware acceleration that underpins many vision processing and inferencing software stacks. They can be divided broadly into two groups, providing a choice of APIs suited to different markets and developer needs. Low-level APIs such as OpenCL and Vulkan provide direct, explicit access to hardware resources for maximum flexibility and control. Vulkan is used for programming GPUs, and OpenCL enables software to use any mix of available heterogeneous processors. Both OpenCL and Vulkan provide parallel acceleration of shaders or kernel programs compiled to Khronos’ SPIR-V intermediate language.

High-level programming frameworks, such as SYCL and OpenVX, focus on streamlined development with effective performance portability across diverse hardware architectures. SYCL provides parallel programming through the standard C++ language, and OpenVX uses a graph abstraction to express vision and inferencing functionality, and can directly import trained neural networks in the Khronos NNEF format. SYCL and OpenVX may be accelerated by the lower level Khronos APIs, but that is a choice left to the implementer.

Figure 3. Khronos Compute Acceleration APIs

The Emerging Need for an Embedded Camera API Standard

Sophisticated camera subsystems are increasingly critical in diverse markets such as robotics, autonomous driving, digital twins, and virtual and augmented reality where vision processing, often utilizing machine learning, is used to understand surroundings, processes, objects, and users. Consequently, an increasing number and diversity of sensors and associated Image Signal Processors (ISPs) are being tightly integrated with vision and inferencing accelerators in self-contained embedded systems.

However, the ongoing lack of interoperable camera API standards results in unnecessarily high integration costs for camera technologies—increasing application development time and maintenance costs while reducing software portability and opportunities for code reuse.

Figure 4. The Need for an Embedded Camera System API

During 2021, The Khronos Group and European Machine Vision Association (EMVA) worked together to convene an Exploratory Group that involved over seventy companies working together to explore industry consensus on the need, terminology, scope, requirements, and design methodology for a new open standard camera system API.  The Exploratory Group determined that an effective cross-platform API could indeed streamline deployment of camera systems by manufacturers and system integrators through enabling control of camera sensors, depth sensors, camera arrays and ISP hardware to generate sophisticated image streams for downstream processing.

The Emergence of the Kamaros API

The requirements generated by the Khronos / EMVA Exploratory Group were the motivation to create the Kamaros Working Group at Khronos in March 2022 focused on the development of an open, cross-vendor standard for camera, sensor, and ISP control. Deliverables will include the Kamaros API specification for use by implementers of the API and developers, an extension namespace registry for Khronos and vendor extensions, an open-source conformance test suite, and an Adopters Program to enable implementations to become officially conformant.

The design goals of Kamaros include portability of application code across diverse platforms, and multiple generations of cameras and sensors, while providing an effective API for the development of applications, frameworks, and libraries that need access to sophisticated camera functionality. Kamaros will leverage many design best-practices from other established Khronos APIs including specification structure and tooling, flexible and powerful extensibility, installable drivers with debug and instrumentation layers, and schema-based generation of multiple API language bindings.

Figure 5. Kamaros is a Cross-Platform Developer-Facing Camera API

Additional key design directions currently being discussed at the Kamaros Working Group include: how best to balance exposing the full power of camera and ISP processing versus a simplified programming model - perhaps through a layered utility library; and to what extent to leverage the proven resource allocation, synchronization, and work submission model of the new generation Vulkan API, perhaps leading to the option of seamless integration of sensor, ISP, and GPU processing within a single runtime.

As the design of the Kamaros API proceeds, Khronos and the EMVA continue their close collaboration and have entered into a Liaison Agreement, enabling ongoing close coordination between the two groups. Designated liaisons have been granted joint member privileges in both organizations. In addition, all EMVA members have a standing invitation to join the Kamaros Advisory Panel at no charge.

Figure 6. Ongoing Cooperation Between Khronos and the EMVA

Get Involved in Khronos and Kamaros

Khronos is committed to providing a safe space for companies to cooperate to create open standards that benefit their own business and the wider industry. All Khronos members are able to participate in any working group, including Kamaros. If your own company would like a voice and vote in any of these standardization activities, or you wish to implement a Khronos standard on your silicon or use a Khronos standard in your system design, Khronos warmly welcomes any company that wishes to participate.