Research and development - Industry 4.0
In recent years, we can observe the dynamic growth and development of cyber-physical systems in many industries. Coupled with communication and machine learning they form something that in 2011 was named for the first time - Industry 4.0. Manufacturers started to invest in modernizing production facilities with smart automation powered by intelligent systems that could collect data and monitor ongoing processes to increase overall efficiency. At the same time, a significant evolution in digital entertainment took place on the consumer market. Computers, consoles and even mobile devices reached or passed the 4K resolution displays and graphics processing with outstanding quality. These devices not only present data but also they can create, capture and stream it on a scale that was never seen before. All of these technologies either rely on or are supplemented by vision systems. Every year quality, resolution and sensitivity requirements are increasing. Better vision systems change the way machine vision can be implemented, change what is achievable by consumers and what kind of content the creators can make. By understanding industry trends, BIVROST engineering team has set the goal of providing next-gen image processing systems. These systems can provide ease of use and integration for standalone or embedded applications and will fit future needs and bring next-gen standards to machine vision.
To understand the need for integrated yet powerful image processing solution let’s take a look at existing imaging systems. Most of the solutions available on the market share common problems. Simple and easy-to-use products suffer from poor sensitivity and low dynamic range with hard-to-handle noise levels. More advanced ones have these parameters at a much better level but still not high enough to provide a level of control required by many applications. Even the most sophisticating products reveal their weaknesses in multi-camera environments: lack of synchronization, rolling shutter artifacts or a nearly non-existent way to control more than one unit. Dedicated multi-camera systems come with a significantly higher price tag and complicated user experience, demanding specialist knowledge for system integration. Time and cost of adopting these systems are very often crucial factors that prevent their use. Sensitivity and resolution are only half of the story. Colour reproduction and the ability to store and process it is key to professional video processing. Many systems today limit colour processing to 8 bits without the option to process RAW or even preserve dynamic range before compressing the image. Transferring high-resolution, high-colour depth image may be impossible and without the configurable onboard processing cause the loss of the majority of colour information.
The search for an appropriate hardware pushed the team to develop designs like the ORB VR camera and the BIVROST Immersive Media Processor (BIMP). A multi-lens camera coupled with BIMP, offering in-place programmable video stitching and processing with high dynamic range and professional level colour grading. The ability to stream video over a network or push it to external recorders gives plenty of space for live production and post-production work. The custom solution provided previously inaccessible global shutter sensors, synchronized to meet the TV standards and cinematic environment compatibility. Setting the whole system required one more element: intermediate mixing devices. BIMP fits in this place with 60Gbits I/O system and additional extension ports.
To address such a wide variety of applications BIVROST designed the Immersive Media Processor with special considerations for latency and resource management. BIMP architecture is based on the zero-copy concept and unified memory model. This results in a significant latency reduction and power efficiency. All crucial components like CPU, GPU or VPU can access video data on all processing stages without the need to push the data back and forth.
Such an optimized pipeline makes a universal platform for live VR and classic video possible. In different scenarios, the steps of video processing can be covered. Main pipeline features are:
- RAW capture and high-quality hardware Bayer demosaic with optional gamma processing and 8 or 10-bit output
- Fisheye and other distortion corrections
- 360/180 VR video stitching as an option for VR setups
- 3D LUT colour grading
- 2D and VR overlay system
- RAW or compressed recording to an internal storage
- Blending and other transitions between multiple sources
- HEVC, H.264 or MJPEG compression option
BIMP performs different roles within a video infrastructure. The same unit can be configured to be a VR stitcher, a multi-stream switcher, a recorder or a streamer. This modular approach has a huge impact on necessary equipment that must be kept in studios. BIMP's can replace multiple devices like dedicated streamers or simpler video consoles while adding professional grading and mixing capabilities. VR video configurations can be converted to 4K/HD streaming platforms without buying two separate video platforms. BIMP lowers entry barriers by complying with industry standards - most of the existing hardware can be connected and used with this media processor. Thanks to a partnership with experienced players like Intel BIVROST keeps product quality and features constantly growing for new and existing customers.
BIMP system is not limited to flat high-resolution video or VR videos format. Software architecture was specially prepared for media processor hardware as components that can be used to build custom video solutions. One of the emerging needs is volumetric video creation. This is part of an ongoing BIVROST research tasks. Console-like optimization for both hardware and software and a modular approach is the fast path to build mesh-based camera clusters for volumetric capture. High computing power in each unit in mesh decentralizing calculations can save bandwidth and reduce infrastructure requirements needed to build and run volumetric capture equipment.
With the edge computing idea, AI systems play an important role. They can initially process data at the lowest level as well as handle complex video analysis at the later stages of the pipeline. For this reason, BIMP can have MOVIDIUS chips installed to greatly accelerate neural computations.
AI acceleration opens the door to computer vision applications in different industries. In entertainment and sports transmissions it can be used to track players, detect unusual situations or create automated video correction systems. Medical applications can detect abnormalities invisible to the human eye. Vision systems for security boosted with AI can keep track of luggage or people increasing safety and protection. Also, industrial quality control is a subject to transform with AI-enabled computer vision.
The scale and complexity of this project prompted BIVROST to open a new R&D center to shorten the time of launching new solutions in the field of image processing, optics-set construction and AI boosted virtual reality video systems. Even with a great team of engineers and prototyping facilities, the participation of customers and technical partners cannot be overestimated. The best solutions require the involvement of both sides to create particularly-suited products. With this thought, BIVROST opened the program to join the research and development team and get early access to components, results and pilot runs of things to come in near future. Please contact BIVROST at: firstname.lastname@example.org for more details.