Why the Realsense line kind of had it coming and why you should think about moving on yourself.
Like many out there, I had a bit of a fright earlier this fall when Intel announced they were canning their Realsense line. While we’ve been assured that the popular D4xx-series depth sensors that have become a staple of many a robot BOM will continue to be made, the F455 and L515 facial recognition and LIDAR devices, respectively, are getting the axe.
There are many possible reasons for this and Intel has been rather tight-lipped about their own thinking, aside from the usual platitudes of “refocusing the business” and “continuing to provide”. Being a veteran of Very Large Companies stomping a product-line you’ve come to know and love, I was also disappointed-but-not-surprised, but the way it was handled kept gnawing at me. Why kill off a well-known and popular product line that they’ve spent the time and brand energy to make it top-of-mind when it comes to depth cameras.
The answer, I think is on my workbench. The Luxonis OAK-D is the latest in an emerging category of devices that would probably best be called a “Vision Processing Device” (VPD). A VPD, quite simply, is a programmable camera. In addition to doing the raw sensing, they can also run a set of user-defined algorithms, neural networks,etc. on the camera itself. The output then, is either a straight-up image, the results of the computation (say, the bounding box of an ROI) or some other user-defined data stream(number of people in an image, distance to the center of an ROI, etc). .
Until the VPD, robot vision has been pretty well divided between sensing and computing. You had a device like a Realsense D435i or Orbbec Astra, etc. sending a firehose of RGB image and depth data to an onboard computer, or in some cases up to a cloud compute resource for processing. This has led to the very popular pairing of a depth sensor for sensing and a high-powered SBC, for processing. While $350 Realsense devices fly off the shelves, they are almost inevitably paired up with a $1400(-ish) Jetson-series SOM, like the Xavier. Intel senses and NVidia processes is one of the current paradigms for robot vision, but it’s one that can’t be sitting too well with the Product or Finance people at Intel.
Look at your depth sensor, now look at my lab, now back to your sensor…
Back to my lab bench. I mentioned there’s an OAK-D vision sensor sitting on it. It’s made by a company called Luxonis and was produced in 2020 as a Kickstarter backed by OpenCV.org. A lot has been written about its capabilities and features, but one piece that’s been overlooked(that plays into the whole “Realsense is Dead, no it’s not” kerfuffle) is the processor inside of it. It’s an Intel Movidius Vector Processor running OpenVINO, their framework for running vision-processing neural networks like ImageNet, PoseNet, YOLO, etc. This lets the OAK-family of devices do all the depth-sensing, object classification,segmentation,sentinment analysis, etc. on the camera. This means that you don’t, by default, need that big Xavier-scale SBC on your robot. You could just as easily get by with a Raspberry Pi or an UP! Board that doesn’t have nearly the power and heat-dissipation requirements. Hello battery life! Hello longer run-times! Hello enhanced mission range!
I think this is why, despite it’s popularity and market penetration, Intel might not care so much about Realsense. It’s willing to trim their product offerings and even float the idea of tanking the whole Realsense line because they see a way out of this product cul-de-sac they’ve found themselves in. The OAK-D is a breakout success in the robotics communities I work and play in, and at $200 USD retail, it’s not hard to see why. Putting on my industry/market-analyst hat again, we see that the Movidius is one component in a device that’s $100 cheaper than the D435i, but it does provides a direct line product-wise into Intel’s own Neural Compute Engine ecosystem and vendors like AAEON, whose UP! board SBCs now also have a device range that take Movidius boards as plug-in modules.
It’s okay if you haven’t heard of VPDs until now. They’ve been simmering as a device class for over a decade, but it’s only been in the past year or two that their capabilities have broken out of the STEM lab and hobbyist realm. They’ve been developed and produced through crowd-funding and/or small word-of-mouth campaigns vs. the traditional Developer Conference/Tech Expo channels that announced the arrival of things like the Microsoft Kinect, Asus Xtion and Intel Realsense.
I’ve been following their development as a device class since roughly 2005 and backed more than a few of the crowd-funding campaigns for these devices. So for me, at least, the arrival of devices like the OAK-D, OpenMVCam, etc. is something I’ve been eagerly awaiting for a while now. I’ve got a separate post warming up on the history of processor-backed robot cameras that I’ll post soon.
There’s more than one way to label your cats…
Lest this come off as a complete ad for Luxonis, The OAK products are not the only VPD game out there. There are number of other devices that add a straight-up 32-bit ARM CPU to their cameras, allowing you to run your machine vision code right on the device. The OpenMVCam H7, for example, has an ARM Cortex V7 hosting an OV7725 image sensor. The onboard firmware is written in C, accessed via an OpenCV-alike API using MicroPython. and support for machine-learning frameworks like TensorFlowLite. It’s Open Source & Open Hardware, with the full electronic schematics and mechanical CAD files on their website. A wide range of adapter boards like a global shutter module and FLIR Lepton carrier are available. An M12-style mount lets you swap out the stock lens for a macro, zoom, fisheye, etc.
Normally, it would be crazy to suggest that companies with robots out in the world and on their drawing board just toss it all for some relatively tiny newcomers. The branch has already creaked once under us Realsense users though and there are some up-and-coming devices that are cheaper and(depending on your application) just as capable as the current D400-series devices put out by Intel. This is really just the first generation of this class of devices and the space is sure to further define and segment itself over the next year or so.
From that angle, I’m honestly trying to think of a good reason not to go out and get a couple of these, even just to check out. In my opinion, if you’re buying Realsenses by the box, you’ve got some cash to throw around and you can pick up one of each of the current crop for what 1 D435i would cost you.
The OpenMVCam H7, for example, goes for $65 on their site. That’s about what a plain old webcam like the Logitech C920 costs. The OAK-D sells for $149-199. More than twice as much as the OpenMVCAM, but still $100-200 less than a Realsense D435i. The Firefly from RockChip has an NPU and comes pre-loaded with a bunch of different face-recognition CNNs starting at $129.
The longer answer is to maybe go back over your current designs and imaging pipeline and see what exactly you’re consuming and producing from your vision sensors. If your data is heavy on the RGB imagery and object recognition, and you don’t need active depth-sensing, I think a VPD is definitely worth a look. If you find that you really need active depth-sensing or need a sensor that can operate in low/no light or have operational & safety requirements that you already have nailed with your current system, keep rolling with what you’re doing, but “watch this space”.
About Familiar Robotics
I started Familiar Robotics not long after I left Willow Garage. It didn’t have that name at the time, but I knew from my experience there that there was a real need for somebody who had a broad knowledge of all the things that went into making robots move. My decades-long background as a System and Network Administrator combined with over a decade of building robots for work and fun gives me a unique perspective to help at both a strategic and tactical level. Tracking, poking and prodding at emerging technology and how that could be implemented in the systems I used and managed is a skill I honed early on in my career and is something I continue to do today for clients who need it.
If you liked this post, need some insight to the robotics space or are simply ” Having Trouble with your Droid” , click here to “Get Familiar” with me and how I’ve helped lots of companies over the past 8 years or so Make Their Things Go.