How a software update could add another amazing Tesla feature

In that sense, a very interesting feature developed by Andrea Ranieri, deep learning researcher at CNR-IMATI (@ 4ndr3r), and posted on Twitter yesterday, could potentially turn Tesla on-board cameras into different types of additional sensors, without any need. add other electronic equipment to vehicles: this would only be a software update (which would obviously require some adjustments and settings beforehand). In this case, we are talking about a deep learning AI, trained with @fastdotai as Ranieri explains, which basically uses ordinary cell phone cameras to detect potholes and cracks in the roads in cement when you walk around. More details will be available as soon as he publishes an article he publishes: code, data and pre-trained models.

This prompted immediate reactions on Twitter: “… the structural damage detection app for buildings, bridges, dams, etc.” caused by earthquakes, hurricanes, etc. Lives. Read up on: Non-destructive testing techniques, “commented Dr JT Kostman (@jt_kostman); to which Ranieri replied” … we’d love to work on that too! We are submitting requests for funding on this subject but so far they have all been rejected. In fact, with a little bit of retraining, the model would be able to detect damage even on concrete and other materials. “

When asked if the technology would work on online or offline videos, he replied “… it works on pictures, so it can work on both offline videos or streaming videos. . This is an offline video and is processed at 7 frames per second on my RTX 2070. But the inference script is still not optimized, there is a lot of room for improvement! ” How many diapers has he used? To which he replied “… this is a DeepLabv3 + with a ResNet-101 as a backbone. The training and video images are 540x540px, resolution matters when you want to see cracks in the asphalt.”

When asked if, by any chance, GPS would mark potholes, Andrea replied that “… no GPS at the moment, but we are working on a semi-permanent installation on a bus which should provide a GPS position. ” Another Twitter user, @GiorgioMantova, asked her if this would work “… so good on dark asphalt too?” (Dark road, dark pothole). “, To which he replied that” … No, on dark asphalt the model has a little more “uncertainty”. I will need to add more dark asphalt images or decrease the brightness as the data increase. “

Tesla Robotics, image courtesy of Tesla, Inc.

Ranieri said he would write an article about it and then do a summary / tutorial to share on Twitter; and that we will have to be patient. Finally, he said that the demo he posted is a video of a normal smartphone but that he would also be happy to try and merge the information from a LIDAR, but that he does not have the hardware. for the moment ; “… maybe for the next project.”

Tesla Home Charging, image courtesy of Tesla, Inc.

Just imagine applying this technology to millions of Teslas (Model 3, Model Y, S, and X, and the upcoming Cybertruck and Roadster) on the road, providing 24/7 information across a variety of categories (that’s i.e. not just potholes or cracks in cement / asphalt roads) and creating a series of giant AI datasets that could then be recycled for a whole range of uses different in machine learning: that would be heaven on earth for Tesla AI engineers, I bet.
What do you think? Please let us know in the comment section below.

All images in this item are courtesy of Tesla, Inc.

Nico Caballero is the vice-president of finance of Cogency Power, specializing in solar energy. He also holds a degree in electric cars from Delft University of Technology in the Netherlands and enjoys researching Tesla and EV batteries. He can be contacted at @NicoTorqueNews on Twitter. Nico covers the latest events from Tesla and EVs at Torque News.