Road blocks: Gary Ogasawara, CTO at object storage systems provider Cloudian, looks at why understanding data gravity is key to AV development and deployment.
While AVs are a big-ticket item on the agendas of some of the world’s best-known and most well-resourced car companies, such as BMW, Daimler and Ford, the world is yet to experience widespread real-life adoption.
The stunning volumes of data involved in the AV sector represent a roadblock to future deployment and adoption. To keep pedestrians, passengers and drivers free from harm, AVs must act as moving computers, processing and managing huge volumes of video and sensor data in order to make real-time decisions.
How can data gravity and edge storage help create a more powerful and durable underlying infrastructure for AVs, helping accelerate their arrival into the mainstream?
Data storage
Many experts, such as analyst house 451 Research, predict that a single consumer AV may generate 12-15TB of data per day.
Based on these numbers, an entire AV fleet could comfortably generate hundreds of terabytes daily. It’s therefore easy to see how organizations might be at a loss about how to store this data efficiently and without incurring excessive costs. This is where understanding the concept of data gravity comes into play. Data gravity recognizes that data has significant mass and can be difficult and costly to move, so it’s better to bring compute, applications and services to where the data is produced.
A hybrid approach
Organizations need to understand that despite the greatly increased capacity that 5G will bring to the table, it will still be unable to deal with the bandwidth requirements of just one AV. Networking infrastructure is simply not capable of handling the data volumes produced.
As a result, data gravity will hamper AV developers who rely too heavily on the public cloud. Most driving decisions cannot wait for the required latency back-and-forth to a public cloud, not to mention the potential unavailability and unreliability of the network communication.
AV developers that can use local data and compute resources are more equipped to develop autonomous capabilities.
It is therefore unrealistic to keep all your AV data in the cloud. Instead a hybrid approach, combining compute and storage at the edge with cloud resources, is needed. This hybrid architecture works with the forces of data gravity and not against them. Edge storage, where data is processed and filtered on the vehicle itself, is a fundamental requirement.
Edge storage
With the colossal scale of AV data being apparent, you would be hard pressed to find a more obvious use case for edge storage. By moving compute and applications closer to the sources of data itself, and respecting the forces of data gravity, implementing edge storage enables organizations to reduce IT costs and simplify their IT processes.
If organizations want to be able to maximize the value of the data that their vehicles collect, they need to find the right balance between what is kept locally on the edge and what is sent either to the data center hub or to the cloud to be analyzed and processed.
This can be achieved by sending the data that is most important and relevant (anything anomalous) to the hub, whereas any ‘noise’ (anything within the boundaries of normality) can be filtered out and stored locally on the edge. Rather than automatically pushing all data to the hub regardless of whether it is relevant, filtering data can ensure that data is always present where it is most needed. Another benefit is that having some data sets stored on the edge allows them to be processed faster, as they do not have to deal with the latency issues associated with the cloud.
EDITOR’S NOTE: This feature was first published in Autonomous Vehicle International October 2020 – one of the world’s biggest magazines dedicated to AV technology. Supported by the world’s leading autonomous and connected car developers and manufacturers, the magazine highlights the latest trends, developments and technological advancements in self-driving vehicle development, testing and implementation.