Multi-Sensor Data Fusion (MSDF) and AI: The Case of AI Self-Driving Cars

By Lance Eliot, the AI Trends Insider

A crucial element of many AI systems is the capability to undertake Multi-Sensor Data Fusion (MSDF), consisting of collecting together and trying to reconcile, harmonize, integrate, and synthesize the data about the surroundings and environment in which the AI system is operating. Simple stated, the sensors of the AI system are the eyes, ears, and sensory input, while the AI must somehow interpret and assemble the sensory data into a cohesive and usable interpretation of the real world.

If the sensor fusion does a poor job of discerning what’s out there, the AI is essentially blind or misled toward making life-or-death algorithmic decisions. Furthermore, the sensor fusion needs to be performed on a timely basis. Any extra time taken to undertake the sensor fusion means there is less time for the AI action planning subsystem to comprehend the driving situation and figure out what driving actions are next needed.

Humans do sensor fusion all the time, in our heads, though we often do not overtly put explicit thought towards our doing so. It just happens, naturally. We do the sensor fusing by a kind of autonomic process, ingrained by our innate and learned abilities of fusing our sensory inputs from the time we are born. On some occasions, we might be sparked to think about our sensor fusing capacities, if the circumstance catches our attention.

The other day, I was driving in the downtown Los Angeles area. There is always an abundance of traffic, including cars, bikes, motorcycles, scooters, and pedestrians that are prone to jaywalking. There is a lot to pay attention to. Is that bicyclist going to stay in the bike lane or decide to veer into the street? Will the pedestrian eyeing my car decide to leap into the road and dart across the street, making them a target and causing me to hit the brakes? It is a free-for-all.

I had my radio on, listening to the news reports, when I began to faintly hear the sound of a siren, seemingly off in the distance. Maybe it was outside, or maybe it was inside the car — the siren might actually be part of a radio segment covering the news of a local car accident that had happened that morning on the freeway, or was it instead a siren somewhere outside of my car? I turned down my radio. I quickly rolled down my driver’s side window.

As I strained to try and hear a siren, I also kept my eyes peeled, anticipating that if the siren was occurring nearby, there might be a police car or ambulance or fire truck that might soon go skyrocketing past me. These days it seems like most driver’s don’t care about emergency vehicles and fail to pull over to give them room to zoom along. I’m one of those motorists that still thinks we ought to help out by getting out of the way of the responders (plus, it’s the law in California, as it is in most states).

Of course, it makes driving sense anyway to get out of the way, since otherwise you are begging to get into a collision with a fast-moving vehicle, which doesn’t seem like a good idea on anyone’s behalf. A few years ago, I saw the aftermath of a collision between a passenger car and an ambulance. The two struck each other with tremendous force. The ambulance ended-up on its side. I happened to drive down the street where the accident had occurred, and the recovery crews were mopping up the scene. It looked outright frightening.

In any case, I was listening intently in the case of my driving in downtown Los Angeles and trying to discern if an emergency vehicle was in my vicinity and warning to be on the watch for it. I could just barely hear the siren. My guess was that it had to be a few blocks away from me. Was it thought getting louder and getting nearer to me, or was it fading and getting further away?

I decided that the siren was definitely getting more distinctive and pronounced. The echoes along the streets and buildings was creating some difficulty in deciding where the siren was coming from. I could not determine if the siren was behind me or somewhere in front of me. I couldn’t even tell if the siren was to my left or to my right. All I could seem to guess is that it was getting closer, one way or another.

At times like this, your need to do some sensor fusion is crucial. Your eyes are looking for any telltale sign of an emergency vehicle. Maybe the flashing lights might be seen from a distance. Perhaps other traffic might start to make way for the emergency vehicle, and that’s a visual clue that the vehicle is coming from a particular direction. Your ears are being used to do a bat-like echolocation of the emergency vehicle, using the sound to gauge the direction, speed, and placement of the speeding object.

I became quite aware of my having to merge together the sounds of the siren with my visual search of the traffic and streets. Each was feeding the other. I could see traffic up ahead that was coming to a stop, doing so even though they had a green light. It caused me to roll down my other window, the front passenger side window, in hopes of aiding my detection of the siren. Sure enough, the sound of the siren came through quite a bit on the right side of my car, more so than the left side of the car. I turned my head toward the right, and in moments saw the ambulance that zipped out of a cross-street and came into the lanes ahead.

This is the crux of Multi-Sensor Data Fusion. I had one kind of sensor, my eyes, providing visual inputs to my brain. I had another kind of sensor, my ears, providing acoustical inputs to my brain. My brain managed to tie together the two kinds of inputs. Not only were the inputs brought together, they were used in a means of each aiding the other. My visual processing led me to listen toward the sound. The sound led me to look toward where the sound seemed to be coming from.

My mind, doing some action planning of how to drive the car, melded together the visual and the acoustic, using it to guide how I would drive the car. In this case, I pulled the car over and came to a near stop. I also continued to listen to the siren. Only once it had gone far enough away, along with my not being able to see the emergency vehicle anymore, did I decide to resume driving down the street.

This whole activity of my doing the sensor fusion was something that played out in just a handful of seconds. I know that my describing it seemed to suggest that it took a long time to occur, but the reality is that the whole thing happened like this, hear siren, try to find siren, match siren with what I see, pull over, wait, and then resume driving once safe to do so. It is a pretty quick effort. You likely do the same from time-to-time.

Suppose though that I was wearing my ear buds and listening to loud music while driving (not a wise thing to do when driving a car, and usually illegal to do while driving), and did not hear the siren? I would have been solely dependent upon my sense of sight. Usually, it is better to have multiple sensors active and available when driving a car, giving you a more enriched texture of the traffic and the driving situation.

Notice too that the siren was hard to pin down in terms of where it was coming from, along with how far away it was. This highlights the aspect that the sensory data being collected might be only partially received or might otherwise be scant, or even faulty. Same could be said about my visually trying to spot the emergency vehicle. The tall buildings blocked my overall view. The other traffic tended to also block my view. If it had been raining, my vision would have been further disrupted.

Another aspect involves attempting to square together the inputs from multiple sensors. Imagine if the siren was getting louder and louder, and yet I did not see any impact to the traffic situation, meaning that no other cars changed their behavior and those pedestrians kept jaywalking. That would have been confusing. The sounds and my ears would seem to be suggesting one thing, while my eyes and visual processing was suggesting something else. It can be hard at times to mentally resolve such matters.

In this case, I was alone in my car. Only me and my own “sensors” were involved in this multi-sensor data fusion. You could have more such sensors, such as when having passengers that can aid you in the driving task.

Peering Into The Fog With Multiple Sensory Devices

I recall during my college days a rather harried driving occasion. While driving to a college basketball game, I managed to get into a thick bank of fog. Some of my buddies were in the car with me. At first, I was tempted to pull the car over and wait out the fog, hoping it would dissipate. My friends in the car were eager to get to the game and urged me to keep driving. I pointed out that I could barely see in front of the car and had zero visibility of anything behind the car.

Not the best way to be driving at highway speeds on an open highway. Plus, it was night time. A potent combination for a car wreck. It wasn’t just me that I was worried about, I also was concerned for my friends. And, even though I thought I could drive through the fog, those other idiot drivers that do so without paying close attention to the road were the really worrisome element. All it would take is some dolt to ram into me or opt to suddenly jam on their brakes, and it could be a bad evening for us all.

Here’s what happened. My buddy in the front passenger seat offered to intently watch for anything to my right. The two friends in the back seat were able to turnaround and look out the back window. I suddenly had the power of six additional eyeballs, all looking for any other traffic. They began each verbally reporting their respective status. I don’t see anything, one said. Another one barked out that a car was coming from my right and heading toward us. I turned my head and swerved the car to avoid what might have been a collision.

Meanwhile, both of the friends in the backseat yelled out that a car was rapidly approaching toward the rear of the car. They surmised that the driver had not seen our car in the fog and was going to run right up into us. I hit the gas to accelerate forward, doing so to gain a distance gap between me and the car from behind. I could see that there wasn’t a car directly ahead of me and so leaping forward was a reasonable gambit to avoid getting hit from behind.

All in all, we made it to the basketball game without nary a nick. It was a bit alarming though and a situation that I will always remember. There we were, working as a team, with me as the driver at the wheel. I had to do some real sensor fusion. I was receiving data from my own eyes, along with hearing from my buddies, and having to mentally combine together what they were telling me with what I could actually see.

When you are driving a car, you often are doing Multi-Target Tracking (MTT). This involves identifying particular objects or “targets” that you are trying to keep an eye on. While driving in downtown Los Angeles, my “targets” included the many cars, bike rides, and pedestrians. While driving in the foggy evening, we had cars coming from the right and from behind.

Your Field of View (FOV) is another vital aspect of driving a car and using your sensory apparatus. During the fog, my own FOV was narrowed to what I could see on the driver’s side of the car, and I could not see anything from behind the car. Fortunately, my buddies provided additional FOV’s. My front passenger was able to augment my FOV by telling me what was seen to the right of the car. The two in the backseat had a FOV of what was behind the car.

Those two stories that I’ve told are indicative of how we humans do our sensor fusion while driving a car. As I earlier mentioned, we often don’t seemingly put any conscious thought to the matter. By watching a teenage novice driver, you can at times observe as they struggle to do sensor fusion. They are new to driving and trying to cope with the myriad of details to be handled. It is a lot to process, such as keeping your hands on the wheel, your feet on the pedals, your eyes on the road, along with having to mentally process everything that is happening, all at once, in real-time.

It can be overwhelming. Seasoned drivers are used to it. But seasoned drivers can also find themselves in situations whereby sensor fusion becomes an outright imperative and involves very deliberate attention and thought. My fog story is somewhat akin to that kind of situation, similarly my siren listening story is another example.

In the news recently there has been the story about the Boeing 737 MAX 8 airplane and in particular two horrific deadly crashes. Some believe that the sensors on the plane were a significant contributing factor to the crashes. Though the matters are still being investigated, it is a potential example of the importance of Multi-Sensor Data Fusion and has lessons that can be applied to driving a car and advanced automation used to do so.

For the Boeing situation as it applies to self-driving cars, see my article: https://www.aitrends.com/selfdrivingcars/boeing-737-max-8-and-lessons-for-ai-the-case-of-ai-self-driving-cars/

For more about the fundamentals of sensor fusion, see my article: https://www.aitrends.com/selfdrivingcars/sensor-fusion-self-driving-cars/

For why one kind of sensor having just is myopic, see my article: https://www.aitrends.com/selfdrivingcars/cyclops-approach-ai-self-driving-cars-myopic/

For my article about what happens when sensors go bad or faulty, see: https://www.aitrends.com/selfdrivingcars/going-blind-sensors-fail-self-driving-cars/

Multi-Sensor Data Fusion for AI Self-Driving Cars

What does this have to do with AI self-driving cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. One important aspect involves the design, development, testing, and fielding of the Multi-Sensor Data Fusion.

Allow me to elaborate.

I’d like to first clarify and introduce the notion that there are varying levels of AI self-driving cars. The topmost level is considered Level 5. A Level 5 self-driving car is one that is being driven by the AI and there is no human driver involved. For the design of Level 5 self-driving cars, the auto makers are even removing the gas pedal, brake pedal, and steering wheel, since those are contraptions used by human drivers. The Level 5 self-driving car is not being driven by a human and nor is there an expectation that a human driver will be present in the self-driving car. It’s all on the shoulders of the AI to drive the car.

For self-driving cars less than a Level 5, there must be a human driver present in the car. The human driver is currently considered the responsible party for the acts of the car. The AI and the human driver are co-sharing the driving task. In spite of this co-sharing, the human is supposed to remain fully immersed into the driving task and be ready at all times to perform the driving task. I’ve repeatedly warned about the dangers of this co-sharing arrangement and predicted it will produce many untoward results.

For my overall framework about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/framework-ai-self-driving-driverless-cars-big-picture/

For the levels of self-driving cars, see my article: https://aitrends.com/selfdrivingcars/richter-scale-levels-self-driving-cars/

For why AI Level 5 self-driving cars are like a moonshot, see my article: https://aitrends.com/selfdrivingcars/self-driving-car-mother-ai-projects-moonshot/

For the dangers of co-sharing the driving task, see my article: https://aitrends.com/selfdrivingcars/human-back-up-drivers-for-ai-self-driving-cars/

Let’s focus herein on the true Level 5 self-driving car. Much of the comments apply to the less than Level 5 self-driving cars too, but the fully autonomous AI self-driving car will receive the most attention in this discussion.

Here’s the usual steps involved in the AI driving task:

Sensor data collection and interpretation
Sensor fusion
Virtual world model updating
AI action planning
Car controls command issuance

Another key aspect of AI self-driving cars is that they will be driving on our roadways in the midst of human driven cars too. There are some pundits of AI self-driving cars that continually refer to a utopian world in which there are only AI self-driving cars on the public roads. Currently there are about 250+ million conventional cars in the United States alone, and those cars are not going to magically disappear or become true Level 5 AI self-driving cars overnight.

Indeed, the use of human driven cars will last for many years, likely many decades, and the advent of AI self-driving cars will occur while there are still human driven cars on the roads. This is a crucial point since this means that the AI of self-driving cars needs to be able to contend with not just other AI self-driving cars, but also contend with human driven cars. It is easy to envision a simplistic and rather unrealistic world in which all AI self-driving cars are politely interacting with each other and being civil about roadway interactions. That’s not what is going to be happening for the foreseeable future. AI self-driving cars and human driven cars will need to be able to cope with each other.

For my article about the grand convergence that has led us to this moment in time, see: https://aitrends.com/selfdrivingcars/grand-convergence-explains-rise-self-driving-cars/

See my article about the ethical dilemmas facing AI self-driving cars: https://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/

For potential regulations about AI self-driving cars, see my article: https://aitrends.com/selfdrivingcars/assessing-federal-regulations-self-driving-cars-house-bill-passed/

For my predictions about AI self-driving cars for the 2020s, 2030s, and 2040s, see my article: https://aitrends.com/selfdrivingcars/gen-z-and-the-fate-of-ai-self-driving-cars/

Returning to the topic of Multi-Sensor Data Fusion, let’s walk through some of the key essentials of how AI self-driving cars undertake such efforts.

Take a look at Figure 1.

I’ve shown my overall framework about AI self-driving cars and highlighted the sensor fusion stage of processing.

Per my earlier remarks about the crucial nature of sensor fusion, consider that if the sensor fusion goes awry, it means that the stages downstream are going to be either without needed information or be using misleading information. The virtual world model won’t be reflective of the real-world surrounding the self-driving car. The AI action planning stage will not be able to make appropriate determinations about what the AI self-driving car actions should be.

One of the major challenges for sensor fusion involves dealing with how to collectively stich together the multitude of sensory data being collected.

You are going to have the visual data collected via the cameras, coming from likely numerous cameras mounted at the front, back, and sides of the self-driving car. There is the radar data collected by the multiple radar sensors mounted on the self-driving car. There are likely ultrasonic sensors. There could be LIDAR sensors, a special kind of sensor that combines light and radar. And there could be other sensors too, such as acoustic sensors, olfactory sensors, etc.

Thus, you will have to stich together sensor data from like-sensors, such as the data from the various cameras. Plus, you will have to stitch together the sensor data from unlike sensors, meaning that you want to do a kind of comparison and contrasting with the cameras, with the radar, with the LIDAR, with the ultrasonic, and so on.

Each different type or kind of sensor provides a different type or kind of potential indication about the real-world. They do not all perceive the world in the same way. This is both good and bad.

The good aspect is that you can potentially achieve a rounded balance by using differing kinds or types of sensors. Cameras and visual processing are usually not as adept at being indicative of the speed of an object as does the radar or the LIDAR. By exploiting the strengths of each kind of sensor, you are able to have a more enriched texturing of what the real-world consists of.

If the sensor fusion subsystem is poorly devised, it can undermine this complimentary triangulation that having differing kinds of sensors inherently provides. It’s a shame. Weak or slimly designed sensor fusion often tosses away important information that could be used to better gauge the surroundings. With a properly concocted complimentary perspective, the AI action planner portion has a greater shot at making better driving decisions because it is more informed about the real-world around the self-driving car.

Let’s though all acknowledge that the more processing you do of the multitude of sensors, the more computer processing you need, which then means that you have to place more computer processors and memory on-board the self-driving car. This adds cost, it adds weight to the car, it consumes electrical power, it generates heat, and has other downsides. Furthermore, trying to bring together all of the data and interpretations is going to take processing time, of which, as emphasized herein many times, the time constraints for the AI are quite severe when driving a car.

For my article about the cognition timing aspects of AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/cognitive-timing-for-ai-self-driving-cars/

For the aspects of LIDAR, see my article: https://www.aitrends.com/selfdrivingcars/lidar-secret-sauce-self-driving-cars/

For the use of compressive sensing, see my article: https://www.aitrends.com/selfdrivingcars/compressive-sensing-ai-self-driving-cars/

For my article about the safety of AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/safety-and-ai-self-driving-cars-world-safety-summit-on-autonomous-tech/

Four Key Ways Approaches to MSDF Assimilation

Let’s consider the fundamental ways that you assimilate together the sensory data from multiple sensors.

Take a look at Figure 2.

I’ll briefly describe the four approaches, consisting of harmonize, reconcile, integrate, and synthesize.

Harmonize

Assume that you have two different kinds of sensors, I’ll call them sensor X and sensor Z. They each are able to sense the world outside of the self-driving car. We won’t concern ourselves for the moment with their respective strengths and weaknesses, which I’ll be covering later on herein.

There is an object in the real-world and the sensor X and the sensor Z are both able to detect the object. This could be a pedestrian in the street, or maybe a dog, or could be a car. In any case, I’m going to simplify the matter to considering the overall notion of detecting an object.

This dual detection means that both of the different kinds of sensors have something to report about the object. We have a dual detection of the object. Now, we want to figure out how much more we can discern about the object because we have two perspectives about it.

This involves harmonizing the two reported detections. Let’s pretend that both sensors detect the distance of the object. And, sensor X indicates the object is six feet tall and about two feet wide. Meanwhile, sensor Z is reporting that the object is moving toward the self-driving car, doing so at a speed of a certain number of feet per second N. We can combine together the two sensor reports and update the virtual world model that there is an object of six feet in height, two feet in width, moving toward the self-driving car at some speed N.

Suppose we only relied upon sensor X. Maybe because we only have sensor X and there is no sensor Z on this self-driving car. Or, sensor Z is broken. Or, sensor Z is temporarily out of commission because there is a bunch of mud sitting on top of the sensor. In this case, we would know only the height and weight and general position of the object, but not have a reading about its speed and direction of travel. That would mean that the AI action planner is not going to have as much a perspective on the object as might be desired.

As a quick aside, this also ties into ongoing debates about which sensors to have on AI self-driving cars. For example, one of the most acrimonious debates involves the choice by Tesla and Elon Musk to not put LIDAR onto the Tesla cars. Elon has stated that he doesn’t believe LIDAR is needed to achieve a true AI Level 5 self-driving car via his Autopilot system, though he also acknowledges that he might ultimately be proven mistaken by this assumption.

Some would claim that the sensory input available via LIDAR cannot be otherwise fully devised via the other kinds of sensors, and so in that sense the Teslas are not going to have the same kind of complimentary or triangulation available that self-driving cars with LIDAR have. Those that are not enamored of LIDAR would claim that the LIDAR sensory data is not worth the added cost, nor worth the added processing effort, nor worth the added cognition time required for processing.

I’ve pointed out that this is not merely a technical or technological question. It is my bet that once AI self-driving cars get into foul car accidents, we’ll see lawsuits that will attempt to go after the auto makers and tech firms for the sensory choices made in the designs of their AI self-driving cars.

If an auto maker or tech firm opted to not use LIDAR, a lawsuit might contend that the omission of LIDAR was a significant drawback to the capabilities of the AI self-driving car, and that the auto maker or tech firm knew or should have known that they were under-powering their AI self-driving car, making it less safe. This is going to be a somewhat “easier” claim to launch, especially due to the aspect that most AI self-driving cars are being equipped with LIDAR.

If an auto maker or tech firm opts to use LIDAR, a lawsuit might contend that the added effort by the AI system to process the LIDAR was a contributor to the car wreck, and that the auto maker or tech firm knew or should have known that the added processing and processing time could lead to the AI self-driving car being less safe. This claim will be more difficult to lodge and support, especially since it goes against the tide of most AI self-driving cars being equipped with LIDAR.

For the crossing of the Rubicon for these kinds of questions, see my article: https://www.aitrends.com/selfdrivingcars/crossing-the-rubicon-and-ai-self-driving-cars/

For the emergence of lawsuits about AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/first-salvo-class-action-lawsuits-defective-self-driving-cars/

For my article about other kinds of legal matters for AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/self-driving-car-lawsuits-bonanza-ahead/

For my Top 10 predictions about where AI self-driving cars are trending, see: https://www.aitrends.com/selfdrivingcars/top-10-ai-trends-insider-predictions-about-ai-and-ai-self-driving-cars-for-2019/

Reconcile

I’d like to revisit the use of sensor X and sensor Z in terms of object detection.

Let’s pretend that sensor X detects an object, and yet sensor Z does not, even though the sensor Z could have. In other words, the object is within the Field of View (FOV) of sensor Z, and yet sensor Z isn’t detecting the object. Note that this is vastly different than if the object were entirely outside the FOV of sensor Z, in which case we would not have any expectation that sensor Z could detect the object.

We have a bit of a conundrum on our hands that needs reconciling.

Sensor X says the object is there in the FOV. Sensor Z says the object is not there in the same FOV intersection. Yikes! It could be that sensor X is correct and sensor Z is incorrect. Perhaps sensor Z is faulty, or obscured, or having some other difficulty. On the other hand, maybe sensor X is incorrect, namely that there isn’t an object there, and the sensor is X is mistaken, reporting a “ghost” of sorts, something that is not really there, while sensor Z is correct in reporting that there isn’t anything there.

There are various means to try and reconcile these seemingly contradictory reports. I’ll be getting to those methods shortly herein.

Integrate

Let’s suppose we have two objects. One of those objects is in the FOV of sensor X. The other object is within the FOV of sensor Z. Sensor X is not able to directly detect the object that sensor Z has detected, rightfully so because the object is not inside the FOV of sensor X. Sensor Z is not able to directly detect the object that sensor X has detected, rightfully so because the object is not inside the FOV of sensor Z.

Everything is okay in that the sensor X and sensor Z are both working as expected.

What we would like to do is see if we can integrate together the reporting of sensor X and sensor Z. They each are finding objects in their respective FOV. It could be that the object in the FOV of sensor Z is heading toward the FOV of sensor X, and thus it might be possible to inform sensor X to especially be on the watch for the object. Likewise, the same could be said about the object that sensor X currently has detected and might forewarn sensor Z.

My story about driving in the fog is a similar example of integrating together sensory data. The cars seen by my front passenger and by those sitting in the backseat of my car were integrated into my own mental processing about the driving scene.

Synthesize

In the fourth kind of approach about assimilating together the sensory data, you can have a situation whereby neither sensor X and nor sensor Z has an object within their respective FOV’s. In this case, the assumption would be that neither one even knows that the object exists.

In the case of my driving in the fog, suppose a bike rider was in my blind spot, and that neither of my buddies saw the bike rider due to the fog. None of us knew that a bike rider was nearby. We are all blind to the bike rider. There are likely going to be gaps in the FOV’s of the sensors on an AI self-driving car, which suggests that at times there will be parts or objects of the surrounding real-world that the AI action planner is not going to know is even there.

You sometimes have a chance at guessing about objects that aren’t in the FOV’s of the sensors by interpreting and interpolating whatever you do know about the objects within the FOV’s of the sensors. This is referred to as synthesis or synthesizing of sensor fusion.

Remember how I mentioned that I saw other cars moving over when I was hearing the sounds of a siren. I could not see the emergency vehicle. Luckily, I had a clue about the emergency vehicle because I could hear it. Erase the hearing aspects and pretend that all that you had was the visual indication that other cars were moving over to the side of the road.

Within your FOV, you have something happening that gives you a clue about what is not within your FOV. You are able to synthesize what you do know and use that to try and predict what you don’t know. It seems like a reasonable guess that if cars around you are pulling over, it suggests an emergency vehicle is coming. I guess it could mean that aliens from Mars have landed and you didn’t notice it because you were strictly looking at the other cars, but I doubt that possibility of those alien creatures landing here.

So, you can use the sensory data to try and indirectly figure out what might be happening in FOV’s that are outside of your purview. Keeping in mind that this is real-time system and that the self-driving car is in-motion, it could be that within moments the thing that you guessed might be in the outside of scope FOV will come within the scope of your FOV, and hopefully you’ll have gotten ready for it. Just as I did about the ambulance that zipped past me.

For my article about dealing with emergency vehicles, see: https://www.aitrends.com/selfdrivingcars/emergency-vehicle-awareness-self-driving-cars/

For the use of computational periscopy and shadow detection, see my article: https://www.aitrends.com/ai-insider/computational-periscopy-and-ai-the-case-of-ai-self-driving-cars/

For the emergence of e-Nose sensors, see my article: https://www.aitrends.com/selfdrivingcars/olfactory-e-nose-sensors-and-ai-self-driving-cars/

For my article about fail-safe aspects of AI self-driving cars, see: https://www.aitrends.com/selfdrivingcars/fail-safe-ai-and-self-driving-cars/

Voting Methods of Multi-Sensor Data Fusion

When you have multiple sensors and you want to bring together in some cohesive manner their respective reporting, there are a variety of methods you can use.

Take a look at Figure 1 again.

I’ll briefly describe each of the voting methods.

Absolute Ranking Method

In this method, you beforehand decide a ranking of sensors. You might declare that the cameras are higher ranked than the radar. The radar you might decide is higher ranked than the LIDAR. And so on. During sensor fusion, the subsystem uses that predetermined ranking.

For example, suppose you get into a situation of reconciliation, such as the instance I described earlier involving sensor X detecting an object in its FOV but that sensor Z in the intersecting FOV did not detect. If sensor X is the camera, while sensor Z is the LIDAR, you might simply use the pre-determined ranking and the algorithm assumes that since the camera is higher ranking it is “okay” that the sensor Z does not detect the object.

There are tradeoffs to this approach. It tends to be fast, easy to implement, and simple. Yet it tends toward doing the kind of “tossing out” that I forewarned is not usually advantageous overall.

Circumstances Ranking Method

This is similar to the Absolute Ranking Method but differs because the ranking is changeable depending upon the circumstance in-hand. For example, we might have setup that if there is rainy weather, the camera is no longer the top dog and instead the radar gets the topmost ranking, due to its less likelihood of being adversely impacted by the rain.

There are tradeoffs to this approach too. It tends to be relatively fast, easy to implement, and simple. Yet it once again tends toward doing the kind of “tossing out” that I forewarned is not usually advantageous overall.

Equal Votes (Consensus) Method

In this approach, you allow each sensor to have a vote. They are all considered equal in their voting capacity. You then use a counting algorithm that might go with a consensus vote. If some threshold of the sensors all agrees about an object, while some do not, you allow the consensus to decide what the AI system is going to be led to believe.