Neuro-symbolic Complex Event Recognition in Autonomous Driving

Summary

Complex Event Recognition (CER) aims to efficiently recognize temporal events and respond accordingly. Sometimes, these events need to be recognized from other modalities rather than simple numerical data, with video being a common modality. One example is in the autonomous driving domain, where different events must be recognized from the perspective of an autonomous vehicle. However, most deep learning methods that solve computer vision tasks function as black boxes, a characteristic unsuitable for such high-risk applications. Also, they cannot use existing domain knowledge, e.g. known patterns to be recognized. In this work, we proposed a neuro-symbolic approach that recognizes complex road events by combining existing explainable CER technology with computer vision methods. We applied our method to the prediction of ‘overtake' incidents and evaluated it on both this complex event and the simpler events it comprises of, comparing it to a purely neural approach. While our proposed method performed better overall, the key finding in our research was its ability to yield good results with smaller and simpler networks, as compared to neural baselines, which required significantly larger networks to achieve the same performance.