Looking-in and looking-out vision for Urban Intelligent Assistance: Estimation of driver attentive state and dynamic surround for safe merging and braking

“Previous studies mainly focus on the driver attention [41] and distraction (either physical distraction or cognitive distraction) [36], driver intention [7], [9], driver styles [42], driver drowsiness and fatigue detection [10]–[12], etc.”

“Our model is an Autoregressive Input-Output Hidden Markov Model (AIO-HMM) that jointly captures the context from both inside and outside the vehicle.”

“These vision algorithms have been used to design assistive features to monitor drivers for drowsiness and attentiveness [35, 41].”

“In another application, anticipating driving maneuvers several seconds in advance [19, 27, 38, 36] enables assistive cars to alert drivers before they make a dangerous maneuver.”

“Point tracking on the other hand can employ temporal information which has a better basis for complementing the detector.”

“They are utilized in order to achieve semantic segmentation of the driving environment [50], target detection and tracking [55], ranging [56,57], driver distraction and fatigue detection [58], and so on [59,60].”

“Understanding, modeling, and evaluation of situational awareness tasks, in particular the understanding of the behavior and intent of agents surrounding a vehicle, is an essential component in the development of such systems [3, 4, 5, 6].”

“[21] found that a car driver’s affective state recognition in terms of the engagement level provides important cues in detecting drowsiness levels while driving.”

“AAMs have also been used to estimate the 3D-pose of a face from a single image [75] and in design of assistive features for driver monitoring [55, 63].”

“In addition to gaze data (Bergasa et al. 2006; Fletcher and Zelinsky 2009; Su et al. 2006; Tawari et al. 2014), a wide variety of other methods have been used to estimate the attentional state of the driver.”

“The concept of a cell’s occupancy is typically implemented based on Bayesian probability [114] or Dempster–Shafer evidence theory [115]; the latter’s advantage is the capability of modeling that a cell has neither been observed as free nor occupied.”

“[275] provided early detection of driver distraction by continuously monitoring driver and surround traffic situation.”

“The driving data is collected from naturalistic, on-road driving using our LISAA testbed [28].”

“Consequently, there is a great potential for human-centric intelligent driver assistance systems [5], [6], [7], [8] to alert or even guide [9] the Authors are with the Laboratory of Intelligent and Safe Automobiles (http://cvrr.ucsd.edu/lisa/), University of California, San Diego, La Jolla, CA, USA…”

“In [14], Tawari et al. used a continuous head motion monitor to predict the driver’s engagement level.”

“Driver safety and crash risk are highly dependent on a driver’s attention levels, especially visual attention, for both traditional and self-driving vehicles [1]–[3].”

“Over the years, the researchers in the field of anticipation for driving scenarios have focused on specific subproblems of this challenging task, such as lane change detection [22, 42], a car’s own driver’s intention [23] or maneuver recognition [11, 13, 12, 24] and pedestrian intention prediction [28, 26, 19, 37].”

“They are utilized in order to achieve semantic segmentation of the driving environment [50], target detection and tracking [55], ranging [56], [57], driver distraction and fatigue detection [58], and so on [59], [60].”

“Tawari et al. [35] developed a "Merge and Lane Change Assist" system in which sensory fusion is achieved by encoding all constraints (spatial, temporal, as well as legal) into a compact probabilistic representation [36].”

“The framework will assist the analysis of naturalistic driving studies by automating the task of data reduction of the observed trajectories.”

“Previous studies mainly focus on the driver attention and distraction (either physical distraction or cognitive distraction) detection [37], driver intention [8] [10], driver drowsiness and fatigue [11]-[13], etc.”

“The types of sensors used in such systems may vary ranging from a single looking camera [657] to a distributed network of cameras for high resolution 3D face reconstruction and tracking [658, 659].”

“…that best leverages these challenges and advantages is key towards a robust face detection system when looking inside the vehicle cabin. in estimating driver’s head pose [6], predicting driver’s intent to change lane [7], monitoring driver’s alterness [8] and estimating driver’s gaze [11], [18].”

“have been quite a few studies that deal with driver distraction and activity modeling [5]–[8], these models are not directly applicable to pedestrians because the forms of distraction and the activities of interest are considerably different.”

“The work reported by [1, 39] falls in this category.”

“Tawari et al. [9] estimate the driver attention state based on its head position.”

“Device-based Head Tracking : Head tracking shows great potential in the automotive industry, including in-vehicle AR/VR and many subsequent applications [13, 27, 36, 42, 43, 51].”

“In the literature, human monitoring systems have been demonstrated to be effective in estimating the head pose [14], correlating the driver’s gaze with road events [7], or analyzing the steering wheel position [10] to gain a better understanding of the driver’s attention.”

“Distinguishing by the research target, [1] and [4] estimate the attentive state of the driver.”

“…detection and feature-based tracking for driver assistance [4], estimating pedestrian orientation for improved path prediction [5], active pedestrian safety by automatic automobile maneuvering [6], driver attention monitoring [7] are some of the most recent progress in enhancing pedestrian safety.”

“Research has led to a multitude of eye gaze-assisted interfaces that use the information gleaned from eye gaze to improve user performance and experience, spanning domains such as driving [Lemercier et al., 2014; Fletcher and Zelinsky, 2009; Tawari et al., 2014], game playing [Kocur et al.”

“Ford Campus [1] 2011 800x600 8 Hz 6 X X X US/Urban TME [2] 2012 1024x768 20 Hz 2 X X X Italian/Hwy Annieway [3] 2012 1382x512 10 Hz 4 X X X German Urban/Hwy LISA-A [4] 2012 1024x522 25 Hz 1 X X X X X US/Hwy Cityscapes [5] 2016 2048x1024 17 Hz 2 X X X European/Urban LISA-T [6] 2016 2704x1440 12 Hz 4 X X US/Hwy This pap…”

“The prediction of the driver’s intention and the anticipation of driving maneuvers are necessary to make the alerts effective [1], [2], [3], [4], [5].”

“[30] multiple cameras, radars, lidar head pose, field of view, attention buffer Real world On-road testing”

“A commercial object tracking module [20] tracks and re-identifies vehicles across LIDAR and radar systems providing vehicle position and velocity in a consistent global frame of reference.”

“f driving conditions. I. INTRODUCTION Driver safety and crash risk are highly dependent on a driver’s attention levels, especially visual attention, for both traditional and self-driving vehicles [1]–[3]. For intelligent vehicles, it is essential to have the ability to monitor the driver’s gaze continuously and use th…”

“In [13], Driver Attention Guard is developed that applies novel learning algorithms for tracking the driver’s head position and rotation using an array of cameras.”

“Besides drowsiness, other researchers looked into studying the driver behavior to measure the level of fatigue [19], inattentiveness/distraction detection [15], and learning the driving behavior (normal, aggressive) [7].”

“For instance, the ability to understand user intentions and to anticipate human error allows ADAS to either alert users or to take over control before potentially dangerous maneuvers take place [3], [4].”

“In [27], the head motion was continuously monitored to estimate the driver’s attention state.”

“To detect a driver’s attention, driver distraction detection systems employ driver cameras and sensors monitoring a driver’s head pose and eye movements [4, 5, 6].”

“…vision sensors looking in at the driver • Data driven processing: construction of features using temporal pyramids and using the random forest algorithm to extract optimal feature subset, where optimal features reveal when and what cue is most relevant for representing the preparatory motions.”

“In automotive applications, the driver’s head pose is useful for safety functions [21].”

“For example, pedestrian protection system Tawari et al. (2014)(implemented in the advanced driver assistant system) and intelligent robot Ćirić et al. (2014) that can follow the specified target are two good examples of such applications.”

“A system, which pays attention to the driver’s activity within the car and also to aspects of the urban road environment, was developed as part of the Urban Intelligent Assist Research Initiative some years ago [18], and since then, other systems with similar, or enhanced capabilities followed suit [19,20].”

“A collaboration between AUDI AG, Volkswagen, and UC San Diego developed a video‐based system for the detection of attention [20, 21].”

“and in design of assistive features for driver monitoring [185, 218].”

“In particular, vehicle detection is a critical subset of object detection in autonomous driving, as surrounding vehicles along with the driver state [6] provides context for vehicle safety tasks such as collision avoidance and lane change assistance [7].”

“Many new datasets and testbeds like LISA-A [12], LISA-Audi[15], LISA-T [2], Waymo[14], nuScenes[1] etc have contributed significantly in terms of data volumes and complexities.”

“A large collection of naturalistic driving data is created using a highly instrumented test vehicle dubbed LISA-A [19].”

“Subsequent research expanded LILO-style systems to include driver pose estimation, gaze tracking, and cognitive state inference using in-cabin cameras and vehicle telemetry [6, 7, 8, 39].”

“For example, Tawari et al. [38] proposed the Driver Attention Guard sys-tem, which could infer the driver’s attention and issue alerts and activate safety systems when the driver was inattentive, thereby e ff ectively reducing the risk of accidents.”

“cle trajectory predictions [1] [2] [3] or combined with driver monitoring data to set up a looking-in and looking-out system [4] to predict driver take-over time[5] [6].”

“…for driving scenarios have focused on specific subproblems of this challenging task, such as lane change detection [Morris et al., 2011; Tawari et al., 2014], a car’s own driver’s intention [Ohn-Bar et al., 2014] or maneuver [Jain et al., 2015, 2016b,a; Olabiyi et al., 2017]…”

“, [3] have conducted an Urban Intelligent Assist (UIA) project which aim is to make urban day-to-day driving safer and stress free.”