Abstract
Real-time detection of distracted driving behavior by drivers can effectively prevent traffic accidents from occurring. This study presents a lightweight distracted driving behavior detection approach leveraging the RT-DETR-R18(real-time detection transformer with ResNet-18)approach to mitigate the real-time performance limitations commonly observed in existing distracted driving detection methods. To address the difficulty of maintaining model accuracy after lightweight improvement,a dual backbone dynamic feature fusion network (DB-DFN)is proposed to replace ResNet-18for early feature extraction and fusion. An additive-based feature fusion module (AddMixer)is introduced to replace attention-based intra-scale feature interaction (AIFI)in the original model for efficient feature interaction and to improve testing speed. The convolution module with an integrated gating mechanism was replaced with the original model's convolution module to improve cnn-based cross-scale feature fusion(CCFM),reducing redundant computation of the model's convolutional layer. This study conducted experiments leveraging the State Farm Distracted Driver Detection dataset. Compared with the RT-DETR-R18 model,the proposed method minimizes the number of parameters by 34.8%,lowers the computational cost by 49. 8%,and increases the inference speed by 34.8 frames per second (FPS).Meanwhile,the mAP50 remains unchanged,indicating that detection accuracy is preserved while real-time performance is significantly enhanced through lightweight optimization.
0 Introduction
With the rapid increase in the number of motor vehicles nationwide,road traffic safety concerns have become progressively more prominent. Among the major factors contributing to traffic accidents,drivers'subjective distracted driving behaviors play a dominant role[1]. At present,detection approaches for dangerous driving behavior can be generally categorized into three types:methods relying on drivers' physiological characteristics,approaches based on vehicle driving states,and computer vision-based action recognition techniques[2]. Compared with the former two approaches,computer vision-based methods demonstrate stronger practicality,greater adaptability,and broader acceptance.
In recent years,extensive research efforts have focused on computer vision-based algorithms for detecting abnormal driving behaviors. In 2023,Zou et al.[3]integrated an SE attention mechanism into the YOLOv5 backbone and further substituted the original CSP module with a lightweight GhostBottleneck design. In addition,the activation function of the network was modified to SiLU,resulting in improved detection accuracy while simultaneously reducing the model parameter size.Shang et al.[4]proposed the FedBiKD framework to address the problem of pseudo-label noise caused by data distribution discrepancies among different drivers in traditional federated learning settings,which often leads to global model performance degradation. Their approach guides local model learning through a global model and mitigates label bias. By integrating bidirectional distillation between local and global models,the framework enhances training stability and accelerates convergence. The experimental evaluation reveals that the proposed method delivers superior results compared with other federated learning methods,particularly with respect to accuracy,communication efficiency,and robustness. Wang et al.[5]introduced a two-stage TLD strategy to enable efficient traffic light detection using lightweight YOLOv5,and further extended the TWSOM module for category recognition. Compared with the original YOLOv5s,the developed model realized a 73.3%shrinkage in model size as well as a 79.2% decline in computational power consumption. Shen et al.[6]fused StarNet with context aggregation and integrated it into the YOLOv8 backbone,replacing the original bottleneck structure with faster blocks and convolutional gated linear unit(CGLU)modules to improve performance. In 2024,Du et al.[7]built an improved YOLOv5-based model named Lightweight GhostConv network to improve detection effectiveness for small-scale objects. They replaced the original PANet with BiFPN to enable multi-level feature fusion and introduced coordinate attention(CA)modules into the detection layers to strengthen spatial and channel-wise information modeling. These modifications markedly decrease the model's parameter volume and overall computational burden,while preserving strong accuracy performance. Zhou et al.[8]carried out structural improvements on YOLOv5to address issues including complicated backgrounds and significant object size diversity in real driving environments. By embedding AFM modules into the backbone and introducing attention-based designs to reinforce same-scale feature correlation,the model's discriminative capability was substantially enhanced.Their results show a 1.8% increase in mAP compared with the baseline YOLOv5. In 2025,Ma[9]proposed a YOLOv11-based approach that employs FasterNet as the backbone,integrates the LSQA attention mechanism within the SPPF module,replaces the C3k2 block in the neck with RSOSA modules that combine channel shuffling,re-parameterized convolutions,and one-shot aggregation strategies,and applies Soft-NMS(non-maximum suppression)to enhance candidate box precision. These improvements effectively enhance detection precision and processing speed. At present,driver behavior analysis in object detection tasks is largely founded on the YOLO series of models. By refining network architectures,activation functions,and training strategies,substantial gains in both accuracy and efficiency can be achieved. However,CNN-based approaches often suffer from excessive suppression of important features in scenarios with complex illumination or low-light conditions. Wang and Li[10]proposed a framework tailored for low-power devices,employing a staged mixed modeling strategy that combines convolutional operations and self-attention mechanisms. Local information modeling layers embedded within FFN modules capture fine-grained feature representations,while a single-head self-attention structure is adopted to reduce redundancy and computational overhead. As a result,the model is lightweight and well-suited for resource-constrained environments. On the State Farm dataset,the method presented in this work attains a top-1 accuracy of 98.5%,with an inference time as low as 4.7 ms,along with a 21.6% gain in computational efficiency and a model compression ratio of 146%. Chen et al.[11]introduced the HCEDSM model,which adopts a compact backbone integrated with EfficientViT and multi-scale attention strategies to improve the representation capability of small-object features. Additionally,a cross-scale aggregation module was designed to enable efficient feature integration,achieving 12.2 frames per second(FPS)and 94. 7% detection accuracy with only 2. 4GFLOPs on the Jetson Nano platform,while also demonstrating strong generalization performance on the LISA and BSTLD datasets.
In 2023,Baidu introduced the RT-DETR(real-time detection transformer)model[12]. It directly predicts class labels and spatial bounding information using an end-to-end real-time training paradigm,thereby eliminating dependence on anchor-based mechanisms and the NMS(non-maximum suppression)procedure. This leads to the model achieving better detection precision and maintaining more stable inference performance. However,within the domain of driver distraction behavior analysis based on object detection,RT-DETR and its variants have been applied only to a limited extent so far.
In this study,we further streamline a high-performance object detection framework and extend it to distracted driving detection scenarios. The core contribution of this work is a lightweight redesign of the RT-DETR-R18 network to alleviate the problems of excessive computational burden and limited real-time capability in distracted driving tasks. The proposed architecture is developed around three key aspects:1)A dual-backbone feature extraction strategy is introduced to alleviate the excessive parameter scale and substantial computational burden associated with the original ResNet-18 feature extractor. Two lightweight convolutional modules with dual branches are constructed to extract features,and a dynamic feature fusion module is incorporated to alleviate the growing feature discrepancy during dual-backbone processing. Through effective feature integration,this module greatly reduces the number of model parameters and the amount of computation,while preserving powerful feature representation ability. 2)To reduce the significant computational overhead caused by multi-head attention in the encoder,the original AIFI(attention-based intra-scale feature interaction)module is substituted with an additive-operation-based feature fusion module,termed AddMixer. Spatial and channel information are extracted independently,and feature fusion is efficiently achieved through depthwise separable convolution combined with additive operations.3)Drawing inspiration from the GConv design,a GConvC3 module is employed to strengthen the fusion block during the process of cross-scale feature aggregation. By replacing the original convolution module with a gated convolution mechanism,the model's representational capacity is further strengthened while computational overhead is reduced. Despite these improvements,certain limitations remain. Although the optimized model achieves reductions in parameter count and computational cost,the lightweight design has not yet been validated on edge devices due to experimental constraints. Moreover,the dataset used in this study is relatively small and may not fully assess the model's ability to perform robustly and adaptively in complex real-world driving scenarios,suggesting the necessity for further exploration.
A brief summary of the main contributions is provided below:In Section 1,the core modules and overall network space structure of RT-DETR-R18 are mainly introduced;In Section 2,specific methods for improving RT-DETR-R18 were introduced,including a dual backbone dynamic fusion network(DB-DFN),improved encoders and decoders;In the third section,the experimental results and analysis for improving the network were elaborated,including an introduction to the dataset,experimental environment,experimental indicators,as well as ablation experiments of the model and comparative experiments with other models.
1 RT-DETR-R18 Network Model
RT-DETR-R18 is an end-to-end real-time object detection framework built upon the transformer architecture. The architecture consists of three core components:a backbone feature extractor,an efficient hybrid encoder,and a transformer-based decoder coupled with a detection head. The efficient hybrid encoder comprises two principal modules,namely the attention-driven intra-scale feature interaction module and the convolutional neural network-based cross-scale feature fusion module. The overall structure is illustrated in Fig.1. Here,Conv𝑁 denotes a convolutional module with 𝑁 output channels,consisting of a convolutional layer,a batch normalization layer,and an activation function;3×3 indicates the kernel size;𝑠 represents the stride,for example,𝑠= 2 means a stride of 2,which is used for feature map downsampling.
2 Recommended Methods
Fig.2provides an overview of the general framework of the proposed method. The overall system is structured into three primary components:the DB-DFN,an enhanced encoder,and a decoder.Within the enhanced encoder,a lightweight feature fusion module,AddMixer,is introduced,which is built upon additive operations. In addition,a GC-CCFM module is incorporated by substituting the original RepC3 block with a gated replacement convolution module,thereby improving the feature fusion capability of the encoder.
2.1 DB-DFN
The feature extraction of RT-DETR-R18 adopts the ResNet-18 model[13],which has a relatively large number of parameters and high computational complexity. This study proposes a feature extraction network using DB-DFN instead of the original model to reduce computational and model parameter complexity. This study proposes the construction of a DB-DFN network based on the network concepts of CSPNet[14]and PP-HGNetv2[15]. Within the DB-DFN feature extraction network,the original image is initially processed by the HGStem module(as illustrated in Fig.3). The HGStem module integrates convolution operations with max-pooling operations to capture low-level visual representations,effectively downsampling the feature maps and this results in a reduction of the computational load on the subsequent modules. The output feature map is simultaneously input into the first branch,consisting of a C2f module and a convolutional layer,and the second branch,consisting of an HGBlock module and a depthwise separable conv(DWConv). These two branches form a parallel structure for deeper level feature extraction.The first branch consists of the CSP-C2F module(as shown in Fig.4)and an ordinary convolutional layer.The second branch consists of the HGBlock module(as shown in Fig. 5)and a depthwise separable convolutional layer.
Fig.
1
RT-DETR-R18 network structure diagram
In order to avoid feature differences between the two branches,we design a dynamic feature fusion module(DF module)to dynamically adjust the feature weights of the two paths,thereby achieving feature fusion(as shown in Fig.6). Firstly,the feature maps 𝑥₁ and 𝑥₂ extracted from the two branches are aligned with the dimensionality of the feature maps in terms of channels through a 1×1 convolutional layer.Then,the features are concatenated through a 3 × 3convolutional layer. The Sigmoid serves as a dynamic weight ω. The split module segments the feature maps into 2 components,𝜔₁and 𝜔₂,based on the number of channels in the feature map output. Subsequently,the two dynamic coefficients are applied to modulate the input features of both branches on a channel-wise basis through a 1×1 convolutional layer. Define two learnable parameters,p₁and p₂,that are updated during training to scale the two feature channels and limit the two parameters to [ 0,1 ] to ensure reasonable parameter sizes. Multiply the weighted dynamic weights with the defined learnable parameters separately,and then add the two parts together to form the final fusion output. The input-output relationship is shown in Eqs.(1)-(4). The dynamic feature fusion module adaptively applies weighted integration to dual input features,along with learnable parameter alignment and an enriched feature representation,which maintains the essential information of the original features while emphasizing salient cues,thereby improving the model's flexibility in handling diverse feature characteristics.
Fig.
2
Improved RT-DETR-R18 structure diagram
Fig.
3
HGStem module
Fig.
4
CSP-C2F module
Fig.
5
HGBlock module
Fig.
6
Dynamic fusion module
(1)
(2)
(3)
(4)
2.2 Lightweight Feature Fusion Module AddMixer
To reduce the computational overhead of the original AIFI module,this study introduces the AddMixer(Fig.7)based on CAS-ViT[16]. As shown in Fig.7,the AddMixer module is applied to the S5-level feature map generated by the DB-DFN network for feature extration. The encoded features are projected via a linear layer and split into 𝐾(key),𝑄(query),and 𝑉(value)branches. Adopting an additive strategy instead of standard self-attention,these branches significantly reduce computational overhead.
The AddMixer module extracts features through spatial operation and channel operation,and uses deep separable convolution and additive operation to complete feature fusion. Through efficient feature enhancement and additive fusion strategies,the AddMixer module employs deep convolutional operations to optimize computational load and parameter scale,ensuring accurate driving behavior classification in complex scenarios while enhancing real-time detection performance.
Fig.
7
AddMixer module
2.3 GC-CCFM
GConv[17]is a convolutional module integrating a gating mechanism. We use the idea of this module to improve the CCFM module of the original network and to introduce a lightweight cross scale feature fusion module GC-CCFM,as demonstrated in Fig. 8.Initially,the input undergoes normalization and is then divided into two separate branches,one dedicated to feature extraction and the other to the gating process. The concepts of pointwise convolution and channel-wise convolution are incorporated into the feature extraction pathway to combine local spatial information with cross-channel feature representations,provide rich features for the gating mechanism,and reduce the number of convolutional layer parameters.The sigmoid activation function is introduced into the gating pathway after point-by-point convolution for cross channel feature interaction to generate gating factors with a range of [0,1] to represent the feature importance of different positions and channel features,so as to enhance or suppress features. Subsequently,the characteristic path and the gating factor are multiplied pixel by pixel,and the multiplication results are fused by a 1×1 point-by-point convolution.At this time,the features of high weight areas are amplified,and the features of low weight areas are suppressed,highlighting the driving behavior information that the network focuses on. Finally,the residual structure is introduced to further ensure the integrity of features.
Fig.
8
GConv module
3 Experiment
3.1 Dataset
This study adopts the State Farm Distracted Driver Detection dataset—commonly used in image classification-based distracted driving detection research—to facilitate comparison of our results with those reported in related work. This dataset was introduced in the 2016 Kaggle image classification competition[18]. The dataset image is an RGB image with a resolution of 640×480 pixels,and the shooting angle is above the copilot,which can completely capture the driver's actions. The dataset contains 10types of distracted driving behaviors,including safe driving,talking on the phone,typing,adjusting the radio,drinking water,talking with the back row,and taking things from the back row. To ensure a more balanced distribution of distracted behavior categories,reduce model complexity,make the model more focused on identifying specific distracted actions,and adhere to the design standards necessary for subsequent real-time detection systems. In this paper,the category of safe driving is removed,and an average of 10 760 pictures is selected from the remaining nine categories. This study selects a subset of images from the original dataset,including 1 445images of right-hand phone calls,1 308 images of right-hand texting,1 661 images of left-hand phone calls,1 633 images of left-hand texting,936 images of adjusting the radio,1 011 images of drinking,1 008 images of talking to the backseat,842 images of reaching behind,and 916 images of applying makeup. The dataset is labeled using Labeling,and the data is finally divided into training,validation,and test subsets with a ratio of 7 ∶ 2 ∶ 1. In order to strengthen both the model's generalization and its resilience,multiple data augmentation strategies were applied to the training dataset prior to initiating the training process. These included hue adjustment,saturation adjustment,brightness adjustment,and horizontal flipping. Specifically,the hue adjustment factor was set to 0.015,the saturation was modified to0. 7,brightness was adjusted to 0. 4,and the probability of horizontal flipping was 0.5.
3.2 Environment
In this study,the experimental environment used Windows 10 as the operating system,and the CPU is an Intel(R)Xeon(R)Platinum 8255C with 12 vCPUs,operating at 2. 50 GHz,the GPU is RTX3080 10G. the experimental settings in this study are configured as follows:the training round is 100rounds,the batch size was chosen to be 8,the image size is 640 × 640,and the AdamW optimizer was utilized,starting with a learning rate of 0.000 1,the weight attenuation coefficient is 0.000 1,the query object of the first 300 features initialization decoder is selected. The image is inverted,cut,flipped,and other operations for data enhancement processing before training.
3.3 Main Model Performance Evaluation Index
In order to conduct a comprehensive evaluation of the performance of the lightweight,enhanced RT-DETR-R18 model,a set of evaluation metrics is employed,including average precision(AP),recall(𝑅),model parameter size(Params),mean average precision(mAP),gigafloating-point operations per second(GFLOPs),and frames per second(FPS).By integrating these metrics,both detection effectiveness and computational efficiency can be comprehensively evaluated[19].
Precision(P)measures the fraction of correctly predicted positive instances among all positive predictions. It is computed using Eq.(5),where TP is defined as the number of samples correctly classified as positive,and FP is defined as the number of samples incorrectly classified as positive.
(5)
Average precision( AP )measures detection performance by calculating the area under the precision-recall(PR)curve across varying confidence thresholds,as defined in Eq.(6). Mean average precision(mAP)is then derived by averaging the AP scores over all target categories,as shown in Eq.(7).
(6)
(7)
Recall(R)is used to evaluate how effectively the model identifies true positive instances among the entire set of actual positive samples. It is calculated using Eq.(8),where FN refers to the set of ground-truth positive samples that are missed by the model during detection.
(8)
3.4 Experimental Results and Analysis
To assess the rationality of the selected number of training epochs,this study reports the mAP50,mAP50-95,and the classification loss curves on the training set,as presented in Fig. 9. As observed from these curves,the model's convergence rate starts to decelerate at approximately 20 epochs,while extending the training process to 100 epochs enables the model to reach a stable and fully converged state.
Fig.
9
mAP50,mAP50-95,cls_loss curve of the training set
To visually and clearly compare the performance of the improved model with that of the baseline model across key evaluation metrics,in this section,the training-stage mAP50 curves for both models are illustrated.
Fig.10 presents the mAP50 curves of both the baseline and the enhanced models over 100 training epochs. Experimental observations show that the baseline model learns features more quickly in the early stages of training. In contrast,owing to the lightweight design,the proposed model shows relatively limited feature representation capacity at the early stage. As training progresses into the intermediate phase,both models demonstrate accelerated convergence behavior. In the later training stage,the mAP50 of the improved model gradually exceeds that of the baseline and maintains a modest but consistent performance gain. These observations suggest that the proposed lightweight optimization strategy does not degrade the model's classification accuracy;Instead,it yields a slight improvement,thereby contributing to overall performance enhancement.
3.4.1 Real time verification of algorithm
To analyze the real-time inference capability of the proposed lightweight model,FPS-based performance testing is conducted on both the improved method and the RT-DETR-R18 model. The associated detection results can be observed in Fig.11.The original model exhibits an overall inference latency of 6.69 ms,achieving an FPS of 173.7,whereas the proposed approach reduces the latency to5.4 ms and increases the FPS to 208.5.
3.4.2 Ablation experiment
To isolate and assess the effect of each proposed component within a consistent experimental framework,ablation studies are conducted on both the validation and test sets with RT-DETR-R18 serving as the baseline architecture. The performance of the enhanced method is quantitatively assessed,and Table 1 offers an overview of the corresponding ablation results. As indicated by the results,the proposed lightweight optimization strategy not only decreases the model's parameter count and computational complexity but also preserves classification accuracy,accelerates inference speed,and substantially improves real-time performance,thereby providing a solid basis for deployment on edge devices.
Fig.
10
mAP50 contrast curve during training process
Fig.
11
The experimental results diagram of FPS
Table
1
Results of ablation experiment
Note:where“√”means adding modules and “-” means not adding modules;"Val"and"Test"represent the experimental data from the validation set and test set,respectively.
3.4.3 Compared with existing methods
To offer a thorough evaluation of the strengths of the presented model,comparative experiments are performed by evaluating against several representative algorithms,with a summary of the results provided in Table 2. The experimental analysis shows that,when compared with DenseNet[20]and ResNet[13],the two mainstream convolutional neural network models,although exhibiting slightly higher computational complexity,achieve notably superior performance in terms of mAP50,Recall,and FPS. In comparison with SSD[21]and YOLOv5m[22],the improved model proposed in this study also outperforms these classical object detection approaches[23-24]across multiple evaluation metrics. Furthermore,when analyzing the baseline model alongside the two enhanced variants,it can be observed that although the recall of the proposed network is marginally lower than that of the baseline,it demonstrates clear advantages in mAP50,mAP50-95,parameter count,computational cost,and inference speed. These results confirm that the three strategies introduced in this work effectively enhance overall model performance,particularly with respect to real-time efficiency.
Table
2
Comparison of different models
The primary objective of this work is to reduce the model complexity while preserving high accuracy.This study conducts a comparison of the detection effects in the self-built test set. In Fig.12,YOLOv5m has a certain degree of false detections and missed detections,and both the RT-DETR-R18 model and its improved version are capable of accurately detecting target behavior.
Fig.
12
Detection results across various models
However,the improved model benefits from the efficient feature representation and integration provided by the DB-DFN network,achieving higher detection confidence. At the same time,based on the average experimental results obtained from multiple experiments,a joint statistical diagram of parameter quantities and FPS for different models was drawn,as shown in Fig.13. It can be readily observed that the enhanced model is positioned closer to the upper-right region of the coordinate space,which means that compared to other models,it has a higher FPS and lower parameter count,and has successfully completed the task of lightweighting the model.
Fig.
13
Statistics of different model parameter counts and FPS
4 Conclusions
In this paper,for the application of distracted driving behavior detection,the lightweight improvement strategy proposed for RT-DETR-R18 not only ensures the accuracy of distracted driving behavior classification,but also realizes the optimization of model parameters and calculational cost to a large extent. However,the method proposed in this article still needs further improvement in complex scene conditions such as low light environments and multi person interactions inside the car. So future research will concentrate on integrating different distracted driving datasets,integrating dark light enhancement networks,integrating physiological signals and vehicle perception data,etc.,to further enhance the robustness of the network across challenging conditions and broaden its applicability to diverse scenarios.
Conflict of Interest
The authors declare no potential conflict of interest.
Consent for publication
Informed consent for publication was obtained from all participants.