REVIEW 1 major objections 1 minor 53 references
A hierarchical transformer fuses RGB and event data to reduce fusion noise in object detection.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-30 08:05 UTC pith:C4TB66M4
load-bearing objection CMTFormer gives a staged RGB-event fusion design with SAM/CEM/LDFM but the mAP gains do not isolate whether those modules actually cut noise or redundancy. the 1 major comments →
CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
CMTFormer marries a transformer architecture with hierarchical cross-modal information interaction, using SAM to fuse low-level features while mitigating disparities, CEM to reinforce middle-level features with texture and edge info, LDFM to aggregate high-level clues via learnable weights, and a Spatial Prior Module for better localization, resulting in superior performance on RGB-event object detection tasks.
What carries the argument
The shallow-to-deep information interaction scheme with Shallow Alignment Module (SAM), Cross-modal Enhancement Module (CEM), and Learnable Deep Fusion Module (LDFM) that enables efficient and stable multimodal collaboration.
Load-bearing premise
The shallow-to-deep interaction scheme mitigates attribute disparities and prevents noisy or redundant integration between RGB and event data without introducing new fusion artifacts.
What would settle it
An ablation study showing that removing the hierarchical modules (SAM, CEM, LDFM) does not degrade performance on the DSEC-Detection benchmark would falsify the central claim.
If this is right
- Surpasses detection counterparts in both uni-modal and multi-modal settings on DSEC-Detection and PKU-DAVIS-SOD benchmarks.
- Prevents noisy or redundant feature integration during cross-modal fusion.
- The learnable weights in LDFM allow adaptive fusion of RGB and event clues.
- Global spatial information from the Spatial Prior Module enhances localization accuracy.
Where Pith is reading between the lines
- Similar hierarchical fusion could be applied to other sensor combinations like RGB and LiDAR.
- The approach may generalize to video object detection where temporal information is key.
- By avoiding new fusion artifacts, it could enable more reliable deployment in challenging environments like low light or fast motion.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CMTFormer, a transformer architecture for RGB-event object detection that employs a shallow-to-deep hierarchical interaction scheme. This includes the Shallow Alignment Module (SAM) for low-level fusion to reduce attribute disparities, the Cross-modal Enhancement Module (CEM) for mutual reinforcement of middle-level features using texture and edge cues, the Learnable Deep Fusion Module (LDFM) for adaptive high-level aggregation via learnable weights, and an additional Spatial Prior Module for improved localization. The central claim is that this design enables efficient multimodal collaboration without noise amplification or redundancy, leading to consistent outperformance over uni-modal and multi-modal baselines on the DSEC-Detection and PKU-DAVIS-SOD benchmarks.
Significance. If the empirical results hold under rigorous validation, the hierarchical fusion paradigm could advance RGB-event detection by addressing modality heterogeneity more systematically than prior fusion techniques. The learnable adaptive fusion in LDFM and the staged interaction represent a structured approach that may generalize to other event-based vision tasks, with the promise of reproducibility noted via planned code release.
major comments (1)
- [Experiments section] Experiments section (and associated tables/figures): The claim that SAM, CEM, and LDFM specifically mitigate attribute disparities and prevent noisy/redundant integration (as stated in the abstract and §1) rests solely on overall mAP improvements versus baselines. No ablation studies isolating each module's contribution, nor direct quantitative analyses (e.g., feature correlation, noise statistics, or redundancy metrics before/after fusion), are described to confirm the modules achieve the stated mitigation rather than other factors such as transformer capacity or training details.
minor comments (1)
- [Abstract] Abstract: No quantitative metrics, baseline names, or specific mAP values are reported, which weakens the standalone readability of the claims.
Simulated Author's Rebuttal
We thank the referee for the constructive comment on the experimental validation of our proposed modules. We address the point below and will revise the manuscript accordingly to strengthen the evidence.
read point-by-point responses
-
Referee: [Experiments section] Experiments section (and associated tables/figures): The claim that SAM, CEM, and LDFM specifically mitigate attribute disparities and prevent noisy/redundant integration (as stated in the abstract and §1) rests solely on overall mAP improvements versus baselines. No ablation studies isolating each module's contribution, nor direct quantitative analyses (e.g., feature correlation, noise statistics, or redundancy metrics before/after fusion), are described to confirm the modules achieve the stated mitigation rather than other factors such as transformer capacity or training details.
Authors: We agree that the current experiments demonstrate overall mAP gains but do not isolate the specific contributions of SAM, CEM, and LDFM through dedicated ablations or direct metrics on disparity mitigation, noise, or redundancy. While the staged design rationale (detailed in §3) and consistent outperformance over comparable baselines provide supporting context, additional targeted experiments are needed to rule out confounding factors such as model capacity. In the revised manuscript, we will add: (i) module-wise ablation tables reporting performance when each component is removed individually, (ii) capacity-controlled variants that retain transformer blocks but omit the hierarchical cross-modal interaction, and (iii) supplementary quantitative analyses (e.g., feature correlation or similarity metrics across modalities before/after fusion) together with qualitative visualizations. These additions will directly address the concern. revision: yes
Circularity Check
No circularity; empirical architecture validated on external benchmarks
full rationale
The paper proposes CMTFormer with descriptive modules (SAM, CEM, LDFM, Spatial Prior) for RGB-event fusion and supports effectiveness solely via end-to-end mAP gains on DSEC-Detection and PKU-DAVIS-SOD against uni- and multi-modal baselines. No equations, derivations, fitted parameters renamed as predictions, or load-bearing self-citations appear. The central claims reduce to experimental comparison rather than any self-referential construction, making the result self-contained against external data.
Axiom & Free-Parameter Ledger
read the original abstract
Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames. However, previous multi-modal fusion techniques typically fail to handle the inherent heterogeneity between RGB frames and event streams, thus easily leading to noise amplification or redundant feature integration during cross-modal fusion. In this paper, we propose a Cross-Modal information inTeraction transFormer, coined as CMTFormer, which hierarchically integrates RGB and event information to achieve efficient and stable multimodal collaboration. Specifically, we design a shallow-to-deep information interaction scheme. In the shallow stage, we present the Shallow Alignment Module (SAM) to achieve an efficient fusion of RGB and event low-level features, which mitigates attribute disparities and prevents noisy information. In the middle stage, we devise the Cross-modal Enhancement Module (CEM) that utilizes texture and edge information to produce mutually reinforced middle-level features. In the deep stage, we present the Learnable Deep Fusion Module (LDFM) which performs high-level information aggregation through learnable weights, thus enabling the network to adaptively fuse RGB and event clues. A Spatial Prior Module is further designed to utilize global spatial information to enhance localization accuracy. Extensive experiments are conducted on two prevalent event-based object detection benchmarks, i.e., DSEC-Detection and PKU-DAVIS-SOD. Our CMTFormer consistently surpasses the detection counterparts in both uni-modal and multi-modal settings, strongly demonstrating the effectiveness of our paradigm. Codes will be available upon publication.
Figures
Reference graph
Works this paper leans on
-
[1]
Z. Zou, K. Chen, Z. Shi, Y . Guo, J. Ye, Object detection in 20 years: A survey, Proceedings of the IEEE 111 (3) (2023) 257–276
2023
-
[2]
Zhou, A yolo-nl object detector for real-time detection, Expert Systems with Applications 238 (2024) 122256
Y . Zhou, A yolo-nl object detector for real-time detection, Expert Systems with Applications 238 (2024) 122256
2024
-
[3]
Y . Hou, Z. Ma, C. Liu, C. C. Loy, Learning lightweight lane detection CNNs by self attention distillation, in: IEEE International Conference on Computer Vision, 2019, pp. 1013–1021
2019
-
[4]
Y . Ma, T. Wang, X. Bai, H. Yang, Y . Hou, Y . Wang, Y . Qiao, R. Yang, X. Zhu, Vision-centric bev perception: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (12) (2024) 10978–10997
2024
-
[5]
W. Li, W. Yang, Y . Hou, L. Liu, Y . Liu, X. Li, Saratr-x: Toward building a foundation model for sar target recognition, IEEE Transactions on Im- age Processing 34 (2025) 869–884
2025
- [6]
-
[7]
Y . Hu, T. Delbruck, S.-C. Liu, Learning to ex- ploit multiple vision modalities by using grafted networks, in: European Conference on Computer Vision, Springer, 2020, pp. 85–101
2020
-
[8]
Chakravarthi, A
B. Chakravarthi, A. A. Verma, K. Daniilidis, C. Fermuller, Y . Yang, Recent event camera inno- vations: A survey, in: European Conference on Computer Vision, Springer, 2024, pp. 342–376
2024
-
[9]
D. Liu, Y . Fan, W. Lu, C. Liu, W. Zhang, Spiking depth: Depth estimation from sparse events with spiking neural networks, Expert Systems with Ap- plications (2025) 129977
2025
-
[10]
D. Kang, D. Kang, An adaptive learning frame- work for event-based remote eye tracking, Expert Systems with Applications (2025) 128038
2025
-
[11]
H. Cao, G. Chen, J. Xia, G. Zhuang, A. Knoll, Fusion-based feature attention gate component for vehicle detection based on event camera, IEEE Sensors Journal 21 (21) (2021) 24540–24548
2021
-
[12]
A. Tomy, A. Paigwar, K. S. Mann, A. Renzaglia, C. Laugier, Fusing event-based and rgb camera for robust object detection in adverse conditions, in: 2022 International Conference on Robotics and Automation (ICRA), IEEE, 2022, pp. 933–939
2022
-
[13]
D. Li, Y . Tian, J. Li, Sodformer: Streaming ob- ject detection with transformer using events and frames, IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (11) (2023) 14020– 14037
2023
-
[14]
Jiang, Y
Y . Jiang, Y . Wang, M. Zhao, Y . Zhang, H. Qi, Nighttime traffic object detection via adaptively integrating event and frame domains, Fundamen- tal Research (2023)
2023
-
[15]
J. Li, S. Dong, Z. Yu, Y . Tian, T. Huang, Event- based vision enhanced: A joint detection frame- work in autonomous driving, in: 2019 ieee interna- tional conference on multimedia and expo (icme), IEEE, 2019, pp. 1396–1401
2019
-
[16]
Jiang, P
Z. Jiang, P. Xia, K. Huang, W. Stechele, G. Chen, Z. Bing, A. Knoll, Mixed frame-/event-driven fast pedestrian detection, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 8332–8338
2019
- [17]
-
[19]
Gehrig, D
D. Gehrig, D. Scaramuzza, Low-latency automo- tive vision with event cameras, Nature 629 (8014) (2024) 1034–1040
2024
-
[20]
Y . Peng, Y . Zhang, P. Xiao, X. Sun, F. Wu, Better and faster: Adaptive event conversion for event-based object detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 37, 2023, pp. 2056–2064
2023
-
[21]
Gehrig, D
M. Gehrig, D. Scaramuzza, Recurrent vision transformers for object detection with event cam- eras, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 13884–13893
2023
-
[22]
Bodden, D
L. Bodden, D. B. Ha, F. Schwaiger, L. Kreuzberg, S. Behnke, Spiking centernet: A distillation- boosted spiking neural network for object detec- tion, in: 2024 International Joint Conference on Neural Networks (IJCNN), IEEE, 2024, pp. 1–9
2024
-
[23]
Schaefer, D
S. Schaefer, D. Gehrig, D. Scaramuzza, Aegnn: Asynchronous event-based graph neural networks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12371–12381
2022
-
[24]
Chitta, A
K. Chitta, A. Prakash, B. Jaeger, Z. Yu, K. Renz, A. Geiger, Transfuser: Imitation with transformer- based sensor fusion for autonomous driving, IEEE transactions on pattern analysis and machine intel- ligence 45 (11) (2022) 12878–12895
2022
-
[25]
Tulyakov, D
S. Tulyakov, D. Gehrig, S. Georgoulis, J. Erbach, M. Gehrig, Y . Li, D. Scaramuzza, Time lens: Event-based video frame interpolation, in: Pro- ceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2021, pp. 16155–16164
2021
-
[26]
P. Duan, Z. W. Wang, B. Shi, O. Cossairt, T. Huang, A. K. Katsaggelos, Guided event filter- ing: Synergy between intensity images and neu- romorphic events for high performance imaging, IEEE Transactions on Pattern Analysis and Ma- chine Intelligence 44 (11) (2021) 8261–8275
2021
-
[27]
Zhang, X
J. Zhang, X. Yang, Y . Fu, X. Wei, B. Yin, B. Dong, Object tracking by jointly exploiting frame and event domain, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13043–13052
2021
-
[28]
Gehrig, M
D. Gehrig, M. Rüegg, M. Gehrig, J. Hidalgo- Carrió, D. Scaramuzza, Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction, IEEE Robotics and Automation Letters 6 (2) (2021) 2822–2829
2021
-
[29]
Y .-F. Zuo, J. Yang, J. Chen, X. Wang, Y . Wang, L. Kneip, Devo: Depth-event camera visual odom- etry in challenging conditions, in: 2022 Interna- tional Conference on Robotics and Automation (ICRA), IEEE, 2022, pp. 2179–2185
2022
-
[30]
L. Gao, Y . Liang, J. Yang, S. Wu, C. Wang, J. Chen, L. Kneip, Vector: A versatile event- centric benchmark for multi-sensor slam, IEEE Robotics and Automation Letters 7 (3) (2022) 8217–8224
2022
-
[31]
M. Liu, N. Qi, Y . Shi, B. Yin, An attention fusion network for event-based vehicle object detection, in: 2021 IEEE International Conference on Image Processing (ICIP), IEEE, 2021, pp. 3363–3367
2021
-
[32]
L. Fan, J. Yang, L. Wang, J. Zhang, X. Lian, H. Shen, Efficient spiking neural network for rgb– event fusion-based object detection, Electronics 14 (6) (2025) 1105
2025
-
[33]
R. L. Graham, Concrete mathematics: a founda- tion for computer science, Pearson Education In- dia, 1994
1994
-
[34]
A. Z. Zhu, L. Yuan, K. Chaney, K. Daniilidis, Un- supervised event-based learning of optical flow, depth, and egomotion, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 989–997
2019
-
[35]
N. F. Chen, Pseudo-labels for supervised learning on dynamic vision sensor data, applied to object detection under ego-motion, in: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018, pp. 644–653
2018
-
[36]
A. I. Maqueda, A. Loquercio, G. Gallego, N. Gar- cía, D. Scaramuzza, Event-based vision meets 14 deep learning on steering prediction for self- driving cars, in: Proceedings of the IEEE confer- ence on computer vision and pattern recognition, 2018, pp. 5419–5427
2018
-
[37]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pat- tern recognition, 2016, pp. 770–778
2016
-
[38]
X. Zhu, W. Su, L. Lu, B. Li, X. Wang, J. Dai, Deformable detr: Deformable transform- ers for end-to-end object detection, arXiv preprint arXiv:2010.04159 (2020)
work page internal anchor Pith review Pith/arXiv arXiv 2010
-
[39]
Kinga, J
D. Kinga, J. B. Adam, et al., A method for stochas- tic optimization, in: International conference on learning representations (ICLR), V ol. 5, Califor- nia;, 2015
2015
-
[40]
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Per- ona, D. Ramanan, P. Dollár, C. L. Zitnick, Mi- crosoft coco: Common objects in context, in: Eu- ropean conference on computer vision, Springer, 2014, pp. 740–755
2014
-
[41]
S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, IEEE transactions on pattern analysis and machine intelligence 39 (6) (2016) 1137–1149
2016
-
[42]
T.-Y . Lin, P. Goyal, R. Girshick, K. He, P. Dollár, Focal loss for dense object detection, in: Proceed- ings of the IEEE international conference on com- puter vision, 2017, pp. 2980–2988
2017
-
[43]
X. Zhou, D. Wang, P. Krähenbühl, Objects as points, arXiv preprint arXiv:1904.07850 (2019)
work page internal anchor Pith review Pith/arXiv arXiv 1904
-
[44]
X. Zhu, S. Lyu, X. Wang, Q. Zhao, Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone-captured sce- narios, in: Proceedings of the IEEE/CVF interna- tional conference on computer vision, 2021, pp. 2778–2788
2021
-
[45]
C.-Y . Wang, A. Bochkovskiy, H.-Y . M. Liao, Yolov7: Trainable bag-of-freebies sets new state- of-the-art for real-time object detectors, in: Pro- ceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2023, pp. 7464–7475
2023
-
[46]
Z. Liu, H. Mao, C.-Y . Wu, C. Feichtenhofer, T. Darrell, S. Xie, A convnet for the 2020s, in: Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2022, pp. 11976–11986
2022
-
[47]
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, B. Guo, Swin transformer: Hierarchical vi- sion transformer using shifted windows, in: Pro- ceedings of the IEEE/CVF international confer- ence on computer vision, 2021, pp. 10012–10022
2021
-
[48]
Q. Su, Y . Chou, Y . Hu, J. Li, S. Mei, Z. Zhang, G. Li, Deep directly-trained spiking neural net- works for object detection, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 6555–6565
2023
-
[49]
Z. Liu, N. Yang, Y . Wang, Y . Li, X. Zhao, F.- Y . Wang, Enhancing traffic object detection in variable illumination with rgb-event fusion, IEEE Transactions on Intelligent Transportation Sys- tems (2024)
2024
-
[50]
H. Cao, Z. Zhang, Y . Xia, X. Li, J. Xia, G. Chen, A. Knoll, Embracing events and frames with hier- archical feature refinement network for object de- tection, in: European Conference on Computer Vi- sion, Springer, 2024, pp. 161–177
2024
-
[51]
Iacono, S
M. Iacono, S. Weber, A. Glover, C. Bartolozzi, To- wards event-driven object detection with off-the- shelf deep learning, in: 2018 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Sys- tems (IROS), IEEE, 2018, pp. 1–9
2018
-
[52]
YOLOv3: An Incremental Improvement
J. Redmon, A. Farhadi, Yolov3: An incremen- tal improvement, arXiv preprint arXiv:1804.02767 (2018)
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[53]
M. Liu, M. Zhu, Mobile video object detection with temporally-aware feature maps, in: Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5686–5695
2018
-
[54]
J. Li, J. Li, L. Zhu, X. Xiang, T. Huang, Y . Tian, Asynchronous spatio-temporal memory network for continuous event-based object detec- tion, IEEE Transactions on Image Processing 31 (2022) 2975–2987. 15
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.