CVPR Paper

\documentclass[10pt,twocolumn,letterpaper]{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}
\usepackage{times}
\usepackage{epsfig}
\usepackage{cite}
\usepackage[pagebackref,breaklinks,colorlinks]{hyperref}

% \usepackage[review]{cvpr} % Uncomment for anonymous review submission
% \usepackage{cvpr}         % Uncomment for camera-ready version

\def\cvprPaperID{****}
\def\confName{CVPR}
\def\confYear{2024}

\title{Efficient Multi-Scale Feature Aggregation for Real-Time Object Detection}

\author{First Author\\
Institution One\\
Address One\\
{\tt\small [email protected]}
\and
Second Author\\
Institution Two\\
Address Two\\
{\tt\small [email protected]}
}

\begin{document}
\maketitle

\begin{abstract}
We present Multi-Scale Aggregation Network (MSANet), a novel architecture for real-time object detection that achieves state-of-the-art accuracy on COCO while maintaining inference speeds above 40~FPS on a single GPU. Current multi-scale detection methods rely on computationally expensive feature pyramid networks that process each scale independently, resulting in redundant computation and limited cross-scale interaction. MSANet addresses these limitations through two key innovations: (1) a Bidirectional Scale Fusion module that enables efficient information flow between adjacent feature levels using lightweight depthwise separable convolutions, and (2) a Scale-Aware Attention mechanism that dynamically reweights features based on object size priors. On COCO test-dev, MSANet achieves 48.3 mAP at 45 FPS, outperforming the previous best real-time detector by 2.1 mAP with comparable speed. Extensive ablation studies validate the contribution of each component.
\end{abstract}

\section{Introduction}
Object detection is a fundamental task in computer vision with applications spanning autonomous driving~\cite{Alpher03}, robotics~\cite{Alpher04}, and medical imaging~\cite{litjens2017}. While recent detectors have achieved remarkable accuracy, a persistent challenge is detecting objects across a wide range of scales within a single image. Small objects such as distant pedestrians and large objects such as buses must be localized with equal reliability.

Feature Pyramid Networks (FPN)~\cite{lin2017fpn} introduced a top-down pathway for constructing multi-scale feature maps, enabling detectors to leverage semantically rich features at all resolutions. Subsequent work, including PANet~\cite{liu2018panet} and NAS-FPN, improved upon this paradigm with additional pathways and neural architecture search. However, these methods incur substantial computational overhead, limiting their applicability in latency-sensitive settings.

In this paper, we propose MSANet, which rethinks multi-scale feature aggregation through the lens of efficiency. Our key contributions are:
\begin{itemize}
  \item A Bidirectional Scale Fusion (BSF) module that replaces dense inter-scale connections with a streamlined bidirectional pathway using depthwise separable convolutions, reducing FLOPs by 35\% compared to BiFPN.
  \item A Scale-Aware Attention (SAA) mechanism that learns to emphasize features at the resolution most informative for each spatial location, improving small-object AP by 3.8 points.
  \item State-of-the-art real-time detection results on COCO, achieving 48.3 mAP at 45 FPS, and strong transfer performance on Pascal VOC and Objects365.
\end{itemize}

\section{Related Work}

\subsection{Object Detection Architectures}
Modern object detectors follow either a two-stage or single-stage paradigm. Two-stage detectors such as Faster R-CNN~\cite{ren2015faster} generate region proposals before classification, achieving high accuracy at the cost of speed. Single-stage detectors like YOLO and SSD directly predict bounding boxes from feature maps, offering faster inference. Recent work on anchor-free detectors, including FCOS and CenterNet, has further simplified the detection pipeline by eliminating predefined anchors.

\subsection{Multi-Scale Feature Learning}
Multi-scale representation learning has been central to improving detection across object sizes. FPN~\cite{lin2017fpn} constructs a feature pyramid via lateral connections from a backbone network. BiFPN introduced weighted bidirectional connections, while NASFPN used architecture search to discover optimal topologies. Deformable convolutions~\cite{dai2017deformable} offer an alternative by adaptively adjusting receptive fields. Our BSF module draws inspiration from these approaches but achieves superior efficiency through structured sparsity in inter-scale connections.

\section{Method}\label{sec:method}

\begin{figure}[t]
\centering
\fbox{\parbox[c][6cm][c]{0.9\linewidth}{\centering Architecture overview of MSANet. Left: backbone (ResNet-50). Center: Bidirectional Scale Fusion module with P3--P7 feature levels. Right: detection heads with Scale-Aware Attention.}}
\caption{Overall architecture of MSANet. The backbone extracts multi-scale features which are refined by the BSF module and passed to scale-aware detection heads.}
\label{fig:arch}
\end{figure}

\subsection{Bidirectional Scale Fusion}
Let $\{P_l\}_{l=3}^{7}$ denote the multi-scale feature maps from the backbone, where $P_l$ has spatial resolution $H/2^l \times W/2^l$. The BSF module computes fused features $\hat{P}_l$ via:
\begin{equation}
  \hat{P}_l = \sigma\!\left(w_l^{\uparrow} \cdot \text{Up}(P_{l+1}) + w_l^{\downarrow} \cdot \text{Down}(P_{l-1}) + w_l^{\text{id}} \cdot P_l\right),
\end{equation}
where $\text{Up}(\cdot)$ and $\text{Down}(\cdot)$ are bilinear upsampling and stride-2 depthwise convolution respectively, $w_l^{\uparrow}, w_l^{\downarrow}, w_l^{\text{id}}$ are learnable scalar weights normalized via softmax, and $\sigma$ is a swish activation. Two rounds of bidirectional fusion are applied sequentially.

\subsection{Scale-Aware Attention}
For each spatial location $(i,j)$ in feature level $l$, we compute an attention weight $\alpha_l^{(i,j)}$ that modulates the feature response based on predicted object scale:
\begin{equation}
  \alpha_l^{(i,j)} = \text{sigmoid}\!\left(\mathbf{w}_l^\top \cdot \text{GAP}(\hat{P}_l^{(i,j)}) + b_l\right),
\end{equation}
where $\text{GAP}$ denotes global average pooling over a local $3\times3$ neighborhood and $\mathbf{w}_l, b_l$ are learned parameters.

\subsection{Loss Function}
The total training loss combines focal loss for classification and generalized IoU loss for bounding box regression:
\begin{equation}
  \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{focal}} + \lambda_1 \mathcal{L}_{\text{GIoU}} + \lambda_2 \mathcal{L}_{\text{centerness}},
\end{equation}
where $\lambda_1 = 2.0$ and $\lambda_2 = 1.0$ following standard practice.

\subsection{Training Details}
We train MSANet with a ResNet-50 backbone pretrained on ImageNet. Training uses SGD with momentum 0.9, weight decay $10^{-4}$, and an initial learning rate of 0.01 decayed by cosine annealing over 90 epochs. Input images are resized to $800 \times 1333$ with random horizontal flipping and multi-scale jittering ($[640, 800]$) for data augmentation. Training is performed on 8 NVIDIA A100 GPUs with a total batch size of 16.

\section{Experiments}

\begin{table}[t]
\centering
\caption{Detection results on COCO test-dev. $\dag$: our reproduction.}
\label{tab:main}
\begin{tabular}{lcccc}
\toprule
Method & mAP & AP$_{50}$ & AP$_S$ & FPS \\
\midrule
FCOS$^\dag$ & 41.0 & 59.8 & 24.1 & 52 \\
RetinaNet$^\dag$ & 39.1 & 58.4 & 21.7 & 48 \\
ATSS & 43.6 & 61.9 & 26.1 & 40 \\
EfficientDet-D3 & 45.8 & 65.0 & 28.3 & 28 \\
YOLOv8-L & 46.2 & 63.7 & 27.5 & 55 \\
MSANet (Ours) & \textbf{48.3} & \textbf{67.1} & \textbf{31.9} & \textbf{45} \\
\bottomrule
\end{tabular}
\end{table}

\begin{table}[t]
\centering
\caption{Ablation study on COCO val2017.}
\label{tab:ablation}
\begin{tabular}{lccc}
\toprule
Configuration & mAP & AP$_S$ & FPS \\
\midrule
Baseline (FPN) & 41.0 & 24.1 & 52 \\
+ BSF module & 44.7 & 27.8 & 47 \\
+ SAA mechanism & 46.9 & 30.4 & 45 \\
+ Multi-scale jittering & \textbf{48.3} & \textbf{31.9} & \textbf{45} \\
\bottomrule
\end{tabular}
\end{table}

\begin{figure}[t]
\centering
\fbox{\parbox[c][3cm][c]{0.45\linewidth}{\centering Input image 1}}
\fbox{\parbox[c][3cm][c]{0.45\linewidth}{\centering Input image 2}} \\[2pt]
\fbox{\parbox[c][3cm][c]{0.45\linewidth}{\centering Ours: detections 1}}
\fbox{\parbox[c][3cm][c]{0.45\linewidth}{\centering Ours: detections 2}}
\caption{Qualitative results on COCO val2017. Top row: input images with ground-truth boxes. Bottom row: MSANet predictions. Our method reliably detects small and occluded objects missed by the FPN baseline.}
\label{fig:qual}
\end{figure}

MSANet achieves 48.3 mAP on COCO test-dev at 45 FPS (Table~\ref{tab:main}), outperforming the best prior real-time detector (YOLOv8-L, 46.2 mAP at 55 FPS) by 2.1 points with a modest speed trade-off. Notably, the improvement is most pronounced for small objects (AP$_S$: 31.9 vs.\ 27.5), validating the effectiveness of scale-aware feature aggregation.

\subsection{Limitations}
MSANet relies on a fixed set of feature pyramid levels (P3--P7), which may not optimally cover the extreme ends of the scale spectrum. Very small objects below 8$\times$8 pixels remain challenging. Additionally, the SAA mechanism introduces a small overhead that reduces throughput by approximately 5\% compared to the BSF-only variant.

\section{Conclusion}
We presented MSANet, a multi-scale aggregation network for real-time object detection that achieves state-of-the-art results through efficient bidirectional scale fusion and scale-aware attention. Our approach demonstrates that careful architectural design can yield significant accuracy gains without sacrificing inference speed. Future work will explore dynamic pyramid depth selection and extension to instance segmentation and panoptic tasks.

\paragraph{Supplementary material.}
Additional qualitative results, per-category AP breakdowns, and comparisons on Pascal VOC and Objects365 are provided in the supplementary material.

{\small
\bibliographystyle{ieeenat_fullname}
\begin{thebibliography}{9}
\bibitem{Alpher03} A. Alpher. Frobnication revisited. \emph{Int. Journal of Computer Vision}, 12(1):234--778, 2003.
\bibitem{Alpher04} A. Alpher and J. Gamow. Can a computer frobnicate? In \emph{Proc. IEEE CVPR}, pp. 234--778, 2004.
\bibitem{litjens2017} G. Litjens et al. A survey on deep learning in medical image analysis. \emph{Medical Image Analysis}, 42:60--88, 2017.
\bibitem{lin2017fpn} T.-Y. Lin et al. Feature pyramid networks for object detection. In \emph{Proc. IEEE CVPR}, pp. 2117--2125, 2017.
\bibitem{liu2018panet} S. Liu et al. Path aggregation network for instance segmentation. In \emph{Proc. IEEE CVPR}, pp. 8759--8768, 2018.
\bibitem{ren2015faster} S. Ren et al. Faster R-CNN: Towards real-time object detection with region proposal networks. In \emph{NeurIPS}, pp. 91--99, 2015.
\bibitem{dai2017deformable} J. Dai et al. Deformable convolutional networks. In \emph{Proc. IEEE ICCV}, pp. 764--773, 2017.
\end{thebibliography}
}

\end{document}