HSTforU: Anomaly Detection in Aerial and Ground-based Videos with Hierarchical Spatio-Temporal Transformer for U-net

Anomaly detection is to identify abnormal events against normal ones within surveillance videos mainly collected in ground-based settings. Recently, the need for drone data processing is rapidly growing as drones find new applications. However, as most aerial videos collected by flying drones contain moving backgrounds and others, it is necessary to deal with their spatio-temporal features in detecting anomalies. This study presents a transformer-based video anomaly detection method whereby we investigate a challenging aerial dataset and three benchmark ground-based datasets. The encoder of our U-net has a four-stage pyramid transformer structure, and each stage has a link to a corresponding spatio-temporal transformer stage. Then, this transformer produces hierarchical feature maps that are conveyed to the decoder as skip connections. Extensive evaluations including several ablation studies suggest that this network outperforms the state-of-the-art on Drone-anomaly dataset and three benchmark datasets. We expect the proposed transformer for U-net can find diverse applications in the video processing area. Code and model are available at https://vt-le.github.io/HSTforU/.