Small human group detection and event representation based on cognitive semantics

To recognize concurrent human activities in videos, we proposed a cognitive semantics based novel event representation for small human group detection and event recognition. Given a video with human detection and tracking results, the video is firstly described by cognitive linguistic primitives, including paths, places, things, actions, and causes. Then the structural and semantic distance of things (i.e. human individuals) in the same place will be calculated, and all the similar things will be merged together to reduce the semantic social entropy with regard to the entire path. Once a group of things (i.e. a human group) is identified, its actions will be classified into atom group activities by their corresponding spatial and temporal semantics. The spatial and temporal similarity of atom group activities is examined and a probabilistic context free grammar is derived from these atom activities based on Minimum Description Length (MDL) criterion. The induced grammar rules will then be used to parse test videos represented by cognitive linguistic primitives. The proposed novel video event representation can be used to describe and recognize complex human activities, including both individual and group actions. The experimental results on the BEHAVE and Collective datasets have demonstrated the effectiveness of the proposed method.