TY - GEN
T1 - Integrating Visual and Textual Affective Descriptors for Sentiment Analysis of Social Media Posts
AU - Dai, Shuanglu
AU - Man, Hong
N1 - Publisher Copyright:
© 2018 IEEE.
PY - 2018/6/26
Y1 - 2018/6/26
N2 - Social media posts often contain a mixture of images and texts. This paper proposes an affective visual descriptor and an integrated visual-textual classification method for sentiment analysis in social media. Firstly a set of affective visual features is explored based on the theory of psychology and art. Secondly, a structured forest is proposed to generate bag of affective words (BoAW) from the joint distribution of ANP. The generated BoAW provides basic "visual cues" for sentiment analysis. Then a set of sentiment part (SSP) feature is introduced to integrate the visual and textual descriptors on multiple statistic manifolds. Multi-scale sentiment classification is finally applied through metric learning on the manifold kernels. In the proposed method, the re-trained class-activation map (CAM) on ILSVRC 2014 is applied and re-trained on an Adjective-Noun-Pair (ANP) labelled affective visual data set. The global average pooling (GAP) layer of CAM is used for discriminative localization, and the fully-connected layer is able to generate objective visual descriptors. 300 tweets with mixed images and texts are manually labelled and evaluated. The proposed structured forest is evaluated on ANP labelled image data set. Promising experimental results have been obtained, which shows the effectiveness of the proposed method for sentiment analysis on social media posts.
AB - Social media posts often contain a mixture of images and texts. This paper proposes an affective visual descriptor and an integrated visual-textual classification method for sentiment analysis in social media. Firstly a set of affective visual features is explored based on the theory of psychology and art. Secondly, a structured forest is proposed to generate bag of affective words (BoAW) from the joint distribution of ANP. The generated BoAW provides basic "visual cues" for sentiment analysis. Then a set of sentiment part (SSP) feature is introduced to integrate the visual and textual descriptors on multiple statistic manifolds. Multi-scale sentiment classification is finally applied through metric learning on the manifold kernels. In the proposed method, the re-trained class-activation map (CAM) on ILSVRC 2014 is applied and re-trained on an Adjective-Noun-Pair (ANP) labelled affective visual data set. The global average pooling (GAP) layer of CAM is used for discriminative localization, and the fully-connected layer is able to generate objective visual descriptors. 300 tweets with mixed images and texts are manually labelled and evaluated. The proposed structured forest is evaluated on ANP labelled image data set. Promising experimental results have been obtained, which shows the effectiveness of the proposed method for sentiment analysis on social media posts.
KW - Affective Image Description
KW - Metric learning
KW - Set of Sentiment Parts
KW - Visual sentiment analysis
UR - http://www.scopus.com/inward/record.url?scp=85050083621&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85050083621&partnerID=8YFLogxK
U2 - 10.1109/MIPR.2018.00011
DO - 10.1109/MIPR.2018.00011
M3 - Conference contribution
AN - SCOPUS:85050083621
T3 - Proceedings - IEEE 1st Conference on Multimedia Information Processing and Retrieval, MIPR 2018
SP - 13
EP - 18
BT - Proceedings - IEEE 1st Conference on Multimedia Information Processing and Retrieval, MIPR 2018
T2 - 1st IEEE Conference on Multimedia Information Processing and Retrieval, MIPR 2018
Y2 - 10 April 2018 through 12 April 2018
ER -