Logo image
Back
SMSF: A signal-driven multimodal sequence fusion framework for sales prediction in short-form video commerce
Journal article   Peer reviewed

SMSF: A signal-driven multimodal sequence fusion framework for sales prediction in short-form video commerce

Xingpeng Xu, Xiapu Liu, Qingfeng Zeng, Songqiao Han and Weiguo Fan
Decision Support Systems, Vol.208, 114719
09/2026
DOI: 10.1016/j.dss.2026.114719

View Online

Abstract

The rapid rise of short-form video commerce has posed new challenges for accurate sales prediction. Previous studies often overlook the sequential features and temporal structure of modality-specific data. Our main contribution is to develop a signal-driven multimodal sequence fusion (SMSF) framework grounded in signaling theory. SMSF employs a hierarchical attention-based fusion strategy that combines self-attention, pairwise cross-modal attention, and joint attention to enhance sequential modeling and enable effective representation of complex and heterogeneous multimodal signals beyond conventional fusion methods. All enhanced signals are fused via dynamic gating for adaptive modality selection, followed by a Mixture-of-Experts module that improves semantic coordination. We use a multimodal dataset of 8084 short-form sales videos from Douyin (TikTok in China) for experiments. The results indicate that SMSF consistently outperforms benchmark models. Ablation experiments further reveal that online comments contribute most to prediction performance. Its signal processing and enhancement methods provide a potential paradigm for multimodal tasks across different scenarios or platforms. •It is the first study for sales prediction in short-form video commerce.•We develop a state-of-the-art Signal-driven Multimodal Sequence Fusion framework.•SMSF introduces hierarchical attention combining self, cross, and joint attention.•SMSF employs dynamic gating fusion and Shared Expert Sparse Mixture-of-Experts.•SMSF provides a paradigm for multimodal tasks across different scenarios or platforms.
 Dynamic gating Mixture-of-experts Multimodal sequence fusion Sales prediction Short-form video commerce Signaling theory

Details

Metrics

1 Record Views
Logo image