Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov submitted 'Transformer-XL: Attentive Language Models beyond a Fixed-Length Context' on January 9, 2019. The paper proposes Transformer-XL, a neural architecture using a segment-level recurrence mechanism and a positional encoding scheme to model dependency beyond a fixed-length context.