zippy/samples/llm-generated/2110.15534_generated.txt

Abstract Predicting linearized Abstract Meaning Rep- resentation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the de- sirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization. Introduction The task of Abstract Meaning Representation (AMR) parsing translates a natural sentence into a rooted directed acyclic graph capturing the se- mantics of the sentence, with nodes representing concepts and edges representing their relations (Ba- narescu et al., 2013). Recent works utilizing pre- trained encoder-decoder language models show great improvements in AMR parsing results (Xu et al., 2020; Bevilacqua et al., 2021). These ap- proaches avoid explicit modeling of the graph struc- ture. Instead, they directly predict the linearized AMR graph treated as free text. While the use of pre-trained Transformer encoders is widely ex- tended in AMR parsing, the use of pre-trained Transformer decoders is recent and has shown to be very effective, maintaining current state-of-the-art results (Bevilacqua et al., 2021). These approaches however lack certain desirable properties. There are no structural guarantees of graph well-formedness, i.e. the model may predict strings that can not be decoded into valid graphs, and post-processing is required. Furthermore, pre- dicting AMR linearizations ignores the implicit alignments between graph nodes and words, which provide a strong inductive bias and are useful for downstream AMR applications (Mitra and Baral, 2016; Liu et al., 2018; Vlachos et al., 2018; Kapa- nipathi et al., 2021; Naseem et al., 2021). On the other hand, transition-based AMR parsers (Wang et al., 2015; Ballesteros and Al-Onaizan, 2017a; Astudillo et al., 2020; Zhou et al., 2021) operate over the tokens of the input sentence, gen- erating the graph incrementally. They implicitly model graph structural constraints through transi- tions and yield alignments by construction, thus guaranteeing graph well-formedness.1 However, it remains unclear whether explicit modeling of structure is still beneficial for AMR parsing in the presence of powerful pre-trained language models and their strong free text generation abilities. In this work, we integrate pre-trained sequence- to-sequence (seq-to-seq) language models with the transition-based approach for AMR parsing, and explore to what degree they are complementary. To fully utilize the generation power of the pre-trained language models, we propose a transition system with a small set of basic actions – a generaliza- tion of the action-pointer transition system of Zhou et al. (2021). We use BART (Lewis et al., 2019) as our pre-trained language model, since it has shown significant improvements in linearized AMR gen- eration (Bevilacqua et al., 2021). Unlike previous approaches that directly fine-tune the model with linearized graphs, we modify the model structure to work with our transition system, and encode parser states in BART’s attention mechanism (Astudillo et al., 2020; Zhou et al., 2021). We also explore dif- ferent vocabulary strategies for action generation. These changes convert the pre-trained BART to a transition-based parser where graph constraints and alignments are internalized. We provide a detailed comparison with top- performing AMR parsers and perform ablation ex- periments showing that our proposed transition sys- tem and BART modifications are both necessary to achieve strong performance. Although BART has great language generation capacity, it still benefits fromstate encoding with hard attention, and can efficiently learn structural output. Our model establishes a new state of the art for AMR 2.0 while maintaining graph well-formedness guarantees and producing built-in alignments. Conclusion We explore the integration of pre-trained sequence- to-sequence language models and transition-based approaches for AMR parsing, with the purpose of retaining the de- sirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization. Introduction The task of Abstract Meaning Representation (AMR) parsing translates a natural sentence into a rooted directed acyclic graph capturing the se- mantics of the sentence, with nodes representing concepts and edges representing their relations (Ba- narescu et al., 2013). Recent works utilizing pre- trained encoder-decoder language models show great improvements in AMR parsing results (Xu et al., 2020; Bevilacqua et al., 2021). These ap- proaches avoid explicit modeling of the graph struc- ture. Instead, they directly predict the linearized AMR graph treated as free text.