Parsing with the Shortest Derivation
Rens Bod
Abstract
Common wisdom has it that the bias of stochastic grammars in favor of shorter derivations of a sentence is harmful and should be redressed. We show that the common wisdom is wrong for stochastic grammars that use elementary trees instead of context-free rules, such as Stochastic Tree-Substitution Grammars used by Data-Oriented Parsing models. For such grammars a non-probabilistic metric based on the shortest derivation outperforms a probabilistic metric on the ATIS and OVIS corpora, while it obtains very competitive results on the Wall Street Journal corpus. This paper also contains the first published experiments with DOP on the Wall Street Journal.
Create a lesson
Related papers
SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification
Yulu An, Yifei Feng, Xiangyang Yin
Balancing hierarchical representation and quantization: A performance trade-off in text embeddings
Kai Ponel, Marco Polignano, Erasmo Purificato et al.
Persona Perspectives: Adopting Viewpoints Reorganizes Representations in LLMs
Austin C. Kozlowski, Avital Mintz, Madison Lee
Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
Peidi Dong, Yunze Xiao, Yu-Lin Tsai et al.
LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models
Lele Yang, Muxi Diao, Zhonghao Yan et al.
SAUCE: Summary Analysis Using Conversation Entailment
Man-Ling Sung, Hemanth Kandula, Jeff Ma et al.