Improving Testsuites via Instrumentation
Norbert Broeker
Abstract
This paper explores the usefulness of a technique from software engineering, namely code instrumentation, for the development of large-scale natural language grammars. Information about the usage of grammar rules in test sentences is used to detect untested rules, redundant test sentences, and likely causes of overgeneration. Results show that less than half of a large-coverage grammar for German is actually tested by two large testsuites, and that 10-30% of testing time is redundant. The methodology applied can be seen as a re-use of grammar writing knowledge for testsuite compilation.
Create a lesson
Related papers
SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification
Yulu An, Yifei Feng, Xiangyang Yin
Balancing hierarchical representation and quantization: A performance trade-off in text embeddings
Kai Ponel, Marco Polignano, Erasmo Purificato et al.
Persona Perspectives: Adopting Viewpoints Reorganizes Representations in LLMs
Austin C. Kozlowski, Avital Mintz, Madison Lee
Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
Peidi Dong, Yunze Xiao, Yu-Lin Tsai et al.
LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models
Lele Yang, Muxi Diao, Zhonghao Yan et al.
SAUCE: Summary Analysis Using Conversation Entailment
Man-Ling Sung, Hemanth Kandula, Jeff Ma et al.