Room P3.10, Mathematics Building

Yves Lepage
Yves Lepage, Waseda University

Measuring the ability of LLMs in analogy between sentences

This presentation will report on a number of studies conducted in my laboratory, mainly on the ability of large language models to solve analogies between sentences. We show that the task does not necessarily require a large number of parameters for fairly formal analogies. By defining constraints and introducing a measure of analogy quality, we are able to collect a large number of more or less robust analogies between sentences. We show that analogy quality prevails over quantity in fine-tuning and demonstrate this in machine translation tasks with low-resource languages. We also mention experiments on solving arithmetic problems by answer verification (binary classification), multiple-choice questions (selection from candidate answers), and generation of answer. The results clearly show that binary classification is the easiest task, and that there is little correlation between the three tasks.