Computational Analogy Making - Revision history

Thiliniiw: /* Publications */

2025-01-20T07:44:40Z

‎Publications

Thiliniiw: /* Publications */

2024-08-06T21:37:28Z

‎Publications

Thiliniiw: /* Publications */

2024-08-06T21:36:22Z

‎Publications

Thiliniiw: /* Publications */

2024-08-06T21:35:44Z

‎Publications

Thiliniiw at 15:08, 10 May 2024

2024-05-10T15:08:13Z

Thiliniiw: /* Publications */

2024-03-27T11:33:38Z

‎Publications

Thiliniiw: /* Publications */

2024-03-27T11:28:44Z

‎Publications

Thiliniiw: /* Publications */

2024-03-27T11:25:03Z

‎Publications

Thiliniiw: /* Publications */

2024-03-27T11:20:09Z

‎Publications

Thiliniiw: /* Publications */

2024-03-21T21:58:11Z

‎Publications

@@ Line 28: / Line 28: @@
 == Publications ==
-'''1. On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models [https://aclanthology.org/2024.findings-eacl.31.pdf] [NEW]'''
+'''1. KnowledgePrompts: Exploring the Abilities of Large Language Models to Solve Proportional Analogies via Knowledge-Enhanced Prompting [https://arxiv.org/pdf/2412.00869] [NEW]'''
 The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in the last decade of language modeling literature. In this work we specifically look at how LLMs’ abilities to capture sentence analogies (sentences that convey analogous meaning to each other) vary with LLMs’ abilities to encode syntactic and semantic structures of sentences. Through our analysis, we find that LLMs’ ability to identify sentence analogies is positively correlated with their ability to encode syntactic and semantic structures of sentences. Specifically, we find that the LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies.

@@ Line 28: / Line 28: @@
 == Publications ==
-''1. On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models [https://aclanthology.org/2024.findings-eacl.31.pdf] '''[NEW]'''''
+'''1. On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models [https://aclanthology.org/2024.findings-eacl.31.pdf] [NEW]'''
 The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in the last decade of language modeling literature. In this work we specifically look at how LLMs’ abilities to capture sentence analogies (sentences that convey analogous meaning to each other) vary with LLMs’ abilities to encode syntactic and semantic structures of sentences. Through our analysis, we find that LLMs’ ability to identify sentence analogies is positively correlated with their ability to encode syntactic and semantic structures of sentences. Specifically, we find that the LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies.
@@ Line 34: / Line 34: @@
 [[File:Thilini_poster_EACL.png|900x900px]]
-''2. Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies? [https://arxiv.org/pdf/2308.01936.pdf Extended Version], [https://www.computer.org/csdl/magazine/ex/2023/05/10269780/1QWO61CnTS8 Abridged Version published in IEEE Intelligent Systems] '''[NEW]'''''
+'''2. Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies? [https://arxiv.org/pdf/2308.01936.pdf Extended Version], [https://www.computer.org/csdl/magazine/ex/2023/05/10269780/1QWO61CnTS8 Abridged Version published in IEEE Intelligent Systems] [NEW]'''
 A hallmark of intelligence is the ability to use a familiar domain to make inferences about a less familiar domain, known as analogical reasoning. In this article, we delve into the performance of Large Language Models (LLMs) in dealing with progressively complex analogies expressed in unstructured text. We discuss analogies at four distinct levels of complexity: lexical analogies, syntactic analogies, semantic analogies, and pragmatic analogies. As the analogies become more complex, they require increasingly extensive, diverse knowledge beyond the textual content, unlikely to be found in the lexical cooccurrence statistics that power LLMs. To address this, we discuss the necessity of employing Neuro-symbolic AI techniques that combine statistical and symbolic AI, informing the representation of unstructured text to highlight and augment relevant content, provide abstraction and guide the mapping process. Our knowledge-informed approach maintains the efficiency of LLMs while preserving the ability to explain analogies for pedagogical applications.

@@ Line 28: / Line 28: @@
 == Publications ==
-'''1. On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models [https://aclanthology.org/2024.findings-eacl.31.pdf] '''[NEW]'''''
+''1. On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models [https://aclanthology.org/2024.findings-eacl.31.pdf] '''[NEW]'''''
 The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in the last decade of language modeling literature. In this work we specifically look at how LLMs’ abilities to capture sentence analogies (sentences that convey analogous meaning to each other) vary with LLMs’ abilities to encode syntactic and semantic structures of sentences. Through our analysis, we find that LLMs’ ability to identify sentence analogies is positively correlated with their ability to encode syntactic and semantic structures of sentences. Specifically, we find that the LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies.
@@ Line 34: / Line 34: @@
 [[File:Thilini_poster_EACL.png|900x900px]]
-'''2. Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies? [https://arxiv.org/pdf/2308.01936.pdf Extended Version], [https://www.computer.org/csdl/magazine/ex/2023/05/10269780/1QWO61CnTS8 Abridged Version published in IEEE Intelligent Systems] '''[NEW]'''''
+''2. Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies? [https://arxiv.org/pdf/2308.01936.pdf Extended Version], [https://www.computer.org/csdl/magazine/ex/2023/05/10269780/1QWO61CnTS8 Abridged Version published in IEEE Intelligent Systems] '''[NEW]'''''
 A hallmark of intelligence is the ability to use a familiar domain to make inferences about a less familiar domain, known as analogical reasoning. In this article, we delve into the performance of Large Language Models (LLMs) in dealing with progressively complex analogies expressed in unstructured text. We discuss analogies at four distinct levels of complexity: lexical analogies, syntactic analogies, semantic analogies, and pragmatic analogies. As the analogies become more complex, they require increasingly extensive, diverse knowledge beyond the textual content, unlikely to be found in the lexical cooccurrence statistics that power LLMs. To address this, we discuss the necessity of employing Neuro-symbolic AI techniques that combine statistical and symbolic AI, informing the representation of unstructured text to highlight and augment relevant content, provide abstraction and guide the mapping process. Our knowledge-informed approach maintains the efficiency of LLMs while preserving the ability to explain analogies for pedagogical applications.

@@ Line 28: / Line 28: @@
 == Publications ==
-'''1. Towards efficient scoring of student-generated long-form analogies in STEM[https://ceur-ws.org/Vol-3389/ICCBR_2022_Workshop_paper_106.pdf]'''
+'''1. On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models [https://aclanthology.org/2024.findings-eacl.31.pdf] '''[NEW]'''''
-Switching from an analogy pedagogy based on comprehension to analogy pedagogy based on production raises an impractical manual analogy scoring problem. Conventional symbol-matching approaches to computational analogy evaluation focus on positive cases and challenge computational feasibility. This work presents the Discriminative Analogy Features (DAF) pipeline to identify the discriminative features of strong and weak long-form text analogies. We introduce four feature categories (semantic, syntactic, sentiment, and statistical) used with supervised vector-based learning methods to discriminate between strong and weak analogies. Using a modestly sized vector of engineered features with SVM attains a 0.67 macro F1 score. While a semantic feature is the most discriminative, out of the top 15 discriminative features, most are syntactic. Combining these engineered features with an ELMo-generated embedding still improves classification relative to an embedding alone. While an unsupervised K-Means clustering-based approach falls short, similar hints of improvement appear when inputs include the engineered features used in supervised learning.
+The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in the last decade of language modeling literature. In this work we specifically look at how LLMs’ abilities to capture sentence analogies (sentences that convey analogous meaning to each other) vary with LLMs’ abilities to encode syntactic and semantic structures of sentences. Through our analysis, we find that LLMs’ ability to identify sentence analogies is positively correlated with their ability to encode syntactic and semantic structures of sentences. Specifically, we find that the LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies.
-[[File:Scoring of student-generated.png|600x600px]]
+[[File:Thilini_poster_EACL.png|900x900px]]
-'''2. ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models [https://aclanthology.org/2023.findings-acl.218/]'''
+A hallmark of intelligence is the ability to use a familiar domain to make inferences about a less familiar domain, known as analogical reasoning. In this article, we delve into the performance of Large Language Models (LLMs) in dealing with progressively complex analogies expressed in unstructured text. We discuss analogies at four distinct levels of complexity: lexical analogies, syntactic analogies, semantic analogies, and pragmatic analogies. As the analogies become more complex, they require increasingly extensive, diverse knowledge beyond the textual content, unlikely to be found in the lexical cooccurrence statistics that power LLMs. To address this, we discuss the necessity of employing Neuro-symbolic AI techniques that combine statistical and symbolic AI, informing the representation of unstructured text to highlight and augment relevant content, provide abstraction and guide the mapping process. Our knowledge-informed approach maintains the efficiency of LLMs while preserving the ability to explain analogies for pedagogical applications.
 Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language models (LLMs), however, are primarily evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE, and there are only a few investigations on whether LLMs can draw analogies between long texts. In this paper, we present ANALOGICAL, a new benchmark to intrinsically evaluate LLMs across a taxonomy of analogies of long text with six levels of complexity – (i) word, (ii) word vs. sentence, (iii) syntactic, (iv) negation, (v) entailment, and (vi) metaphor. Using thirteen datasets and three different distance measures, we evaluate the abilities of eight LLMs in identifying analogical pairs in the semantic vector space. Our evaluation finds that it is increasingly challenging for LLMs to identify analogies when going up the analogy taxonomy.
@@ Line 41: / Line 46: @@
 [[File:Analogical poster.png|900x900px]]
-'''3. Why Do We Need Neuro-symbolic AI to Model Pragmatic Analogies? [https://arxiv.org/pdf/2308.01936.pdf Extended Version], [https://www.computer.org/csdl/magazine/ex/2023/05/10269780/1QWO61CnTS8 Abridged Version published in IEEE Intelligent Systems]'''
+Switching from an analogy pedagogy based on comprehension to analogy pedagogy based on production raises an impractical manual analogy scoring problem. Conventional symbol-matching approaches to computational analogy evaluation focus on positive cases and challenge computational feasibility. This work presents the Discriminative Analogy Features (DAF) pipeline to identify the discriminative features of strong and weak long-form text analogies. We introduce four feature categories (semantic, syntactic, sentiment, and statistical) used with supervised vector-based learning methods to discriminate between strong and weak analogies. Using a modestly sized vector of engineered features with SVM attains a 0.67 macro F1 score. While a semantic feature is the most discriminative, out of the top 15 discriminative features, most are syntactic. Combining these engineered features with an ELMo-generated embedding still improves classification relative to an embedding alone. While an unsupervised K-Means clustering-based approach falls short, similar hints of improvement appear when inputs include the engineered features used in supervised learning.
-A hallmark of intelligence is the ability to use a familiar domain to make inferences about a less familiar domain, known as analogical reasoning. In this article, we delve into the performance of Large Language Models (LLMs) in dealing with progressively complex analogies expressed in unstructured text. We discuss analogies at four distinct levels of complexity: lexical analogies, syntactic analogies, semantic analogies, and pragmatic analogies. As the analogies become more complex, they require increasingly extensive, diverse knowledge beyond the textual content, unlikely to be found in the lexical cooccurrence statistics that power LLMs. To address this, we discuss the necessity of employing Neuro-symbolic AI techniques that combine statistical and symbolic AI, informing the representation of unstructured text to highlight and augment relevant content, provide abstraction and guide the mapping process. Our knowledge-informed approach maintains the efficiency of LLMs while preserving the ability to explain analogies for pedagogical applications.
+[[File:Scoring of student-generated.png|600x600px]]
-−
-[[File:Pragmatic analogies.png|650x650px]]
-−
-'''4. On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models [https://aclanthology.org/2024.findings-eacl.31.pdf]'''
-−
-The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in the last decade of language modeling literature. In this work we specifically look at how LLMs’ abilities to capture sentence analogies (sentences that convey analogous meaning to each other) vary with LLMs’ abilities to encode syntactic and semantic structures of sentences. Through our analysis, we find that LLMs’ ability to identify sentence analogies is positively correlated with their ability to encode syntactic and semantic structures of sentences. Specifically, we find that the LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies.
-−
-[[File:Thilini_poster_EACL.png|900x900px]]
 == Talks ==

@@ Line 1: / Line 1: @@
 == Background and motivation ==
@@ Line 15: / Line 26: @@
 {{#ev:youtube|https://www.youtube.com/watch?v=hl_dGTSgTN4&list=PLqJzTtkUiq577Rc1HpX4iE1_ntNeuppzA&index=9|500|}}
 </center>
 == Publications ==

@@ Line 51: / Line 51: @@
 The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in the last decade of language modeling literature. In this work we specifically look at how LLMs’ abilities to capture sentence analogies (sentences that convey analogous meaning to each other) vary with LLMs’ abilities to encode syntactic and semantic structures of sentences. Through our analysis, we find that LLMs’ ability to identify sentence analogies is positively correlated with their ability to encode syntactic and semantic structures of sentences. Specifically, we find that the LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies.
-[[File:Thilini_poster_EACL_no_blur.png|900x900px]]
+[[File:Thilini_poster_EACL.jpg|900x900px]]
 == Talks ==