Revision as of 21:39, 15 September 2014

Clustering

The FACES approach generates faceted entity summaries that are both concise and comprehensive. Conciseness is about selecting a small number of facts. Comprehensiveness is about selecting facts to represent all aspects of an entity that improves coverage. Diversity is about selecting facts that are orthogonal to each other so that the selected few facts enrich coverage. Hence, diversity improves comprehensiveness when the number of features to include in a summary is limited. Conciseness may be achieved by following various ranking and filtering techniques. But creating summaries that satisfy both conciseness and comprehensiveness constraints simultaneously is not a trivial task. It needs to recognize facets of an entity that features represent so that the summary can represent as many facets (diverse and comprehensive) as possible without redundancy (leads to conciseness). Number and nature of clusters (corresponding to abstract concepts) in a feature set is not known a priori for an entity and is hard to guess without human intervention or explicit knowledge. Therefore, a supervised clustering algorithm or unsupervised clustering algorithm with prescribed number of clusters to seek cannot be used in this context. To achieve this objective, we have adapted a flexible unsupervised clustering algorithm based on Cobweb [2][3] and have designed a ranking algorithm for feature selection.

Hierarchical conceptual clustering

We use Cobweb [2] algorithm as the hierarchical conceptual clustering algorithm in our problem. " Cobweb is an incremental system for hierarchical clustering. the system carries out a hill-climbing search through a space of hierarchical classification schemes using operator that enable bidirectional travel through this space." Cobweb uses a heauristic measure called category utility to guide search. Category utility is a tradeoff between intra-class similarity and inter-class dissimilarity of objects (attribute-value pairs). Intra-class similarity is the conditional probability of the form P(A_i = V_ij| C_k), where A_i = V_ij is an attribute-value pair and C_k is a class. When this probability is larger, more memebrs from the class share more values. Inter-class similarity is the conditional probability P(C_k| A_i = V_ij). When this probability is larger, fewer objects sharing similar values are in other classes. Therefore, categori utility (CU) is defined as the product of intra-class and inter-class similarities. For a partition {C₁, C₂,...., C_n}, CU is defined as follows,

Finally, they define CU as the gaining function as below.

Cobweb algorithm is explained in Figure 1 and its four operators are explained in Figure 2 (taken from [3]).

Figure 1. Cobweb algorithm

Figure 2. Auxiliary Cobweb operators

Evaluation

For this evaluation we evaluate FACES against RELIN and SUMMARUM. We chose DBpedia as the dataset as it was used in RELIN and is a huge dataset containng multi-domain entities. We extracted 50 entities randomly from English DBpedia version 3.9. We asked 15 human judges to create length 5 and 10 entity summaries (ideal summaries) and used them as the gold standard. We also made sure that each entity gets at least 7 ideal summaries. We could not use experiment data of RELIN as authors of RELIN confirmed that the data are not available. We avoid processing properties such as owl:sameAs, rdf:type, db:wordnet_type. db:wikiPageWikiLink, db:wikiPageExternalLink, db:wikiPageUsesTemplate, db:wikiPageRevisionID, db:wikiPageID, dc:subject, and db:Template. In the sample dataset, there are at least 17 ditinct properties and 19 - 88 distinct features per entity.

Experiment 1

Results of our evaluation with the Gold standard are listed in Table 1. It shows that FACES clearly outperforms other systems. RELINM is a modification of RELIN to have the capability to pick filter duplicate properties in the summary. Table 2 shows that our change of search service to RELIN does not affect the final outcome. We tested this using randomly selected entities of size 5.

Experiment 2

This experiment measures FACES ability to identify facets. In other words, its ability to identify conceptually similar groups in comparison to ideal summaries. For this purpose we just ealuate property name overlap between computer generated summaries and ideal summaries. The reults are shown in Table 3.

Experiment 3

In this experiment, we asked 69 users to vote for the best summary that helps them to identiy the entity. In the first usecase, we used FACES and RELIN side by side. Then we used all three systems side by side and results are shown in table 4. Users were not given information of the systems that produced summaries.

The following graph presents statistics of the blind-folded user evaluation where user preference for randomly selected 10 entities are recorded. SUMMARUM performed better than FACES in only two entities and on average, FACES outperformed other systems.

Figure 3. User preference of system summaries for randomly selected 10 instances

Dataset

Evauation data is available for download

References

[1] Cheng, Gong, Thanh Tran, and Yuzhong Qu. "RELIN: relatedness and informativeness-based centrality for entity summarization." In The Semantic Web–ISWC 2011, pp. 114-129. Springer Berlin Heidelberg, 2011.
[2] Fisher, D.H.: Knowledge acquisition via incremental conceptual clustering. Machine learning 2(2), 139–172 (1987)
[3] Gennari, J.H., Langley, P., Fisher, D.: Models of incremental concept formation. Artificial intelligence 40(1), 11–61 (1989)

to be added in the paper
[4] Nenkova, Ani, and Kathleen McKeown. "A survey of text summarization techniques." In Mining Text Data, pp. 43-76. Springer US, 2012.
[5] Das, Dipanjan, and André FT Martins. "A survey on automatic text summarization." Literature Survey for the Language and Statistics II course at CMU 4 (2007): 192-195.
[6] Li, Xuan, Liang Du, and Yi-Dong Shen. "Update summarization via graph-based sentence ranking." Knowledge and Data Engineering, IEEE Transactions on 25, no. 5 (2013): 1162-1174.
[7] Erkan, Günes, and Dragomir R. Radev. "LexRank: Graph-based lexical centrality as salience in text summarization." J. Artif. Intell. Res.(JAIR) 22, no. 1 (2004): 457-479.

-->