research-article

A new benchmark dataset with production methodology for short text semantic similarity algorithms

Authors:

Keeley CrockettAuthors Info & Claims

ACM Transactions on Speech and Language Processing (TSLP), Volume 10, Issue 4

Article No.: 19, Pages 1 - 63

https://doi.org/10.1145/2537046

Published: 03 January 2014 Publication History

Abstract

This research presents a new benchmark dataset for evaluating Short Text Semantic Similarity (STSS) measurement algorithms and the methodology used for its creation. The power of the dataset is evaluated by using it to compare two established algorithms, STASIS and Latent Semantic Analysis. This dataset focuses on measures for use in Conversational Agents; other potential applications include email processing and data mining of social networks. Such applications involve integrating the STSS algorithm in a complex system, but STSS algorithms must be evaluated in their own right and compared with others for their effectiveness before systems integration. Semantic similarity is an artifact of human perception; therefore its evaluation is inherently empirical and requires benchmark datasets derived from human similarity ratings. The new dataset of 64 sentence pairs, STSS-131, has been designed to meet these requirements drawing on a range of resources from traditional grammar to cognitive neuroscience. The human ratings are obtained from a set of trials using new and improved experimental methods, with validated measures and statistics. The results illustrate the increased challenge and the potential longevity of the STSS-131 dataset as the Gold Standard for future STSS algorithm evaluation.

References

[1]

Achananuparp, P., Hu, X., Zhou, X., and Zhang, X. 2008. Utilizing semantic, syntactic, and question category information for automated digital reference services. In Proceedings of the 11^th International Conference on Asian Digital Libraries: Universal and Ubiquitous Access to Information. 203--214.

Digital Library

[2]

Agirre, A. G. 2012. Exploring semantic textual similarity. Master's dissertation, University of the Basque Country (UPV/EHU).

[3]

Agirre, E., Cer, D., Diab, M., and Gonzalez-Agirre, A. 2012a. Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of the 1^st Joint Conference on Lexical and Computational Semantics (SEM'12). Association for Computational Linguistics, 385--393.

Digital Library

[4]

Agirre, E., Cer, D., Diab, M., and Gonzalez-Agirre, A. 2012b. Task description | Semantic textual similarity. http://www.cs.york.ac.uk/semeval-2012/task6/.

[5]

Agirre, E., Cer, D., Diab, M., Agirre, A. G., and Guo, W. 2013. Sem 2013 shared task: Semantic textual similarity. In Proceedings of the 2^nd Joint Conference on Computational Semantics (SEM'13). Association for Computational Linguistics, 32--43.

[6]

Al-Mubaid, H. and Nguyen, H. A. 2006. A cluster-based approach for semantic similarity in the biomedical domain In Proceedings of the 28^th IEEE EMBS Annual International Conference, A. Hielscher, Ed. 2713--2717.

[7]

Almarsoomi, F., O'Shea, J., Bandar, Z., and Crockett, K. 2012. Arabic word semantic similarity. World Acad. Sci. Engin.Technol. 70, 87--95.

[8]

Aqa, 2010. Aqa languages. http://web.aqa.org.uk/qual/lang_gate.php.

[9]

Bär, D., Zesch, T., and Gurevych, I. 2011. A reflective view on text similarity. In Recent Advances in Natural Language Processing, R. Mitkov and G. Galia Angelova, Eds., 515--520.

[10]

Bär, D., Biemann, C., Gurevych, I., and Zesch, T. 2012. Ukp: Computing semantic textual similarity by combining multiple content similarity measures. In Proceedings of the 1^st Joint Conference on Lexical and Computational Semantics (SEM'12). Y. Marton, Ed., Association for Computational Linguistics, 435--440.

Digital Library

[11]

Barzilay, R. and McKeown, K. 2005. Sentence fusion for multidocument news summarization. Comput. Linguist. 31, 3, 297--328.

[12]

Battig, W. F. and Montague, W. E. 1969. Category norms for verbal items in 56 categories: A replication and extension of the connecticut category norms. J. Exper. Psychol. Monographs 80, 3, 1--46.

[13]

Bernstein, A., Kaufmann, E., Buerki, C., and Klein, M. 2005. How similar is it&quest; Towards personalized similarity measures in ontologies. In Proceedings of the Internationale Tagung Wirtschaftsinformatik (WI'05). 1347--1366.

[14]

Cai, X. and Li, W. 2011. Enhancing sentence-level clustering with integrated and interactive frameworks for theme-based summarization. J. Amer. Soc. Inf. Sci. Technol. 62, 10, 2067--2082.

Digital Library

[15]

Capitani, E., Laiacona, M., Mahon, B. Z., and Caramazzaz, A. 2003. What are the facts of semantic categoryspecific deficits&quest; A critical review of clinical evidence. Cogn. Neuropsychol. 20, 213--261.

[16]

Caramazza, A. and Shelton, J. R. 1998. Domain-specific knowledge systems in the brain: The animate-inanimate distinction. J. Cogn. Neurosci. 10, 1, 1--34.

Digital Library

[17]

Chafe, W. L. 1970. Meaning and the Structure of Language. University of Chicago Press, Chicago, IL.

[18]

Charles, W. G. 2000. Contextual correlates of meaning. Appl. Psycholinguist. 21, 505--524.

[19]

Cook, W. 1979. Case Grammar: Development of the Matrix Model (1970--1978). Georgetown University Press, Washington, DC.

[20]

Cook, W. A. 1989. Case Grammar Theory. Georgetown University Press, Washington, DC.

[21]

Corley, C., Csomai, A., and Mihalcea, R. 2007. A knowledge-based approach to text-to-text similarity. In Recent Advances in Natural Language Processing, John Benjamins Publishers, Amsterdam, 197--206.

[22]

Crockett, K., Bandar, Z., O'Shea, J., and McLean, D. 2009. Bullying and debt: Developing novel applications of dialogue systems. In Proceedings of the 6^th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems. 1--9.

[23]

Crystal, M., Baron, A., Godfrey, K., Micciulla, L., Tenney, Y., and Weischedel, R. 2005. A methodology for extrinsically evaluating information extraction performance. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 652--659.

Digital Library

[24]

Davidson, G. 2004. Roget's Thesaurus of English Words and Phrases. Penguin Reference, London.

[25]

Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., and Harshman, R. 1990. Indexing by latent semantic analysis. J. Amer. Soc. Inf. Sci. 41, 6, 391--407.

[26]

Dethlefs, N., Cuayahuitl, H., Richter, K.-F., Andonova, E., and Bateman, J. 2010. Evaluating task success in a dialogue system for indoor navigation. In Proceedings of the 14^th Workshop on the Semantics and Pragmatics of Dialogue (SemDial'10). P. Lupkowski and M. Purver, Eds., 143--146.

[27]

Devlin, J. T., Russell, R. P., Davis, M. H., Price, C. J., Moss, H. E., Fadili, M. J., and Tyler, L. K. 2002. Is there an anatomical basis for category-specificity&quest; Semantic memory studies in pet and fmri. Neuropsychologia 40, 1, 54--75.

[28]

Dixon, R. M. W. 1991. A New Approach to English Grammar, on Semantic Principles. Oxford University Press: Clarendon Paperbacks.

[29]

Dolan, W. B. and Brockett, C. 2005. Automatically constructing a corpus of sentential paraphrases In Proceedings of the 3^rd International Workshop on Paraphrasing (IWP'05). M. Dras and K. Yamamoto, Eds., Asia Federation of Natural Language Processing, 9--16.

[30]

Ediger, D., Jiang, K., Riedy, J., Bader, D. A., and Corley, C. 2010. Massive social network analysis: Mining twitter for social good. In Proceedings of the 39^th International Conference on Parallel Processing. W.-C. Lee and X. Yuan, Eds., 583--593.

Digital Library

[31]

Erk, K. and Padó, S. 2010. Exemplar-based models for word meaning in context. In Proceedings of the ACL Conference. P. Koehn and J.-S. Chang, Eds., Association for Computational Linguistics, 92--97.

Digital Library

[32]

Erkan, G. and Radev, D. R. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. J. Artif. Intell. Res. 22, 457--479.

[33]

Farah, M. J. and McClelland, J. L. 1991. A computational model of semantic memory impairment: Modality specificity and emergent category specificity. J. Exper. Psychol. General 120, 4, 339--357.

[34]

Fattah, M. A. and Ren, F. 2009. Ga, mr, ffnn, pnn and gmm based models for automatic text summarization. Comput. Speech Lang. 23, 1260--144.

Digital Library

[35]

Feng, J., Zhou, Y., and Martin, T. 2008. Sentence similarity based on relevance. In Proceedings of the Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU'08). 832--839.

[36]

Fenton, N. and Pfleeger, S. 1998. Software Metrics: A Rigorous and Practical Approach. PWS Publishing Company, Boston, MA.

Digital Library

[37]

Ferri, F., Grifoni, P., and Paolozzi, S. 2007. Multimodal sentence similarity in human-computer interaction systems. In Proceedings of the 11^th International Conference on Knowledge-Based Intelligent Information and Engineering Systems (KES'07). Lecture Notes in Artificial Intelligence, vol. 4693. Springer, 403--410.

Digital Library

[38]

Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, S., Wolfman, G., and Ruppin, E. 2002a. The wordsimilarity-353 test collection. http://www.cs.technion.ac.il/&sim;gabr/resources/data/wordsim353/.

[39]

Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., and Ruppin, E. 2002b. Placing search in context: The concept revisited. ACM Trans. Inf. Syst. 20, 1, 116--131.

Digital Library

[40]

Foltz, P. W., Britt, M. A., and Perfetti, C. A. 1996. Reasoning from multiple texts: An automatic analysis of readers' situation models. In Proceedings of the 18^th Annual Cognitive Science Conference. G. W. Cottrell, Ed., Lawrence Erlbaum, 110--115.

[41]

Forde, E. M. E., Francis, D., Riddoch, M. J., Rumiati, R. I., and Humphreys, G. W. 1997. On the links between visual knowledge and naming: A single case study of a patient with a category-specific impairment for living things. Cogn. Neuropsychol. 14, 3, 403--458.

[42]

Funnell, E. and Sheridan, J. S. 1992. Categories of knowledge&quest; Unfamiliar aspects of living and nonliving things. Cogn. Neuropsychol. 9, 2, 135--153.

[43]

Gabrilovich, E. and Markovitch, S. 2007. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI'07). M. M. Veloso, Ed., 1606--1611.

Digital Library

[44]

Gainotti, G. and Silveri, M. C. 1996. Cognitive and anatomical locus of lesion in a patient with a category-specific semantic impairment for living beings. Cogn. Neuropsychol. 13, 3, 357--389.

[45]

Gliozzo, A., Strapparava, C., and Dagan, I. 2009. Improving text categorization bootstrapping via unsupervised learning. ACM Trans. Speech Lang. Process. 6, 1, 1--24.

Digital Library

[46]

Grabin, C. 2013. General statistics. http://psych.unl.edu/psycrs/statpage/regression.html.

[47]

Guo, W. and Diab, M. 2012. Modeling sentences in the latent space. In Proceedings of the 50^th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 864--872.

Digital Library

[48]

Gurevych, I. and Strube, M. 2004. Semantic similarity applied to spoken dialogue summarization. In Proceedings of the 20^th International Conference on Computational Linguistics. 764--770.

Digital Library

[49]

Gurevych, I. and Niederlich, H. 2005. Computing semantic relatedness in german with revised information content metrics. In Proceedings of the Ontologies and Lexical Resources Workshop (OntoLex'05). 28--33.

[50]

Hassan, S. and Mihalcea, R. 2011. Semantic relatedness using salient semantic analysis. In Proceedings of the 25^th AAAI Conference on Artificial Intelligence. W. Burgard and D. Roth, Eds., AAAI Press.

[51]

Hatzivassiloglou, V., Klavans, J. L., Holcombe, M. L., Barzilay, R., Kan, M.-Y., and McKeown, K. R. 2001. Simfinder: A flexible clustering tool for summarization. In Proceedings of the Annual Meeting of the North American Association for Computational Linguistics: Workshop on Automatic Summarization (NAACL'01). 41--49.

[52]

Herz, R. S., Eliassen, J., Beland, S., and Souza, T. 2004. Neuroimaging evidence for the emotional potency of odorevoked memory. Neuropsychologia 42, 3, 371--378.

[53]

Ho, C., Azrifah, M., Murad, A., Kadir, R. A., and Doraisamy, S. C. 2010. Word sense disambiguation-based sentence similarity. In Proceedings of the 23^rd International Conference on Computational Linguistics: Posters (COLING'10). Q. Lu and T. Zhao, Eds., 418--426.

Digital Library

[54]

Inkpen, D. 2007. Semantic similarity knowledge and its applications. Studia Universitatis Babes-Bolyai Informatica. 11--22. http://www.site.uottawa.ca/&sim;diana/publications/studia_d1.pdf.

[55]

Islam, A. and Inkpen, D. 2007. Semantic similarity of short texts. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP'07). 227--236.

[56]

Islam, A. and Inkpen, D. 2008. Semantic text similarity using corpus-based word similarity and string similarity. ACM Trans. Knowl. Discov. Data 2, 2, 1--25.

Digital Library

[57]

Jackendoff, R. 1972. Semantic Interpretation in Generative Grammar. MIT Press, Cambridge, MA.

[58]

Jijkoun, V. and de Rijke, M. 2005. Recognizing textual entailment using lexical similarity. In The PASCAL RTE Challenge. 73--76. http://dare.uva.nl/document/18001.

[59]

Jimenez, S., Becerra, C., and Gelbukh, A. 2012. Soft cardinality: A parameterized similarity function for text comparison. In Proceedings of the 1^st Joint Conference on Lexical and Computational Semantics (SEM'12). Y. Marton, Ed., Association for Computational Linguistics, 449--453.

Digital Library

[60]

Jin, H. and Chen, H. 2008. Semrex: Efficient search in a semantic overlay for literature retrieval. Future Gener. Comput. Syst. 24, 475--488.

Digital Library

[61]

Joun, S., Yi, E., Ryu, C., and Kim, H. 2003. A computation of fingerprint similarity measures based on bayesian probability modeling. In Computer Analysis of Images and Patterns. Lecture Notes in Computer Science, vol. 2756, Springer, 512--520.

[62]

Kennedy, A. and Szpakowicz, S. 2008. Evaluating roget's thesauri. http://aclweb.org/anthology//P/P08/P08-1048.pdf.

[63]

Kiebel, S. J. and Holmes, A. P. 2003. The general linear model. In Human Brain Function, Academic Press.

[64]

Kimura, Y., Araki, K., and Tochinai, K. 2007. Identification of spoken questions using similarity-based tf. Aoi. Syst. Comput. Japan 38, 10, 81--94.

Digital Library

[65]

Klein, D. and Murphy, G. 2002. Paper has been my ruin: Conceptual relations of polysemous senses. J. Memory Lang. 47, 4, 548--570.

[66]

Lee, M. D., Pincombe, B. M., and Welsh, M. B. 2005. An empirical evaluation of models of text document similarity In Proceedings of the 27^th Annual Conference of the Cognitive Science Society. Cognitive Science Society, 1254--1259.

[67]

Levin, B. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago, IL.

[68]

Li, Y., Bandar, Z., and McLean, D. 2003. An approach for measuring semantic similarity between words using multiple information sources. IEEE Trans. Knowl. Data Engin. 15, 4, 871--882.

Digital Library

[69]

Li, Y., Bandar, Z., McLean, D., and O'Shea, J. 2004. A method for measuring sentence similarity and its application to conversational agents. In Proceedings of the 17^th International Florida Artificial Intelligence Research Society Conference (FLAIRS'04). V. Barr and Z. Markov, Eds., AAAI Press, 820--825.

[70]

Li, Y., Bandar, Z., McLean, D., and O'Shea, J. 2006. Sentence similarity based on semantic nets and corpus statistics. IEEE Trans. Knowl. Data Engin. 18, 8, 1138--1150.

Digital Library

[71]

Lin, C. Y. and Och, F. J. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42^nd Annual Meeting on Association for Computational Linguistics (ACL'04). O. Rambow and S. Sergi Balari, Eds.

Digital Library

[72]

Litman, D. J. and Pan, S. 2002. Designing and evaluating an adaptive spoken dialogue system. User Model. User-Adapt. Interact. 12, 111--137.

Digital Library

[73]

Little, W., Fowler, H. W., and Coulson, J. 1983. The Shorter Oxford English Dictionary. Book Club Associates, London.

[74]

Lord, P. W., Stevens, R. D., Brass, A., and Goble, C. A. 2003. Semantic similarity measures as tools for exploring the gene ontology. In Proceedings of the 8^th Pacific Symposium on Biocomputing. 601--612.

[75]

Lund, K. and Burgess, C. 1996. Producing high-dimensional semantic spaces from lexical co-occurrence. Behav. Res. Methods Instrument. Comput. 28, 203--208.

[76]

Madnani, N. and Dorr, B. J. 2010. Generating phrasal and sentential paraphrases: A survey of data-driven methods. Comput. Linguist. 36, 3, 341--387.

Digital Library

[77]

Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., and Miller, K. 1990. Introduction to wordnet: An on-line lexical database. Int. J. Lexicography 3, 4, 235--244.

[78]

Miller, G. A. and Charles, W. G. 1991. Contextual correlates of semantic similarity. Lang. Cogn. Process. 6, 1, 1--28.

[79]

Mitchell, J. and Lapata, L. 2008. Vector-based models of semantic composition. In Proceedings of the Human Language Technology Conference (HLT'08). J. Joakim Nivre and N. A. Smith, Eds., Association for Computational Linguistics, 236--244.

[80]

Mitchell, T. M., Shinkareva, S. V., Carlson, A., Kai-Min Chang, K.-M., Malave, V. L., Mason, R. A., and Just, M. A. 2008. Predicting human brain activity associated with the meanings of nouns. Sci. 320, 5880, 1191--1195.

[81]

Mohler, M. and Mihalcea, R. 2009. Text-to-text semantic similarity for automatic short answer grading. In Proceedings of the 12^th Conference of the European Chapter of the ACL. D. Schlangen and K. Kemal Oflazer, Eds., Association for Computational Linguistics, 567--575.

Digital Library

[82]

Montgomery, D. C. and Runger, G. C. 1994. Applied Statistics and Probability for Engineers. Wiley.

[83]

O'Donaill, E. and Ni Churraighin, D. 1995. Now You're Talking: Multi-Media Course in Irish for Beginners. Gill and Macmillan Ltd.

[84]

O'Shea, J. 2008. Pilot short text semantic similarity benchmark data set: Full listing and description. http://www2.docm.mmu.ac.uk/STAFF/J.Oshea/TRMMUCCA20081_5.pdf.

[85]

O'Shea, J. 2010. A framework for applying short text semantic similarity in goal-oriented conversational agents. Tech. rep. Manchester Metropolitan University.

[86]

O'Shea, J., Bandar, Z., and Crockett, K. 2010a. A machine learning approach to speech act classification using function words. In Proceedings of the 4^th International Symposium on Agent and Multi-Agent Systems: Technologies and Applications (KES'10). Lecture Notes in Artificial Intelligence, vol. 6071, Springer, 82--91.

Digital Library

[87]

O'Shea, J., Bandar, Z., Crockett, K., and McLean, D. 2008. A comparative study of two short text semantic similarity measures. In Proceedings of the 2^nd KES International Conference on Agent and Multi-Agent Systems: Technologies and Applications (KES-AMSTA'08). Lecture Notes in Artificial Intelligence, vol. 4953, Springer, 172--181.

Digital Library

[88]

O'Shea, J., Bandar, Z., Crockett, K., and McLean, D. 2010b. Benchmarking short text semantic similarity. Int. J. Intell. Inf. Database Syst. 4, 2, 103--120.

Digital Library

[89]

Oppenheim, A. N. 1992. Questionnaire Design, Interviewing and Attitude Measurement. Continuum, London, UK.

[90]

Osathanunkul, K., O'Shea, J., Bandar, Z., and Crockett, K. 2011. Semantic similarity measures for the development of thai dialog system. In Proceedings of the 5^th KES International Conference on Agent and Multi-Agent Systems: Technologies and Applications (KES-AMSTA'11). Lecture Notes in Artificial Intelligence, vol. 6682, Springer, 544--552.

Digital Library

[91]

Pouratian, N., Bookheimer, S. Y., Rubino, R., Martin, N. A., and Toga, A. W. 2003. Category-specific naming deficit identified by intraoperative stimulation mapping and postoperative neuropsychological testing. J. Neurosurgery 99, 1, 170--176.

[92]

Quarteroni, S. and Manandhar, S. 2008. Designing an interactive open-domain question answering system. Natural Lang. Engin. 15, 1, 73--95.

Digital Library

[93]

Quirk, R., Greenbaum, S., Leech, G., and Svartik, J. 1985. A Comprehensive Grammar of the English Language. Addison Wesley Longman Ltd., Harlow, UK.

[94]

Resnik, P. 1999. Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. J. Artif. Intell. Res. 11, 95--130.

[95]

Resnik, P. and Diab, M. 2000. Measuring verb similarity. In Proceedings of the 22^nd Annual Meeting of the Cognitive Science Society (COGSCI'00). 399--404.

[96]

Rice, J. A. 1994. Mathematical Statistics and Data Analysis. Duxbury Press.

[97]

Rieck, K. and Laskov, P. 2007. Linear-time computation of similarity measures for sequential data. Adv. Neural Inf. Process. Syst. 19, 1177--1184.

[98]

Rossell, S. L., Shapleske, J., and David, A. S. 1988. Sentence verification and delusions: A context specific deficit. Psychol. Med. 28, 5, 1189--1198.

[99]

Rubenstein, H. and Goodenough, J. 1965. Contextual correlates of synonymy. Comm. ACM 8, 10, 627--633.

Digital Library

[100]

Sahami, M. and Heilman, T. D. 2006. A web based kernel function for measuring the similarity of short text snippets. In Proceedings of the 15^th International Conference on World Wide Web (WWW'06). 377--386.

Digital Library

[101]

Salton, G., Wong, A., and Yang, C. S. 1975. A vector space model for automatic indexing. Comm. ACM 18, 11, 613--620.

Digital Library

[102]

Santos, L. R. and Caramazza, A. 2002. The domain-specific hypothesis. In Category Specificity in Brain and Mind, E. M. E. Forde and G. W. Humpreys, Eds., Psychology Press, Sussex, UK.

[103]

Saric, F., Glavas, G., Karan, M., Snajder, J., and Basic, B. D. 2012. Takelab: Systems for measuring semantic text similarity. In Proceedings of the 1^st Joint Conference on Lexical and Computational Semantics (SEM'12). Y. Marton, Ed., Association for Computational Linguistics, 441--448.

Digital Library

[104]

Sartori, G., Miozzo, M., and Job, R. 1993. Category-specific naming impairments&quest; Yes. Quart. J. Exper. Psychol. 46A, 3, 489--504.

[105]

Schwering, A. and Raubal, M. 2005. Spatial relations for semantic similarity measurement. In Proceedings of the 24^th International Conference on Perspectives in Conceptual Modeling. 259--269.

Digital Library

[106]

Searle, J. R. 1999. Mind, Language and Society. Weidenfield and Nicholson, London, UK.

[107]

Simpson, J. and Weiner, E. 1989. The Oxford English Dictionary. Clarendon Press, Oxford, UK.

[108]

Sinclair, J. 2001. Collins Cobuild English Dictionary for Advanced Learners. HarperCollins, Glasgow, UK.

[109]

Sparck-Jones, K. 1972. A statistical interpretation of term specificity and its application in retrieval. J. Document. 28, 11--21.

[110]

Steiger, J. H. 1980. Tests for comparing elements of a correlation matrix. Psychol. Bull. 87, 2, 245--251.

[111]

Steyvers, M., Shiffrin, R. M., and Nelson, D. L. 2004. Word association spaces for predicting semantic similarity effects in episodic memory. In Experimental Cognitive Psychology and its Applications: Festschrift in honor of Lyle Bourne, Walter Kintsch, and Thomas Landauer, 237--249.

[112]

Thomson, A. J. and Martinet, A. V. 1969. A Practical English Grammar. Oxford University Press, Oxford, UK.

[113]

Tranel, D., Logan, C. G., Frank, R. J., and Damasio, A. R. 1997. Explaining category-related effects in the retrieval of conceptual and lexical knowledge for concrete entities: Operationalization and analysis of factors. Neuropsychologia 35, 10 1329--1339.

[114]

Tsatsaronis, G., Varlamis, I., and Vazirgiannis, M. 2010. Text relatedness based on a word thesaurus. J. Artif. Intell. Res. 37, 1--39.

[115]

Tukey, J. W. 1977. Exploratory Data Analysis. Addison-Wesley, Reading, MA.

[116]

Tversky, A. 1977. Features of similarity. Psychol. Rev. 84, 4, 327--352.

[117]

Uitenbroek, D. G. 2013. Simple statistical correlation analysis online. http://www.quantitativeskills.com/sisa/statistics/correl.htm.

[118]

Valcourt, G. and Wells, L. 1999. Mastery: A University Word List Reader. The University of Michigan Press.

[119]

Van Der Pligt, J. and Taylor, C. 1984. Trait attribution: Evaluation, description and attitude extremity. Euro. J. Social Psychol. 14, 2, 211--221.

[120]

Van Valin, R. D. 1993. A synopsis of role and reference grammar. In Advances in Role and Reference Grammar. R. D. Van Valin, Ed., Benjamins, Amsterdam, 1--164.

[121]

Vigliocco, G., Vinson, D., Lewis, W., and Garrett, M. 2002. Representing the meanings of object and action words: The featural and unitary semantic space hypothesis. Cogn. Psychol. 48, 422--488.

[122]

Vinson, D. P., Vigliocco, G., Cappa, S., and Siri, S. 2003. The breakdown of semantic knowledge: Insights from a statistical model of meaning representation. Brain Lang. 86, 3, 347--365.

[123]

Volokh, A. and Neumann, N. 2012. Dfki-lt - task-oriented dependency parsing evaluation methodology. In Proceedings of the 13^th IEEE International Conference on Information Reuse and Integration. 132--137.

[124]

Walker, M. A., Litman, D. J., Kamm, C. A., and Abella, A. 1997. Paradise: A framework for evaluating spoken dialogue agents. In Proceedings of the 35^th Annual Meeting of the Association for Computational Linguistics. R. Mitkov and B. Boguraev, Eds., 271--280.

Digital Library

[125]

Warrington, E. K. and Shallice, T. 1984. Category-specific semantic impairments. Brain 107, 3, 829--853.

[126]

Witten, I. H. and Eibe, F. 2005. Data Mining: Practical Machine Learning Tools and Techniques. Elsevier.

Digital Library

[127]

Yeh, J.-Y., Ke, H.-R., and Yang, W.-P. 2008. Ispreadrank: Ranking sentences for extraction-based summarization using feature weight propagation in the sentence similarity network. Expert Syst. Appl. 35, 1451--1462.

Digital Library

[128]

Yokote, K.-I., Bollegala, D., and Ishizuka, M. 2012. Similarity is not entailment— Jointly learning similarity transformations for textual entailment. In Proceedings of the 26^th National Conference on Artificial Intelligence (AAAI'12). J. Hoffmann and B. Selman, Eds., Association for the Advancement of Artificial Intelligence, 1720--1726.

[129]

Yuan, X. and Chee, Y. S. 2005. Design and evaluation of elva: An embodied tour guide in an interactive virtual art gallery. Comput. Animation Virtual Worlds 16, 2, 109--119.

Digital Library

Cited By

Kwon H(2023)Dual-Targeted Textfooler Attack on Text Classification SystemsIEEE Access10.1109/ACCESS.2021.312136611(15164-15173)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2021.3121366
Adel NCrockett KLivesey DCarvalho J(2022)An Interval Type-2 Fuzzy Ontological Similarity MeasureIEEE Access10.1109/ACCESS.2022.319451010(81506-81521)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3194510
Adel NCrockett KCarvalho JCross V(2021)Fuzzy Influence in Fuzzy Semantic Similarity Measures2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ45933.2021.9494535(1-7)Online publication date: 11-Jul-2021
https://doi.org/10.1109/FUZZ45933.2021.9494535
Show More Cited By

Index Terms

A new benchmark dataset with production methodology for short text semantic similarity algorithms
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
  2. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Cluster analysis
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Evaluating semantic similarity and relatedness over the semantic grouping of clinical term pairs

Display Omitted Objective: develop a method to quantify the similarity and relatedness of biomedical and clinical term pairs.Semantic similarity and relatedness measures exploit information extrapolated from the Unified Medical Language System.Evaluates ...
A semantic similarity measure for linked data

Linked Data allows structured data to be published in a standard manner so that datasets from diverse domains can be interlinked. By leveraging Semantic Web standards and technologies, a growing amount of semantic content has been published on the Web ...
Towards a hybrid semantic similarity measure to set the conceptual relatedness in a hierarchy

Assessment of semantic similarity between concepts is of great importance in many applications dealing with textual data, such as natural language processing, knowledge acquisition, document semantic annotation and information retrieval systems. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Speech and Language Processing

ACM Transactions on Speech and Language Processing Volume 10, Issue 4

December 2013

206 pages

ISSN:1550-4875

EISSN:1550-4883

DOI:10.1145/2560566

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 January 2014

Accepted: 01 September 2013

Revised: 01 September 2013

Received: 01 June 2012

Published in TSLP Volume 10, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
936
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)7

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kwon H(2023)Dual-Targeted Textfooler Attack on Text Classification SystemsIEEE Access10.1109/ACCESS.2021.312136611(15164-15173)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2021.3121366
Adel NCrockett KLivesey DCarvalho J(2022)An Interval Type-2 Fuzzy Ontological Similarity MeasureIEEE Access10.1109/ACCESS.2022.319451010(81506-81521)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3194510
Adel NCrockett KCarvalho JCross V(2021)Fuzzy Influence in Fuzzy Semantic Similarity Measures2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ45933.2021.9494535(1-7)Online publication date: 11-Jul-2021
https://doi.org/10.1109/FUZZ45933.2021.9494535
Rosu RStoica APopescu PMihaescu M(2020)NLP based Deep Learning Approach for Plagiarism DetectionInternational Joural of User-System Interaction10.37789/ijusi.2020.13.1.413:1(48-60)Online publication date: 2020
https://doi.org/10.37789/ijusi.2020.13.1.4
Little CMclean DCrockett KEdmonds B(2020)A Semantic and Syntactic Similarity Measure for Political TweetsIEEE Access10.1109/ACCESS.2020.30177978(154095-154113)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3017797
Adel NCrockett KCrispin AChandran DCarvalho J(2018)FUSE (Fuzzy Similarity Measure) - A measure for determining fuzzy short text similarity using Interval Type-2 fuzzy sets2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ-IEEE.2018.8491641(1-8)Online publication date: Jul-2018
https://doi.org/10.1109/FUZZ-IEEE.2018.8491641
Noori ZCrockett KBandar ZAl-Mousa M(2018)An Arabic Word Similarity Measure for Semantic Conversational Agents2018 IEEE 2nd International Workshop on Arabic and Derived Script Analysis and Recognition (ASAR)10.1109/ASAR.2018.8480252(119-123)Online publication date: Mar-2018
https://doi.org/10.1109/ASAR.2018.8480252
Amir STanasescu AZighed D(2017)Sentence similarity based on semantic kernels for intelligent text retrievalJournal of Intelligent Information Systems10.1007/s10844-016-0434-348:3(675-689)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1007/s10844-016-0434-3
Augello ACuzzocrea APilato GSpiccia CVassallo G(2016)An Innovative Similarity Measure for Sentence Plagiarism DetectionComputational Science and Its Applications – ICCSA 201610.1007/978-3-319-42092-9_42(552-566)Online publication date: 1-Jul-2016
https://doi.org/10.1007/978-3-319-42092-9_42
Chandran DCrockett KMclean D(2014)On the creation of a fuzzy dataset for the evaluation of fuzzy semantic similarity measures2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)10.1109/FUZZ-IEEE.2014.6891571(752-759)Online publication date: Jul-2014
https://doi.org/10.1109/FUZZ-IEEE.2014.6891571

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents