Revision as of 23:30, 25 February 2024

Neurosymbolic AI Overview

Humans interact with the environment using a combination of perception transforming sensory inputs from their environment into symbols, and cognition - mapping symbols to knowledge about the environment for supporting abstraction, reasoning by analogy, and long-term planning. Human perception-inspired machine perception, in the context of artificial intelligence (AI), refers to large-scale pattern recognition from raw data using neural networks trained using self-supervised learning objectives such as next-word prediction or object recognition. On the other hand, machine cognition encompasses more complex computations, such as using knowledge of the environment to guide reasoning, analogy, and long-term planning. Humans can also control and explain their cognitive functions. This seems to require the retention of symbolic mappings from perception outputs to knowledge about their environment. For example, humans can follow and explain the guidelines and safety constraints driving their decision making in safety-critical applications such as health care, criminal justice, and autonomous driving.

Embodying intelligent behavior in an artificial intelligence system must involve both perception, processing raw data and cognition, using background knowledge to support abstraction, analogy, reasoning, and planning. Symbolic structures represent this background knowledge explicitly. Although neural networks are a powerful tool for processing and extracting patterns from data, they lack explicit representations of background knowledge, hindering the reliable evaluation of their cognition capabilities. Furthermore, applying appropriate safety standards while providing explainable outcomes guided by concepts from background knowledge is crucial for establishing trustworthy models of cognition for decision support.

Outcomes Achieved So far

1. The rapid improvement in language models suggests that they will achieve almost optimal performance levels for large-scale perception. Knowledge graphs are suitable for symbolic structures that bridge the cognition and perception aspects because they support real-world dynamism. Unlike static and brittle symbolic logics, such as first-order logic, they are easy to update. In addition to their suitability for enterprise use cases and established standards for portability, knowledge graphs are part of a mature ecosystem of algorithms that enable highly efficient graph management and querying. This scalability allows for modeling large and complex datasets with millions or billions of nodes.
2. We find that combining language models and knowledge graphs are most effective in current implementations. However, it also suggests that future knowledge graphs have the potential to model heterogeneous types of application- and domain-level knowledge beyond schemas. This includes workflows, constraint specifications, and process structures
3. Combining such enhanced knowledge graphs with high-capacity neural networks would provide the end user with an extremely high degree of algorithmic- and application-level utility. The concern for safety is behind the recent push to withhold further rollout of generative AI systems such as GPT* as current systems could significantly harm individuals and society without additional guardrails. We believe that guidelines, policy, and regulations can be encoded via extended forms of knowledge graphs, which in turn can provide explainability accountability, rigorous auditing capabilities, and safety. Encouragingly, progress is being made on all these fronts swiftly, and the future looks promising

Journal Publications

1. Jaimini, Utkarshani, Tongtao Zhang, Georgia Olympia Brikis, and Amit Sheth. "iMetaverseKG: Industrial Metaverse Knowledge Graph to Promote Interoperability in Design and Engineering Applications." IEEE Internet Computing 26, no. 6 (2022): 59-67. (Link: https://www.computer.org/csdl/magazine/ic/2022/06/10018009/1JYZ2E4yUZG)
2. Roy, K., Gaur, M., Rawte, V., Kalyan, A., & Sheth, A. (2023). ProKnow: Process Knowledge for Safety Constrained and Explainable Question Generation for Mental Health Diagnostic Assistance., Frontiers Big Data, 09 January 2023 (Link : https://www.frontiersin.org/articles/10.3389/fdata.2022.1056728/full)
3. Sheth, A., Roy, K., & Gaur, M. (2023). Neurosymbolic Artificial Intelligence (Why, What, and How). IEEE Intelligent Systems, 38(3), 56-62.
4. Sheth, A., & Roy, K. (2024). Neurosymbolic Value-Inspired AI (Why, What, and How). IEEE Intelligent Systems.

Conference Publications

1. Pallagani, V., Roy, K., Muppasani, B., Fabiano, F., Loreggia, A., Murugesan, K., ... & Sheth, A. (2024). On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS). International Conference on Automated Planning and Scheduling (ICAPS). (Link: https://arxiv.org/pdf/2401.02500)
2. Zi, Y., Veeramani, H., Roy, K., & Sheth, A. (2024). RDR: the Recap, Deliberate, and Respond Method for Enhanced Language Understanding. AAAI Workshop on Neuro-Symbolic Learning and Reasoning in the era of Large Language Models. (Link: https://openreview.net/forum?id=hNQJI0KS3T)
3. Zi, Y., Roy, K., Narayanan, V., & Sheth, A. (2024). Exploring Alternative Approaches to Language Modeling for Learning from Data and Knowledge. AAAI Spring Symposium on Empowering Machine Learning and Large Language Models with Domain and Commonsense Knowledge. (Link: https://scholarcommons.sc.edu/cgi/viewcontent.cgi?article=1619&context=aii_fac_pub)
4. Roy, K., Oltramari, A., Zi, Y., Shyalika, C., Narayanan, V., & Sheth, A. (2024). Causal Event Graph-Guided Language-based Spatiotemporal Question Answering. AAAI Spring Symposium on Empowering Machine Learning and Large Language Models with Domain and Commonsense Knowledge. (Link: https://scholarcommons.sc.edu/cgi/viewcontent.cgi?article=1618&context=aii_fac_pub)
5. Roy, Kaushik, et al. "Knowledge-Infused Self Attention Transformers." KDD Workshop on Knowledge Augmented Methods for NLP (2023). (Link: https://arxiv.org/pdf/2306.13501)
6. Roy, Kaushik, et al. "Process Knowledge-infused Learning for Clinician-friendly Explanations." AAAI Second Symposium on Human Partnership with Medical Artificial Intelligence (2023). (Link: https://arxiv.org/pdf/2306.09824)
7. Roy, Kaushik, and Vipula Rawte. "TDLR: Top Semantic-Down Syntactic Language Representation." NeurIPS'22 Workshop on All Things Attention: Bridging Different Perspectives on Attention. 2022. (Link: https://attention-learning-workshop.github.io/2022/papers/rawte-tdlr_top_semanticdown_syntactic_language_representation.pdf)
8. Keyur Faldu, Amit Sheth, Prashant Kikani, Hemang Akbari. KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding. arXiv:2104.08145 [cs.CL] https://doi.org/10.48550/arXiv.2104.08145
9. Gaur, M., Kursuncu, U., Alambo, A., Sheth, A., Daniulaityte, R., Thirunarayan, K., & Pathak, J. (2018). “Let Me Tell You About Your Mental Health!”: Contextualized Classification of Reddit Posts to DSM-5 for Web-based Intervention. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 753–762. https://doi.org/10.1145/3269206.3271732
10. Sheth, A., Yip, H. Y., & Shekarpour, S. (2019). Extending Patient-Chatbot Experience with Internet-of-Things and Background Knowledge: Case Studies with Healthcare Applications. IEEE Intelligent Systems, 34(4), 24–30. https://doi.org/10.1109/MIS.2019.2905748
11. Kadariya, D., Venkataramanan, R., Yip, H. Y., Kalra, M., Thirunarayanan, K., & Sheth, A. (2019). kBot: Knowledge-Enabled Personalized Chatbot for Asthma Self-Management. 2019 IEEE International Conference on Smart Computing (SMARTCOMP), 138–143. https://doi.org/10.1109/SMARTCOMP.2019.00043
12. Gaur, M., Alambo, A., Sain, J. P., Kursuncu, U., Thirunarayan, K., Kavuluru, R., … Pathak, J. (2019). Knowledge-aware Assessment of Severity of Suicide Risk for Early Intervention. The World Wide Web Conference, 514–525.https://doi.org/10.1145/3308558.3313698
13. Bhatt, S., Padhee, S., Sheth, A., Chen, K., Shalin, V., Doran, D., & Minnery, B. (2019). Knowledge Graph Enhanced Community Detection and Characterization. Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, 51–59. https://doi.org/10.1145/3289600.3291031
14. Alambo, A., Gaur, M., Lokala, U., Kursuncu, U., Thirunarayan, K., Gyrard, A., Sheth, A., Welton, R. S., Pathak, J. (2019). Question Answering for Suicide Risk Assessment Using Reddit. 2019 IEEE 13th International Conference on Semantic Computing (ICSC), 468-473. Retrieved from https://doi.org/10.1109/ICOSC.2019.8665525
15. Yadav, S., Pallagani, V., & Sheth, A. (2020). Medical Knowledge-enriched Textual Entailment Framework. ArXiv:2011.05257 [Cs]. http://arxiv.org/abs/2011.05257
16. Wickramarachchi, R., Henson, C., & Sheth, A. (2020). An evaluation of knowledge graph embeddings for autonomous driving data: Experience and practice. Proceedings of the AAAI 2020 Spring Symposium on Combining Machine Learning and Knowledge Engineering in Practice (AAAI-MAKE 2020)
17. Purohit, H., Shalin, V. L., Sheth, A. P., & Sheth, A. (2020). Knowledge graphs to empower humanity-inspired AI systems. IEEE Internet Computing, 24(4), 48–54. https://doi.org/10.1109/MIC.2020.3013683
18. Kursuncu, U., Gaur, M., & Sheth, A. (2020). Knowledge infused learning (K-IL): Towards deep incorporation of knowledge in deep learning. Proceedings of the AAAI 2020 Spring Symposium on Combining Machine Learning and Knowledge Engineering in Practice (AAAI-MAKE).
19. Gaur, M., Desai, A., Faldu, Keyur, & Sheth, A. (2020). Explainable AI using knowledge graphs. ACM CoDS-COMAD Conference. https://aiisc.ai/xaikg/
20. Yadav, S., Lokala, U., Daniulaityte, R., Thirunarayan, K., Lamy, F., & Sheth, A. (2021). “When they say weed causes depression, but it’s your fav antidepressant”: Knowledge-aware attention framework for relationship extraction. PLOS ONE, 16(3), e0248299. https://doi.org/10.1371/journal.pone.0248299
21. Xia, K., Saidy, C., Kirkpatrick, M., Anumbe, N., Sheth, A., & Harik, R. (2021). Towards semantic integration of machine vision systems to aid manufacturing event understanding. Sensors, 21(13), 4276. https://doi.org/10.3390/s21134276
22. Wickramarachchi, R., Henson, C., & Sheth, A. (2021). Knowledge-infused Learning for Entity Prediction in Driving Scenes. Frontiers in Big Data, 4.https://doi.org/10.3389/fdata.2021.759110
23. Sheth, A., & Thirunarayan, K. (2021). The duality of data and knowledge across the three waves of AI. IT Professional, 23(3), 35–45. https://doi.org/10.1109/MITP.2021.3070985
24. Sheth, A., Shalin, V. L., & Kursuncu, U. (2021, April). Defining and detecting toxicity on social media: Context and knowledge are key. Neurocomputing. https://doi.org/10.1016/j.neucom.2021.11.095
25. Roy, K., Zhang, Q., Gaur, M., & Sheth, A. (2021). Knowledge infused policy gradients for adaptive pandemic control. Proceedings of the AAAI 2021 Spring Symposium on Combining Machine Learning and Knowledge Engineering (AAAI-MAKE 2021).
26. Roy, K., Zhang, Q., Gaur, M., & Sheth, A. (2021). Knowledge infused policy gradients with upper confidence bound for relational bandits, In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 35-50. Springer, Cham, 2021..
27. Roy, K., Lokala, U., Khandelwal, V., & Sheth, A. (2021). “Is depression related to cannabis?”: A knowledge-infused model for entity and relation extraction with limited Supervision. Proceedings of the AAAI 2021 Spring Symposium on Combining Machine Learning and Knowledge Engineering (AAAI-MAKE 2021).
28. Jaimini, U., & Sheth, A. (2021). Personalized digital phenotype score, healthcare management and intervention strategies using Knowledge enabled Digital Health Framework for pediatric asthma. In Asthma. IntechOpen. https://doi.org/10.5772/intechopen.97430
29. Gaur, M., Roy, K., Sharma, A., Srivastava, B., & Sheth, A. (2021, August). “Who can help me?”: Knowledge Infused Matching of Support Seekers and Support Providers during COVID-19 on Reddit. In 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI) (pp. 265-269). IEEE.
30. Gaur, M., Faldu, K., & Sheth, A. (2021). Semantics of the black-box: Can knowledge graphs help make deep learning systems more interpretable and explainable? IEEE Internet Computing, 25 (1), Jan-Feb 2021, pp. 51–59. https://doi.org/10.1109/MIC.2020.3031769
31. Gaur, M., Aribandi, V., Alambo, A., Kursuncu, U., Thirunarayan, K., Beich, J., Pathak, J., & Sheth, A. (2021). Characterization of time-variant and time-invariant assessment of suicidality on Reddit using C-SSRS. PloS one, 16(5), e0250448. https://doi.org/10.1371/journal.pone.0250448
32. Dolbir, N., Dastidar, T., & Roy, K. (2021). NLP is not enough - Contextualization of user input in chatbots. KGC Workshop on Knowledge-infused Learning. https://aiisc.ai/KiL2021/papers/KiL2021.pdf
33. Sawhney, R., Neerkaje, A., & Gaur, M. (2022, May). A Risk-Averse Mechanism for Suicidality Assessment on Social Media. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 628-635).

Application Paper Publications

1. Roy, K., Khandelwal, V., Surana, H., Vera, V., Sheth, A., & Heckman, H. (2024). GEAR-Up: Generative AI and External Knowledge-based Retrieval Upgrading Scholarly Article Searches for Systematic Reviews. AAAI Conference on Artificial Intelligence. (Link: https://arxiv.org/pdf/2312.09948)
2. Roy, K., Khandelwal, V., Goswami, R., Dolbir, N., Malekar, J., & Sheth, A. (2023). Demo Alleviate: Demonstrating artificial intelligence-enabled virtual assistance for telehealth: The mental health case. Thirty-Seventh AAAI Conference on Artificial Intelligence. (Link: https://scholarcommons.sc.edu/aii_fac_pub/566/)

Keynotes, Tutorials and Talks

1. Sheth, A., Knowledge-infused NLU for Addiction and Mental Health Research, Keynotes at the MAISoN21, Aug 2021, and the ASONAM21, Sept 2021.Presentation-GDriveVideo-MAISoN@IJCAI
2. Sheth, A., Don’t Handicap AI without Explicit Knowledge, Keynotes at the IEEE Services, Sept 2021, and the DEXA2021, Oct 2021. Presenation-GDriveVideo IEEE SErvicesVideo-DEXA
3. Amit Sheth, Semantics of the Black-Box: Towards Knowledge-infused Learning --- Keynote talk at the Semantic Machine Learning Workshop in conjunction with the 15th IEEE International Conference on Semantic Computing, February 2021, https://youtu.be/cx-l0XDk9Tw
4. Sheth, A., Augmented Personalized Health: an explicit knowledge enhanced neural symbolic Health approach to patient empowerment for managing chronic disease burden. Keynote at the SWAT4HCLS (Semantic Web Applications and Tools for Healthcare and Life Sciences), ​​January 2022. SlidesharePresentation-GDriveVideo
5. Amit Sheth, From NLP to NLU: Why we need varied, comprehensive, and stratified knowledge (Neuro-symbolic AI), Keynote at KnowledgeNLP at AAAI2023, February 2023. Slides Abstract Video
6. Roy, K. (2024). Knowledge-infused Neurosymbolic Artificial Intelligence for Mental Healthcare. Intelligent Clinical Care Center, University of Florida. (Slides: https://lnkd.in/eVCzinAk)

Links to previous years' publications: http://wiki.aiisc.ai/index.php/Advancing_Neuro-symbolic_AI_with_Deep_Knowledge-infused_Learning, and https://wiki.aiisc.ai/index.php?title=EAGER:_Knowledge-guided_neurosymbolic_AI_with_guardrails_for_safe_virtual_health_assistants

Funding

• NSF Award #: 2335967
• Award Period of Performance:   Start Date: 10/01/2023     End Date: 09/30/2025
• Project Title: EAGER: Knowledge-guided neurosymbolic AI with guardrails for safe virtual health assistants
• Award Amount: $200,000

Personnel

• Faculty: Amit Sheth (PI), Vignesh Narayanan (Co-PI)
• Graduate Research Assistants: Yuxin Zi, Kaushik Roy