Smart manufacturing

The utilization of AI in manufacturing processes has ushered in a new era of data-driven optimization and problem-solving. AI is a pivotal cornerstone, especially in Industry 4.0 and 5.0 applications, yielding advanced technologies to guide a new era of heightened efficiency, productivity, and refined decision-making. In this capacity, AI emerges as an indispensable force driving transformative advancements within the manufacturing domain, where data-driven insights and smart automation catalyze innovation and shape the future of industrial processes. The assessment of SoA approaches in the manufacturing realm helped us identify three concrete use cases for further investigation: (1) Rare event prediction, (2) Anomaly detection using process knowledge workflow, and (3) Explaining the root cause of an event through causal knowledge.

PROJECTS

Project-1:Rare Event Prediction

There are different types of events that can be observed within autonomous systems. For example, our consideration on this topic led us to explore “rare events” as they are particularly important for smart manufacturing applications. Rare events are occurrences that take place with a significantly lower frequency than more common regular events. In manufacturing, predicting such events is particularly important, as they lead to unplanned downtime, shortening equipment lifespan, and high energy consumption. The occurrence of events is usually considered rare if observed in 5-10% of all instances, while the occurrence of less than 1% is considered extremely rare. The rarity of events is inversely correlated with the maturity of a manufacturing industry. Typically, the rarity of events causes the multivariate data generated within a manufacturing process to be highly imbalanced, which leads to bias in predictive models. To this end, we conducted a thorough survey to investigate prior art along four dimensions: (i) data acquisition methods, (ii) data preprocessing techniques, (iii) algorithmic approaches, and (iv) evaluation/assessment approaches. Figure X below summarizes the methods we looked at within each dimension

Regarding modeling rare event prediction, we first evaluated the role of data enrichment approaches combined with supervised machine-learning techniques. Predominantly, we explored using three data enrichment approaches; data augmentation, sampling and imputation. Our study is based on open real-world datasets we obtained from public data sources. Figure 1 includes the approaches to learning from rare event data that we exemplified from the comprehensive survey.

Project-2:RI2AP: Robust and Interpretable 2D Anomaly Prediction in Assembly Pipelines

Predicting anomalies in manufacturing assembly lines is crucial for reducing time and labor costs and improving processes. For instance, in rocket assembly, premature part failures can lead to significant financial losses and labor inefficiencies. With the abundance of sensor data in the Industry 4.0 era, machine learning (ML) offers potential for early anomaly detection. However, current ML methods for anomaly prediction have limitations, with F1 measure scores of only 50% and 66% for prediction and detection, respectively. This is due to challenges like the rarity of anomalous events, scarcity of high-fidelity simulation data (actual data are expensive), and the complex relationships between anomalies not easily captured using traditional ML approaches. Specifically, these challenges relate to two dimensions of anomaly prediction: predicting when anomalies will occur and understanding the dependencies between them. This paper introduces a new method called Robust and Interpretable 2D Anomaly Prediction (RI2AP) designed to address both dimensions effectively. RI2AP is demonstrated on a rocket assembly simulation, showing up to a 30-point improvement in F1 measure compared to current ML methods. This highlights its potential to enhance automated anomaly prediction in manufacturing. Additionally, RI2AP includes a novel interpretation mechanism inspired by a causal-influence framework, providing domain experts with valuable insights into sensor readings and their impact on predictions. Finally, the RI2AP model was deployed in a real manufacturing setting for assembling rocket parts. Results and insights from this deployment demonstrate the promise of RI2AP for anomaly prediction in manufacturing assembly pipelines.

Project-3:Multimodal Hybrid Fusion for Robust and Explainable Anomaly Prediction in Assembly Pipelines

In modern assembly pipelines, identifying anomalies is crucial to ensuring product quality and operational efficiency. In complex manufacturing environments, contemporary methods frequently prove inadequacy in managing the intricacies of multi-modal data. This project proposes a novel hybrid fusion approach for multi-modal anomaly prediction in assembly pipelines. Our research builds upon two primary approaches in multi-modal learning: time series-image fusion learning models and joint models for time series and image representation learning. We review existing baselines, including feature-level fusion, where vision and time-series data are combined at the beginning using a unified model, and decision-level fusion, where separate models for each modality are fused mid-process. We introduce a hybrid fusion mechanism that leverages both feature-level and decision-level fusion techniques. The proposed approach involves a two-stage process: data preprocessing and hybrid fusion modeling. We demonstrate and evaluate the new method using the multi-modal dataset that we have made public and conduct comprehensive ablation studies to evaluate the impact of our preprocessing and hybrid fusion stages compared to traditional baselines. This study demonstrates that a hybrid fusion approach can effectively harness the complementary strengths of time-series and image data, offering a robust solution for anomaly prediction in assembly pipelines and provide higher accuracy in predicting anomalies.

Project-4:SmartPilot: A Custom-Compact Neurosymbolic Co-Pilot for Next-Gen Manufacturing

In the dynamic landscape of modern manufacturing, achieving efficiency, precision, and adaptability is paramount. This project introduces SmartPilot, a custom-compact co-pilot designed to transform manufacturing operations that use neurosymbolic AI.

Custom-Compact Neurosymbolic Approach

The core of SmartPilot lies in a custom, compact, neurosymbolic approach tailored specifically for manufacturing. This approach addresses the unique challenges faced by the industry, focusing on solving specific problems with targeted solutions. Custom models are designed to be lightweight and cost-effective, making them ideal for deployment on edge devices where real-time, on-site decision-making is crucial. By integrating curated data, manufacturing knowledge, and human expertise, SmartPilot ensures that its insights are not only data-driven but also contextually aware and reliable.

SmartPilot is powered by three specialized agents, each targeting a critical aspect of the manufacturing process:

1. Anomaly/Rare Events Detection/Prediction Agent: This agent identifies and predicts anomalies before they occur, allowing for proactive intervention and minimizing disruptions in the production line.

2. Demand Forecasting Agent: By analyzing product data, this agent anticipates fluctuations in demand, providing foresight into potential supply chain challenges and ensuring smooth operations.

3. Documentation Agent: Serving as an intelligent Q&A chatbot, this agent is equipped to handle domain-specific queries, generating accurate and contextually relevant responses to support manufacturing personnel.

Robust Architecture

These agents are integrated into a comprehensive architecture that seamlessly connects with existing manufacturing systems. By leveraging HPE's Common Metadata Framework, SmartPilot offers robust global lineage and traceability, versioning and identifying all key artifacts such as datasets, machine learning models, and performance metrics, ensuring the highest levels of transparency and accountability.

Datasets and Knowledge Sources

SmartPilot draws upon a variety of datasets, including sensor data, production data, and quality data, forming the foundation for its predictive and forecasting capabilities. Additionally, it incorporates knowledge sources such as process guidelines and expert insights, ensuring that decisions are both informed and contextually appropriate.

Seamless Coordination and AI Safety

The coordination between the agents ensures that their interactions are smooth and that the manufacturing process remains trustworthy, safe, and efficient. With a focus on AI safety and up-to-date knowledge, SmartPilot is designed to keep pace with the evolving needs of the manufacturing industry.

Project-5:Process Knowledge Workflow using PDDL

We proposed Process Knowledge Workflow (PKW) using Planning Domain Definition Language (PDDL) to detect anomalies in the assembly line (Figure 2&3). Specifically, the PKW uses multimodal data from robot sensors, conveyor belt sensors and camera to detect anomalies. The images from the camera will be processed using state-of-the-art object detection techniques to identify the objects of interest and the interactions among them. The data from the sensors will be utilized to detect the movement of the desired objects. Using this information, PKW will be constructed for the expected process. Then the PKW can serve as an orchestrator to identify anomalies in the sensor values. Based on the sensor values, the PKW will also have the ability to define and name the anomalies. We conducted an extensive survey of the list of sensors in the assembly line from neXt Future Factories lab and studied the type and range of data being generated. Using this, the first version of the PKW ontology will be constructed. A demonstration of PKW can be found in Figure 2.

Project-6:Explaining the root cause of an event through causal knowledge

Our goal is to utilize the process knowledge workflow to construct a sequential causal Bayesian network (Figure 4) that captures both the input-output sequence of the process and the agent information. Specifically, we aim to create a model that can track the cause-and-effect relationships between the input commands, robot actions, and pipeline outputs in the assembly process. By doing so, we will be able to predict the impact of a given input/output and action combination on the assembly pipeline. If an anomaly occurs, we will use the causal Bayesian network to identify the possible root cause and explain the event.

Project-7:AI Empowered Personalized Education

We have surveyed the latest AI technologies for personalized education and have identified synergies with some active projects on using AI for education at AIISC. There is growing use of KGs along with statistical AI (i.e., neural network based) techniques for EduTech solutions using AI for Education. One use of KG is for capturing domain knowledge. For example, the EduTech company Embibe.com has developed comprehensive KG for all subjects and courses it supports. Another use of KG is for creating personalized KG to capture the history of student’s interaction with an AI based education application (e.g., what are all the topics students has learned and how well so that the AI based application can guide the next learning objective and help teachers and the students with personalized assessment). At AIISC we are using AI for education through gaming (e.g., teach strategies for solving Rubrics cube) and support analogy-based pedagogy to improve learning about complex topics with the aid of analogy with more familiar concepts. We identify Dynamic KG to be another interesting application for personalized education. Specifically, the use of a Dynamic KG would allow us to keep the course materials up to date with the latest developments while catering to the emerging demands of students’ learning goals. We believe our active project on AI supported Analogy based personalized learning has an interesting use-case for effectively explaining difficult manufacturing concepts to students. For example, consider the case where students are learning about a new concept: Data Interoperability. Using the analogy about the role of a translator in multilingual communication, we can explain the need for data interoperability to ensure seamless communication between different computer systems and software applications. This approach was tested in the workforce development context as part of the Summer AI Camp for High School students. See (https://aiisc.ai/safari/).

TEAM MEMBERS

Advised By - Dr.Amit Sheth

AIISC Collaborators:

1. Ruwan Wickramarachchi
2. Chathurangi Shyalika
3. Utkarshani Jamini
4. Revathy Venkatramanan
5. Vishal Pallagani
6. Yuxin Zi
7. Kaushik Roy
8. Renjith Prasad

External Collaborators:

1. Fadi El Kalach

Professors:

1. Dr.Ramy Harik
2. Dr.Vignesh Narayanan

JOURNAL PUBLICATIONS

1. [NEW ARTICLE] Shyalika, C., Wickramarachchi, R., El Kalach, F., Harik, R., & Sheth, A. (2024). Evaluating the Role of Data Enrichment Approaches towards Rare Event Analysis in Manufacturing. Sensors, 24(15), 5009, https://doi.org/10.3390/s24155009
2. El Kalach, F., Yousif, I., Wuest, T., R. Harik & A. Sheth, Cognitive manufacturing: definition and current trends. Journal of Intelligent Manufacturing (2024), doi.org/10.1007/s10845-024-02429-9
3. Jaimini, U., Henson, C., and Sheth, A., (2024). Causal Neurosymbolic AI: A Synergy Between Causality and Neurosymbolic Methods. IEEE Intelligent Systems, 39(3), pp.13-19. (Link: https://www.computer.org/csdl/magazine/ex/2024/03/10570374/1Y2h54VNhTy)
4. Shyalika, C., Roy, K., Prasad, R., Kalach, F. E., Zi, Y., Mittal, P., Narayanan, V., R. Harik & A. Sheth, (2024). RI2AP: Robust and Interpretable 2D Anomaly Prediction in Assembly Pipelines. Sensors, 24(10), 3244, https://doi.org/10.3390/s24103244
5. Shyalika, C.; Roy, K.; Prasad, R.; Kalach, F.E.; Zi, Y.; Mittal, P.; Narayanan, V.; Harik, R.; Sheth, A. RI2AP: Robust and Interpretable 2D Anomaly Prediction in Assembly Pipelines. Preprints 2024, 2024041959. https://doi.org/10.20944/preprints202404.1959.v1
6. Shyalika, C., Wickramarachchi, R., & Sheth, A., A Comprehensive Survey on Rare Event Prediction. 2023, arXiv preprint "https://arxiv.org/pdf/2309.11356.pdf"
7. R. Venkataramanan, A. Tripathy, M. Foltin, H. Y. Yip, A. Justine & A. Sheth, "Knowledge Graph Empowered Machine Learning Pipelines for Improved Efficiency, Reusability, and Explainability," in IEEE Internet Computing, vol. 27, no. 1, pp. 81-88, 1 Jan.-Feb. 2023, doi: 10.1109/MIC.2022.3228087.
8. Venkataramanan, Revathy; Padhee, Swati; Rao, Saini Rohan; Sundarajan, Anirudh; Kaoshik, Ronak; and Amit Sheth. “Ki-Cook: Knowledge infused Multimodal Cooking Representation Learning”. Frontiers Big Data. 2023 Jul 24;6:1200840. doi: 10.3389/fdata.2023.1200840. PMID: 37554262; PMCID: PMC10406211. Full text - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10406211/
9. Kalach, F. E., R. Wickramarachchi, R. Harik & A. Sheth, "A Semantic Web Approach to Fault Tolerant Autonomous Manufacturing," in IEEE Intelligent Systems, vol. 38, no. 1, pp. 69-75, 1 Jan.-Feb. 2023, doi: 10.1109/MIS.2023.3235677.
10. Jaimini, U., Tongtao, Z., Georgia, B., & Sheth, A., "iMetaverseKG: Industrial Metaverse Knowledge Graph to Promote Interoperability in Design and Engineering Applications." IEEE Internet Computing 26, no. 6 (2022): 59-67.
11. Jaimini, U., & Sheth, U., "CausalKG: Causal Knowledge Graph Explainability using interventional and counterfactual reasoning." IEEE Internet Computing 26, no. 1 (2022): 43-50.
12. Wickramarachchi, R., C. Henson & A. Sheth, "Knowledge-Based Entity Prediction for Improved Machine Perception in Autonomous Systems," in IEEE Intelligent Systems, vol. 37, no. 5, pp. 42-49, 1 Sept.-Oct. 2022, doi: 10.1109/MIS.2022.3181015.

CONFERENCE PUBLICATIONS

1. [NEW ARTICLE]Jaimini, U., Henson, C., Sheth, A., and Harik, R., 2024. Causal Neuro-Symbolic AI for Root Cause Analysis in Smart Manufacturing, 23rd International Semantic Web Conference (ISWC) 2024.
2. [NEW ARTICLE]Jaimini, U., Henson, C., and Sheth, A., 2024. Causal Neuro-Symbolic AI based Causal Entity Prediction in Autonomous Driving, 23rd International Semantic Web Conference (ISWC) 2024.
3. [NEW ARTICLE]Jaimini, U., Henson, C., and Sheth, A., 2024. Visual Causal Question and Answering with Knowledge Graph Link Prediction, 23rd International Semantic Web Conference (ISWC) 2024.
4. [NEW ARTICLE] Prasad, R.*, Shyalika, C.*, Zand, R., Kalach, F. E., Venkataramanan, R., Harik, R., & Sheth, A. (2024). AssemAI: Interpretable Image-Based Anomaly Detection for Manufacturing Pipelines. arXiv preprint arXiv:2408.02181.
5. Venkataramanan, R., Roy, K., Raj, K., Prasad, R., Zi, Y., Narayanan, V., & Sheth, A. (2023, October). Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 981-986). IEEE. (Link: https://ieeexplore.ieee.org/abstract/document/10394432)
6. Wickramarachchi, R., Henson, C., & Sheth, A., CLUE-AD: A Context-Based Method for Labeling Unobserved Entities in Autonomous Driving Data. 37th AAAI Conference on Artificial Intelligence (AAAI), February, 2023.
7. Jaimini, U., Henson C., & Sheth, A., "An Ontology Design Pattern for Representing Causality" ISWC 2023 Workshop on Ontology Design Pattern.

PRESENTATIONS

1. Shyalika, C., Roy, K., Prasad, R., Zi, Y., Mittal, P., Kalach, F. E., Narayanan, V., R. Harik & A. Sheth, “RI2AP: Robust and Interpretable 2D Anomaly Prediction in Assembly Pipelines”. DiscoverUSC-2024, University of South Carolina, April 19, 2024. [Honoured with a "Best Poster" award]
2. Shyalika, C., Roy, K., Prasad, R., Zi, Y., Mittal, P., Kalach, F. E., Narayanan, V., R. Harik & A. Sheth, “RI2AP: Robust and Interpretable 2D Anomaly Prediction in Assembly Pipelines”. CSE Research Symposium, University of South Carolina, March 15, 2024.
3. Shyalika, C., Wickramarachchi, R., & Sheth, A., Towards Rare Event Prediction in Manufacturing Domain. CSE Research Symposium, University of South Carolina, April 14, 2023.
4. Shyalika, C., Wickramarachchi, R., & Sheth, A., Multivariate Data Augmentation for Rare Event Prediction in Manufacturing. CRA-WP Grad Cohort for Women, April 21, 2023.
5. Shyalika, C., Wickramarachchi, R., Harik, R., & Sheth, A., Evaluating the Role of Multivariate Data Augmentation Towards Rare Event Prediction in Manufacturing. Discover USC Symposium, University of South Carolina, April 21, 2023.

BLOG POSTS

1. Shyalika, C., Wickramarachchi, R., & Sheth, A., A Comprehensive Survey on Rare Event Prediction. AIHub, Available at "https://aihub.org/2023/11/22/a-comprehensive-survey-on-rare-event-prediction/", 2023

FUNDING

• NSF Award#: 2119654
• RII Track 2 FEC: Enabling Factory to Factory (F2F) Networking for Future Manufacturing
• Timeline: Oct 2021 - Sept 2025
• Award Amount: $739,239

---

• NSF Award#: 2133842
• EAGER: Advancing Neuro-symbolic AI with Deep Knowledge-infused Learning
• Timeline: 01 July 2021 - 30 June 2022
• Award Amount: $139,999