Difference between revisions of "CyberInfrastructure Proposal For EarthCube Community"
| (13 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | This proposal presents a cyberInfrastructure for | + | This proposal presents a cyberInfrastructure for sharing and discovery of long tail of science data. |
== Objective == | == Objective == | ||
| − | The proposed system will | + | The proposed system will facilitate scientists to upload their data and annotate them using community developed vocabularies such as GCMD and AGI. It will also enable refining or adapting automatically suggested annotations. Once published, the data can be discovered using keywords, topic terms and attribute value pairs in a faceted search. The system will provide tools for flexible search and for harmonizing structural, content and semantic heterogeneity. |
| − | We identified three | + | We have identified three different workflows to cater to the different requirements for publishing long tail of science data. To accommodate variations in the nature of the data, and the expertise of a user, the three workflows present different trade-offs between convenience of data sharing and breadth of data analysis to be performed. |
=== Publishing Data in Original Form/Legacy Data === | === Publishing Data in Original Form/Legacy Data === | ||
| − | This will allow long tail | + | This workflow will allow long tail of science data to be published in its original form which can be legacy data, or unstructured text, images and tables as found in technical papers, or available separately. We will provide tools to read the data in its original form, annotate the data and index it to make it searchable. Initially, the focus will be on processing and indexing only captions for images and tables. |
=== Publishing Data in Digital Format === | === Publishing Data in Digital Format === | ||
| − | This workflow will serve the | + | This workflow will serve the long tail of science data that is in a digital format such as EXCEL, CSV and relational data files. We will provide additional capabilities to the data publishers by suggesting annotations for structured data such as by annotating columns and cell values in their data. |
=== Publishing Data in Linked Data === | === Publishing Data in Linked Data === | ||
| − | Linked Data initiative http://linkeddata.org/ emerged from Semantic Web technologies | + | Linked Data initiative http://linkeddata.org/ emerged from Semantic Web technologies and provides a new paradigm for publishing and querying structured data on the Web. LOD has changed the way data is shared, linked and reused on the Web. Currently it consists of 295 data sets with 31 billion RDF triples and covers a broad range of domains such as Life Sciences, Geography, Government, Media, Education, Publication and so on. |
| − | + | ||
| − | + | ||
| + | The scientists will have the capabilities to convert their data into the RDF format and publish it as Linked Open Data. This will enable standardization and rich interlinking of existing datasets. This makes their data available for more advanced intelligent applications such as using integrated or federated querying. Even though there are existing tools to convert data into RDF and publish it, these are still not easy to use for domain experts. | ||
== Architecture == | == Architecture == | ||
The following image illustrates the architecture of the proposed system. | The following image illustrates the architecture of the proposed system. | ||
| − | [[File: | + | [[File:Archi7.2.png]] |
| − | + | ||
* '''Data Registry''' | * '''Data Registry''' | ||
| − | Data publishers will register their data through the data registry | + | Data publishers will register their data through the data registry, providing provenance information such as author, affiliation, and location. A sample of the form of data can be registered is given later. |
* '''Annotator''' | * '''Annotator''' | ||
| − | Registered data will be annotated using standard vocabularies such as | + | Registered data will be annotated using standard vocabularies such as GCMD and AGI index which is stored in a vocabulary registry. Annotation tools will suggest annotations to the user and enable them to modify it if desired. Annotations will be stored in the Meta Data Store. |
| − | Kino http://wiki.knoesis.org/index.php/Kino is an integrated suite of tools that enables scientists to annotate Web documents | + | Kino http://wiki.knoesis.org/index.php/Kino is an integrated suite of tools that enables scientists to annotate Web documents. we plan to extend and customize this for Geo-sciences domain. |
* '''Indexer''' | * '''Indexer''' | ||
| − | Collected data and its associated | + | Collected data and its associated metadata will be indexed for search. |
* '''Simple Search''' | * '''Simple Search''' | ||
| − | Simple Search facilitates | + | Simple Search facilitates keyword based queries to which the system will respond with a ranked list of results. |
* '''Faceted Search''' | * '''Faceted Search''' | ||
| − | In addition to the Simple Search functionality system will provide | + | In addition to the Simple Search functionality, the system will provide Faceted Search capability where users can provide attribute value pairs to enable more expressive search and discover. Users can also query using provenance information. |
* '''Mapping to RDF''' | * '''Mapping to RDF''' | ||
| − | As defined in the third | + | As defined in the third workflow, data publishers can transform their data into RDF using existing tools and convert into standard form. |
* '''Data Publisher''' | * '''Data Publisher''' | ||
| − | This component will upload the RDF converted data into Linked Open Data | + | This component will upload the RDF converted data into Linked Open Data. The LOD can be accessed and queried globally. |
* '''Semantic Browsing''' | * '''Semantic Browsing''' | ||
| − | Semantic Browsing will allow | + | Semantic Browsing will allow navigation through the RDF datasets. iExplore http://knoesis.wright.edu/iExplore/ is a tool we have developed for Semantic Browsing. |
== Form of Data == | == Form of Data == | ||
Latest revision as of 16:21, 21 August 2012
This proposal presents a cyberInfrastructure for sharing and discovery of long tail of science data.
Contents
Objective
The proposed system will facilitate scientists to upload their data and annotate them using community developed vocabularies such as GCMD and AGI. It will also enable refining or adapting automatically suggested annotations. Once published, the data can be discovered using keywords, topic terms and attribute value pairs in a faceted search. The system will provide tools for flexible search and for harmonizing structural, content and semantic heterogeneity.
We have identified three different workflows to cater to the different requirements for publishing long tail of science data. To accommodate variations in the nature of the data, and the expertise of a user, the three workflows present different trade-offs between convenience of data sharing and breadth of data analysis to be performed.
Publishing Data in Original Form/Legacy Data
This workflow will allow long tail of science data to be published in its original form which can be legacy data, or unstructured text, images and tables as found in technical papers, or available separately. We will provide tools to read the data in its original form, annotate the data and index it to make it searchable. Initially, the focus will be on processing and indexing only captions for images and tables.
Publishing Data in Digital Format
This workflow will serve the long tail of science data that is in a digital format such as EXCEL, CSV and relational data files. We will provide additional capabilities to the data publishers by suggesting annotations for structured data such as by annotating columns and cell values in their data.
Publishing Data in Linked Data
Linked Data initiative http://linkeddata.org/ emerged from Semantic Web technologies and provides a new paradigm for publishing and querying structured data on the Web. LOD has changed the way data is shared, linked and reused on the Web. Currently it consists of 295 data sets with 31 billion RDF triples and covers a broad range of domains such as Life Sciences, Geography, Government, Media, Education, Publication and so on.
The scientists will have the capabilities to convert their data into the RDF format and publish it as Linked Open Data. This will enable standardization and rich interlinking of existing datasets. This makes their data available for more advanced intelligent applications such as using integrated or federated querying. Even though there are existing tools to convert data into RDF and publish it, these are still not easy to use for domain experts.
Architecture
The following image illustrates the architecture of the proposed system.
- Data Registry
Data publishers will register their data through the data registry, providing provenance information such as author, affiliation, and location. A sample of the form of data can be registered is given later.
- Annotator
Registered data will be annotated using standard vocabularies such as GCMD and AGI index which is stored in a vocabulary registry. Annotation tools will suggest annotations to the user and enable them to modify it if desired. Annotations will be stored in the Meta Data Store.
Kino http://wiki.knoesis.org/index.php/Kino is an integrated suite of tools that enables scientists to annotate Web documents. we plan to extend and customize this for Geo-sciences domain.
- Indexer
Collected data and its associated metadata will be indexed for search.
- Simple Search
Simple Search facilitates keyword based queries to which the system will respond with a ranked list of results.
- Faceted Search
In addition to the Simple Search functionality, the system will provide Faceted Search capability where users can provide attribute value pairs to enable more expressive search and discover. Users can also query using provenance information.
- Mapping to RDF
As defined in the third workflow, data publishers can transform their data into RDF using existing tools and convert into standard form.
- Data Publisher
This component will upload the RDF converted data into Linked Open Data. The LOD can be accessed and queried globally.
- Semantic Browsing
Semantic Browsing will allow navigation through the RDF datasets. iExplore http://knoesis.wright.edu/iExplore/ is a tool we have developed for Semantic Browsing.
Form of Data
Please click on the images to enlarge.
Table
Image
Unstructured Data
