Twarql
Twarql: Twitter Feeds through SPARQL
NOTICE: Twarql is in pre-alpha stage.
We are currently refactoring some parts of the code and stabilizing our REST API. We plan to release a first version of the source code and API by June 7th, 2010.
Contents
[hide]Introduction
Our approach encompasses the following steps:
- extract content (entity mentions, hashtags and URLs) from microposts;
- encode content in a structured format (RDF) using shared vocabularies (FOAF, SIOC, MOAT, etc.);
- enable structured querying of microposts (SPARQL);
- enable subscription to a stream of microposts that match a given query (Concept Feeds);
- enable scalable real-time delivery of streaming data (SparqlPuSH).
Demonstration We have two demonstration videos. The first video demonstrates the user perspective, interacting with the system to formulate a query and obtain microblog posts that match that query. The second video focuses on the server side and demonstrates the modules of our architecture at work, distributing the microposts via pubsubhubbub.
Architecture
See the workflow between the components of the architecture:
Tweet Annotation
- extract content (entity mentions, hashtags and URLs) from microposts;
- encode content in a structured format (RDF) using shared vocabularies (FOAF, SIOC, MOAT, etc.);
Concept Feeds
- enable structured querying of microposts (SPARQL);
- enable subscription to a stream of microposts that match a given query;
SPARQLPuSH
- More about the SPARQL Push implementation: Introducing sparqlPuSH, open source implementation
Twarql API
REST Endpoints
Summary:
- URL scheme: http://<base-url>/<operation>/?<parameter>=<value>&...&output=<output format>
- Base URL: http://knoesis1.wright.edu/twarql
- Operations: search, register, stream, query
- Output formats: twitter-json, sparql-json, entities
Parameters:
- http://knoesis1.wright.edu/twarql/search?keyword=k1,...,kn&output=<output type>
- input: keywords, output type (twitter-json, entities, sparql-json)
- output: tweets, entities, triples
- http://knoesis1.wright.edu/twarql/search?keyword=k1,...,kn&output=<output type>
- D2R
- http://knoesis1.wright.edu/twarql/register?query=<sparql query>&id=<concept feed id>
- #id
- http://knoesis1.wright.edu/twarql/stream?keyword=k1,...,kn&output=<output type>
Output Formats
We also provide output according to the format presented on the twitter-api-announce message.
{
"text" : "hey @raffi tell @noradio to check out http://dev.twitter.com #hot",
...
"entities" : {
"user_mentions" : [
{
"id" : 8285392,
"screen_name" : "raffi",
"indices" : [4, 9]
},
{
"id" : 3191321,
"screen_name" : "noradio",
"indices" : [16, 23]
}
],
"urls" : [
{ "url" : "http://dev.twitter.com",
"indices" : [38, 64]
},
],
"hashtags" : [
{ "text" : "#hot",
"indices" : [66, 69]
"url" : "http://search.twitter.com/search?q=%23hot"
}
]
}
...
}
Error/Warning Messages
ERROR
- Unknown Stream: You are requesting a stream id that was not registered.
- Invalid Query: You are trying to register an invalid SPARQL query.
- Unsupported Content-type: The requested content type is not supported.
WARNING
- No Results: There are no results for the query.
Supported Clients
- SPARQL Protocol-compliant Clients
- Cuebee is a SPARQL query formulation and results exploration engine. We provide a TweetExplorer that can be directly plugged into Cuebee.
- RSS/Atom clients
- View SparqlPuSH
People
You may contact us if you have any questions about the implementation or API. We try list our major contributions below our names so that you know to whom you should direct your question.
- Pablo Mendes
- Architecture, SPARQL client (Cuebee), Social Sensor (Extraction, Annotation), API, Documentation
- Pavan Kapanipathi
- Application Server, Semantic Publisher, Streaming SPARQL, API Content Negotiation
- Alex Passant
- SparqlPuSH, Annotation Vocabularies
