Twarql

Twarql: Twitter Feeds through SPARQL

Introduction

Our approach encompasses the following steps:

• extract content (entity mentions, hashtags and URLs) from microposts;
• encode content in a structured format (RDF) using shared vocabularies (FOAF, SIOC, MOAT, etc.);
• enable structured querying of microposts (SPARQL);
• enable subscription to a stream of microposts that match a given query (Concept Feeds);
• enable scalable real-time delivery of streaming data (SparqlPuSH).

Architecture

• See the workflow between the components of the architecture

Tweet Annotation

• extract content (entity mentions, hashtags and URLs) from microposts;
• encode content in a structured format (RDF) using shared vocabularies (FOAF, SIOC, MOAT, etc.);

Concept Feeds

• enable structured querying of microposts (SPARQL);
• enable subscription to a stream of microposts that match a given query;

SPARQLPuSH

• More about the SPARQL Push implementation: Introducing sparqlPuSH, open source implementation

Twarql API

REST Endpoints

Summary:

• URL scheme: http://<base-url>/<operation>/?<parameter>=<value>&...&output=<output format>
• Base URL: http://knoesis1.wright.edu/twarql
• Operations: search, register, stream, query
• Output formats: twitter-json, sparql-json, entities

Parameters:

• http://knoesis1.wright.edu/twarql/search?keyword=k1,...,kn&output=<output type>
  • input: keywords, output type (twitter-json, entities, sparql-json)
  • output: tweets, entities, triples

• http://knoesis1.wright.edu/twarql/search?keyword=k1,...,kn&output=<output type>
  • D2R

• http://knoesis1.wright.edu/twarql/register?query=<sparql query>&id=<concept feed id>
  • #id

• http://knoesis1.wright.edu/twarql/stream?keyword=k1,...,kn&output=<output type>

Output Formats

We also provide output according to the format presented on the twitter-api-announce message.

{
"text" : "hey @raffi tell @noradio to check out http://dev.twitter.com #hot",
...
"entities" : {
 "user_mentions" : [
 {
   "id" : 8285392,
   "screen_name" : "raffi",
   "indices" : [4, 9]
 },
 {
   "id" : 3191321,
   "screen_name" : "noradio",
   "indices" : [16, 23]
 }
],
"urls" : [
 {      "url" : "http://dev.twitter.com",
   "indices" : [38, 64]
 },
],
"hashtags" : [
 {      "text" : "#hot",
   "indices" : [66, 69]
   "url" : "http://search.twitter.com/search?q=%23hot"
 }
]
}
...
}

Supported Clients

• SPARQL Protocol-compliant Clients
  • Cuebee is a SPARQL query formulation and results exploration engine. We provide a TweetExplorer that can be directly plugged into Cuebee.

• Regular Twitter clients
  • To support plain Twitter clients, we may have to implement their style to content negotiation http://www.zienit.nl/blog/2010/01/rest/control-jax-rs-content-negotiation-with-filters

• RSS/Atom clients