Monday, 31 December 2012

Linking ORBIS to Pelagios

Parameterized route finding in ORBIS
ORBIS: The Stanford Geospatial Network Model of the Roman World has recently joined the Pelagios network, thanks to the encouragement and patient assistance of Leif, Elton and Rainer.

Our initial steps were these: (a) create an API allowing a few simple RESTful queries of the ORBIS database (orbis.stanford.edu/api), (b) publish an RDF file which maps our URIs for 402 of ORBIS’ 751 sites to Pleiades places, and (c) build a simple dynamic map landing page for the URIs.

Project background

The initial phase of the ORBIS project was completed with a web site launch in May, 2012. Development has largely paused for several months, but will resume in early 2013. The goal of the project is to model Roman communication costs in terms of both time and expense, and to permit scholars and the general public to interact with that model, by means of a route-finding web map and an interactive distance cartogram. By simulating movement along the principal routes of the Roman road network, the main navigable rivers, and hundreds of sea routes in the Mediterranean, Black Sea and coastal Atlantic, the model reconstructs the duration and financial cost of travel in antiquity. The purpose, development and use of the model and web application are described in considerable detail on the site.

Interactive distance cartogram in ORBIS, showing ‘grain-distance’ to Rome during June, in denarii
The ORBIS project is led by Walter Scheidel (Professor of Classics and History); the novel geospatial model, database and cartogram were developed by Elijah Meeks (Digital Humanities Specialist). Yours truly (Karl Grossner, Digital Humanities Research Developer) arrived to the project in time to become its web developer.

So, what does ORBIS add to the growing graph of annotations to Pleiades places that Pelagios enables? The initial answer is that for each of 402 places we can tell you which Pleiades places are directly connected to it by a single network segment. In the coming weeks we will be extending both our annotations and the range of API queries, and report that here.

ORBIS places

The ORBIS database includes 751 sites, most of them urban settlements but also including important promontories and mountain passes; 268 of these served as sea ports. ORBIS sites are drawn, with a few adjustments, from the Barrington Atlas of the Greek and Roman World (Talbert 2000). The Barrington map key categorization for size and importance of settlements is maintained, in five rank classes (in ORBIS: increments of 10, 60 to 100). Of the 751 sites, 402 have been identified as identical with Pleiades places, and this is the basis for ORBIS’ connection to the Linked Data cloud.

The road network encompasses 84,631 kilometers of roads and desert tracks, and 28,272 kilometers of navigable rivers and canals. This multi-modal terrestrial network is described in the spatial database in terms of 625 distinct road segments and 107 distinct river or canal reaches. As an aside, these segments and reaches are places as well—certainly events have occurred at, on, or near them. Granted the first order of business in our collective effort at large-scale digital historical gazetteers is point locations, principally for settlements. However, named and unnamed paths (e.g. roads, rivers) and potential or actual routes taken upon them (journeys or trajectories), are entities of interest as well—as sites of historical activity and events.

Creating an API

The ORBIS routing application calculates routes between sites, along a multimodal network comprising the terrestrial segments mentioned above, and a 0.1 degree lattice laid over the Mediterranean and Black Seas and North Atlantic Ocean. The route calculation accounts for 8 parameters, each with multiple possible values: start (639), destination (639), month (12), priority (3), network type (5! = 120), river vessel (2), sea vessel (2), land vehicle/mode (9), and price (3). Obviously, results for these billions of distinct combinations can’t be (and needn’t be) pre-calculated and published; a simplified API can deliver results programmatically. So, what to offer in an API? We have begun simply. You can request for example,
  • A dump of all 402 sites mapped to Pleiades places, in JSON format
    http://orbis.stanford.edu/api/sites
  • A listing of sites one network hop from a given site (using either Pleiades place ID or ORBIS site ID) by road or river, with segment length, e.g
    http://orbis.stanford.edu/api/sites/108772
  • A route between two places. There are four required parameters: start, destination, month and priority (fastest or shortest route). For example,
    http://orbis.stanford.edu/api/route_pl/79299/108772/6/1
    requests the shortest path between Aquae Sulis and Augusta Suessionum in June using Pleiades IDs—and returns a JSON object with the estimated trip distance and duration, and a sequence of 7 segments (6 road, 1 coastal sea). For each segment, distance, duration and type are returned as well.

API technical details

Because ORBIS uses PHP for database access, and the syntax we preferred uses slashes (like orbis.stanford.edu/{param1}/{param2}/{param3}, I found the Slim API framework (http://www.slimframework.com/) to be simple and effective. In our case this meant a central index.php referring to separate PHP scripts for each of our three offerings, and a single mod_rewrite statement added to our Apache web server configuration. The API queries are simplified version of existing queries in the ORBIS application. For further details, just contact me.

RDF publication

Pelagios required first and foremost a web-published RDF document describing (minimally) mappings between ORBIS sites and Pleiades places, in the form suggested by the Open Annotation Collaboration (openannotation.org ). One such entry is as follows:

<rdf:Description rdf:ID="orbis_50012">
    <rdf:type rdf:resource="http://www.openannotation.org/ns/Annotation"/>
    <oac:hasBody rdf:resource="http://pleiades.stoa.org/places/108751"/>
    <oac:hasTarget rdf:resource="http://orbis.stanford.edu/api/site/50012"/>
    <dcterms:creator rdf:resource="http://orbis.stanford.edu/"/>
    <dcterms:title>The Roman era place, Ara Agrippinensium</dcterms:title>
</rdf:Description>


Generating this document was straightforward for us; the ORBIS sites table has always included a Pleiades ID field, and we had already made 402 mappings there. A SQL query that concatenated the necessary XML markup with site numbers and Pleiades IDs was a simple matter. The file can be freely downloaded, but is presently useful only to Pelagios.

Web landing page

The URIs published in the above file have to resolve to a meaningful location. Navigating to http://orbis.stanford.edu/api/site/50012 loads a page with a small map with markers for that site and the sites directly connected by a single segment to it.

A few comments

The ORBIS site does offer the option to export lists of calculated routes as KML for Google mapping, or comma-delimited format (CSV) for more general use. This is good as far as it goes, but we are intrigued at the prospect of ORBIS routes and related data being mashed up in other applications, and see a potential ‘multiplier’ effect in this capability. The ORBIS project will also be a partner collection in Anvil Academic’s Built Upon digital publishing initiative, which may present novel opportunities for linked data applications.

Pelagios allows anyone referring to a Pleiades place to discover other annotations for it, now including those of ORBIS. However, given the parametric nature of ORBIS data, what can a simple static URI provide?  For the time being, this is only the set of Pleiades places connected to that place by a single terrestrial segment. This seems to me of limited value, although as a Linked Data devotee I understand my job is simply to publish data, enabling applications we can’t necessarily predict. The API we’ve developed to return parameterized routes appears more immediately useful, and in fact we know of one pending application related to paths between coin finds and nearby mints. The annotation provided through Pelagios could be extended, but how? For example, would it be useful to provide, for a given place, distances and durations to all places of a given size or importance? Suggestions are most welcome.

______________________

Talbert, R. J. A. (ed.) 2000. Barrington Atlas of the Greek and Roman World. Princeton.

Friday, 5 October 2012

Bringing the “Book of the Dead” Places to Pelagios

The Book of the Dead is a collection of spells which accompanied the deceased in the realm of the dead. The spells supplied information about the residents, places and incidents in the afterlife which helped the dead person to avert danger and to be accepted among the gods. Altogether, the Book of the Dead - as a text corpus - comprises c. 200 spells. The actual composition of a single book, in contrast, varies, so that each source is unique.


The instances of the Book of the Dead are transmitted on papyri, mummy wrappings, shrouds, coffins etc. In total, there are nearly 3000 objects. Their records and photographs (~ 20.000) have been gathered and worked on within the Book of the Dead Project which started at the University of Bonn in the 1990s. Recently, they have been integrated into a digital archive, in cooperation with the Cologne Center for eHumanities (CCeH).

There are two kinds of place references included in the data records: references to the object's current location (country, place and institution) and to their provenance (place of origin and specific locality). The latter are being integrated into the Pelagios network. 

Alignment to the Pleiades Gazetteer


In terms of granularity, the places of origin have been chosen for the mapping as their level of granularity corresponds roughly to the level of the precisely located places identified in Pleiades (e.g. Athribis or El Kurru).
The first step in the alignment process was to import the Pleiades+ dataset via Oxygen and convert it to XML. This was done because the digital archive is based upon an eXist database and the data thus reside entirely in the XML cosmos. An XSLT script was written to map the Book of the Dead place occurrences to the places in the Pleiades+ dataset. That way, about half of the places could be mapped automatically. The results were checked in consideration of the geographic coordinates given in the archive. There was only one mismap: "Theben" was mapped to the Greek Thebes instead of the Egyptian Thebai.
Most of the remaining places were identified manually by means of their corresponding Greek place names. In the archive, a great deal of the place names are German transliterations of Arabic names. Up to now about 90% of the place names have been identified. 


Annotation and Dataset Metadata Creation


A main component of the digital archive's data model is an external knowledge base simply called "Wissen" (knowledge) bringing together additional information such as geographic coordinates, year dates for periodization, canonical and selective lists for spells, degrees of kinship. Part of this and relevant to the Pelagios alignment is a distinctive list of places of origin, to which the Pleiades IDs have been assigned.


An XQuery module was created to provide single annotations, a dataset dump and a dataset metadata description in RDF/XML. Single annotations and the dataset dump are created on the fly by joining the data record's place occurrences to the Pleiades IDs in the knowlegde base. So far, the annotations are organized as one single dataset as their number is relatively small (currently, there are 1346 annotations).
As a next step the Pelagios tools and widgets will be explored in order to deploy them on the Book of the Dead project website.

Tuesday, 2 October 2012

The Portable Antiquities Scheme joins Pelagios

Hacking Pelagios rdf in the ISAW library, June 2012
Earlier in 2012, the excellent Linked Ancient World Data Institute was held in New York at the Institute for the Study of the Ancient World (ISAW). During this symposium, Leif and Elton convinced many participants that they should contribute their data to the Pelagios project, and I was one of them.

I work for a project based at the British Museum called the Portable Antiquities Scheme which encourages members of the public within England and Wales to voluntarily record objects that they discover whilst pursuing their hobbies (such as metal-detecting or gardening). The centrepiece of this projects is a publicly accessible database which has been on-line in various guises for over 13 years and the latest version is now in the position to produce interoperable data much more easily than previously.

Image of the finds.org.uk database
The Portable Antiquities Scheme database

Within the database that I have designed and built (using Zend Framework, jQuery, Solr and Twitter Bootstrap), we now hold records for over 812,000 objects, with a high proportion of these being Roman coin records (175,000+ at the time of writing, some with more than 1 coin per record). Many of these coins have mints attached (over 51,000 are available to all access levels on our database, with a further 30,000 or so held back due to our workflow model.) To align these mints with a Pleiades place identifier was straightforward due to the limited number of places that are involved, with the simple addition of columns to our database. Where possible, these mints have also been assigned identifiers from Nomisma, Geonames and Yahoo!'s WOEID system (although that might be on the way out with the recent BOSS news), however some mints I haven't been able to assign - for instance 'mint moving with Republican issuer' or 'C' mint which has an unknown location.

Once these identifiers were assigned to the database, it allowed easy creation of  RDF for use by the Pelagios project and it also facilitated use of their widgets to enhance our site further. To create the RDF for ingestion by Pelagios, our solr search index dumps XML via a cron job cUrl request, which is transformed by XSLT every Sunday night to our server and uses s3sync to send the dump to Amazon S3 (where we have incremental snapshots). These data grow at the rate of around 100 - 200 coins a week, depending on staff time, knowledge and whether the state of the coin allows one to attribute a mint (around 45% of the time.) The PAS database also has the facility for error reporting and commenting on records, so if you use the attributions provided through Pelagios and find a mistake, do tell us!

At some point in the future, I plan to try and match data extracted from natural language processing (using Yahoo geo tools and OpenCalais) against Pleiades identifiers and attempt to make more annotations available to researchers and Pelagios.

For example, this object WMID-3FE965, the Staffordshire Moorlands patera or trulla (shown below):

Has the following inscription with place names:

This is a list of four forts located at the western end of Hadrian's Wall; Bowness (MAIS), Drumburgh (COGGABATA), Stanwix (UXELODUNUM) and Castlesteads (CAMMOGLANNA). it incorporates the name of an individual, AELIUS DRACO and a further place-name, RIGOREVALI. Which can further be given Pleiades identifiers as such:
  1. Bowness: 89239
  2. Drumburgh: 89151
  3. Stanwix: 967060430
  4. Castlesteads: 89133

Integrating the Pelagios widget and awld.js

Using Pleiades and Nomisma identifers allows the PAS database to enrich records further via the use of rdfa in view scripts and by the incorporation of the Pelagios widget and the ISAW javascript library on a variety of pages. For example, the screenshot below gives a view of a gold aureus of Nero recorded in the North East of England with the Pelagios widget activated:
The pelagios widget embedded on a coin record:  DUR-B4E094 
The javascript library by Nick Rabinowitz and Sebastian Heath also allows for enriched web pages, this page for Nero shows the libary in action:

These emperor pages also pull in various resources from third party websites (such as Adrian Murdoch's excellent talking head video biographies of Roman emperors), data from dbpedia, nomisma, viaf and the site's internal search engine. The same approach is also used, but in a more pared down way for all other issuer periods on our website, for example: Cnut the Great.


Integrating Johan's map tiles

Following on from Johan's posting on the magnificent set of map tiles that he's produced for the Pelagios project (and as seen in use over at the Pleiades site and OCRE), I've now integrated these into our mapping system. I've done it slightly differently to the examples that Johan gave; due to the volume of traffic that we serve up, it wasn't fair to saddle the Pelagios team with extra bandwidth. Therefore, Johan provided zipped downloads of the map tiles and I store these on our server (if you're a low traffic site, feel free to use our tile store):
Imperium map layer, with parish boundary. Zoom level 10.
The map zoom has been set to the level (10 for Great Britain) at which we decided site security was ensured for the discovery points (although Johan has made tiles available to level 11). This complements the other layers we use:

  • Open Street Map
  • terrain 
  • satellite
  • soil map
  • Stamen map watercolor
  • Stamen map toner 
  • NLS historic OS maps
Each find spot is also reverse geocoded for a WOEID and Geonames identifier to be produced, elevation to obtained and subsequently we link to Aaron Straup Cope's excellent woedb for further enhancement of place data.  We also serve up boundaries derived from the Ordnance Survey Opendata BoundaryLine dataset, split from shapefiles and converted to KML by ogr2ogr scripts. The incorporation of this layer allows researchers (over 300 projects currently use our data) to interpret the results that they get from searches on our database against the road network and settlement data much more easily and has already gathered many positive comments from our staff and research colleagues.

By contributing to the Pelagios project, we hope that people will find our resources more easily and that we in turn can promote the efforts of all the fantastic projects that have been involved in this programme. What we've managed to implement from joining the Pelagios project already outweighs the time spent coding the changes to our system. If you run a database or website with ancient world references, you should join too!