Initial evaluation of user needs for the Pelagios apps and widgets

This is the second in a series of posts about users’ requirements for Pelagios apps and widgets. The first post set the scene, and this one describes some of our initial findings from the "super user" group.

The questions we asked were about

Thanks to docpop for the pic.

• the target audience(s) for the apps and widgets
• what the Pelagios apps and widgets could do
  e.g. specific goals they could help people achieve, tasks they should help people carry out
• what sort of things different types of users might do with the apps/widgets
  If there are different audiences, what will these different audiences do
• barriers to exploiting Pelagios partners data
• technical and legal considerations

This post summarises the findings about the target audience(s), the widgets, and the perceived barriers.

The target audience(s) for the apps and widgets
Some respondents identified "super users" as the primary target audience for the apps and widgets. However, the majority of responses suggested that non-technical users should be catered for.
Suggestions on this theme included groups of people with both a professional and/or an  amateur interest in the ancient world:
- tourists who want to know about the cultural background of their vacation spot,
- historians that want to have an overview of their local history in terms of research,
- archaeologists that are interested in the findings and citations in literature related to the area of their excavations, -   fiction writers who want to get abetter historical background of a setting for a story
-   educators who want to give their students some background knowledge about the ancient history of a place they are looking at
- researchers that want to find things by place
and
"anybody who uses the web".
What the Pelagios widgets should do
Respondents suggestions varied from general guidance about the nature of the widgets (e.g. 'nice, compact, easy-to-transit view of related linkages', 'allow people to easily add some "Pelagios context" to their own place-related web pages') to some more specific suggestions.
The specific suggestions include:

• focus  on a topographical spot so that once embedded into a website the widget will display contextual data for a Pleiades place, e.g. showing any literature mentioning Isica  on a map, or as a list.
  
• interest in using Pelagios with the Concorcodia vocabulary
  
• enable the user to search on modern place names (e.g. Mainz instead of Mogontiacum)
  
• an API that outputs RDF/xml or plain XML of the entire Pelagios dataset, input parameters would be Pleiades-id or search string of place name
  
• a person is viewing a web page that mentions a place; the  widget that displays "other cool stuff" from the Pelagios universe related to that place
  

Barriers to exploiting Pelagios partners data
Concerns raised included

• the projects that are linked by Pelagios use different languages
• the PELAGIOS annotation spec is not complete and published. The format may change, thereby breaking any applications or queries we build around it.

The next post will be about requirements for the Pelagios apps, and technical and legal issues that were raised by respondents. By the way  if you have been invited to fill out the requirements questionnaire please do so, we are still analying responses but would like some more before the 9th of March :-)

That's all for now
Andrew

Pelagios Visualization Service: user requirements for the Pelagios apps and widgets

This is the first in a series of posts about users’ requirements for the Pelagios visualization service. The visualization service will be implemented as a set of web apps and widgets that will "enable providers, end-users and developers to find, visualize and build upon geospatial connections between openly available resources" (JISC Pelagios2 project summary). To put this post in context, the project plan adds a bit more detail indicating that the web apps and widgets will provide end users with intuitive and user friendly tools to visualize data available through APIs produced by WP1 (webcrawling and indexing service) and WP2 (place/space-based APIs and contextualisation service). The end users may be subject specialists, but they will not necessarily have the technical expertise to make use of this data directly. Overall, there are three key groups of users that we will target to gather requirements from:

• "Super users" - e.g. developers, digital humanities specialists typified by the Pelagios2 project partners


• "End users" - people with an interest in Pelagios data, but without the technical skills necessary to exploit it e.g. ancient history students and
  teachers, folk who visit ancient sites on holiday


• "Web site admins" - people who own administer sites that could embed Pelagios widgets. These could be museum or history related sites, and it's their technical requirements and restrictions that are the initial concern.

This first post sets out to discriminate between web apps and widgets, with the intention (and hope) that explaining the meaning of these terms will help the interpretation of future posts about the users' requirements. We're also thinking that to gather requirements from non-techie end users it will be useful to give some examples of apps and widgets that they may already be familiar with.

What is a web app? What is a web widget?

‘Web app’ and ‘web widget’ are shorthand terms to summarise some features of web based tools and applications.

We are using the term ‘app’ to refer to a application that would typically be accessed in its own web page, and allows the user to manipulate data and relationships between items of data. For example, the Flickr web app shown in this screen grab enables the Flickr user to drag pictures from the group at the bottom of the screen into the working area in the middle of the screen, and then edit data about the selected photos (title, location time taken etc.), and to group the photos into sets. (Please click on the image to open a clearer view of it).

In contrast, a ‘widget’ is an interface to a web application that can be embedded within a web page. A widget will typically offer fewer options, and less functionality, than an app to the user, because often it will occupy less screen space. A widget may meet one specific need, whereas an app may meet several, and usually a widget allows data to be displayed, whereas an app enables data to be displayed and edited.. The Flickr slide show shown below is an example of a widget. This allows the end user to embed a copy of a set of photos in their site. In this case the 'end user' is the person embedding the slide show, not the person who created it.


For media objects such as photos, videos and music, web apps typically provide a way to upload, tag and describe individual items, or how items relate to one another, or to edit the items themselves. Widgets typically provide the functionality to embed a media object in another page (or a set of objects), and allow access to a subset of the descriptive information that is available via the full app user interface.
It may be the case in Pelagios, as in other areas, that there will be a grey area between apps and widgets.
Next steps
We have begun by surveying super users (including Pelagios partners) using a questionnaire, and in the next post I will describe initial feedback from this group about their requirements for Pelagios web apps and widgets. By the way, if you have received an invitation to respond to the questionnaire, we do want more responses so please do complete it and return it!
Thanks
Andrew (the inept photographer responsible for the images embedded in this post)

Pelagios usability testing results

In my earlier posts, Evaluating Pelagios' usability and Evaluating usability: what happens in a user testing session?, I promised I'd share some preliminary results. Because my last posts were getting rather long, I'll keep this short and sweet.

I had two main questions going into the user testing: could a user discover all the functionality of Pelagios, and could they make sense of what they were shown?  In short, the answer to the first question is 'no', and the answer to the second is 'yes'.

I'm a big fan of testing the usability of a site with real users.  Test participants not only give you incredibly useful insights into your site or application, but they help you clarify your own thoughts about the design. It was exciting to see test participants realise the potential of the site - particularly the map and data views, which is a cue to make them more prominent when the site first loads - but it was clear that the graph interface needs improvements to make the full range of actions available for selecting, linking and exploring datasets more visible to the user.  The test participants also used search heavily when looking for particular resources, so this would be a key area for future work.

If you've been working on a project, user testing is wonderful and painful in equal measures.  It's definitely easier to test someone else's project, not least because it's easier to prioritise tasks from the users' point of view when you don't have to deal with the details of implementing the changes.

The overall goal of the usability testing was to produce a prioritised list of design and development tasks to improve the usability of the Pelagios visualisation for a defined target audience (non/semi-specialist adults with an interest in the ancient world), and this user testing was really successful in giving the team a clear list of future tasks.

Evaluating usability: what happens in a user testing session?

In my last post I talked about the test plan for assessing the usability of the Pelagios 'graph explorer' for the project's (deep breath) 'non/semi-specialist adults with an interest in the ancient world' audience. Before I get into the details of what happens in a usability test session, I thought I'd introduce you to our design persona, Johanna.

Image credit: @ANDYwithCAMERA

Johanna is 21, and is a third year History student. She moved from her native Germany to the UK three years ago for university. Her goal is to get a First so she has more options for future academic work, perhaps in the Classics. She's slightly swotty, and is always organised and methodical, but finds that she's easily distracted by Facebook and chat when she's working on the computer. She can often be found having coffee or in the pub with friends, at her part-time job in a clothing store, or in the library (her shared house is often noisy when she's trying to work). She dislikes distractions when she's trying to study, and hates rude customers at work. She likes her bike, RomComs and catching up with friends. Her favourite brands are Facebook, MacBook, Topshop, Spiegel Online and The Body Shop. Her most important personal belongings are her laptop, her mobile phone, and photos of friends and family from Hamburg and college.

Johanna is technically competent, and prefers to learn through trial and error rather than reading manuals or instructions. But she also has limited patience and will give up on interfaces that are too difficult. Johanna is a heavy user of social networks and also uses online research databases and library catalogues.

Johanna has an assignment on inscriptions due in a month. She hates the emphasis on big battles and big men in the subject, and finds inscriptions dry, but has been told they can also convey interesting social history and cultural values. She's not convinced (and she's not sure whether she'll be able to make much of the language of the inscriptions) so she wants to find an ancient place that also has other historical material about it to make the assignment more relevant to her own interests.

To create our persona and design the test tasks, I quizzed Elton on the types of questions people ask when they find out he's a Classicist to get a sense of common (mis)perceptions and interests, and about the types of students he's encountered.

So, onto the usability tests themselves. The time and venue for each test was organised directly with the participant, with the restriction that we had to be able to get online, be in an environment where it was ok to talk aloud, and ideally we'd meet somewhere the participant would feel comfortable.

In my last post I mentioned writing and testings some set tasks for the usability test, a short semi-structured interview, and an introductory script. Once the participant had arrived, and was settled with a cup of tea or whatever, I'd introduce myself and explain how I came to be working with the project. I've included the basic introductory script below so you can get a sense of how a test session starts:

Thank you for agreeing to help us test the usability of the current interface for Pelagios.

We'll be using these tests to produce a prioritised list of design and development tasks to improve the Pelagios visualisation for people like you.

The session will take up to an hour and will start with a short interview, then your initial impressions of the site, and finally we'll go through some typical tasks on the site. I'll ask you to 'think aloud' as you use the site - a running stream of thoughts about what you're seeing and how you think it works. I might also ask you questions to clarify or explore interesting things that come during the session.

I want you to know that you're not being tested! We're testing the interface - anything that goes wrong is almost definitely its fault, not yours! Also, I haven't been involved in the project design, so you don't need to worry about hurting my feelings - be as direct as you like about what you're seeing.

I won't be recording this, but I will be taking notes as we go, and summarising them to pass them on the project team.

You can stop for a break or questions at any time.

Do you have any questions before we begin?

The next phase of the test session was the short interview. Again, I've included the questions below:

• Demographic data: what is your age, gender, educational level, nationality/cultural background?
• What websites do you use regularly (on a daily/weekly basis)?
• What's your favourite website, and why?
• What websites do you use in your research/daily work?
• Have you seen sites like [Guardian, Gapminder, etc] that feature interactive visualisations?
• How would you describe your level of experience with the classics? (e.g. a lot, a little). Do you focus on any particular area?
• What is your definition of the classics? (Geographical, chronological scope)

Once the questionnaire was over, and any questions that had arisen had been discussed, the test began. The first part of the test covered first impressions of the 'look and feel' of the site, what they thought the site might be about, and what content it would include, and what they thought the 'blobs' that are the first view of the graph visualisation represented. I was also observing the kinds of interactions participants tried with the visualisation, whether single or double mouse-clicks, dragging, right-clicking, etc, because I wanted to know how much of the functionality of the site was intuitively discoverable.

The first formal task was: "find all the resources related to Cyrene" [or a place related to their own interests]. I'd note the actions the participants took along with their comments as they 'thought aloud'. Sometimes I'd ask for more information about why they were doing certain things, or remind them to tell me about the options they were considering. I also noted the points where the participant expressed confusion or frustration, or gave up on a task, though I didn't time the tasks or record a qualitative count of errors.

After the task, I'd ask (if it hadn't already come up):

• What do you think these resources are?
• How do you think they relate to your actions?
• What contextual information might you need to make sense of these resources?

These questions were based on the team's review of the site and were aimed at making sure we understood the participant's 'mental model' of the site. If there's a mismatch between the users' mental model and what your site actually does, you need to help users develop a more appropriate mental model.

The second task, "Are there links between [Place 1, Place 2]? If so, what are they and how many are there?" was more open-ended and designed to see how participants managed small result sets on the site. Again, I had questions prepared as prompts in case they hadn't already been answered during the task:

• What do you think you're looking at here?
• What does the screen tell you?
• What do you think the links mean/are?
• What do you think the movements on the screen mean?
• How do you interpret the results?
• How do you think they're selected?

Finally, I asked some questions aimed at giving the project some metrics to measure improvement in the usability of the site after design updates: 'would you use the site again?', 'how likely are you to recommend it to a friend?'.  The final questions were: 'what would you suggest as first priority?' and 'any final comments?'.

After running each test, I'd tidy up my notes and summarise the key points for the team so they could prioritise the next items of design or development work. Which leads me onto my next post, which will include some preliminary results...

Evaluating Pelagios' usability

Hello!  I'm Mia, and I was drafted into the Pelagios project to run some usability testing on the 'graph explorer'. (These days I'm working on a PhD in Digital Humanities in the department of History at the Open University, but until quite recently I worked as an analyst/programmer and user experience designer, mostly in museums, and in early 2011 I completed City University London's MSc in Human-Computer Interaction).

There's a range of usability methods we could have used to evaluate Pelagios' usability, but the 'gold standard' is user testing (basically, showing the site to typical users and gathering their feedback as they complete set tasks). It requires more resources to set up and run the user tests than other usability techniques, but it's particularly useful for 'novel' interfaces like the Pelagios visualisation.  Other common methods are testing with paper prototypes (e.g. if you haven't got a working site), card sorting, or having experts review the site according to usability checklists (AKA 'heuristic evaluation').

Quite a bit of work goes into preparing and piloting user testing. Once I'd written a usability testing plan, I worked with Elton, Leif and Rainer to define the key audiences ('subject specialists' and 'non-specialists with an interest in the classics/ancient world' - specialist is a relative term in this context) and create a persona to represent the specific target audience we were going to test for, the snappily-titled 'non-specialist adults with an interest in the classics/ancient world'.  In addition to focusing our minds on the usability requirements of this audience and typical tasks they might undertake on the site, this persona would be used in future design processes to ensure the project delivers user-centred designs.

We also reviewed the available functionality and interfaces to design test tasks that would help us understand where improvements were needed to make the graph visualisation more useful to its audiences.  The trick is to write tasks that make sense to the test participants that will also lead them to use key areas of the site.  I also included an initial open-ended question to elicit overall impressions of the site, as it's a good way to gather feedback on the overall design and get a sense of what the participant thinks the scope of the site might be.

I also designed a short semi-structured interview - a set list of questions to ensure consistent data collection, with the flexibility to explore interesting issues that will provide insight into user requirements, expectations and mental models as they arise.  I included questions on other sites the participants use regularly for research or leisure, as these will give some idea of their expectations of other sites. It's helpful to order your questions so the easy ones are first, as it gives the participant a chance to relax and get used to the situation.  User tests use the 'think aloud' protocol, where the participant, well, thinks aloud, sharing the thoughts and questions that are running through their minds as they use the site and go through their tasks.

Meanwhile, Elton was recruiting participants - we were aiming to include people interested (but not yet specialist) in the Classics or ancient world, to match our target audience as closely as possible. Once I had the test tasks and interview in order, I wrote a short introductory script to read at the start of each testing session.  This script helps the tester remember to give everyone the same information so the tests are consistent and the participant has a positive experience.  I then ran a pilot test with a volunteer participant, including the interview and intro script - this is one of the most important stages, because it helps you refine your language so it's clear to people new to the project, check the timing of tasks and make sure everything works as expected.

In my next post I'll explain what happens in a usability test session, and share our design persona with you. In the post after that, I'll share some of the results... Post below if you have any questions or comments!

Pelagios Project Plan Part 5: Project Team Relationships and End User Engagement

Pelagios is made up of an international consortium of projects leading digital research on the ancient world, the more established of which already have large user communities: in short, these groups are the end users of the Pelagios project. Each partner has at least one prominent member on the Pelagios working group to represent them. So, welcome to the team:

Elton Barker (GAP, The Open University) Principal Investigator

Elton is the non techie on the project: he’s a lecturer in Classical Studies at the OU with a specialism in all things Greek (Homer, tragedy, historiography...). But he’s slowly slipping into the murky world of Digital Humanities having been Principal Investigator of HESTIA - a project investigating spatial concepts in Herodotus - and now GAP. (He’s also assisting in the promotion and understanding of DH at the OU.) Elton is in overall charge of all things Pelagios related, like overseeing progress of the work packages and making sure that the team delivers on what it promised: in other words, it's his neck on the JISC chopping-block if things go pear-shaped - not that he's worried. He’s also responsible for GAP’s contribution to project documentation (meaning that he’ll try to put the techie talk into plain English).

Leif Isaksen (GAP, Southampton) Co-Investigator

Leif, a strange hybrid - part philosopher, part classicist, part archaeologist, and total computer geek - is a Research Fellow at the Archaeological Computing Research Group (Southampton). Having been the technical consultant on HESTIA, Leif has moved on to be Co-I for both GAP and Pelagios. Leif is the go-to man for all the technical components of the project, working in conjunction with Development Partners to ensure infrastructural outputs are delivered. He’s also responsible for GAP’s data contribution to the project (i.e. the techie stuff).

Tom Elliott, Sean Gillies and Sebastian Heath (Institute for the study of the Ancient World) Development Partners

Tom is managing editor of Pleiades, an innovative ancient world online atlas, which was established with a grant from the U.S. National Endowment for the Humanities and which he has since developed as Associate Director for Digital Programs in the Institute for the Study of the Ancient World at New York University.
Sean Gillies is Pleiades’ chief engineer and representative in open source web and GIS initiatives. Together, he and Tom will provide support and expertise in mapping ancient place references to Pleiades’ uniform resource identifiers for each location (so called URIs). As part of this task they are helping to develop the core ontology by which different projects can point their place reference data to the Pleiades URIs.

Sebastian Heath is a Ceramics specialist with expertise in Linked Data and also responsible for nomisma.org.

Mathieu d’Aquin (LUCERO, The Open University) Dev Partner

Mathieu is a researcher at the OU’s Knowledge Media Institute and project director of the JISC-funded LUCERO project, which is exploring the means of integrating Linked Data practices for research and education in higher-educational organizations. As our ontology guru, Mathieu is looking to inject some Linked Data goodness into our project: he’s responsible for implementing the core semantic infrastructure, and will assist all the partners in its delivery and documentation.

Rainer Simon (DME, Austrian Institute of Technology) Dev Partner

Rainer is presently involved in the EuropeanaConnect project, where he leads research and development activities concerned with semantic media annotation, including the development of demonstrators for annotating and linking digitised maps and geospatial content. Rainer, then, has the coolest job of us all - to come up with funky ways of experimenting with and visualizing what Pelagios can do for you. He’s also responsible for DME’s data and documentation contributions to the project.

Greg Crane (Perseus, Tufts) Data + Doc Partner

Greg is Editor-in-Chief of the Perseus project, a trail-blazer in the world of Digital Humanities for its programme of digitizing classical texts - it currently hosts the world’s largest classical online library - with a particular focus on organising the data to meet user needs. Among supporters of Perseus are: the Annenberg/CPB Projects; the Digital Library Initiative Phase 2; the National Endowment for the Humanities and the National Science Foundation; the Institute for Museum and Library Services; the Mellon Foundation; and Google. Overseeing this vast digitised Classical empire, Greg is responsible for Perseus’ data and documentation contributions to the project.

Reinhard Foertsch (Arachne, Cologne) Data + Doc Partner
Reinhard is director of the CoDArchLab (computing archaeology) at Cologne University and leads the Arachne project, the central object-database of the German Archaeological Institute (DAI), which has over 6,000 registered users and approximately 981,000 scanned images and documentation for roughly 270,000 sites accessible free of charge and, helpfully, in English. Arachne is a partner of CLAROSnet and has recently won, like GAP and Perseus, a Google Digital Humanities Research Award. Reinhard is responsible for Arachne’s data and documentation contributions to the project.

Mark Hedges (SPQR, KCL) Data + Doc Partner

Mark is Deputy Director of the Centre for e-Research at KCL and Director of SPQR, which is not a project about the famous phrase describing the constituent part of the Roman Republic (Senatus Populusque Romanus) but rather one that adopts Semantic Web and Linked Data approaches to allow researchers to formalise resources and the links between them more flexibly. Mark is responsible for SPQR’s data and documentation contributions to the project.

Neel Smith (Ptolemy, Holy Cross) Data + Doc Partner

Neel is Associate Professor of Classics at the College of the Holy Cross (Worcester, Mass.), an architect of the Homer Multitext project, a principal designer of the Center for Hellenic Studies' CITE architecture, and is editing a geographically-aware digital edition of Ptolemy's Geography. He has been involved in digital scholarship since the early 1980s and still uses ed for quick edits.

Sebastian Rahtz (CLAROS, Oxford) Data + Doc Partner
Sebastian is the head of the Information and Support Group of Oxford University Computing Services. In this capacity he provides the technical support and development for CLAROS. He also directs the Lexicon of Greek Personal Names and works on and for the Text Encoding Initiative.

Other key project members are:

Andreas Geissler & Karin Hoehne (Arachne), Tobias Blanke + Gabriel Bodard (SPQR), Kate Byrne + Eric Kansa (GAP), Alex Dutton (CLAROS)

End-User Engagement
Pelagios has received feedback from a range of national and international organisations and initiatives including Google, EDINA, ADS, STELLAR, EnAKTing, UKLP, GeoNames, GeoDia, the European Commission, Open Context, Orbis, Regnum Francorum Online, Digital Atlas of Roman and Medieval Civilizations (Harvard) and Open Encyclopaedia of Classical Sites (Princeton). Many of these groups will be participating in the Pelagios workshop at KCL on 24 March. This event is open to ALL. Anyone wishing to attend should contact Elton at e.t.e.barker@open.ac.uk.

In addition to this blog, we are also archiving all group email discussions under pelagios-project@googlegroups.com. At the end of the project (October 2011) we will wrap up with a Web-launch of the ontology, tools and services to maximise impact.

Pelagios has consulted closely with JISC in finalizing the plan, budget and website and will also interact with, and solicit feedback from, the JISC Geospatial Working Group.