It's All One Big Opportunity

Joanne Cook, Senior IT Support and Development Officer, Oxford Archaeology/OA Digital

Abstract:Location runs all the way through archaeology- from the position of an individual find to the distribution of settlements in the landscape. We do mobile GIS, augmented reality, and high-end spatial analysis, and it's all open source. Going open has not only helped our company survive in very difficult circumstances, but has provided new opportunities for innovation, improved efficiency, and helped us better meet our company goals. We prove that it's possible to run a professional company in this way, but also to innovate and succeed in an economic downturn. Our experiences may be of value to other companies faced with a similar situation- can you also turn adversity into opportunity?

Introduction:In this paper I'm going to work through a hypothetical archaeological excavation, explaining how we achieve, or are trying to achieve, the various steps with open source software. I will talk about the challenges and misconceptions we have overcome, and the opportunities and benefits we have gained from our approach. I will try and place these challenges, misconceptions and opportunities into a more general concept, and highlight ways in which I believe other companies can learn from our experiences.

What's this archaeology thing about- is it like Time Team?The formal definition of archaeology is the study of past societies through their human remains. The purpose of archaeology is not only to tell us about our ancestors, but to provide insights into how we got to where we are now. For example, how did our ancestors deal with (or cause) warfare, climate change, or epidemics? Archaeology allows us to put our own lives in context- we are who we are, and live in the society that we live in, because of everything that has gone before.

A few common misconceptions remain about archaeology. Firstly we don't (often) encounter dinosaurs, and secondly, unlike on Time Team we usually take considerably more than three days to study a site. We usually require more than a single piece of pottery to reconstruct an entire Roman villa too! Thirdly, archaeology is no longer generally the pursuit of elderly retired gentlemen antiquarians or adventurers, but by highly qualified professionals, often with post-graduate degrees in their chosen specialism.

In the UK, archaeology is generally undertaken as part of the planning process prior to building work. Some level of archaeological intervention, be that a simple desk-based investigation of historic maps through to a full-scale open area excavation, has been a legal requirement of the planning process since 1990. The majority of archaeological work is undertaken by commercial units, via a highly competitive tendering process overseen by the county councils, although some university-based research also takes place.

Archaeology therefore forms part of the complex chain of contractors, sub-contractors and deadlines that make up the average construction job. Though a very small part of the process, delays in the archaeological phase, such as from the discovery of something important or unexpected, can lead to massive delays and therefore penalties later in the scheme. Consequently, as a result of the tendering process and of integration in the overall construction scheme, archaeologists need to work to extremely tight budgets, to strict deadlines, in a harsh environment, and to liaise and share data with many other contractors. Not only that, but the type of data we gather and use is extremely varied. As a result, our IT requirements can be quite

AGI GeoCommunity '10: Opportunities in a Changing World'Innovate - Connect - Succeed'

demanding, but we are a niche software market, unattractive to traditional software or hardware manufacturers.

Oxford Archaeology are the largest commercial archaeological unit in Europe. We have 400 staff, spread over three offices in the UK, and two in France, and are called on to provide expertise at an international level, in countries as diverse as China, Iraq, Turkey and Tibet.

The requirement for opennessOur data is often the only record that remains of a site, once it has been developed. Our understanding of the site is necessarily incomplete, due to the limited size of the excavation trenches, and it is not uncommon for findings to be contradicted by later work, often decades later. At the very least, later work may place our findings into a wider context. Consequently our primary data must be kept in perpetuity, so that later researchers can access it, and can repeat the analytical processes that led to the final conclusions. With a paper archive, this is relatively straightforward. However, with digital data and devices, the situation is quite different. Either it is necessary to maintain devices and packages capable of reading the data, and spend time curating the archive, ensuring that it remains readable, or to choose open, published formats and programs that will always be readable. The long-term impact of using a non-standard format, readable only by a particular software package or device, however suitable that is for the recording task, will be to render our archives useless or inaccessible. Moreover, for the analytical process to be truly repeatable, at any later point, the algorithms used should be accessible, or the process is a “black-box”, repeatable only by researchers with the same version of the same software running on the same operating system.

The Oxford Archaeology “Open Ethos”Our approach to the requirement for openness outlined above is called “open archaeology”. This has three parts:

Open access to our data, Open standards for file formats, Open source for our software.

As part of this, we are trying to create a work-flow for excavating a site that uses only open source software.

The archaeological processIn broad terms, the process of excavating an archaeological site has the following phases: Excavation, Analysis and Publication.

ExcavationInitially, trenches are laid out across the site with professional-grade survey equipment. Within a trench, buried features manifest themselves as changes in colour and texture of the soil, known as contexts. Every context is planned (drawn to scale) and recorded before being removed, and an average excavation may contain several thousand contexts. Artefacts are generally recorded in relation to the context in which they are found, as that relationship is one of the key ways in which the development of a site is understood during the analytical phase of the work. Previously this process would have been almost entirely paper-based, with site records often being measurable in metres of paper rather than number of sheets.

The search for a device for digitally recording data on site, to replace the paper archive, has been going on for decades. Approaches to the device problem include electronic paper, ruggedised laptops and pdas. Devices must be capable of surviving in a dirty, wet environment, preferably be operated by cold or gloved fingers, and have good battery life. Old favourites include Psion series 5s, which can run for days on standard AA batteries and will work inside a plastic bag, but these are now difficult to acquire, technologically limited, and only solve part of the site recording problem.

AGI GeoCommunity '10: Opportunities in a Changing World'Innovate - Connect - Succeed'

Our aim is for a digital recording system that collects tabular data in a database, spatially locates it, and allows photos to be attached. The database needs to synchronise with a server-based master version, needs to support multiple users, and also allow device-based data storage in areas where there is little or no data signal. Having trialled a number of devices over the years, we have discounted single-function devices in favour of multi-function devices such as pdas and now smart phones, to keep our overall costs down.

One device that seemed originally seemed very promising was the openmoko phone. This is a totally open source phone (both hardware and software), with built-in GPS, accelerometer, USB host capabilities, a very readable screen, and a robust bump-proof body. Unfortunately for openmoko, the revolution in Android phones pretty much took away their target market, and the phones are no longer being developed.

Our current target device is a Motorola milestone phone. We issue these to most of our staff, for use as their phone, and as a digital recording device. Getting good enough batter life from the milestone is a challenge, but we are investigating solar charging options and it is relatively easy to provide spare batteries. Bump proof cases, and waterproof bags protect the phones on site. The increase in cost of the phones themselves (actually not as bad as it seems as part of a company-wide contract) is offset by savings in data collection costs, both on site and later in the analytical phase of the project. There are many additional benefits to this approach. Smart phones mean staff can check their email on the move, which is important as many of our site staff may never visit one of the main company offices. This helps them stay engaged and feel part of the company. They are also a conspicuous sign of company investment in our staff, which again helps with engagement and morale and ensures that staff take good care of them- all important factors in the present economic climate!

A promising data collection program is Epicollect (http://www.epicollect.net/), that meets most, if not all, of our data collection requirements. Staff can take geospatially enabled photos of all features and finds, and immediately attach them to their database entry. This package is being paired with gvSIGmini (http://confluence.prodevelop.es/display/GVMN), which provides a feature-rich GIS client. The master database is in PostgreSQL, spatially enabled with PostGIS. Full enterprise level database functionality ensures our data is secure, and structurally robust.

AnalysisThis phase is often the longest part of an excavation. It begins with the transfer of all the records gathered on site into a database. When site archives are mainly paper-based, this process can take months, and often errors are introduced simply due to typographic mistakes. When a site archive is created digitally in the first place, this phase of work can be much reduced. Specialists then use the archive to reconstruct the developmental sequence of the site, examine the finds, and place the site in the context of it's wider geographic and historic surroundings. Often this phase of the project is a high-level analytical process. Examining thousands of sherds of pottery, or pieces of animal bone for example, requires statistical techniques. Increasingly, geospatial analytical techniques are used as part of this process. For example, when examining prehistoric sites where the evidence is mainly artefactual rather than physical, the changes in density of flint scatters across a site can help pin-point areas of specific activity such as tool manufacture or animal skinning.

Previously, Microsoft Access was used as the site database, with specialists (who are often self-employed and working remotely) being sent their own copy of the database, leading to currency problems, and a total lack of control over the data structure being used. However, replacing Microsoft Access completely is not advisable as staff often have many years experience in using it to analyse data, and are not familiar with alternatives. Our current system is to use Microsoft Access as the front-end to a PostgreSQL database. In this way, we maintain control and security over the data, we can ensure all staff are seeing the most up to date version, can control who has

AGI GeoCommunity '10: Opportunities in a Changing World'Innovate - Connect - Succeed'

the ability to delete or change records at a very fine-grained level, but also allow staff and specialists the familiar interface to the data.

Geospatial analysis was previously done using ArcGIS, but the cost of licensing and training meant that few staff had regular access and the skills to use it. Until current versions, it has also not been possible to connect to PostgreSQL databases without yet another expensive plugin. We have now migrated almost entirely to Quantum GIS and GvSIG as our desktop GIS packages, both of which have arguably a more powerful set of geospatial analytical tools (GRASS and Sextante respectively), and natively connect to PostgreSQL.

PublicationThe final phase of a site is to check that all of the conclusions are logically consistent, and bring everything into a single coherent narrative. The report of the excavation is published either as a basic site report (commonly known as “grey literature”, held by the County at a Historic Environment Record (HER), or as an academic monograph, or even sometimes as a “popular publication” aimed at the general reader.

The various levels of publication have different requirements. “Grey Literature” reports can be created in a word processing package, with illustrations generated in the GIS as necessary. Monographs and “Popular Publications” require a full desktop publishing package. We are in the process of converting to Open Office for our word processing/spreadsheet needs, and for some cases, to Inkscape and GIMP for creating illustrations.

ChallengesIt would be naïve (and false) to suggest that there haven't been any challenges along the way- and some still to overcome.

Changing the work-flows of a company of approximately 400 staff, many of whom have been doing their job for decades is not easy. It is necessary to win hearts and minds, and to get engagement at all levels. Simply making a financial argument is not enough, as people see the process causing them more work, or forcing them to learn new skills, and abandon old ones (the flip-side of this is that it highlights the difference between real skills and simply knowing which button to press).

Misconceptions of staff, clients and other contractors that the open source approach leads to bad data. We have encountered a complete lack of understanding of the difference between software and data at all levels of the archaeological and construction process. Often specific software packages are mandated in tenders for work, when what is actually required is a specific software format. At the very least this requires us to word our submissions extremely carefully in order to avoid losing work because we do not use ArcGIS!

Problems are often blamed on the open source software where previously they would have been blamed on something else (or not blamed at all). Software crashes all the time, everyone knows this. However people tend to be a lot more critical of open source software than it's closed source compatriots. The reason for this is unclear- it may simply be due to the perception that open source software is somehow not as high quality as proprietary software (it can't be, otherwise it calls into question why one would pay for the proprietary alternative). Furthermore with packages such as Open Office, every formatting issue is the fault of the software. Every previous incident of this kind with Microsoft Office is conveniently forgotten.

Changing work-flows does need an investment of time (and therefore money). From software installation to staff training, this process has taken time, and has been criticised as a result. However, many of these so-called “additional costs” are comparable to those that would be entailed by moving from (say) one version of Microsoft Office to another, or

AGI GeoCommunity '10: Opportunities in a Changing World'Innovate - Connect - Succeed'

from ArcView to ArcGIS. Many of these costs are also offset by the lack of licensing costs, and by efficiencies that re-examining the work-flow brings.

Dead ends. Particularly with the search for a device for digital recording on site, there have been a number of false starts. The openmoko phone is an example of this- although since it is possible to run the latest Android operating system on them, they make excellent development devices!

Limitations of the software. Some of the open source packages we use do have rough edges, as they have been designed to meet a particular need and time has been spent improving the functionality rather than the polished interface. Unfortunately this leads to an undeserved impression that the software is somehow not up to scratch. GvSIG is an example of this. In it's native form it is a Spanish package, and the translation to English is not as good as it could be, but functionally it is a superb piece of software. Part of our task in choosing it as one of our desktop GIS packages has been to improve the translation. Tools without a GUI, such as ogr2ogr, are shunned because of an innate fear of the command-line (perhaps a throwback to the DOS years). Quantum GIS and GvSIG both lack the ability to create really high-quality cartographic output, but we circumvent that by exporting the basic figure as a PDF or SVG and finishing it in InkScape. Initially this process added to the amount of time required to create a figure, but as staff familiarity with the software has improved this is no longer an issue.

Lack of open source cad packages. While a traditional site suits a GIS-based approach, for some of our work, such as building survey, a CAD-based approach is more suitable. Furthermore many of the engineers that we work with on road schemes and other large-scale construction projects use CAD rather than GIS. While there are a number of open source alternatives, we are yet to find one that fulfils all of our requirements, and keeps up with the annual change in the DWG format.

Opportunities Greater staff engagement. The elimination of licensing costs and restrictions has meant

that we can roll the software out to a much larger proportion of our staff, and afford to train them. Working through issues, learning a new package, and helping to develop work-flows have caught people's enthusiasm, and made them more committed members of staff, considerably better at their job.

Consultancy. The open source geospatial market is still small, and lacks any major players to dominate the market. Oxford Archaeology's open approach has garnered enough attention that we are called upon to provide support and advice to other companies looking to incorporate some open source software into their work-flow. We are of course living proof that it is possible to work in a high-pressure collaborative environment and to use open source software, with only a small number of hiccups (see above).

Our consultancy arm doesn't just provide support for open source software, however. For example, we are using the new technologies that the company is investing in, such as the Android phones, to investigate augmented reality applications, for doing historical “virtual tours” based on medieval maps and archaeological evidence.

Cost savings. There are enormous financial savings to be made with an open source approach for many companies, although these can be hard to quantify due to the costs of implementing the new software, training new staff, providing support, and so on.

More control- no licensing changes. For educational charities the proprietary software market is becoming a lot more strict about licensing. Over the last couple of years, many software providers have changed the terms of their licensing agreements to bar educational charities from receiving a discount. It is a moot point whether or not this is fair,

AGI GeoCommunity '10: Opportunities in a Changing World'Innovate - Connect - Succeed'

but at the very least it leads to a large increase in the software budget, often at very short notice- simply to continue using the software that they have. Moving to open source software does at least ensure that this will not happen in the future.

ConclusionsFor Oxford Archaeology the open source approach has been, and continues to be a success, helping us to survive in a very difficult industry during an economic downturn. It has also led to many positive factors, such as more engaged and skilled staff, and opportunities, such as our flourishing consultancy business. We do not believe that our approach will work for everyone, in fact our position, even within the archaeological industry, is unusual. However, we do believe that many companies in different industries can learn something from our experiences, particularly in the current economic climate.

For example, large-scale change of software and work-flows might seem untenable, expensive, and difficult for staff. However, it is an opportunity for a truthful assessment of what functionality is actually needed- as prior assumptions about how much of a particular package are used, and why, may be wrong. Are staff skilled in achieving tasks with a particular software package, or do they have a true understanding of their area of expertise, with flexibility and transferable skills? If the former, then even upgrading to a later version of the same software may be costly.

Furthermore, true ownership costs should be taken into account when considering a move to a different software package, and it's fair to say that these are often under-estimated when open source software advocates promote their favourite software. However, the cost of remaining with key proprietary packages who perhaps change file format every couple of years, or who demand better and better computer specifications as a minimum, should also be considered.

Our key advice is that people should question their software use, licensing, costs and requirements to ensure that they truly get the best value, both from the software, and their staff. In many cases, an honest assessment of requirements may lead to a decision to investigate open source alternatives. Our success over the last few years proves that it is possible to use open source software in a high-pressure environment, doing advanced analysis, and we would like to see other companies reap the same rewards.

AGI GeoCommunity '10: Opportunities in a Changing World'Innovate - Connect - Succeed'