Workflow systems or curatorial tools

One of the goals of PoWR is to make current trends in digital preservation meaningful and relevant to information professionals with the day-to-day responsibility for looking after web resources. Anyone coming for the first time to the field of digital preservation can find it a daunting area, with very distinct terminology and concepts. Some of these are drawn from time-honoured approaches to managing things like government records or Institutional archives, while others have been developed exclusively in the digital domain.

PoWR workshops

The Project ran three workshops: in London, Aberdeen and Manchester. The workshops, organised by UKOLN, were a mixture of presentations and break-out groups, where a great deal of useful discussion took place and many ideas were generated. Much valuable and interesting input was gleaned from the mixture of professionals who participated, including people from a records management background, web managers, and other information professionals with an interest in web preservation, or experience of the difficulties and issues.

The PoWR blog

A blog was built (http://jiscpowr.jiscinvolve.org/) at the very start of the project in April 2008. Several key chapters of the Handbook (see below) originated on this blog, many of them starting life as a series of ‘what if’ scenarios or actual case studies, focusing on various challenging aspects of web content and the actual use made of systems in an HFE context. The resulting discussions and comments provided a great deal of content to assess and assimilate.

The Handbook

The Handbook, written by ULCC staff, is a distillation and synthesis of the material gathered via the workshops and blog; it also draws heavily on the expertise of the PoWR team in the areas of website management, records management, digital preservation, etc. The Handbook aims to provide suggestions for best practice and advice specifically for UK higher and further educational institutions, to enable the preservation of websites and web-based resources.

The Guide

This Guide was developed using content from the Handbook and turning it into a practical guide for all those who would benefit from knowing how to go about preserving web resources.

A note on the authors

This Guide has contributions from a range of people but was primarily written and reviewed by the JISC PoWR team:

University of London Computer Centre (ULCC), Senate House, South Block, Malet Street, London WC1E 7HU (http://www.ulcc.ac.uk/)

• Ed Pinsent
• Richard Davis
• Kevin Ashley

UKOLN, University of Bath, Bath BA2 7AY (http://www.ukoln.ac.uk/)

• Brian Kelly
• Marieke Guy

This Guide was edited by:

Susan Farrell Consulting Ltd (http://www.farrellconsulting.co.uk/)

• Susan Farrell

Issues

Many wikis have a backup option which will enable the capture of wiki content, such as Wetpaint, Wikidot, Mediawiki, and Confluence. However all these options produce imperfect results. In contrast to this, if a spidering engine like HTTrack or Wget (see Appendix C) is used to harvest the site remotely, a working local copy of the wiki, looking much as it does on the web, will be the result. This might be an attractive option if a record of what it looked like on a certain date is required.

However, it may not be necessary to gather every web page since the wiki contains many automatically generated pages: versioning, indexing, admin etc. So a selection decision is needed. For example, the edit history and discussion pages may be excluded as the user community only wants to look at the finished content.

The change history is important to the current owner-operators of the wiki, however is this really needed for long term (or even permanent) preservation. Indeed, could their access requirement be satisfied merely by allowing the wiki (presuming it is reasonably secure, backed-up etc.) to go on operating the way it is, as a self-documenting collaborative editing tool?

Approaches

All this suggests some basic questions to ask when setting up a wiki for a project:

• What aspects of the wiki do we want to preserve and for how long?
• Is there a business need to capture the wiki change history, and for how long?
• Will it need preserving at intervals, or at a completion date?
• Is it more important to preserve text content, complete functionality, or its look?
• Should we back it up? If so, what should we back up?
• Does the wiki provide backup features? If so, what does it back up (e.g. attachments, discussions, revisions)?
• Once ‘backed up’, how easily can it be restored?
• Will the links still work in our preservation or backup copy?
• If the backup includes raw wiki markup, do you have the capabilities to re-render this as HTML?

Issues

• Should the option be open to students to have their resources persist on Institutional servers after they leave – perhaps as part of an Alumni programme?
• Should this be an opt-in or opt-out process, and should fees be involved?
• Does an Institution have permission to archive the content of blogs? This might include permission not only from the blog’s author but also from creators of third party content.
• Is it possible to excise potentially offending material, or is the risk (probably negligible) that an Institution might be sued for copyright breaches acceptable?
• Are Institutional staff and students well-informed about the issues of online copyright? Is it possible to include a default Creative Commons licence in the terms of use of the system?
• Is it more sustainable for the Institution to host and manage a blogging service or to use third party providers such as Blogger.com or WordPress.com?

Approaches

• Decide that the issue is predominantly one of policy, not of in-house hosting versus third party hosting. If an educational Institution is encouraging the use of blogs to support reflection, discourse and deep learning, it has a responsibility to make that online environment as safe as it tries to make its physical campus.
• Institutions could recommend the use of mature hosted blogging services for students who will normally only be at the Institution for a short period. Third party hosting might be a reasonable alternative to the costs of service development and maintenance, but the Institution must examine the terms and conditions and functionality very carefully to ensure they meet standards it can recommend to those in its charge.
• Seek permission from the owners of the blog content before making copies, investigate wider application of Creative Commons licences and work towards resolving third party issues.

Issues

If your resources are located on the main Institutional website (usually in the .ac.uk second level domain, managed by JANET), then your domain is unlikely to disappear unless there are major changes affecting your institution.

If, however, you are using an alternative domain name (such as .org, .org.uk, .co.uk or .com) then care is needed in managing domain registrations. Internal administrative management procedures will need to be in place to ensure that the domain name is renewed prior to the expiry.

You may ask why anyone would wish to make use of a non-.ac.uk domain in light of such possible dangers. JANET does not sell off its domains to the highest bidder. Instead it has strict eligibility guidelines that may not be met by short-term, collaborative or cross-sectoral projects and services. Equally within Institutions, the allocation of fourth level sub-domains (e.g. specialproject.london.ac.uk) is often tightly controlled or subject to considerable bureaucracy.

Approaches

• Carry out an audit of the Institution’s use of non- .ac.uk domains.
• Ensure that such domains have adequate administrative processes in place to ensure that the domain name is not lost if, for example, project funding ceases and staff involved in the project leave the Institution.
• Carry out a risk assessment of the dangers of losing such domains, and the costs your Institution may be willing to pay to claim back the domain.

Approaches

Capturing Twitter tweets is straightforward as they are easily downloaded from the RSS feeds that the service generates. These could be converted into text documents or web pages, or even imported into a database or blogging software. A number of solutions are available for tweet preservation such as Twapper Keeper (http://twapperkeeper.com/index.php), FriendFeed (http://friendfeed.com/), the WordPress Lifestream plugin (http://wordpress.org/extend/plugins/lifestream/), What the Hashtag (http://wthashtag.com/Main_Page) and Tweetdoc (http://www.tweetdoc.org/). More significant is the need to define an Institutional decision about the value of these resources. Why keep Twitter tweets?

• Corporate record: Is a Twitter a digital resource that information professionals should be interested in capturing or preserving? At what point does a Twitter turn into a record that requires the attention of the records manager?
• Scholarly record: can it be demonstrated that tweets are part of the scholarly record that is not captured in other forms?
• Legal reasons: Is Twitter being used to deliver learning? Is there a legal requirement to record what has been sent, as part of the assessment record?

Possible outcomes from the decision:

• It is agreed that Twitter posts are transient and do not need to be preserved. Records of corporate activity are something Institutions consider within their archiving policies, and tweets could be considered part of that corporate record. The decision needs to be reviewed as some information / communication media may take over the role of others.
• As a recognised official Institution publication, the posts need to be subject to Quality Assurance and editorial processes, which include keeping a record of the posts.
• An informal log of posts is kept, in order to have a record of topics that have been covered, audit the number of posts, be able to identify any significant impact from these services, etc.

Issues

• How has your Institutional home page changed over time?
• Have you kept records of the changes and the decisions which were made (and how they were made)?
• If you needed to do this for your Institution, do you feel you would be able to deliver a solution? How far back could you go?

Approaches

• The Internet Archive (see Chapter 7) has taken snapshots of websites since 1996 and may have captured web pages from your Institution. The University of Bath used the snapshots of its home page captured by Internet Archive to illustrate how it had changed between 1997 and 2007: an animated visualisation of the changes, linking to the IA’s snapshots, is available on UKOLN’s website. However, there is no guarantee that the Internet Archive will have captured every iteration of your Institution’s website, nor that the copies it has are complete and fully functional.
• Even if there are few, or no, surviving copies of previous versions of your website, there is no time like the present to start making sure snapshots are kept, either by taking your own copies, or ensuring the Internet Archive takes a copy. You can use an online form to nominate a site for crawling by the Internet Archive. It is also possible to nominate your site for capture by the UK Web Archive (see Chapter 7).
• Another approach is to build a compiled online history. The University of Virginia maintains a web page detailing 14 years of its website history. It includes fascinating statistical information based on analysis of the web server logs. Copies of the website are not available before 1996 and the image of the website in 1996 is taken from the Internet Archive. All subsequent snapshots are hosted on the main U.Va website, in subdirectories (/virginia1999, etc.). Some years are missing: whether because the changes were insignificant, or no copy survives, is not clear. Although there are broken links in the archived sites, or there are anachronistic links to current versions of pages, the archived snapshots provide a valuable view of the evolution of the Institution’s web presence.

• All existing policies which may have an impact on web preservation should be located and assessed.
• A policy review should be carried out.

It is unlikely that any Institution will have a single policy or mission statement that governs everything that should be happening in relation to websites and web resources. Any relevant Institutional statements are probably scattered across several places and departments; further, any guidance relating to the creation, storage and preservation of web-based materials may only be implied rather than made explicit.

However, the first step is to investigate the following sources (where available) as they may prove helpful.

• Institutional mission statement.
• Legal or legislative mandate.
• Regulatory requirements.
• Change management policy and procedures.
• Terms and conditions of website use, website privacy statement, accessibility policy, disclaimer, and copyright notice.
• Acceptable use policy / regulations concerning use of Institutional computing.
• Code of conduct for work areas and use of software.
• Web publishing policies and guidelines.
• IT security policy.
• System administration code of practice.
• Blogging terms and conditions.
• Records management policy.
• Archivist’s collection and preservation policies.
• Digital library guidelines.
• Institutional Repository deposit agreements.
• E-learning object repository policies.
• Any institutional or departmental policies governing information management, asset management, or knowledge management.

It may also be useful to locate the Minutes of any Committees or Advisory Groups in the Institution which formulate web development strategies or advise on policy and current development activities.

Assessing policies

Once all relevant policies are located, the following should be considered:

• Do any policies refer explicitly to web resources?
• Do the policies refer to the three proposed web resource classes? See Chapter 2.
• Do the policies suggest any action with regard to keeping web resources?
• Is there any scope for influencing the behaviour of those who create and use web resources?
• Is there any scope for assigning responsibilities for creation, capture and management of web resources to individuals?
• Would these policies allow the carrying out of preservation actions?
• Would these policies prevent the carrying out of preservation actions?

For example, a records manager’s retention schedules may not explicitly mention web resources by name as the content of the record tends to be identified rather than describing the form it is in. Therefore these, and other policies, will need to be translated, or adapted, to address the preservation of web resources. This includes all web resources and must stand the test of time without the need for endless revision.

Policy review

Reviewing policies and procedures is vital and should be embedded as a continual review action within the preservation process itself. As part of its long-term and evolving strategy, the Institution should:

• Strive to define technology-neutral policies. The policies should not be dependent on a choice of software, nor the format of the resource.
• Apply the policies to emerging systems.
• Make sure that its web resources and their management are explicitly covered by appropriate policies.
• Separate decisions about what policy says would be ideal, from what is achievable using current resources and technology.

Decisions made at these stages should be taken at Institutional level, so that ways can be found of embedding the decisions in practice, or matching them up to existing policies.

• Four main operational models can be considered.
• The quick win approaches include domain harvesting, carrying out pilot projects and considering using the EDRMS.
• Strategic approaches include information lifecycle management, adapting records management approaches, and the continuity approach.

Operational models

There are various operational models for developing and implementing a web preservation programme. Selecting a model will depend on balancing such factors as costs, risks, priorities, and available resources and infrastructure.

In-house: the programme is resourced, managed and implemented within the Institution. This approach offers the most flexibility and control, assuming the necessary skills and resources are available. In the USA, the Harvard Web Archive Collection Service (WAX) is an example of a successful in-house project that relied heavily on the active participation of the creators of the resources. 48 Harvard websites were put into the collection, representing Departments, Committees, Schools, Libraries, Museums, and educational programmes.

Contracted-out: all or some of the work is performed by a contractor. It is unlikely a single contractor will have all the skills and resources to perform the entire package, and Institutions may need to look at contracting out services for e.g. hosting and storage or website hosting, while retaining the curational and selection elements in-house.

Collaborative: one or more Institutions work together, pooling skills and resources towards a common goal. JISC often fund such projects which bring together initiatives from two or more HFE Institutions, particularly for collaborative tools and e-learning projects. However, as yet, no funded programme for website preservation exists.

Consortium: the programme is implemented by a consortium of organisations, using some shared resource or infrastructure. The IIPC is an example of the consortium approach. In this country, the UK Web Archive used to fill this role, although its role as a consortium appears not to be as active. The activities of the Digital Preservation Coalition’s Web Archiving and Preservation Task Force will also prove valuable in this collaborative area. It aims to ‘provide a mutually supportive environment for continued policy development for members and a mechanism through which non-members can engage with web archiving policy’.

Approaches

Once the operational model is decided, there are two main classes of approach:

1) Quick win: This can be used to protect resources identified as being most at risk and the approaches include domain harvesting, remote harvesting, pilot projects, and using an EDRMS. They may be attractive because they are quick, and some of them can be performed without involving other people or requiring changes in working. However, they may become expensive to sustain if they do not evolve into strategy.

2) Strategic: This includes longer-term solutions which take more time to implement, involve some degree of change, and affect more people in the Institution. These approaches are adapted from Lifecycle Management and records management, and also may involve working with external organisations that will do the work (or some of it) for the Institution. The pay-off may be delayed in some cases, but the more these solutions become embedded in the workflow, the more web archiving and preservation becomes a matter of course.

Quick win

Domain harvesting refers to two possible approaches:

• The Institution conducts its own domain harvest, sweeping the entire domain (or domains), using appropriate web crawling tools.
• The Institution works in partnership with an external agency to do domain harvesting on its behalf.

Domain harvesting is only ever a partial solution to the preservation of web content as there are limitations to the systems which currently exist. Too much content may be gathered, including that which does not need to be preserved. Conversely, content which ought to be collected may be missed including hidden links, secure and encrypted pages, external domains, database-driven content, and databases. Simply harvesting the material and storing a copy of it may also not address all the issues associated with preservation.

Instead of trying to solve the web resource problem for an entire Institution, a pilot project could be carried out from which a visible result will be seen quite quickly. This may make it more persuasive for other departments to participate and will add credibility to the programme. Pilot projects can also generate useful reports about lessons learned that can be used to make the next project even more successful. In addition fewer stakeholders may be involved, if the project is scoped tightly enough, thus saving time on consulting users and owners of the content. Pilot projects could be targeted as outlined in Chapter 5.

Migration of resources from one operating system to another or from one storage/management system to another is a form of preservation. This may raise questions about emulation and performance. Can the resource be successfully extracted from its old system, and behave in an acceptable way in the new system?

It is not yet known how feasible it is to use an EDRMS (Electronic Document and Records Management System) for the management of web resources. These systems seem to work best with static documents; authors of reports, for example, understand that a good time to declare their report as a record is when the final approved version has been accepted. Yet one of the distinctive features of Web 2.0 content is that the information is very fluid, and often there is no obvious point at which to draw this line and fix content.

It is technically feasible, for example, to capture Instant Messaging outputs as text or HTML files which could be saved into an EDRMS. The question remains whether there is a defined policy that supports doing this; one that recognises use of IM as a legitimate record-keeping tool, and as a practice that is acceptable to the Institution. The attraction of storing certain web-based output in an EDRMS is that then such resources could be managed in line with agreed retention schedules; and that related records are filed together, like with like.

Strategic Approaches

These include Information Lifecycle Management, adapting records management approaches, or following the web continuity methodology. Any approach taken should ensure that web resources are protected from careless or wrongful destruction, deletion, or removal of the resource.

Information Lifecycle Management (ILM) involves recognised professional standards and practices, leading to better management of information, and is one possible approach. If a lifecycle model can be applied to web resources, they will be created, managed, stored and disposed of in a more efficient and consistent way. It can also assist with the process of identifying what should and should not be retained, and why; and that in turn will help with making preservation decisions. ILM makes no assumptions about software or IT systems, nor does it assume that all information will be managed through a single software tool; rather, it is a conceptual framework to help ensure consistency within an organisation. It can be especially helpful when introducing new systems, or reviewing existing ones.

Information moves through a series of phases over time. JISC’s approach to ILM proposes four distinct phases: creation; active use; semi-active use; and final outcome.

Information should be managed throughout each phase, and there are pertinent issues which apply. ILM can also be aligned very closely to the records management programme. An ILM approach always takes a start-to-finish, cradle-to-grave view. A model can be adapted according to your Institutional needs. The model should have a chronological structure, clearly defined phases, user identification, and consistency.

There is a lot of literature available including the JISCInfoNet published guidance, Managing The Information Lifecycle, which is geared towards the HFE sector. (See http://www.jiscinfonet.ac.uk/infokits/information-lifecycle.)

If a records model can be applied to web resources, the same benefits associated with ILM apply: web resources will be created, managed, stored and disposed of in a more efficient and consistent way. The RM programme will already be established, and through the agreed retention schedules it can assist with the process of identifying what should and should not be retained, and why. All of that in turn will help with making preservation decisions. Under records management, these things will take place within a legislative and regulatory framework that enables and obliges the creation and disposal of records.

The ‘Web Continuity’ project involves a ‘comprehensive archiving of the government web estate by The National Archives’, and aims to address both ‘persistence’ and ‘preservation’ in a way that is seamless and robust. Web continuity offers concepts and ways of working that may be adaptable to a web preservation programme in an Institution, particularly as a main area of focus is the integrity of website links. The project’s use of digital object identifiers (DOIs) can marry a live URL to a persistent identifier. To achieve persistency of links, a redirection component derived from open source software is used. This component will ‘deliver the information requested by the user whether it is on the live website, or retrieved from the web archive and presented appropriately’. Of course, this redirection component only works if the domains are still being maintained, but it will do much to ensure that links persist over time.

Part if the project involves building a centralised registry database of Government websites, and is a means of auditing the website crawls that are undertaken. Such a registry approach is worth considering on a smaller scale for an Institution as would the project’s methodology for rolling out XML site maps across government. These site maps can help preservation because they help to expose hidden content that is not linked to by navigation, or dynamic pages created by a CMS or database.

The intended presentation method will make it much clearer to users that they are accessing an archived page instead of a live one and helps to address any potential liability issues arising from members of the public acting upon outdated information.

• The Institution must take ownership of the web preservation programme at the highest level so that staff and students are motivated to take part.
• There are some national and international initiatives which might help.

Success with the preservation of web resources will potentially involve the participation and collaboration of a wide range of experts: information managers, asset managers, web managers, IT specialists, system administrators, records managers, and archivists. Each of the participants will have an interest derived from their role which means some may be more driven in terms of the process than others.

For example, the interest of records managers will be in legal compliance, long-term RM goals, retention, disposal, and classification of those web resources classed as records. This is a central activity in their role, whereas web managers may be less driven as their role is more about managing current content and new technological initiatives. Each needs to consider the other’s view of preservation and work together to make the web preservation programme a success.

Records managers need to:

• recognise that the web is a potential place where records can occur, and identify where, how, when and by what agency this is happening;
• rethink some of their traditional models as centralised control over web resources is not possible;
• overcome any fear of IT, and forge relations with people like the web manager, system administrator, and IT manager.

Web managers:

• need to recognise that the web is a potential place where records can occur, and that they have some responsibility to ensure they are protected;
• should think twice before deleting everything or disabling an account;
• should exploit software for better ways to capture and manage the content;
• consider preservation-friendly software for the next purchase of web tools.

As well as those with the specific roles identified above the preservation programme must also involve the stakeholders identified during the appraisal process (see Chapter 5).

The Institution as owner

It is vital that the Institution takes ownership at the highest level as this will motivate all Institutional staff and students to take part in the web preservation programme. Also, effective web preservation needs to be policy-driven as it is about changing behaviour, and consistently working to policies. A clear policy is needed that states the importance and value of the Institution’s web resources, and makes it clear why some of them are being preserved. There should be a sense of corporate ownership of the Institution’s website, the web publication programme, web resources that have value and need to be preserved, and all other issues associated with making the resources ready for preservation such as capture, storage and management.

There are many drivers for an Institution to take ownership of the web preservation programme and these are discussed in Chapter 3. However, the strategic, legal, financial and contractual obligations coupled with the risk issues should provide enough impetus for the Institution to take action.

Technology is not the answer

Web resources have existed in UK HFE Institutions for many years, and so have the tools that would help an Institution capture, manage and store those resources. The fact that these tools are not being widely used is an indicator that web preservation is not primarily a technological problem: the solution does not lie in buying new software or more software. There is no single tool that addresses all possible web preservation issues (behaviour, dynamic content, scripts, versioning, etc.), so any programme of work associated with web preservation needs to be properly resourced, with team-based and collaborative approaches drawn from across more than one discipline.

Can other people do it for you?

The Internet Archive (http://www.archive.org), also know as the ‘Wayback Machine’, is unique in that it has been gathering pages from websites for so long that it holds web material that cannot be retrieved or found anywhere else, and would have been lost.

The Internet Archive has ways which allow anyone to submit a website for inclusion in the Archive including a subscription service called Archive-It (http://www.archive-it.org/). The advantage of this service is that distinct web archives (collections) are created containing only the content selected for harvesting and at the chosen frequency. These can be catalogued and managed directly by the subscriber. The assumption is that archived copies will be made public, via the Internet Archive, although arrangements can be made to keep them private.

Additionally, people are encouraged to use the Internet Archive as a sort of ‘People’s Repository’. By registering, it is possible to upload images, texts, moving images, and audio material, thus making use of IA’s considerable storage capacity. Again, in return for free storage, it is expected that your resource is made public.

A few caveats about the suitability of this solution to UK HFE Institutions:

• IA lacks an explicit preservation principle or policy, and has no real mandate to capture websites outside of a societal desire to see it happening and to share the results with the public. This may cause severe problems to HFE Institutions; as it may not cover everything the Institution needs to do within its remit.
• There is potential for legal difficulties and litigation. IPR issues may not be adequately dealt with by Creative Commons and the IA’s ‘notice and take down’ approach.
• IA may not have a sustainable funding model and its continuance is largely dependent on the generosity of its creator, Brewster Kahle.

There are additional caveats about the technical failings of Internet Archive:

• IA will not capture all web-based assets and cannot capture any site or service that depends on a database, or a login.
• IA cannot guarantee capture to a reliable depth, or reliable quality. Note that this applies to dynamic content.
• There can be large gaps between capture dates.
• The image assets in IA are always smaller than archive quality copies.
• IA may not be preserving the resources they capture to OAIS standards.
• There is little in the way of contextual information in their catalogues.

A number of Institutional assets are missed out by the IA approach including library catalogues, image collections, e-print collections with a database, and interactive teaching materials.

The UK Web Archive (UKWA) is a British Library initiative which started gathering and curating websites in 2004. The archive is free to view, accessed directly from the web itself and has collected thousands of websites. UKWA’s approach is selective, and determined by written selection policies. The Archive contains UK websites that publish research, reflect the diversity of lives, interests and activities throughout the UK, and demonstrate web innovation.

It may be possible to nominate a Institutional website for capture with UKWA but it may not be selected or archived. The current position on this process is that owners of UK website are especially encouraged to nominate their sites but that UKWA reserves the right to decide whether to include the nominated sites.

Institutions may also want to bear the following in mind:

• The capture will be a snapshot of the website at a certain date and time.
• Certain resources will be beyond the reach of the Heritrix crawler (e.g. databases, secure and password-protected pages, and hidden links).
• Similarly, if the website depends heavily on server-side architecture, then remote capture may fail.

If the website is selected by UKWA, it will involve a few practical things:

• Signing a permissions agreement to allow remote harvesting and copying.
• Agreeing to have the archived copy made publicly available.
• Allowing the remote harvester to ignore robot exclusions.

A UKWA solution is better than nothing but there are limitations, and it may not constitute a quality solution to preservation of all the Institution’s web resources. (http://www.webarchive.org.uk/ukwa/)

The International Internet Preservation Consortium (IIPC) will not help to harvest an Institution’s website, but they are an internationally-recognised body of excellence for website preservation. The mission of the IIPC is to acquire, preserve and make accessible knowledge and information from the Internet for future generations everywhere, promoting global exchange and international relations.

The goals of the consortium are (http://www.netpreserve.org/about/index.php):

• To enable the collection, preservation and long-term access of a rich body of Internet content from around the world.
• To foster the development and use of common tools, techniques and standards for the creation of international archives.
• To be a strong international advocate for initiatives and legislation that encourage the collection, preservation and access to Internet content.
• To encourage and support libraries, archives, museums and cultural heritage institutions everywhere to address Internet content collecting and preservation.

Other external initiatives, mostly library-based and sponsored at a National level, aim to complete selective web collections, often based on the aim of archiving the entire ‘national’ domain. They are provided here as they may be able to offer some useful lessons learned. However, they will not be able to assist with the archiving of an Institution’s website. Some of the National Library collections are not open to the public.

MINERVA, Library of Congress (http://lcweb2.loc.gov/diglib/lcwa/html/lcwa-home.html). A selective web archive based on themes of national importance, i.e. national political elections, wars and terrorism.

PANDORA (Preserving and Accessing Networked Documentary Resources of Australia) (http://pandora.nla.gov.au/). National Library of Australia with nine other Australian libraries and cultural collecting organisations.

UK Government Web Archive (http://www.nationalarchives.gov.uk/webarchive/). A selective collection of UK Government websites, archived at regular intervals from August 2003, developed by The National Archives using Internet Archive.

Austrian On-line Archive (AOLA), Austrian National Library and Technical University of Vienna (http://www.ifs.tuwien.ac.at/~aola/).

DAMP (Digital Archive for Web Publications) (http://www.nsk.hr/DigitalLib.aspx?id=80), University of Zagreb and the National and University Library (NUL) in Zagreb, Croatia.

Kulturarw3 – KB Web Archive, Royal Library – the National Library of Sweden.

The State and University Library and the Royal Library, Denmark (http://netarchive.dk/).

WebArchiv, National Library of the Czech Republic and Masaryk University in Brno (http://en.webarchiv.cz/).

• The capture of web resources can be carried out within the authoring system or server, at the browser or with a crawler. Each process has its advantages and disadvantages.
• Tools are available for each type of capture.

Tools

A number of tools are available for capturing web resources:

• Tools for capturing web resources.
• Workflow systems and curatorial tools.
• Snapshot tools.

For more see Appendix C. Netpreserve.org and Harvard University Library also have lists of tools.

Point of capture

There are three points in the journey of a web page, from server to user, where its capture is likely to be most feasible.

1. Capture within the authoring system or server: This involves retrieving web pages directly at their point of origin, usually the content management system, or the server on which web pages are held. It is possible to get all the content (HTML, CSS, GIFs, JPEGs, etc) however, increasingly web pages are formatted ‘on-the-fly’ to suit the specific needs of the user that is requesting them (e.g. type of browser, small screen or large screen, desktop device or mobile phone). So this method raises the question of which of these possible versions should be captured.

2. Capture at the browser: This could also be described as capture post-rendering, or at the point of the HTTP transaction. It implies something of a snapshotting approach, and such a snapshot is going to result in frozen content.

3. Capture with a crawler: Using a crawler is going to resolve some of the problems of other methods but not all. Crawlers are unlikely to succeed totally as they miss other external sources such as document servers, databases and datafeeds, internal databases, subscription databases, and file management platforms. Web content management systems, access methods, protocols, and security and logins, may also present barriers.

Many crawlers, including Heritrix, are also prone to the ‘collateral harvesting’ problem. This means they can gather lots of content which is not needed, by blindly following links. There are ways of setting exclusion filters to prevent this, but its behaviour can still be unexpected.

• The appraisal process looks at the location, use and ownership of the web resources, and particularly aims to ensure that web resources containing unique information are preserved.
• Resources which are managed elsewhere (e.g. in asset collections or Institutional repositories), or are of little or no value, can be omitted from the web preservation programme.
• The MoSCoW method enables prioritisation of resources.
• The selection approach can be unselective (domain harvesting) or selective (criteria-based or event-based).
• Which aspects of resources (information and experience), and which elements (content, appearance and behaviours) to preserve must be considered.

Careful appraisal and selection of web resources will make the task of web preservation more manageable as this will make it clear what should, and should not, be included in the scope of the web preservation programme. Also, within that list, each web resource can be assigned a priority for action.

Some issues to bear in mind when deciding which of the resources to preserve are:

• The Institutional structure and its aims.
• The policies and drivers for preservation.
• The legal record-keeping and audit requirements for the Institution.
• The potential reuse value of resources.
• Is the resource needed by staff to perform a specific task?
• Has the resource been accessed in the last six months?
• Does the resource represent a significant financial investment in terms of staff cost and time spent creating it?

It is particularly important to facilitate the survival of web resources which contain unique information such as those:

• which only exist in web-based form – for example, teaching materials designed as web pages;
• which do not exist anywhere else but on the website;
• whose ownership or responsibility is unclear, or lacking altogether;
• that constitute records, according to the definitions in Chapter 2;
• that have potential archival value, according to definitions supplied by the archivist.

Appraisal and selection

Appraisal identifies those web resources which constitute records, publications and artefacts (as defined in Chapter 2) for preservation. The web resources selected should provide information about, and evidence of, what the Institution has done and why, what it and its staff and students have achieved, and its impact locally and in the wider world.

In simple terms, appraisal of HFI resources for preservation should focus on:

• substantive functions (i.e. teaching, research, academic award administration) and;
• substantive elements (e.g. strategy development, policy development) of facilitative functions (e.g. governance, estate management, public relations).

(From JISC Infonet (2007) Guidance on Archival Appraisal)

The questions that need to be answered for appraisal relate to: location; use; and ownership. However there are some resources which can immediately be excluded from the selection list.

As indicated in Chapter 2, web resources can be found on web server(s), in managed systems or in less well-managed systems which may be hosted externally. The locations of these servers and systems must be discovered so consulting the web manager and the IT manager is the first step. This should result in finding out the:

• number and names of domains and sub-domains being used, including staff and student intranets and portals, funded project websites and any others. It may prove difficult to track them all down as departments and other areas of the Institution may register their own domains;
• number and location of web servers. Again this may be difficult to do because of the possible autonomy of departments but the IT Security staff should be able to help as they will manage the firewall;
• managed systems and their locations;
• less managed systems (where known). Discussions with stakeholders are most likely to track these down;
• backup schedules for all relevant systems and servers;
• resources with external dependencies.

Once the locations of the web resources are known, each should be appraised to find out its purpose and how it is being created. It is particularly important to find out if original resources are being created (i.e. those that are not available in any other format).

Finding out the owners of the web resources will identify the stakeholders of the web preservation programme. Each owner will be able to provide valuable help in finalising the programme as they can say which resources they would like to be kept and why, and for how long they need to be kept. This approach will also help get people on board and start to embed a culture of good practice and web management within the Institution.

However it is important to be objective when considering the information provided by the stakeholders as it is likely that they may want to keep everything indefinitely. Conversely, it is all too easy to sweep resources away simply because the stakeholder is not around to defend their interests.

There are some resources which can be immediately excluded from the preservation programme including those that are already being managed elsewhere, and those which have little or no value.

Asset Collections. For some asset collections, or e-resource collections, the web is often just an access tool for the underlying information resource so preservation actions are best concentrated directly on that resource. This class might include: digitised images; research databases; electronic journals; ebooks; digitised periodicals; examples of past examination papers; and theses.

Institutional repositories (examples include DSpace, eprints or Fedora). Institutional repositories are web-based tools, but the materials stored in an IR are already being managed as there are elements such as metadata profiling, secure and managed storage, backup procedures, audit trails of use, and recognised ownership. A well-managed IR therefore already constitutes a recognised digital preservation method in itself.

Duplicate copies. In some cases, the website is a pointer to resources that are stored and managed somewhere else. Or the resource has been uploaded from a drive which is owned and maintained by another department. If it is ascertained that the ‘somewhere else’ is already being preserved, then it may not be necessary to keep the website copies.

Institutional web-based applications which deliver a common service. In this case the web application is an incidental component used in the management of such services. Quite often the important record component in such instances is actually stored or managed elsewhere, for example in a database of underlying data.

Services which do not generate any informational material of lasting value to the Institution. Some examples of this are room booking systems, systems which allow automated submission of student work for assessment, or circulation of examination results.

Resources which clearly fall outside the scope of an agreed records retention policy, or an archival selection policy. Examples of this might include Twitter and Instant Messaging, unless evidence can be found of a strong Institutional driver to retain and manage such outputs. (See case study on page 13.)

Prioritisation is fundamental to successful preservation – keeping everything is rarely possible. So, when considering what to preserve and what not to preserve, the MoSCoW method can be used as this classifies the requirements as one of the following:

• M: resources which must be preserved.
• S: resources which should preserved, if at all possible.
• C: resources which could be preserved, if it does not affect anything else.
• W: resources which won’t be preserved.

Even after carrying out the MoSCoW method, it is possible that the list of web resources in the M and S categories will be long and potentially unmanageable. So the MoSCoW method can then be carried out again with just those in the M category to find out the order in which the resources should be tackled.

Selection approaches

Three main approaches (one unselective and two selective) have arisen from the work carried out by National Libraries and these can be adapted to the requirements of an HFE Institution.

1. Unselective approach – bulk / domain harvesting: This could involve harvesting the entire website, and/or all its associated domains. Some argue that it is cheaper and quicker to be unselective than to go through the time-consuming selection route; that it is demonstrably less ‘subjective’ and will produce a more accurate picture of the web resource collections; and that since it is technically feasible, why not?

However, aspects of those arguments are more applicable to a digital archive or repository trying to scope its collection within certain affordable and pragmatic boundaries. Secondly, there is no point in capturing ‘everything’ if it has already been established that there are significant quantities of web resources in the Institution that do not even need capture, let alone preservation. In running a frequent domain-wide harvest of Institutional networks, there is a risk of creating large amounts of unsorted and potentially useless data, and committing additional resources to its storage.

2. Selective approach – criteria-based selection: This could entail selecting web resources according to a pre-defined set of criteria, for example:

• All resources owned by one Department.
• One genre of web resource (e.g. all blogs).
• Resources that share a common subject, or related subjects.
• All resources that affect students or staff only.
• All funded projects with web-based deliverables.
• All resources thought to be at risk of loss.
• All records or all publications.
• Resources that would most benefit an external user community (e.g. alumni).
• Resources covering a pre-determined theme.

3. Selective approach – event-based: Consider if there would be value in taking ‘before and after’ snapshots of certain web pages, if agents of change are known to be at work. The sort of time-based events which might trigger a decision to capture are:

• End or beginning of term, or beginning of a new academic year.
• Appointment, or departure, of a senior official.
• Completion of a major piece of research.
• Publication of the new prospectus.
• Purchase of new authoring software which affects web content.
• Corporate or Institutional re-branding.
• Formation of a new department.

Having decided on the approach, the next step is to decide which aspects and elements of web resources must be captured.

Aspects

It is possible to make a distinction between preserving an experience (experience of accessing web content including all its attendant behaviours and aspects) and preserving the information (all meaningful content including words, figures, images, audio) which the experience makes available. Both are valid preservation approaches and both achieve different ends.

Deciding which aspects of web resources to capture can be informed to a large extent by the Institutional drivers, and the agreed policies for retention and preservation.

A few examples are given below.

Evidential and record-keeping: As well as the content, this would involve preserving some form of change history, with as much contextual information as possible. This may not apply to all the web resources, just to those which are needed for legal purposes, to protect the Institution, where decision-making is involved, etc. For such resources the following should be captured and preserved:

• an audit trail of changes and a change history;
• contextual information about people (authors, users etc), and dates and times (creation, change and publication dates etc);
• the content, appearance and behaviour of the resource.

Repurposing and reuse: For web resources which are being reused and potentially repurposed in a different context (or even on a different server), it would make sense to preserve:

• the content, appearance and behaviour of the resource;
• contextual metadata about its creation, its original location, its authorship, its access rights, etc.

Social history: For web resources which are not needed for evidential purposes, but are being preserved to retain something about the history of the Institution, the capture requirements may not be as exacting. For example, if it was decided to preserve a sample of student home pages, appearance of the resource could be preserved to demonstrate how home pages looked five years ago.

Elements

The elements of web resources that need to be considered are content, appearance and behaviour.

Content: This is just the words. No links, no behaviour, no framesets, no stylesheets, no images – just plain text.

Appearance: This is the look and feel of the page including navigational devices, images, page layout etc.

Behaviour: This covers web resources which have dynamic or animated features. One example of a website with a mixture of behaviours would be a blog, which might have behaviours such as a live feed, comments which can change, site administration features, and bookmarking and tagging features.

If the preservation of content, appearance and behaviour is required, the job of preservation becomes more complex. It may not be feasible or desirable to capture all of these elements therefore it is important to specify which are most significant for preservation.

DPC decision tree

A potentially useful tool is the Decision Tree produced by the Digital Preservation Coalition. It is intended to help build a selection policy for digital resources, although it should be pointed out that it was intended for use in a digital archive or repository. The Decision Tree may have some value for appraising web resources if it is suitably adapted.

It may prove more successful for an Institution to manage the individual stages as a series of clearly defined projects, with owners. Indeed, some of the suggested actions fall into the category of generic project management actions, and do not have a corresponding chapter in the Guide.

Resources required

The resources required are: staff; hardware and infrastructure; and software.

Chapter 7 covers which staff should be involved in the programme but it useful to bear in mind that the skills required are:

• technicians who understand websites;
• selection and curator skills (archivists or librarians);
• digital preservation skills;
• permissions management.

The following in terms of hardware and infrastructure may be required:

• dedicated servers for data storage;
• sufficient web server capacity for delivery of resources;
• high speed and high bandwidth Internet connection for capture;
• sufficient connection bandwidth for users.

The following software may be required:

• collection tools (See Appendix C);
• harvesting methodologies;
• database archiving;
• specialised software for access and delivery;
• specialised software for metadata management.

• The drivers for carrying out web preservation are strategic, legal, financial, contractual and reputational.
• Web preservation also has a role to play in business continuity planning.
• The espida project provides a useful methodology for quantifying the value of web preservation which is helpful when developing a business case for a web preservation project.

There are many internal and external drivers for undertaking web preservation within an HFE Institution and some of these are summarised below.

Strategic, legal, financial and contractual obligations

Institutional websites contain evidence of Institutional activity which is not recorded elsewhere and may be lost if the website is not archived or regular snapshots taken. This loss could be construed as a threat to the business continuity of the organisation as reference to this content may be required for the checking of strategic, legal, financial, contractual or scholarly information. For example, if certain information is not recorded or protected the Institution is in danger of failing to comply with legal acts such as Freedom of Information and Data Protection. Also the Institution may be breaking contractual and auditing obligations, and putting itself at risk.

Reputational risk

The Institution’s reputation can be put at risk by poor website continuity, broken links, and missing resources but more damaging is the risk that website users may make decisions based on misleading or out-of-date content.

Supporting the Institutional Mission

The case can be strengthened for carrying out web preservation by showing what it delivers to support the Institution’s mission statement, and its associated strategies in the areas of research, teaching and learning, information, libraries, and records management. A policy for preserved web resources could feasibly assist in supporting Institution-wide aims and generic objectives, even when they do not explicitly mention digital resources by name, for example ‘attracting a wide variety of students’.

Risk management and risk analysis

Possible risks associated with websites and web resources include: loss of data, records or resources; failure to be information compliant; the risk of litigation from students or the public; and the risk of breaching copyright.

Bringing web preservation into line with business continuity planning may help to change Institutional practice. The risks associated with possible IPR infringements are put in perspective by Charles Oppenheim’s lightweight risk formula: R = A x B x C x D, where:

• A: probability that you are illegal;
• B: probability that you are found out;
• C: probability that action will be taken against you;
• D: extent of financial risk.

The aim is to keep all of these values as low as possible, but it is also the case that if any of these is zero, the overall risk is effectively nullified.

If a risk management strategy is required to enable web preservation, using the JISC InfoKit on Risk Management is a good place to start. The kit takes the view that Risk Management is an essential part of project management.

Saving money

Web resources cost money to create and store; failure to repurpose and reuse them would constitute a waste of money. Although web preservation may have an initial cost, once the process has begun the savings can be great. Having a good strategy in place (which means selection, retention, and deletion where appropriate) will save both money and energy in the long run. The website may also contain digital assets and electronic resources – assets which may be of continued value which may increase in value through sharing and repurposing.

Responsibility to staff and students

The Institution has a responsibility to the people who use the resources. Students and staff may make serious choices about their academic careers or their jobs based on website information, and the Institution has a responsibility to make sure a record is kept of the publication programme.

Responsibility to users

The Institution has a responsibility to the people who may need to use the resources in the future. Many of the resources which the Institution publishes are unique, and deleting them may mean that invaluable scholarly, cultural and scientific resources (heritage records) will not be available to future generations.

Gaining a competitive edge

Starting a web preservation programme will make the Institution look ‘forward thinking’ and therefore give them an edge over their competitors. The Institution could be one of the first to start an official web preservation programme which will be great marketing fodder. Embedding web preservation strategies will also help the Institution think about the continuity of resources, broken links and other aspects of web management.

Quantifying the value of web preservation

The espida project in Glasgow (http://www.gla.ac.uk/espida/) offers a useful methodology which could be used to quantify the value of web preservation. It takes a pragmatic view of the way that HFE Institutions operate in the real world recognising that preservation activities will continue to vie with other services for funds.

espida can help:

• demonstrate the value of websites and web resources;
• communicate the intangible benefits of web preservation and web resource preservation to senior management, and articulate those benefits;
• make a case for a web preservation programme, based on a formalised and transparent communication process between the proposer and the funder;
• identify costs and benefits of web preservation, using scorecards and cost templates;
• produce a decision-making process that is transparent and based on all relevant information.

The results of the process will enable the development of a business case which not only answers the question ‘how much does web preservation cost?’, but also ‘why do we need web preservation?’ and ‘why should we spend money on web preservation, rather than on the primary business of the organisation?’.

• Web resources are those delivered by a web browser, and can be found on web servers, in managed systems (including content management systems, Institutional repositories and digital collections), and less well-managed systems (such as Web 2.0 applications and services).
• To help with determining whether they should be preserved, web resources should be categorised into records, publications or artefacts.

Web resources are those delivered by a web browser so they may appear not only on the Institutional website but also on other websites and web systems. Some examples are:

• Institutional and departmental records, with legal and business requirements governing their retention and good maintenance.
• Content affecting students, such as prospectuses and e-learning objects.
• Administrative, research, teaching and project outputs.
• Evidence of other activities (e.g. conferences).

Many of the resources of an Institutional website may be stored in a CMS or within well-established managed systems, such as:

• Those for managing assessments and examinations.
• Online libraries.
• Virtual Learning Environments (VLEs).
• Online teaching courses and course content.
• Institutional repositories.
• Digital collections used for study.
• E-learning objects and teaching materials.
• E-portfolios.

In these cases, the resources are simply being accessed or delivered by a web browser so, except for those in a CMS, the preservation should be of the system rather than the pages as they appear on the website. In the case of a CMS both the system and the pages produced need to be preserved.

But the increasing use of Web 2.0 services and applications means that many resources exist in less well-managed systems, many of which are hosted outside the Institution, including:

• Blogs Blogger (e.g. WordPress, Edublogs, Warwick Blogs).
• Wikis (e.g. Mediawiki, Wetpaint, Tiddlywiki).
• Social bookmarking (e.g. Delicious, CiteULike, Connotea).
• Media sharing services (e.g. Flickr, Slideshare, YouTube. (Scribd, DeviantArt)).
• Social networking systems (e.g. Facebook, Twitter, Ning, Elgg, Crowdvine, LinkedIn).
• Collaborative editing tools (e.g. Google Docs).
• Syndication and notification technologies (e.g. Netvibes, Technorati).
• Instant messaging (e.g. Facebook Chat, Google Chat, Skype, Jabber, Windows Messenger).

Which web resources need to be preserved?

To determine which web resources to preserve it is helpful to consider which of three categories best describes the resource – record, publication or artefact. If it is a record or a publication, the resource should be considered in the context of existing policies and procedures for these types of document.

A record is ‘recorded information, in any form, created or received and maintained by an organisation or person in the transaction of business or conduct of affairs and kept as evidence of such activity.’ (http://www.recordsmanagement.ed.ac.uk/InfoStaff/RMstaff/RM_framework.htm)

A web resource can be considered a record if it:

• constitutes evidence of business activity that needs to be referred to again;
• is evidence of a transaction;
• needs to be kept for legal reasons.

Some examples:

• The website contains the only copy of an important record. Web resources should not be removed or deleted without establishing if they are the only copy.
• The website, or a set of web pages, in itself constitutes evidence of Institutional activity. The history of this evidence is visible through the various iterations and changes of the website.
• The website is in itself evidence of the publication programme, or has such evidence embedded within its systems. If it is necessary to show, as evidence, that the Institution published a particular document on a certain date, then the logs in the CMS constitute an evidential record. In some cases, this may be needed to protect against liability.
• A transaction of some sort that has taken place through the website (transaction does not just mean money has changed hands). If these transactions need to be kept for legal or evidential reasons, then they are records too. The transaction may generate some form of documentation (e.g. automated email responses), which may in turn need to be captured out of the process and stored in a place where it can be retrieved and accessed.

A publication is ‘a work is deemed to have been published if reproductions of the work or edition have been made available (whether by sale or otherwise) to the public’. (National Library of Australia http://www.nla.gov.au/services/ldeposit.html)

A web resource might be considered a publication if it is:

• a web page that is exposed to the public on the website;
• an attachment to a web page (e.g. a PDF or Word Document) that is exposed on the website;
• a copy of a digital resource, e.g. a report or dissertation that has already been published by other means.

Some examples:

• Websites containing the only copy of an important publication.
• Web pages constituting a version of information that is available elsewhere. By version, it is meant that it has been rendered in some way to bring it into the website. This rendering may include, for example, the addition of navigation elements that make it different from the original source.
• A web page constituting a mix of published information. For example, a page of original Institutional material combined with an RSS feed from outside the Institution.

An artefact is ‘anything else that isn’t a record or a publication by the above definitions, but which is still worth preserving, can be understood as an artefact’.

A web resource might be considered an artefact if, for example, it:

• has intrinsic value to the Institution for historical or heritage purposes;
• is an example of a significant milestone in the Institution’s technical progress, for example the first instance of using a particular type of software.

Preserved artefacts could include image collections (still and moving), databases, e-learning objects, digitised objects or research objects.

• Definition of ‘web preservation’.
• To ensure that everyone in the Institution agrees on what should be preserved and how, a web preservation programme should be developed.
• All resources must be managed in order to preserve them.
• There are issues to bear in mind which are specific to web resources, Web 2.0 resources and content management systems.

For the purposes of this Guide, we define web preservation as ‘the capture, management and preservation of websites and web resources’. Web preservation must be a start-to-finish activity, and it should encompass the entire lifecycle of the web resource.

Another definition to consider in this context is JISC’s definition of ‘digital preservation’ – ‘the set of processes and activities that ensures long-term, sustained storage of, access to and interpretation of, digital information’.

Institutional views of preservation requirements may vary so it is important for an Institution to agree on a web preservation programme which defines the web resources which will be preserved. When considering this bear in mind that:

• Resources must be managed in order to preserve them.
• Preservation will not apply to all web resources: a selective approach is recommended.
• Preserving every version of every resource is not always necessary.
• Permanent preservation (as defined by the OAIS model) is not the only viable option. Short-term protection of a resource from loss or damage is an acceptable form of preservation.
• Preservation actions do not have to result in a ‘perfect’ solution.

When considering web resources there are a number of specific preservation issues which apply. In addition, Web 2.0 and content management systems present unique issues.

Web resource preservation issues

1 – Frequency of change: Web resources change to a greater or lesser extent every day, and periodically change dramatically because of events such as re-branding, the implementation of a content management system, or changes to content providers.

2 – Quantity and range of resources: The quantity and range of resources potentially needing preservation are so large it is vital to: know what resources there are; where they are; and what to do about them.

3 – Continuity: Because of the ease with which websites and pages can be edited, the possible impact on users expecting continuity in web resources can be overlooked. For example, a page may stay the same, but no longer be available from the same URL, or it may remain at the same URL but its content changes. So the issues are: persistence of resources at a given URL; and persistence of resources within a domain.

Ideally it should be possible to support versioning across a whole site, so that old versions of a page link to their associated contemporary versions, but this represents a large overhead.

4- Integrity of web resources: Websites and pages need to be protected from careless or wrongful amendment, deletion, or removal, whether by malevolent hackers/crackers, or well-intentioned Institutional staff.

5 – Ownership: There may be issues of ownership resulting from web resources being managed by many different departments or members of staff, or by sub-sites sometimes being temporary or ad hoc (for example, a project site).

6 – Databases and deep websites: Databases present particular issues because preserving an underlying database may not preserve the user’s experience on the web. Also database-driven websites are not always easy to capture by remote harvesting.

7 – Streaming and multimedia: The quantity and quality of data, and the range of formats, can cause issues when dealing with multimedia. In addition, these resources can be hosted elsewhere and therefore the same set of issues applies as for Web 2.0 applications (see below).

8 – Personalised websites: Some websites offer users customisable features. This raises the issue of whether every possible combination of every user’s custom view should be preserved.

9 – Appraisal and selection: Appraising and selecting which web resources should be preserved raises many questions which are dealt with in Chapter 5.

10 – Provising access: Once preserved it has to be considered how access will be provided to the web resources and how to deal with issues of IPR and ownership.

11 – Resources for preservation: Both personnel and technical resource issues also have to be considered. Preservation work can be an overhead on day-to-day web and records management activities so assigning people to the preservation work needs to be balanced with routine web and records management.

In technical terms, it is necessary to estimate how much storage space will be required to store the old web resources and where this will be located.

Web 2.0 preservation issues

The two most important issues with Web 2.0 software and applications are ownership and retention.

1 – Ownership and responsibility: Often individuals create and manage their own Web 2.0 resources such as external (personal) accounts for Flickr, Slideshare or WordPress.com. So it is possible for academics to conduct a significant amount of Institutional business outside any known Institution network. In these cases, the Institution either does not know this activity is taking place, or ownership of the resources is not recognised officially. In such a scenario, it is likely the resources are at risk.

2 – Retention of ‘master copies’: Third party sites such as Slideshare or YouTube are excellent for dissemination, but they cannot be relied on to preserve materials permanently. So, if a resource is created on one of these third party sites and it requires retention or preservation, arrangements must be made within the Institution for the ‘master copy’.

Content management system preservation issues

With digital preservation in mind, the features of particular value which content management systems (CMSs) may offer are:

• Version control – when changes are made to items in the CMS, the previous version is kept.
• Change logging – when changes are made to items in the CMS, the system records who made the change and when.
• Rollback/reversion – the facility to restore the website, or a part of it, to a previous state.
• Creating a snapshot of the website at a particular point in time.

Many CMSs offer one or more of these features but the extent to which they can easily be used to reinstate older versions of a website, or find what changes happened when, varies dramatically. Version control information is easy to create and store, but less easy to put to practical use. Discussions with web managers suggest that these features are rarely tested very vigorously.

The particular preservation issues of CMSs are:

• Page names and numbers.
• Rollback function is limited.
• Lifespan of system.
• Compatibility between systems.

Page names and numbers: Some CMSs may present problems to a remote harvesting engine, or crawler, as pages that are identified with numerical tags instead of page names, for example, may not be recognised, and hence may be missed by the remote harvester. This is especially true if the CMS generates pages dynamically. The severity of this behaviour may also depend on how the site was built in the first place.

Rollback function is limited: A rollback may not be the same as restoring a full snapshot as it will tend to focus on a particular page or content element, but not its entire context. Web pages usually have many objects that they relate to – for example embedded images and stylesheets – so the rollback cannot be used to view the content of the whole page as seen by the user. The content is held in the database as layers of time-stamped pages and a script is required to retrieve it. It is therefore not clear to what extent the rollback functions and version control tools produce useful, tangible outputs that could be captured, managed or preserved.

Compatibility between systems: A CMS may not be supported indefinitely so the question arises about whether the new version will be compatible with the old version. Also, the Institution may decide to change the CMS and, as CMS internal management of content, data and metadata tends to be application-specific, this may mean that moving large quantities of interlinked website content between CMS packages is likely to be a manual and intensive process.

Backing up is not enough: A CMS is a database full of content, but simply backing up the database will not constitute preservation of the content. The backup action would capture a change history of the website for as long as it was kept in that CMS; it would not constitute a usable collection of page snapshots, or an archived website.

Metadata: The change history metadata would be extremely useful for records management and preservation purposes, but access to that metadata is not guaranteed: it would need to be exportable in a form that could be preserved.

The project handbook was published in November 2008. Since then we have seen a growing awareness of the importance of digital preservation in general and in the preservation of web resources (including web pages, web-based applications and websites) in particular. The current economic crisis and the expected cuts across public sector organisations mean that a decade of growth and optimism is now over – instead we can expect to see reduced levels of funding available within the sector which will have an impact on the networked services which are used to support teaching and learning and research activities.

The need to manage the implications of these cutbacks is likely to result in a renewed interest in digital preservation. We are therefore pleased to be able to publish this new guide, based on the original PoWR: The Preservation of Web Resources Handbook, which provides practical advice to practitioners and policy makers responsible for the provision of web services.

Brian Kelly, UKOLN, Project Director, JISC PoWR project