Web archives as a data resource for digital scholars 

 

Selection of content for web archives 
 
1.  How is web archiving framed by the law? 

    i.        legal deposit legislation 

   ii.        legislation on public archives 

2.  How is web archived content selected? 

    i.        broad crawls (covering top-level domain crawls and relevant content outside of the national domain(s)) 

   ii.        selective crawls (thematic or events-based collections) 

Consultation, access and ease of use of web archives 

 

1.  How to consult and access web archived content? 

    -     Granting access to the public must comply with the rules of copyright law 

    -     In a number of web archives, only specific parts of the collections are freely accessible. 

    -    Some web archives, which are not publicly available, are only accessible from specific workstations in the                 library premises. 

2. What makes a web archive easy to use? 

    -    Most web archive interfaces have a URL search method. 

    -    Full-text search is also supported by most web archives. 

3. Tools used in web archiving 

    -   HTTrack (Roche 2018) copies the website(s) to the disc so that the user can simply    
         access it in the browser. 

   -   Wget (Free Software Foundation 2017) and the related Wpull tool (Foo 2016) are 
        powerful command line tools with built-in web crawl features comparable to HTTrack. 

   -   F(b)arc (Fbarc GitHub 2018) is a command line tool that can be used to archive data 
        through the Facebook Graph interface. 

   -   Twarc (Twarc GitHub 2018) is a command line and library application that makes it 
         easy to use the Twitter APIs. 

   -   Social Feed Manager (Social Feed Manager 2018) can harvest data from Twitter, 
        Tumblr, Flickr, and Sina Weibo. 

   -   Tools to view the archived websites include Webrecorder Player (Webrecorder Player 

        for Desktop GitHub 2018), OpenWayback (IIPC 2018), pywb (Pywb GitHub 2018) and 
         WAIL (Web Archiving Integration Layer) (Kelly 2017). 

1.     Policy, Act or Agreement govern by the WA implementation 

 

i)                Policy 

·       Vision and Objectives 

The institution must explain the purpose of their chosen web archiving, where they want to finish in doing so and how those steps linked to the broader mission of the institution. Steps in this cycle occur mainly because institutions are initially planning their programs. However, institutions tend to review and redefine 

their web archiving objectives throughout the life of the program. 

 

·       Resources and Workflow 

It can be defined in many ways. In some context of the WALCM’s outer circle, institution observe the resources and workflow which will be beneficial to maintain or create an entire institution’s web archiving program. It can be defined similar to policy, as they can used in various fields of web archiving life cycle model. It is also considered a general program management term that can be used for each element in the model's inner ring. In this context, resources and workflows become part of daily web archiving activities. 

 

·       Access/ Use/ Reuse 

The institution makes decisions about whether and how to provide access to their collections and monitor how patrons use the content. Most of Archive-It partners have publicly archived content, although the growing volume requires some content to be restricted for a period of time, whether a specific URL or domain, individual collection, or their entire account with various collections. 

 

·       Preservation 

The institution makes decisions about how they want to preserve 

the data they assemble in their web archiving activities. This includes both data files and metadata. Many institutions working with the Archive-It service rely on the Internet Archive for storage and maintenance of WARC files and related metadata 

 

·       Risk Management 

The institution considers their approach to risk in creating the web archive the program, they see copyright and permissions as well as access. The decision can also be seen in the choice institutions making when decide on which sites to archive. It can be managed and reduced preemptively, and sometimes institutions can need to address possible problems that arise after content archiving has occurred. 

 

2.     Citation 

 

Brag, M., & Hanna, K. (2013). THE WEB ARCHIVING LIFE CYCLE MODEL. THE WEB ARCHIVING LIFE CYCLE MODEL.The Archive-It Team Internet Archive. 

 

Web archives as a data resource for digital scholars 

How is web archiving framed by the law? 

There are two legal ways to enable web archiving: 

1.    legal deposit legislation 

2.    legislation on public archives 

How is web archived content selected? 

1.    broad crawls 

2.    domain crawls 

How to consult and access web archived content? 

1.    Some of the web archives are freely accessible online such as Arquivo.pt. in Portugal or the web archive developed by the National Library of Ireland. 

2.    Some web archives, which are not freely available, are only accessible on the premises of the library from specific workstations. 

Tools used in web archiving: 

1.    HTTrack (Roche 2018) copies the website(s) to disk so the user can simply open it in a browser. 

2.    Wget (Free Software Foundation 2017) 

3.    Wpull 

4.    F(b)arc (Fbarc GitHub 2018) is a command line tool that can be used to archive data using the Facebook Graph interface. 

5.    Twarc (Twarc GitHub 2018) is a command line tool and library that makes using the Twitter APIs easy. 

6.    Heritrix (Webarchive.jira.com, July 2016) is a general purpose web crawler designed with web archiving in mind. 

7.    Brozzler (Brozzler GitHub 2018) 

Tools to view the archived websites: 

1.    Webrecorder Player (Webrecorder Player for Desktop GitHub 2018), 

2.    OpenWayback (IIPC 2018), pywb (Pywb GitHub 2018) 

3.    WAIL (Web Archiving Integration Layer) (Kelly 2017) 

Reference: 

Bragg, M. & Hanna, K. (2013). The web archiving life cycle model. Retrieved from  https://archiveit.org/static/files/archiveit_life_cycle_model.pdf 

WEB ARCHIVING LIFE CYCLE 

1)      FIRST OUTER LAYER 

·         POLICY 

 

2)      SECOND OUTER LAYER 

·         VISION AND OBJECTIVES 

·         RESOURCES AND WORKFLOW 

·         ACCESS / USE/ REUSE 

·         PRESERVATION 

·         RISK MANAGEMENT 

Refference

Bottis, M., Papadouulos,M. et. al.(2019). On the Eve of Web-Harvesting and WebArchiving for Libraries in Greece. No(2).