# data[].attributes.headline
Introducing the EmailKnowledge Extraction with Social Network Analysis (EKESNA) tool for discovering an organisation's expertise network (2015)

# data[].attributes.subject
Introducing the EmailKnowledge Extraction with Social Network Analysis (EKESNA) tool for discovering an organisation's expertise network (2015)

# data[].attributes.headline
Blogpost: PDF or HTML for public documents?

# data[].attributes.subject
Blogpost: PDF or HTML for public documents?

# data[].attributes.attachment_link.readable_text
Should official online documents be PDF files? Many institutions say they obviously should, but the format has some clear disadvantages. An article on the UK's Government Digital Service site argues that HTML, not PDF, is the right format for UK government documents. Its arguments, to the extent that they're valid, apply to lots of other documents.

It makes a plausible case against PDF. The trouble is that the case against HTML is even stronger in some ways.

The limitations of PDF

The original idea of PDF was that it would define a document with a consistent appearance using any computer and software. The emphasis is on appearance, not content. In the nineties, that was enough. Today, the Internet's emphasis is on flexible content delivery. Information may appear on a small screen or a big one, it may be printed, or the user may hear it read aloud. Fonts, colors, and sizes can change to suit the user's needs and preferences.

In the general case, PDF does a poor job of presenting a document's content. The sequence of text in a file may have little to do with its logical structure. As the article notes, this creates difficulties for accessing and extracting information.

Using tagged PDF and PDF/A to impose order and exclude certain features greatly improves the usability of documents. The article doesn't mention these profiles but refers to them by implication. They require working from an original document, such as an OpenOffice file, to get good results; tagging a PDF after the fact is possible but usually won't reflect the document's structure very well.

The article does have a point. PDF files are inflexible and often don't serve online needs well. How well it stacks up against HTML is another question.

Issues with HTML

HTML has its own problems for official documents. It doesn't necessarily present a consistent view of a document. In fact, HTML5 is designed so that the same document can have a completely different appearance, even showing different content, in different environments.

Most HTML is broken. Files violate their own syntax rules, and browsers piece them back together. It's not unusual for a file to fail to render at all on some browsers.

HTML files routinely have dependencies on other files, often from different hosts. Even a simple file with no JavaScript usually depends on image files and CSS for its appearance.

The exact appearance of a legally significant document is important. When the resolution of legal disputes depends on tiny details, differences in what the user sees can affect how courts rule.

Document-quality HTML?

Neither HTML nor PDF is entirely satisfactory for presenting official documents on the Internet. One is too free-wheeling, and the other emphasizes appearance over content. Perhaps what's needed is a standard which defines a restricted HTML that's comparable to PDF/A. It would guarantee consistent content.

This "HTML/A" would be free of JavaScript and third-party files. Conforming files would strictly follow HTML5 syntax. Validation would apply to a set of files, including images and CSS. Only files of specified types would be allowed. To be complete, it would need to specify what those files had to conform to.

It would be nice to have a standard like this. HTML that conformed to it would have a reliable appearance on any browser, and it wouldn't be hard to make it backward compatible to many older browser releases. Creating the standard would be a major job, though, and I don't think anyone with influence has been inclined to tackle it.

"Pure" HTML, with no JavaScript or CSS and with validated compliance, would be one way to go. That wouldn't produce responsive pages, though. Images would remain the same size regardless of the viewing environment. This isn't what the Government Digital Service is looking for.

For all its problems, PDF/A has some strong advantages over HTML for documents that need stable and unambiguous content. There's no fully satisfactory solution.

# data[].attributes.attachment_link.description
Should official online documents be PDF files? Many institutions say they obviously should, but the format has some clear disadvantages. An article on the UK's Government Digital Service site argues that HTML, not PDF, is the right format for UK government documents. Its arguments, to the extent that they're valid, apply to lots of other...

# data[].attributes.headline
Book: Theory & Craft of Digital Preservation (2017)

# data[].attributes.subject
Book: Theory & Craft of Digital Preservation (2017)

# data[].attributes.body
Trevor Owens

# data[].attributes.attachment_link.readable_text
This weekend I'm submitting the full draft of the manuscript for my book The Theory and Craft of Digital Preservation to the publisher, Johns Hopkins University Press.

Update: to make it easier to read, I've shared a PDF preprint of the whole draft.

I've had a lot of fun working on this on nights and weekends over the last year. I have also learned a ton from everyone who has read drafts of the work in progress.

I've had a few folks reach out to me after reading parts of drafts and say things like "I'd love to read more of this. When will it be out?" I'm not sure exactly how long it will take for the next round of review and all the improvements that will come from working with a great press. With that said, drafts of the entire book are now online. Instead of having folks pick through my previous blog posts with the links, I figured I would put them all together in order in this post.

So to that end, below you can find an index to the eight chapters and the intro and conclusion. I'm going to leave this up with all the comments in them. I went through and resolved comments offline in my own copies of these but thought it would be fun to leave up the messy original drafts and a record of all the great input and ideas that folks have offered up to improve the text.

Table of Contents

Introduction Beyond Digital Hype & Digital Anxiety (7 pages)
Section One: Theory of Digital Preservation
Ch 1: Preservation's Divergent Lineages (14 pages)

Ch 2: Understanding Digital Objects (12 pages)

Ch 3: Challenges & Opportunities for Digital Preservation (11 pages)
Section Two: The Craft of digital Preservation
Ch 4: The Craft of Digital Preservation (6 pages)

Ch 5: Preservation Intent & Collection Development (13 pages)

Ch 6: Managing Copies & Formats (15 pages)

Ch 7: Arranging & Describing Digital Objects (19 pages)

Ch 8: Enabling Multimodal Access & Use (18 pages)

Conclusion: Tools for Looking Forward (9 pages)

Advance Twitter Praise for the Book

I pulled out a few fun tweets from folks responding to the book that I thought were fun to share.

Gulped down these 3 draft chapters. Really like them and idea of the book: an accessible explanation of digital preservation that https://t.co/nMcQGrXBRb

- Alan Liu (@alanyliu) February 15, 2017

https://twitter.com/save4use/status/877128435696619520

axioms by @tjowens are gold. i want to print them out and show up to meetings with annotated copies #digipres https://t.co/i99LDJQ2XT

- Elvia Arroyo-Ramirez (@elvia_arroyo) June 20, 2017

We all shared what our favorite numbers were today at UNC. This is great, thank you!

- julesies (@julesies) June 21, 2017

Really looking forward to the book! #s 3, 11, and 12 had me cheering out loud. Which I realize may be a bit much, but hey.

- Ryan MacMichael (@supalaze) June 20, 2017

# data[].attributes.attachment_link.title
Full Draft of Theory & Craft of Digital Preservation

# data[].attributes.attachment_link.description
This weekend I'm submitting the full draft of the manuscript for my book The Theory and Craft of Digital Preservation to the publisher, Johns Hopkins University Press. Update: to make it easier to read, I've shared a PDF preprint of the whole draft.

# data[].attributes.body
Relevance to ESRC:  while we host older OHRM's and websites, Google's memory hole results in a risk - if we don't regularly update content, then old content will fall off Google

# data[].attributes.attachment_link.readable_text
I think Google has stopped indexing the older parts of the Web. I think I can prove it. Google's competition is doing better.

Evidence · This isn't just a proof, it's a rock-n-roll proof. Back in 2006, I published a review of Lou Reed's Rock n Roll Animal album. Back in 2008, Brent Simmons published That New Sound, about The Clash's London Calling. Here's a challenge: Can you find either of these with Google? Even if you read them first and can carefully conjure up exact-match strings, and then use the "site:" prefix? I can't.
[Update: Now you can, because this piece went a little viral. But you sure couldn't earlier in the day.]
Why? · Obviously, indexing the whole Web is crushingly expensive, and getting more so every day. Things like 10+-year-old music reviews that are never updated, no longer accept comments, are lightly if at all linked-to outside their own site, and rarely if ever visited... well, let's face it, Google's not going to be selling many ads next to search results that turn them up. So from a business point of view, it's hard to make a case for Google indexing everything, no matter how old and how obscure.

My pain here is purely personal; I freely confess that I'd been using Google's global infrastructure as my own personal search index for my own personal publications. But the pain is real; I frequently mine my own history to re-use, for example in constructing the current #SongOfTheDay series.

Competition · Bing can find it! DuckDuckGo can too! Both of them can find Brent's London Calling piece, too.

What Google cares about · It cares about giving you great answers to the questions that matter to you right now. And I find that if I type in a question, even something complicated and obscure, Google often surprises me with a timely, accurate answer. They've never claimed to index every word on every page.

My mental model of the Web is as a permanent, long-lived store of humanity's intellectual heritage. For this to be useful, it needs to be indexed, just like a library. Google apparently doesn't share that view.

What I'm going to do · When I have a question I want answered, I'll probably still go to Google. When I want to find a specific Web page and I think I know some of the words it contains, I won't any more, I'll pick Bing or DuckDuckGo.

I am an employee
of Amazon.com, but
the opinions expressed here
are my own, and no other party
necessarily agrees with them.

A full disclosure of my
professional interests is
on the author page.

# data[].attributes.attachment_link.description
I think Google has stopped in­dex­ing the old­er parts of the We­b. I think I can prove it. Google's com­pe­ti­tion is do­ing bet­ter. Ev­i­dence · This isn't just a proof, it's a rock-n-roll proof. Back in 2006, I pub­lished a re­view of Lou Reed's Rock n Roll An­i­mal al­bum.

# data[].attributes.headline
Meeting: Designing Storage Architectures for Digital Collections 2014

# data[].attributes.subject
Meeting: Designing Storage Architectures for Digital Collections 2014

# data[].attributes.body
Slides, notes, readings

# data[].attributes.attachment_link.keywords[]
national digital information infrastructure and preservation program

# data[].attributes.attachment_link.title
Designing Storage Architectures for Digital Collections 2014 Meeting & Events - Digital Preservation (Library of Congress)

# data[].attributes.attachment_link.description
Information about Designing Storage Architectures 2013.

# data[].attributes.headline
Best Practice Data Life Cycle Approaches for Life Sciences (2017)

# data[].attributes.subject
Best Practice Data Life Cycle Approaches for Life Sciences (2017)

# data[].attributes.body
Peter Neish

# data[].attributes.attachment_link.readable_text
Abstract

Throughout history, the life sciences have been revolutionised by technological advances; in our era this is manifested by advances in instrumentation for data generation, and consequently researchers now routinely handle large amounts of heterogeneous data in digital formats. The simultaneous transitions towards biology as a data science and towards a 'life cycle' view of research data pose new challenges. Researchers face a bewildering landscape of data management requirements, recommendations and regulations, without necessarily being able to access data management training or possessing a clear understanding of practical approaches that can assist in data management in their particular research domain. Here we provide an overview of best practice data life cycle approaches for researchers in the life sciences/bioinformatics space with a particular focus on 'omics' datasets and computer-based data processing and analysis. We discuss the different stages of the data life cycle and provide practical suggestions for useful tools and resources to improve data management practices.

# data[].attributes.attachment_link.title
Best Practice Data Life Cycle Approaches for the Life Sciences

# data[].attributes.attachment_link.description
Throughout history, the life sciences have been revolutionised by technological advances; in our era this is manifested by advances in instrumentation for data generation, and consequently researchers now routinely handle large amounts of heterogeneous data in digital formats. The simultaneous transitions towards biology as a data science and towards a 'life cycle' view of research data pose new challenges.

# data[].attributes.headline
[READ] Blogpost: On Hearing the Learned Astronomer

# data[].attributes.subject
[READ] Blogpost: On Hearing the Learned Astronomer

# data[].attributes.headline
Blogpost: Getting Started in Bagit in 2018

# data[].attributes.subject
Blogpost: Getting Started in Bagit in 2018

# data[].attributes.attachment_link.readable_text
Take two!

In December, I hastily wrote an update to an old post about BagIt, the Library of Congress' open-source specification for hierarchical packaging of files to support safe data storage and transfer. The primary motivation for the update was some issues that the Video Preservation course I work with encountered with my instructions for installing the bagit-python command-line tool, so I wanted to double-check my process there and make sure I was guiding readers correctly. I also figured that it had been a couple years and I could write about new implementations while I was at it. A cursory search turned up a BagIt-for-Ruby library, so I threw that in there, posted, *then* opened up a call for anything I'd missed.

It was at this point, as I sifted through the various scripts, apps, tools and libraries that create bags in some way that I realized I had lost the thread of what I was even trying to summarize or explain.

Every piece of software using the BagIt spec ever? That, happily, is a fool's errand - the whole point of the spec is that it's super easy and flexible to implement, no matter how short the script. So...there's a lot of implementations.

Every programming language with an available port/module for creating bags according to the BagIt spec? Mildly interesting for hybrid archivist/developers, but probably of less practical use for preservation students, or the average user/creator just trying to take care of their own files, or archivists that are less programming-inclined. A Ruby module for BagIt is objectively cool and useful - for those working/writing apps and scripts in Ruby. Given that setting up a Ruby dev environment requires some other command-line setup that I didn't even get into, someone's likely not heading straight to that module right out of the gate.

"Using BagIt" was/is the wrong framework. Too broad, too undefined, and as Ed Summers pointed out, antithetical to the spirit in which a simple, open source specification is made in the first place: to allow anyone to use it, anywhere, however they can - not according to one of four or five methods proscribed in a blog post.

So I am rewriting this post from the mindset, not of "here's all the forms and tools in which BagIt exists", but rather, "ok, so I'm learning what a bag is and why's it useful - how can I make one to get started?"

Because the contents of a specification are terrific and informative, but in my experience nothing reinforces understanding of a spec like a concrete example. And not only that, but one step further - *making* an example. Technical concepts without hands-on labwork or activities to solidify them get lost - and budding digital preservationists told to use the BagIt spec need somewhere to start.

So whether you're just trying to securely back up your personal files to a cloud service, or trying to get a GLAM institution's digital repository to be OAIS-compliant, validation and fixity starts at square one. Let me do that as well.

What's a bag?

Just for refresher's sake, I'm going to re-post here what I wrote back in 2016 - so that this post can stand alone as a primer:

One of the big challenges in digital archiving is file fixity - a fancy term for checking that the contents of a file have not been changed or altered (that the file has remained "fixed"). There's all sorts of reasons to regularly verify file fixity, even if a file has done nothing but sit on a computer or server or external hard drive: to make sure that a file hasn't corrupted over time, that its metadata (file name, technical specs, etc.) hasn't been accidentally changed by software or an operating system, etc.

But one of the biggest threats to file fixity is when you move a file - from a computer to a hard drive, or over a server. Think of it kind of like putting something in the mail: there are a lot of points in the mailing process where a computer or USPS employee has to read the labeling and sort your mail into the proper bin or truck or plane so that it ends up getting to the correct destination. And there's a LOT of opportunity for external forces to batter and jostle and otherwise get in your mail's personal space. If you just slap a stamp on that beautiful glass vase you bought for your mother's birthday and shove it in the mailbox, it's not going to get to your mom in one piece.

So a "bag" is a kind of special digital container - a way of packaging files together to make sure what we get on the receiving end of a transfer is the same thing that started the journey (like putting that nice glass vase in a heavily padded box with "fragile" stamped all over it).

Sounds great! How do I make a bag?

At its core, all you need to make a bag out of a digital file or group of files is an editor capable of making plain text files (.txt) and an ability to generate MD5 checksums. An MD5 generator takes *any* string of digital information - including an entire file - and encodes it into a 128-bit fingerprint; that is, a 32-character string of seemingly "random" letters and numbers. Running an MD5 generator on the same file will always produce the same 32-character string. If the file changes in some way (even some change or edit invisible to the user), the MD5 string will change as well. So this process of generating and checking strings allows you to know whether a file is exactly the same on the receiving end of a transfer as it was at the beginning.

BagIt bags facilitate this process via a "tag manifest" - a text file including all the digital files contained in the bag (the "data" in question) and their corresponding MD5 checksums. Packaged together (along with some meta information on the BagIt spec and the bag itself), this all allows for convenient fixity checking.

Convenient, though, in the sense of easing automation. While you *can* put together a bag by hand - generating checksums for each file, copying them into text files to create the manifests, structuring the data and manifests together in BagIt's dictated hierarchy -that is a copy/paste nightmare, and not exactly going to encourage the computer-shy into healthier digipres practice.

This is why simple scripts and tools and apps are handy. Down the line, when you're creating your own archival workflow, you may want to find or tweak or make your own process for creating bags - but for your first bag, there's no need to reinvent the wheel.

I'm going to cover intro tools here, for either the command line or GUI user.

Command Line Tools

1. this Bash scriptA simple shell script by Ed that just requires just two arguments: the directory you want to bag, and an output directory (in which to put the bag).Hit the green "Download" button in the corner of the GitHub page, select the ZIP file, then unzip the result. Move the "bagit.sh" file inside to a convenient/accessible location in your computer.Once in Terminal, you can run this bash script by navigating to wherever you put this script, then executing it with:
or

(the "./" or "bash" commands do the same thing - indicating to the Bash terminal to execute the bagit.sh script)

The "/path/to/directory" should be a folder containing all the files you want to be in the bag. Then you will specify the output path for the bag with "/path/to/bag". Both can be accomplished with drag-and-dropping folders from the Finder.

2. bagit-pythonBagit-python is the Library of Congress's officially-supported command line utility for making and working with bags. It requires a working Python interpreter on your computer, plus Python's package manager, "pip". By default, macOS comes with a Python interpreter (2.7.10), but not pip. So we go to the popular command-line Mac package manager Homebrew to put this all together.Sigh. OK. So one of the reasons this post didn't come out last week is that, literally in that same time frame, Homebrew went through....something with regards to their Python packages and how they behaved with Python 2.x vs Python 3.x vs the Python installation that comes with your Mac. (they've locked/deleted a lot of the conversations and issues now, but it was really the dark side of FOSS projects in there for a bit). I kept trying to check my instructions were correct, and meanwhile, every "$ brew update" was sending my python installs haywire. It seems like they've finally settled, but, I'd still now generally recommend giving this page a once-over before working with python-via-homebrew.

But to summarize: if you want to work with Python 3.x, you install a *package* called "python" and then invoke it with python3 and pip3 commands. If you want to use Python 2.x, you install a package called "python@2" and then invoke with either python and pip or python2 and pip2 commands.

...got it?

For the purposes of just using the bagit-python command-line tool, at least, it doesn't matter whether you choose Python 2.x or 3.x. It'll work with both. But stick with one or the other through this installation process. So either:

+

or:

+

That's it! It's just making sure you have a version of python installed through Homebrew, then use the python package/module installer "pip"to install the bagit-python tool. I highly recommend using admin privileges with "sudo" to globally install and avoid some weird permissions issues that may arise from trying to run python scripts and tools like bagit-python otherwise.

One installed, look over the help page with

to see the command syntax - and all the features that you can cover! Including using different hash generators (rather than MD5), adding metadata, validating existing bags rather than creating new ones, etc.

*** a note about bagit-java***
If you are using Homebrew and just run

it will install the bagit-java 4.12.3 library and command-line tool. The LOC no longer supports and doesn't recommend this tool for command line use, and the -help instructions that come with it don't even actually reflect the command syntax you have to use to make it work. So! This isn't a recommendation but just a note for Homebrew users who might get confused about what's happening here.

GUIs

Again, the LOC's official graphical utility program for creating and validating bags. Following the instructions from their GitHub repository linked above, you're going to download a release and then run on macOS by finding and clicking on the "bagger.jar" file (you'll need a working Java install as well).Inside Bagger, once you choose the "Create a Bag" option, Bagger will ask you to choose a "profile" - these just refer to the metadata fields available for inserting more administrative information about your bag and the files therein, within the bag itself. These are really useful for keeping metadata consistent if you're creating a whole bunch of bags, but choosing "<no profile>" is also totally acceptable to get started (you can always re-open bags and insert more metadata later!)"Create Bag in Place" is also a useful option if you don't want (or digital storage limitations even *prevents*) to have two copies of your files (the original + the copy inside the "data" folder in your bag). Rather than copying and creating the bag in a new directory elsewhere, it'll just move around/checksum/restructure the files according to the BagIt spec within the original directory.

2. Exactly

A GUI developed by AVP and the University of Kentucky that combines the bagging process with file transfer - which is the presumed end-goal of bagging in any case. To that end, Exactly doesn't "bag in place" - you always have to pick a source file/directory (or sources - Exactly will bundle them all together into one bag) and a destination for the resulting, created bag. Like Bagger, you can also add metadata via custom-designed fields or import templates/profiles. Added support for FTP or SFTP transfers to remote servers (in addition to locally-attached network storage units like a Samba or Windows share) make it a simple starter option for file delivery.

***************************

If you're getting started with the BagIt spec, these are the places I'd begin. But as to what implementation *you* can come up with from there, based on your personal/institutional needs...that's up to you!

# data[].attributes.attachment_link.description
Take two! In December, I hastily wrote an update to an old post about BagIt, the Library of Congress' open-source specification for hierarchical packaging of files to support safe data storage and transfer.

# data[].attributes.body
https://forensicswiki.org/wiki/Tools - focuses on disk imaging and analysis

http://www.digipres.org/tools/ - extensive, but still incomplete.

https://libguides.mit.edu/digitalarchivestools - a good rundown of some of the primary tools used by an org with a mature digital archives program.
Lachlan: there are some 'core' tools that seem to come up over and over again in workflows because they're established and well supported, such as Teracopy, Jhove and DROID. Perhaps the most important thing in all this is to evaluate the tool against the gap in your workflow and be able to justify it's use. Richard Lehane did this really nicely in his Siegfried demo when he was talking about the distinctions between Fido, DROID and SF.