Archives — Phil Cifone

Where this started

In 2020 I was the digital asset manager at a photographer’s studio that was about to merge two locations. The archive was 265 TB on a shared storage network, and the only safe way to move it was to make another complete copy first. So before anything left the building, it all went to cloud storage. The vendor’s write-up quotes me saying I’m not an IT guy. That was true when the project started. It wasn’t by the end. Everything I’ve done since is the same job at a smaller scale: making sure the only copy of someone’s work is never the only copy.

Copies: 3-2-1

The oldest rule in backups is three copies of everything, on two different kinds of storage, with one somewhere else. A fire, a theft or a failed drive can take one copy, but not all three.

Two things that get missed:

A sync is not a backup. Dropbox, iCloud or a mirrored drive copies your mistakes as faithfully as your work. Delete a folder and it’s gone everywhere. A backup keeps history, so you can go back to last Tuesday.
A copy nobody has checked is a guess. More on that below.

On my own systems the main archive lives on a NAS. It’s snapshotted every hour and replicated to a backup server and to my desktop. The irreplaceable parts also go to a Mac mini with its own drive, and from there offsite to Backblaze. ZFS housekeeping has the details, including the parts I got wrong. For a client it’s usually simpler: a small NAS, a second machine or drive that pulls from it, and a cloud copy with versioning turned on.

Checking the copies

Archivists call this fixity: proof that a file is still, bit for bit, the file you stored. Drives fail quietly. A sector goes bad, and a photograph that opened fine five years ago comes back with a grey band across it. Nothing warns you until someone opens it.

The usual approach is to record a checksum for every file and recalculate it on a schedule. A checksum is a short fingerprint calculated from a file’s contents. If the fingerprint changes, the file has.

I use ZFS, which does this all the time rather than on a schedule:

Every block has a checksum, and ZFS checks it every time the block is read. If it doesn’t match and there’s a redundant copy, ZFS repairs the bad block from it.
A scrub reads and checks every block in the pool, including files nobody has opened in years.
Copy-on-write means changes go to a fresh spot on the disk, and the old version is released only after the new one is safely written. A power cut halfway through a save can’t leave a half-written file. It’s also why snapshots are instant.

Every morning a script emails me the health of every pool. That email is how I found out, one Sunday, that a pool had been suspended halfway through a scrub. The write-up shows what checks like this are for.

The gap is that ZFS’s checksums stay inside ZFS. When files leave, for a cloud bucket, a client’s hard drive or a museum taking in a collection, the fingerprints have to travel with them. The library world’s answer is BagIt: a folder of files plus a plain-text list of every file’s checksum, so whoever receives it can confirm nothing changed on the way. It’s a simple open standard, and it’s what I’m building into handovers.

Formats that last

A format lasts when it’s openly documented, widely supported and doesn’t need one company’s software to open it. The working rule: keep a master in a durable format and make everything else from it.

What	Keep (the master)	Share (copies)
Camera originals	The raw file as shot, plus a DNG	JPEG
Scans and finished photographs	TIFF, uncompressed or lossless	JPEG, AVIF
Images on a website	Made from the master	AVIF, with a WebP or JPEG fallback
Documents	PDF/A	PDF
Catalogues and lists	CSV, next to the spreadsheet	—

Raw files are the hard case. Most cameras still record their maker’s own raw format, such as Canon’s CR3, Nikon’s NEF or Sony’s ARW. Each one stays readable only as long as software keeps supporting it. DNG is Adobe’s openly published raw format, and the Library of Congress accepts it for digital photographs. Keep the original raw anyway: it’s what the camera actually wrote, and converting can drop data only the maker’s software understands. A DNG alongside it is cheap insurance.

AVIF is my favourite format for the web. It’s open, royalty-free and noticeably smaller than WebP or JPEG at the same quality. This site uses it, and so do my own sites. But it’s a format for showing work, not keeping it. At the settings websites use it’s lossy, so it should never be the master or the only copy.

Prints, negatives and other physical work

Handling and storage:

Hold prints and negatives by the edges, wearing nitrile gloves. Cotton gloves are out of date: they snag, they carry dirt and they make things easier to drop.
Use enclosures made for photographs. That means paper or plastic that has passed the Photographic Activity Test (PAT), or polyester and polypropylene sleeves. Never PVC. No rubber bands, paper clips or tape.
Keep it cool, dry and steady. Not the attic, not the basement, not against an outside wall. Swings in temperature and humidity do as much harm as heat itself.
Separate negatives that smell of vinegar. Older acetate film breaks down that way, and the gas speeds up the same decay in its neighbours. Digitize those first.

Digitizing:

Camera scanning, with a camera on a copy stand, is faster than a flatbed for most prints and negatives, and gentler on them. Even light, a colour target in the first frame, raw capture.
The capture is the master. Cropping, colour correction and dust removal happen on copies, so the untouched capture is always there to go back to.
FADGI is the yardstick. These US federal guidelines set the resolution, colour accuracy and lighting expected at each quality level. A small archive doesn’t need the top rating, but it should know which one it’s aiming for.

Cataloguing

A file nobody can find is lost, however well it’s preserved.

Names that sort and survive a move. A date or catalogue number first (2024-03-11_studio-view.tif, RB-0047.tif), no spaces, and nothing that only makes sense inside the folder it’s in today.
Metadata inside the file. Title, creator, date, caption and rights, written into the image itself (IPTC and XMP), so they go wherever the file goes.
One catalogue. A spreadsheet with a row for every work: title, date, medium, real dimensions, where the physical piece is and which files belong to it. Saved as CSV too, because a CSV will open in anything.

Ronnie Bookbinder’s site is built from a catalogue like this. Every painting’s real size is recorded, which is why the site can show each one in proportion.

The things around the archive

An archive also depends on things that aren’t files. These are what get lost when someone dies, retires or leaves a job:

The domain name. The artist or organization should be the registrant, the legal owner, at a fairly priced registrar. Whoever looks after the website can be the technical contact without owning it.
Accounts and passwords. Registrar, hosting, cloud storage and email, written down somewhere the family or the next director can actually find them.
The hardware and network. What’s in the building, how it’s connected and what’s still under warranty.

Checking all of this is the first part of the assessment I offer.

Words institutions use

I work at the scale of a studio, not a national library. These are the standards larger institutions use. I borrow their ideas where they fit, and they’re useful shared words when an organization already works with them.

OAIS: The reference model for an archive (ISO 14721). Material comes in, is kept and checked, and goes back out to people who need it, and each stage has its own package of files and description.
BagIt: A folder of files plus a checksum list, so a transfer can be checked at the other end. Started at the Library of Congress, now an internet standard (RFC 8493).
PREMIS: A dictionary for preservation metadata: what happened to a file, when, and who did it. Checksums verified, formats migrated, copies made.
FADGI: The US federal guidelines for digitization quality, mentioned above.
NDSA Levels of Digital Preservation: A one-page chart that grades an archive from level 1 to 4 in five areas: storage, integrity, control, metadata and content. It's a good shape for an assessment.

The notes are the write-ups behind this page, from my own systems. If you have an archive or a collection that needs looking after, write to [email protected].