Magic Cap Overview

Storage is boring — and we aim to keep it that way.

Magic Cap keeps private information encrypted and gives you simple tools to control access.

No accounts, no identities, safely store user data on untrusted providers.

Written in Rust, we provide libraries to read and write data from various sources and a command-line tool for experimentation.

Caution

This is a release-early library that has not yet received cryptographic (or other) audits. We do appreciate feedback, but you own both pieces if you deploy to production :)

We can ignore most details and look at the high-level view of this tool as two diagrams. Encrypting:

overview diagram of magic-cap turning plaintext into encrypted data + a read-capability

Decrypting:

overview diagram of magic-cap turning Data + Read Cap back into plaintext

There is a less-powerful “Verify Cap” that cannot see the plaintext but can confirm all the ciphertext is available and correct:

diagram of magic-cap Verify Cap confirming the validity of ciphertext

Note that the Verify Cap cannot decrypt anything.

Capabilities Without Accounts

Plaintext is transformed into an encrypted Data and a corresponding Read Cap. A Read Cap may be transformed (offline) into a Verify Cap. The size of the Data is a few percent larger than the plaintext. Read and Verify Caps look like this:

mcap0r3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8EAnU9NkykdwXfOg6VdQ7v
mcap0v3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8

Anyone with both a Read Cap and the corresponding Data can turn it back into the plaintext.

Anyone with both a Verify Cap and the corresponding Data can confirm the ciphertext is correct and well-formed (but cannot see the plaintext)

This allows many use-cases, facilitating direct peer interactions since no server interaction is needed to create, transform or share Read Caps (or Verify Caps).

Magic Cap is inspired by the core ideas of “capability theory” embedded in Tahoe-LAFS.

Magic Cap currently implements two kinds of “data capabilities” (or “Caps” for short).

Immutable Read Cap: can be exchanged for the plaintext (alongside a corresponding Data)
Immutable Verify Cap: used to check whether a corresponding Data is valid (but not see the plaintext)

Plaintext is transformed into two pieces: encrypted Data and an associated Read Cap. Re-combining these two pieces later allows the software to recover the plaintext. The Read Cap is a short string (approximately 73 bytes). The size of the encrypted Data is the same as the plaintext (plus a little overhead)

Where and how the Data is stored, moved or transmitted is up to the application. Similarly, where and how the Read Cap is kept is up to the application – its small size allows for storage in TPM or other secure storage or even printed out or transcribed hard-copy.

This gives users of this library a lot of choice, while keeping the core concepts straightforware to reason about.

Networking

Although we have a couple ideas embedded in the command-line application currently, how to store and fetch Data is very open-ended (on purpose). This allows many different uses and experimentation.

It is also possible to layer additional access controls on top, if your threat-model or use-case demands that. For example, Data may be on an SSH-accessible server accessed via keys or passwords. One may use S3 to store data, taking advantage of additional IAM services – or not!

The current methods built-in to the command-line tool are:

Local Files: Data is stored on the file-system, with familiar permissions-based read/write access-controls
HTTP ReST-like API: The mcap publish subcommand can export local filesystem data into a format suitable for direct access via HTTP, simulating a proof-of-concept ReST-style API that can be statically-hosted.

See the “Hands-On Examples” section for more.

Use Case Examples

Note that these are ideas about how this technology might be used.

We haven’t built these applications, and highly recommend developing your own security model.

Traditional App

Lots of application need to store user data. A common pattern is to host the data on servers the application operators or developers control (e.g. Amazon S3).

Traditionally, these same developers have access to all of this data.

Using Magic Caps allows this user data to be truly private to the user – the “Data” piece is stored on the servers, while the Read Cap is stored on user-controlled secure storage (e.g. the TPM of a smartphone or an encrypted hard-drive of a laptop).

(Read about “Verify Caps” that allow the service to confirm data integrity without having access to the plaintext).

Timed Release of Large Digital Artifact (game, movie)

You have a multi-gigabyte digital artifact to release at a particular time.

Using Magic Caps, you can do this:

Encrypt the artifact, producing the large Data file and small Immutable Read Cap string
Upload the large Data file to any storage system you like (Web server, Torrent)
Instruct fans or users to download the Data

Then, when the actual release time arrives, you distribute the Read Cap string (via email, text, SMS, Signal, a Web site update).

This separates the distribution from the actual “release”, reducing server bandwidth requirements, etc.

Shared Organizational Data

An organazation often has lots of data, often with different visibility requirements.

A system centered around Magic Caps (along with the Catalog and Anthology concepts) could allow members of the organization to carry offline copies of all the data while only allowing access to particular pieces of it.

While Magic Cap itself has no concept of identity or users, application developers may layer this on top.

So, members of an organzation could all have a complete copy of all organization Data items (and keep in sync periodically via rsync or similar).

Since all these Data items are encrypted, members need a Read Cap to actually decrypt any of them. Thus, particular members could be given Read Caps as they require them. (Using the Anthology concept makes it easier to share many items with one Read Cap instead of many).

Source Code, Documentation

Crate: https://docs.rs/magic_cap/latest/magic_cap/
CLI: https://docs.rs/magic_cap_cli/latest/magic_cap_cli/
Code: https://github.com/magic-cap/magic-cap
Prose Documentation: https://magic-cap.readthedocs.io/en/latest/
API Documentation: https://docs.rs/magic_cap/latest/magic_cap/

Next Steps

This tool can be used via the available CLI, which exposes most of the Rust library functionality.

In the next sections, we explore more of the details and how to access them via the CLI.

Diving Deeper

The Overview glosses over or doesn’t discuss several other details.

Locating and Retriving Data: Locating the correct Data to go with a Read Cap will be required by many applications
Retrieving Data via Network: An API to retrieve Data for a particular Read Cap from an HTTP server
Catalogs and Anthologies: Two different ways to consider groups of encrypted Data.
Serialization Format: The shape and details of the on-disk Data format.

Locating and Retrieving Data

Applications require a method to associate a Data to a particular Read Cap.

Since the Read Cap itself gives access to the plaintext, we can’t use that. Therefore, we have a way to derive an Immutable Identifier from either a Read Cap or a Verify Cap.

This Identifier doesn’t reveal which Read Cap it goes with – that can only be done by a device holding the Read Cap (or Verify Cap). Internally, we use the Identifier in the Catalog.

The Identifier is a 32-byte value derived via tagged-hash (with tag "magic_cap_storage_index_v1"). This Identifier is essentially the same as the “storage index” used in Tahoe-LAFS.

When locating Data stored in files on disk, that Catalog implementation uses the Identifier to put the files in a structure very similar to a Git “objects/” subdirectory.

In the HTTP implementation of Catalog, the Identifier is used in the URL to ask for the ciphertext (and metadata) separately.

Catalogs and Anthologies

When considering a group or collection of related items, we see two main possibilities.

Catalogs

One way is sort of like a library: a whole bunch of different titles, some perhaps related and others less so. Before the Internet, real-world libraries often used a “card catalog” to index the collection. So, for this sort of collection in Magic Cap, we name it a “Catalog”.

In this analogy, the “title” corresponds to the Identifier and the result is the ciphertext and associated metadata for the Read Cap (or Verify Cap) the Identifier is derived from.

Using the CLI, the top-level --catalog option configures Catalogs to use.

You can ask the CLI to output the Identifier with mcap debug locator which will output the hex-string of the Identifier on stdout.

Anthologies

Another kind of collection is when we have two or more tightly-related items. For example, a collection of pictures to share with friends. Or a group of HTML, CSS and images that comprise a Web site or blog post. Anything that one might typically use a Zip file for.

We name this sort of collection an “Anthology” to stay with the book analogy: several related books or stories or poems are collected together in a single anthology.

An Anthology in Magic Cap is essentially just a list of names and an associated Read Cap. That is, a “Anthology Read Cap” is just a normal Read Cap with its contents being in a particular format which refers to other Read Caps. All the Data for a single Anthology should usually be located in a single Catalog.

The CLI has a mcap anthology sub-command group to create and list Anthologies.

Retrieving via HTTP

On the filesystem, seeking inside a file is fairly inexpensive. We’d like to avoid doing range-request and other less-standard HTTP behavior (although this works too).

The HTTP API is rooted at some base URI. (On the command-line, this is the --catalog-url option). Following this, we have some additional valid sub-URLs:

GET /magic-cap-catalog: JSON describing this Catalog. Currently contains simply {"version": 0}
GET /<identifier>/metadata: the encoded metadata (including the encrypted portion) for the given Identifier.
GET /<identifier>/ciphertext: the ciphertext blocks, one after the other. A client that wishes to do random-access to particular blocks could use HTTP Range: Requests to do so. The block size is encoded in the metadata.

Serialization Format

Magic Cap Strings

Looking at the anatomy of a Magic Cap string. Here is a Verify Cap:

diagram of on-disk data format

The front bit is a static tag, version and format: mcap0r means a Magic Cap version 0 “Read Cap. Following that is Web-safe Base64-encoded data. The amount and interpretation of the data is specific to the Magic Cap kind and version.

For version 0, we have:

Verify Cap: Contains a 32-byte hash, wrapping up the metadata which includes the Merkle root of all the ciphertext blocks
Read Cap: Contains the same 32-byte hash as the Verify Cap, plus a 16-byte symmetric key used to decrypt the ciphertext.

The Read Cap is a secret: it contains a secret key. Anyone holding it may decrypt the corresponding Data, assuming they can locate it.

To aid in location, a Verify Cap or Read Cap may be converted to an “Identifier” (which is a tagged hash of the 32-byte hash containted in the caps). Thus, this Identifier does not reveal which Read or Verify Cap it “comes from” and serves as a unique id for storage servers (see also the Network API and Catalog).

On Disc Format

A serialization of Magic Cap Data to disk looks like this.

diagram of on-disk data format

Highlighted in brown is the static “mcap” tag bytes, followed by 4 bytes of unsigned int (u32) represending the version (which is “1” the only existing version).

Next, we look at the blue highlighted data and the end of the file – the last 8 bytes. This is a u64 telling us the offset where the metadata begins. The metadata is all the bytes from this offset to the begining of the offset.

In this diagram, the metadata is highlighted in purple. The remainder of the file is ciphertext blocks.

In the network API, the metdata is delivered by one endpoint, and the ciphertext by another. Regardless, the metadata is encoded the same way on-disk an in the network API.

Metadata Encoding

TODO

Command Line Interface

To allow rapid experimentation with this library, many aspects of it are exported via the mcap CLI tool.

This is a sub-command based tool. There are a few top-level options and several sub commands (even some “sub-sub commands”).

Common Top-Level Options

These options apply to every command.

--loglevel, -l: Specify the logging level, one of ERROR, WARN, INFO, DEBUG, TRACE.

Core Functionality

The main thing this library does is encrypt and later decrypt data.

mcap encrypt: Turn plaintext into the ciphertext Data and a Read Cap. We need somewhere to write the Data; the Read Cap is printed to stderr (in case you chose to write the Data to stdout instead of a file or Catalog).
mcap decrypt: Turn a ciphertext Data and associated Read Cap back into the plaintext.

These two commands know how to use data inside a Catalog, which can live on disk or behind an HTTP interface. These are controlled with the --catalog and --catalog-url options. Currently, we can only write to an on-disk catalog.

mcap verify: Confirm that a given Data exists and correctly corresponds to the Read Cap that the specified Verify Cap is derived from. Basically this computes the merkle root of the ciphertext blocks and confirms it matches the root in the Cap. This needs access to the ciphertext, so uses options for at Catalog similar to the prior two commands.
mcap reduce: Turn a Read Cap into its associated Verify Cap.

mcap publish: Turn a local on-disk Catalog into a format suitable for static-hosting. This “static hosted” data corresponds to the read-only ReST API. An example workflow could be to mcap publish your Catalog periodically, and rsync it to a static Web host.

Anthologies

Highly related collections of data are called an “Anthology” in Magic Cap. This is somewhat like a Zip archive.

The result is a single Read Cap that itself points to other Read Caps. Local directories are used by these commands: you can turn (recursively!) a directory into an Anthology. Usually you’d use this to encode a bunch of things into a particular Catalog.

When decrypting, you can decrypt everything in the Anthology to a local directory, or use mcap anthology list to choose particular items of interest.

mcap anthology create: Turn a local directory (and all its descendants) into a collection of Read Caps, culminating in a top-level Anthology Read Cap that is printed out. Anyone with access to the appropriate Catalog and the Anthology Read Cap can access all the data inside it. You may also choose to share just the Read Cap for an item in an Anthology (e.g. a sub-directory or single file) – the receiver of such a Read Cap cannot determine if this is part of any Anthologies (unless they also have the Anthology Read Cap).
mcap anthology list: Itemize everything in a given Anthology. Does not access or decrypt any of these items.

Hands-On Examples

Once you have the software installed and working, it can be useful to explore the CLI directly.

All the examples below assume you’ve:

used git clone (or Jujutsu as we do) to obtain the source;
used cargo build to compile it;
and thus cargo run to operate the “mcap” command-line tool.

Run all examples from the root of the above directory, using example Data and other things from the repository.

If all the above works, you should be able to run the help:

$ cargo run -- --help
        ┏┳┓┏━┓┏━╸╻┏━╸        ┏━╸┏━┓┏━┓
O------ ┃┃┃┣━┫┃╺┓┃┃    ---   ┃  ┣━┫┣━┛ ------O
        ╹ ╹╹ ╹┗━┛╹┗━╸        ┗━╸╹ ╹╹

Create, read and verify Magic Cap strings and encrypted data.

Data is a file containing encrypted data and associated metadata.

A Read Cap is a string containing secret information to decrypt a corresponding Data.

A Verify Cap has only the power to confirm that the data is correct.
Any Read Cap may be turned into a Verify Cap offline.

Anyone with both the Data and corresponding Read Cap may re-create the plaintext.


Usage: mcap [OPTIONS] [COMMAND]

Commands:
  encrypt    turn plaintext into a Read Cap + ciphertext
  decrypt    turn a Read Cap + ciphertext into plaintext
  verify     Confirm a ciphertext is valid
  reduce     Turn a Read Cap into a less-powerful Verify Cap
  publish    Turn a disc Catalog into one suitable for static hosting (as the REST API).
  debug      Debugging tools. Be careful copy-pasting any of these from untrusted sources
  anthology  Tools to create and manipulate Anthologies
  help       Print this message or the help of the given subcommand(s)

Options:
  -l, --loglevel <LOGLEVEL>  ERROR, WARN, INFO, DEBUG, TRACE in that order [default: INFO]
  -h, --help                 Print help
  -V, --version              Print version

Basic Decryption

Inside the repository is kitten.mcap which is a Data that can be decrypted using the Magic Cap in the README. By default, mcap decrypt writes to stdout, and we know this is an image so:

$ cargo run -- decrypt --local-file kitten.mcap mcap0r3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8EAnU9NkykdwXfOg6VdQ7v > brunnhilde.jpeg
$ display brunnhilde.jpeg

What is happening here is that the software is fetching the Data from a “local file” source (kitten.mcap) and then using the Magic Cap to (attempt) decryption. If this Data doesn’t correspond to the Magic Cap, no decryption takes place.

Verification

We can turn the above Magic Cap into a less-powerful Verify Cap:

$ cargo run -- reduce mcap0r3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8EAnU9NkykdwXfOg6VdQ7v
mcap0v3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8

This Verify Cap (mcap0v3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8) cannot decrypt kitten.mcap but can verify that all the ciphertext is there, and valid. We can ask the software to do this:

$ cargo run -- verify --ciphertext kitten.mcap mcap0v3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8
$ echo $?
0

Notice that there’s no output by default; the exit-code indicates success or failure.

Exercise: try changing some of the file and see if it still verifies correctly.

Catalogs

It quickly becomes annoying to have to create .mcap files, and then remember which Read Cap is for which file. We have the concept of a Catalog (think “card-catalog from a library”) to help with this.

The remaining examples will all use the existing, local Catalog located in ./data/root.

This is referenced by most commands with --local-catalog ./data/root

When encoding Data, it will be written into the Catalog according to its “Identifier”. Similarly, Data is found by searching for the correct “Identifier”. The Identifier is deterministically derived from a Read Cap or a Verify Cap.

As per the README, we also have an example Catalog containing several kittens at --local-catalog ./kitten-catalog.

Let’s decrypt one of those:

$ cargo run -- decrypt --local-catalog ./kitten-catalog mcap0rPa8OKdDYv9ts53IGzK1aBFIpMXOaJ-_xxtC3Hcw9aaxxdsyw1qQTBQ6L57da22z_ | display

Note that we’re piping the output from stdout (which is JPEG data) into the “display” program from the Image Magick packages. If you see a “broken pipe” error it’s likely you don’t have this program, or it can’t run for some reason. Replace | display with > kitten.jpeg to output to a file instead.

We can also use these on the encoding side, so for example we can encode the README.org file into a local Catalog at ./data/root like this:

$ mkdir data/root
$ cargo run -- encrypt --local-catalog ./data/root README.org
mcap0rtKJjQeHHgPPzj5vFAX0-BqaZYAdMuvZtTHgLUyi6W2Ud8hzN7utr_1-8uVVt6M1d

Note that the Read Cap printed when you run this will be different because a random key is used. Regardless, you should then be able to immediately decrypt this like this:

$ cargo run -- decrypt --local-catalog ./data/root mcap0rtKJjQeHHgPPzj5vFAX0-BqaZYAdMuvZtTHgLUyi6W2Ud8hzN7utr_1-8uVVt6M1d
...

The original README.org contents should be printed. If instead you see an error, check that you copy-pasted your Magic Cap (which will be different from the example above).

Exercise: try encoding and decrypting other files. Try using mcap reduce to create Verify Caps from your Read Caps (and use mcap verify to confirm the Data can be found).

Remote / Hosted Catalogs

To facilitate network interactions, we’ve made some rudimentrary tools so you can export a local Catalog and host it (on any static Web host).

This is the mcap publish tool, which transforms a local Catalog into a directory suitable for publishing.

$ cargo run -- publish ./data/root data/published
publish "./data/root" to "data/published"
  ./data/root/fc/fc8319d66125b6eaffbacb4ddcfe7d491844cb89e6ea38e0d8f8f6b3625801db: 12740 bytes, 00004 blocks.

Your output may differ; this shows that we had just a single Magic Cap in the Catalog when the above was run. Looking at ./data/published:

$ find data/published
data/published/
data/published/fc8319d66125b6eaffbacb4ddcfe7d491844cb89e6ea38e0d8f8f6b3625801db
data/published/fc8319d66125b6eaffbacb4ddcfe7d491844cb89e6ea38e0d8f8f6b3625801db/metadata
data/published/fc8319d66125b6eaffbacb4ddcfe7d491844cb89e6ea38e0d8f8f6b3625801db/ciphertext
data/published/README
data/published/magic-cap-catalog

This intends to simulate the ReST-Style Network API we’ve mocked up. If you put the above Catalog at a static Web host, you can then pass that URL to --url-catalog.

The fc9319...801db is an Identifier; we can’t tell which Read Cap that derived from unless we also have that Read Cap (or its corresponding Verify Cap). You may fetch the .../metadata or ciphertext blocks (.../ciphertext) for an Identifier.

The root-level README and magic-cap-catalog JSON explain what this directory contains.

Tip

You should turn off directory-listing when hosting this. Failing to do so allows someone without and Magic Caps to determine how many (and what size) Data you are hosting in the Catalog

Glossary

Read Cap

A short base64-encoded string which can be used to decrypt Data. For example, mcap0r3LsgJf1LYZtRc_BGOzhx8j_FVDmFROmoBhDHGNTfXq8EAnU9NkykdwXfOg6VdQ7v

In Rust, this is represented by the ImmutableReadCap type.

Data

The metadata & ciphertext that can unlocked with a “Read Cap”. Note that some basic metadata (block-size, total size, ciphertext merkle root and the merkle leaves) is unencrypted; all other metadata, including all application-provided keys and values is encrypted.

Verify Cap

A short base64-encoded string that can verify the validity of Data (but not decrypt it). This is related to the Read Cap and can be derived (offline) from it in a deterministic way.

A “Read Cap” (or “Verify Cap”) “corresponds to” a given Data if it may decrypt (or only verify) it. In Rust, this is represented by the ImmutableVerifyCap type.

Identifier or Locator

A 32-byte string (usually base32 encoded) derived from a “Read Cap” (or “Verify Cap”) that uniquely identifies it (e.g. within a Catalog). This is used by parts of the system to associate a Data to a Read Cap (without revealing the Read Cap or Verify Cap to the storage system).

Catalog

A collection of Data organized by “Identifier” (at some root directory or URL, for example).

This allows a client holding a Read Cap to derive the Identifier and then ask the storage system if that Data is available. For example, an on-disk storage system may simply ask if a particular file exists.

Anthology

A way to group multiple different Read Caps together. For example, all the assets for a blog post. This allows you to have a single “top-level” Read Cap that references other Read Caps – sort of like a Zip file.

Keyboard shortcuts

Magic Cap