Plaid Manual

In Plaid a project is a collection of documents which all have the same data model. Plaid’s data model is configurable on a per-project basis, allowing you to have as many or as few annotation types as you would like. For example, perhaps in one project you might only annotate a single part-of-speech tag per word, while in another you might want a POS tag, a lemma, and a gloss for each word. These projects would have different configurations in order to appropriately accommodate the annotation needs of each.

The means by which this configuration is performed is the layer. (If you have used ELAN before, note that this is similar to ELAN’s tiers.) Loosely speaking, a layer corresponds to a single "type" of annotation: for instance, in the example above where we want to collect a POS tag, lemma, and gloss for each word, we would have three span layers associated with the project. We will discuss layers more shortly.

Plaid offers a built-in user system which allows individuals to log in with a password. By default, each project is private, and one of three permissions levels is needed in order to interact with one:

Additionally, a global admin role exists, which allows the user to see and edit all data in the Plaid instance.

Plaid’s functionality is exposed for development as a REST API. This allows you to use Plaid from any programming language with an HTTP client library.

Additionally, we provide official clients for two programming languages, in plaid-client-py/ (Python) and plaid-client-js/ (JavaScript). These clients provide an API for interacting with Plaid that is idiomatic for each programming language and frees you from concern about low-level details of HTTP requests. Here is an example of how to use the Python client:

The same, in JavaScript:

Every write to the database is recorded in an append-only audit log which provides an account of who performed every change. Audit history is retained indefinitely — it is both the forensic record and the source for document time travel (see below), so pruning it trades away history. Operators monitor disk via the /health endpoint’s audit block.

The log can be queried like so:

Plaid offers a simple system for real-time communication on a per-project basis. This is intended to support two purposes:

This functionality is exposed in two simple functions in the client. The send_message/sendMessage function allows a client to broadcast a message to all clients in the project:

Note that the second positional argument, the body, can be any JSON value.

On the other end, a client may listen like so. Note that there are two arguments for the message. event_type is "message" for data sent via send_message/sendMessage by another client, and "audit-log" for audit log notifications. Consider an example of listener setup:

After the send_message invocation we just saw, this on_event function would produce the following output:

And suppose that another client executed the following code:

The listener’s code above would print this:

A service is an external program that registers itself on a project and performs work on request. The motivating case is NLP: you might have a Python program that runs a parser, a tokenizer, or a large language model, and you want annotators to be able to invoke it from the editor with the click of a button. Because a service is just another client speaking to Plaid’s REST API, it can be written in any language and run anywhere that can reach the server—it authenticates with an ordinary token (typically a named API token; see Clients) and needs writer access to the project.

There are two sides to the system: a program that serves (offers to do work) and a client that requests work. The server sits in the middle: it tracks which services are currently connected, routes each request to the one service it names, and relays that service’s replies back to the requester. Nothing is broadcast—a request reaches only its target service, and a reply reaches only the client that made the request.

Suppose we’re working on a project where all we are doing is POS-tagging. The configuration of the project’s layers (in a simplified JSON representation) would look something like this:

This layer structure prescribes the structure of individual documents. Consider a document where we have POS tagged the sentence "Fido barks":

Notice the following:

We will continue discussing this example in more detail below.

A project is the root of a layer configuration and has a name.

A project has many documents, and each has a name and a unique ID:

For each text layer, each document may have at most one text, which consists of a single string. This string holds all the text which is to be analyzed in dependent layers. A text object looks something like this:

For each token layer, each document may have many tokens, which are defined as substrings of a text:

Note the following:

Token layers are unique in that their invariants are configurable. In addition to the default configuration, it is possible to constrain them further: for example, a token layer may require that no tokens be overlapping in extent of begin and end. For full details, see Token Layer Constraints below.

Tokens are intended to serve as the basic units for further linguistic analysis using spans and relations.

For each span layer, each document may have many spans, which are groupings of one or more tokens which have a single value:

There are no restrictions on spans, other than that they must hold at least one token, and that they all must belong to the span layer’s parent token layer.

For each relation layer, each document may have many relations, which are directed edges between two spans with a label. Both spans must belong to the relation layer’s parent span layer. For example, if we wanted to extend the example above with a syntactic dependency relation between "Fido" and "barks" expressing that "Fido" is the subject, we could have a relation like this:

The four basic layer types (text, token, span, and relation) are all project-specific and cannot be used in more than one project. The fifth layer type, the vocab layer, can be used across projects. As its name suggests, this layer is intended for recording occurrences of lexical entries.

The vocab layer itself has a name:

The vocab layer has vocab items, which represent lexical entries, each with a canonical form:

Finally, vocab links are used to indicate occurrences of lexical entries. Recall the tokens from before:

We can create links between them and the above vocab items with vocab links like so:

Notice that multiple tokens may be specified, allowing for multi-word and non-contiguous lexical items.

It is often desirable to enrich an entity with additional information—for instance, you might want to record some information about the annotator’s confidence in whether a certain span value is correct. Additionally, you might want to do the same with a layer in order to e.g. specify what values are acceptable for spans in a given layer. To accommodate this, Plaid allows arbitrary data to be stored in the config attribute for layer types (project, text layer, token layer, span layer, relation layer, vocab layer) and in the metadata attribute for data types (document, text, token, span, relation, vocab item, vocab link).

You might use a config to store legal tags for a given layer:

As for metadata, the canonical use is recording whether a human or a machine produced an annotation — Plaid standardizes this as the Provenance convention, including where a model’s confidence and prediction details go:

In the previous section, we noted the constraints which Plaid enforces on each data type. Plaid guarantees that the database will never violate these, no matter what, by ensuring that invalid entities are never created, and often by deleting structures which are indirectly rendered invalid by another change. Consider these examples:

These invariants have been incorporated into Plaid because of their broad desirability in linguistic annotation. However, some invariants will vary by annotation framework. For example, it is quite common to want a span layer’s spans to be in one-to-one correspondence with tokens in the parent token layer.

For token layers, you can ask Plaid to enforce some commonly needed additional invariants for you, server-side: these are described in Token Layer Constraints below. For everything else, Plaid provides three further mechanisms—strict mode, locking, and atomic batches—which help you coordinate writes from the client so that you can enforce your own data integrity constraints without needing to write any code outside of your client.

The constraints in the previous section are always on. Token layers additionally support a set of opt-in, server-side constraints which capture some of the most common token-level invariants in linguistic annotation. Because they are enforced by the server on every write, there is no need for extra measures on the client side in order to maintain them. They are configured per token layer at creation time, and are immutable thereafter.

Multiple users may edit the same document simultaneously, and in some cases, undesirable conflicts may occur as users fail to take into account each other’s work. Suppose, for example, that one user is editing a sentence’s lemmas, and the other is editing a sentence’s POS tags. If the lemma editor doesn’t know that a certain POS tag has changed, they might make the wrong decision about which lemma to assign. Plaid clients' optional strict mode causes edits to fail when someone other than the current user has made an edit. Consider this exact scenario in code:

When client 1 executes the second lemma span’s value, unless they happened to have loaded the document anew after client 2’s change, they will not be aware of the new POS tag for the second word.

Strict mode causes requests to fail when someone other than the user in strict mode has edited a document since strict mode began. If client 1 had initiated strict mode at the beginning, then the second request would have failed:

This failure gives client 1 the opportunity to reload the document only when it is necessary, allowing them to reconsider the current state of the document before making changes.

Sometimes a more heavyweight solution is needed. A lock gives a user exclusive permission to write to a document, preventing all other users from changing its contents. Locks have a 60 second expiration timer by default, and they may be released early or renewed by either explicit renewal or any write to the locked document. Consider:

Locks are useful for situations where a concurrent edit by another user could yield an invalid state with respect to data integrity constraints beyond what is enforced in Plaid’s core. They should be used only where necessary in order to minimize the risk of invariant violations stemming from concurrent modifications.

Finally, you may also submit multiple requests in batches. Batches are atomic meaning that we guarantee that either they will all succeed or all fail. This is a very useful guarantee whenever you have sophisticated data integrity requirements that must be orchestrated using more than one request.

Here is an example of how to submit a batch using the JavaScript client:

Or in Python:

Owing to some implementation details, no other writes may be executed while a batch is being processed, so only use them where necessary.

For all constraints which you might attempt to enforce using the three mechanisms described above, there is, of course, nothing stopping a malicious user from circumventing them and submitting changes which invalidate your formalism-specific data integrity constraints. For example, if you have a document where you want all spans to be in one-to-one correspondence with tokens, a malicious user could simply circumvent your UI code entirely and craft a malicious request to e.g. create multiple spans associated with a single token. Since there is no server-side validation for this constraint, the server will happily execute it and advance into a database state that does not violate any core data integrity constraints but does violate your formalism-specific constraints.

This is a fundamental limitation of Plaid which was deliberately adopted in order to support frontend-only development. You therefore must only grant write privileges only to users who you trust not to circumvent client-side validation guardrails. Fortunately, we think this is not an onerous imposition in most real-world circumstances.

For most linguistic annotation applications built on top of Plaid, the layers can be divided into two categories. First are substrate layers: these are the primary text layer and some token layers, including the token layer for sentences, words, and maybe morphemes. There are many different apps that would be interested in working with these.

Then there are annotation layers: these hold information that probably only exactly one app cares about, such as a Universal Dependencies part-of-speech tag. These annotations typically do not need to be shared between apps.

Now, recall from Metadata and Config that each app writes its configuration under its own namespace. This configuration can help two different apps, such as one used for UD and another used for IGT, to track only the annotations each cares about: the UD app would use configuration cues to know where to look for its tag and dependency layers, and similarly, the IGT app can use configuration to find the gloss layers it uses, which can happily sit side by side on the same tokens. For annotations, each app is oblivious to the other, because each app only ever looks at the layers it created.

The substrate is the part that the apps do share, but even here they will only agree up to a point. A sentence is a sentence in both apps, and the same is usually true of a word. Below the word, though, the apps may have different needs, and one and the same word may be divided in two different ways. In the UD app, the layer beneath the word holds CoNLL-U syntactic words, which sometimes disagree with the CoNLL-U tokens above them: the English token teacher’s, for example, is a multiword token holding the two syntactic words teacher and 's. In the IGT app, the layer that is closest in purpose to UD’s syntactic word layer holds morphemes, which split on all morphology, so that this same word, teacher’s, is instead divided into the three morphemes teach, -er, and 's. It is therefore not always possible to share all token layers across apps.

A reasonable rule of thumb, then, is to share the substrate as far down as the apps agree, and to let them branch below that point. In our example, the two apps would share the text, the sentence layer, and the word layer, and then, beneath the word layer, the UD app would add its syntactic-word layer while the IGT app adds its morpheme layer. (Recall from Token Layer Hierarchy that a single token layer may have more than one child layer, so both of these can sit side by side under the shared word layer.) The apps agree all the way down to the word, and only then go their separate ways. If two apps genuinely do mean the same thing by a layer, so that their morpheme segmentation really is identical, then they are free to share it as well; but this should always be a deliberate decision rather than a silent assumption.

Suppose that a third app comes along that has no knowledge of the UD or IGT apps, and that it wants to find out what the extant layers mean. While there is no perfect solution to this, Plaid provides a small shared vocabulary of roles recorded in each layer’s configuration under the plaid namespace. A layer carries at most one role, and the inventory is deliberately small:

Plaid itself attaches no meaning to these values. These names are rather blessed conventional names for communicating the meaning of a layer that apps are expected to honor when present. The app that first sets the project up might record the roles of its sentence and word layers like this:

The plaid namespace is reserved for conventions that every app agrees on, which makes it the right home for these shared role tags. An app’s own private configuration, such as its vocabularies, its colors, and the meanings of its fields, belongs instead under that app’s own namespace, where no other app will mistake it for shared information. Keeping these two kinds of configuration apart is what allows several apps to coexist on one project.

We also advise app developers to consider the absence of a role tag to be a meaningful signal: if a layer is not tagged with a role, then as a matter of convention, no other app should read or modify it.

Annotation projects routinely mix human labor with machine output—a parser fills in a first pass, an auto-linker proposes vocabulary links, an importer carries another tool’s analysis—and it matters enormously which is which: machine output is cheap and replaceable, while human judgment is expensive and must never be silently destroyed. Plaid standardizes this distinction as the provenance convention: flat metadata keys on annotation entities (spans, relations, vocab links, and optionally tokens) that put every entity in one of three states.

provSource names the producer: service:<serviceId> for services, rule:<name> for built-in rule algorithms, or an app-specific id such as gloss:doc-frequency or flex-import. The presence of the prov key—not its particular value—is what marks an entity as machine-made; inferred is simply the only value defined today. The keys are deliberately flat scalars, which the query system’s metadata filters match well, so questions like "show me every unverified machine annotation in this document" are ordinary queries.

A producer may also record prediction extras—how confident it was, and what else it considered—in two further reserved slots, split along the queryability line:

Prediction extras describe the machine’s original prediction. They are kept even after a human edits the entity—the history is valuable for audit and retraining—which means a consumer must not present provProb as confidence in the current value once the entity is verified: any human edit verifies (rule 3 below), so provConfirmed is exactly the flag to check before displaying it.

Of the bundled services, the Whisper transcriber records its per-segment scores this way (provDetail.avgLogprob / provDetail.noSpeechProb / provDetail.model—and no provProb, since a log-probability is not a calibrated probability), and the Stanza parser records provDetail.model and provDetail.language (its pipeline exposes no per-prediction probabilities; a parser that has them would add provProb plus a top-k distribution in the detail map). On the read side, the UD editor consumes distributions a parser records under provDetail.uposProbs / provDetail.xposProbs / provDetail.deprelProbs (each a flat {label: probability} map): the top-k floats above the rest of the vocabulary in the corresponding cell’s dropdown as a "Parser suggestions" group, and machine-made cells describe their origin (producer, model, provProb) in their hover tooltip.

The convention is made operational by a write contract that every machine writer (service, built-in algorithm, importer) is expected to follow:

Rule 3 is what lets editors render unverified machine output distinctly (Plaid’s apps use violet italics) and have the marking dissolve naturally as a human works through the material—each touched annotation graduates to verified, while everything still violet remains fair game for the next machine pass.

For service authors, both clients ship helpers: stampInferred(source) / stamp_inferred(source) builds the fragment to merge into everything you create (every bulk create accepts per-item metadata) and takes the prediction extras as options (stampInferred(source, {prob, detail}) / stamp_inferred(source, prob=, detail=)), serviceSource(serviceId) / service_source(service_id) builds the canonical producer id, and isProtected(metadata) / is_protected(metadata) implements the rule-2 check against material you are about to destroy. Born-verified material—an import carrying upstream human approval, or a guess written only upon explicit user confirmation—uses confirmedInferred / confirmed_inferred instead.

In Data Integrity, we discussed various ways of maintaining app-specific data model invariants within a certain app. Here, we consider the more difficult matter of one app updating a shared substrate layer in a way that invalidates a different app’s data model invariants.

Suppose that a sentence has been annotated for UD, and that later on, someone opens the same sentence in the IGT app and splits the sentence. From the IGT app’s perspective, this is a totally unproblematic change to make. However, on the UD side, it is a strict requirement that all dependency relations must never cross sentence boundaries, and this change in the IGT app at the shared sentence tokenization layer has caused an invariant violation in the UD app.

There is no way for Plaid itself to know about this invariant, because it is application-specific. Our only option, then, as long as this layer is shared, is to somehow have the UD app deal with this violation the next time that document is opened.

The discipline that resolves this is easy to state, and in the interest of smooth interoperability, we encourage all app developers to adhere to it:

Every time an app opens a document, it should validate all its invariants and automatically repair any violations that may have been introduced by other apps.

In our running example, the next time the document is opened in the UD app, the UD app would validate the data and realize that a relation is straddling a sentence boundary. The simplest thing to do at this point would be to delete the offending relation. Although information is lost this way, we are able to restore data integrity without requiring any human intervention. Ideally, the app would also loudly tell the user what happened, and perhaps even encourage or require review. And indeed, in general, some invariant violations in other situations might require human intervention in order to avoid worse consequences.

A simple query specifies what to find, and lists the conditions a match must satisfy in where:

Read that where line as a sentence: "a span, call it ?s, on the pos layer, whose value is NOUN." The ?s is a variable — any name you invent, written with a ? — and every match comes back as a row holding one id per name in find.

Here is a more interesting query, where we search for a noun immediately followed by a verb:

When multiple clauses are provided to where, all must be satisfied in order to yield a match. In the latter three clauses, we use relationship clauses, which are used to enforce structural relations between entities. covers requires that a span have a token as one of its constituent tokens, and precedes requires that one token come immediately before another in linear order. These are two of several relationship clauses; the reference catalogs the rest.

By default a match is a tuple of ids. Different return types are available, identified by return:

Normal permissions restrictions apply to queries: a query submitted by a user is only executed over data that user has permission to read. In order to further restrict visibility to e.g. a single project, the scope parameter may be provided in a query:

Queries always read the current state of your data: there is no time-travel parameter (one passed as asOf is refused). For a document’s history, see the Audit Log.

See the query language reference for comprehensive information about how queries work.