1. Introduction

Plaid is a platform, which means that it is not a complete app on its own. Rather, it provides an implementation of the "boring" parts of a linguistic annotation app, including a user system, a database layer, and more. Building an app with Plaid allows you to focus your attention on just the interesting parts of your app including the UI and logic which is specific to your annotation framework.

The purpose of this document is to teach you how to think about and use Plaid. If you’d rather just dive into a real app programmed with Plaid, check out the UD editor example app.

2. Key Concepts

2.1. Projects and Layers

In Plaid a project is a collection of documents which all have the same data model. Plaid’s data model is configurable on a per-project basis, allowing you to have as many or as few annotation types as you would like. For example, perhaps in one project you might only annotate a single part-of-speech tag per word, while in another you might want a POS tag, a lemma, and a gloss for each word. These projects would have different configurations in order to appropriately accommodate the annotation needs of each.

The means by which this configuration is performed is the layer. (If you have used ELAN before, note that this is similar to ELAN’s tiers.) Loosely speaking, a layer corresponds to a single "type" of annotation: for instance, in the example above where we want to collect a POS tag, lemma, and gloss for each word, we would have three span layers associated with the project. We will discuss layers more shortly.

2.2. Users and Permissions

Plaid offers a built-in user system which allows individuals to log in with a password. A user’s id is their email address, compared without regard to case, Unicode composition, or invisible characters at either end. So Ana@Example.org and ana@example.org name one account. By default, each project is private, and one of three permissions levels is needed in order to interact with one:

  • Maintainers have full privileges for working with projects: they may edit documents and also modify the project’s configuration.

  • Writers may edit and delete documents belonging to a project, but may not modify the project’s configuration.

  • Readers may only read documents belonging to a project—they may not make any edits to the project’s documents or configuration.

Additionally, a global admin role exists, which allows the user to see and edit all data in the Plaid instance.

A project’s roles also say what each person may do with the project’s vocabularies (see Vocabs).

Admins and the maintainers of any project may list and search every account on the server (GET /api/v1/users), so that they can add members. Anyone may create a project and so become a maintainer, which means every user can search the accounts.

A request for a project, document or vocabulary that does not exist answers 404 to an admin. Anyone else gets the same 403 they would get for one that exists but is not theirs. When the 403 is for an unknown id of something inside a project or vocabulary, such as a document, a token or a vocabulary entry, its body also carries "unresolved": true, which a client can read as "changed or removed by someone else". An unknown project or vocabulary id gets the 403 without it.

Permissions say what a person may do, not how far their work is trusted. A project may also mark members whose work needs review, whatever their role, as described under Provenance.

Each user may upload a profile picture, which appears wherever they are named in the apps. The server crops it to a square, scales it to [media] avatar_size_px, and strips its camera metadata. You may set or remove your own picture, and an admin may set or remove anyone’s.

2.2.1. Tokens and sign-out

A sign-in or API token is read from the Authorization header only, never from a URL. An image or audio element cannot send that header, so the URL of a picture or a recording carries a short-lived link token instead. The clients build these URLs for you:

  • users.avatarUrl(id, avatarHash) (users.avatar_url(id, avatar_hash) in Python) gives a URL for any user’s picture, using a token from POST /api/v1/avatar-link.

  • POST /api/v1/documents/:id/media/link answers {url, expires-at}, a URL that opens one document’s recording.

A link token opens nothing else, lasts at most [media] avatar_link_ttl_seconds or [media] link_ttl_seconds, and stops working when the credential that made it does.

Signing out of an app forgets the sign-in in that browser only. To end every sign-in of your account on every device, call client.auth.logout_everywhere() (client.auth.logoutEverywhere() in JavaScript). Named API tokens are not affected, and are revoked one at a time with api_tokens.revoke (apiTokens.revoke). Each user mints their own named API tokens on the profile page of any app, which shows a new token once, to be copied.

A named API token acts as its owner, but since it never expires, it cannot do the few things that would outlive revoking it. It cannot change a password, mint or revoke API tokens, or mint an invite. An admin’s named token also cannot create, promote, deactivate or reactivate accounts, clear login rate limits, or write another user’s private data. These answer 403 "This needs a signed-in session, not an API token."

2.2.2. Private user data

Each user also has a small private key/value store (/users/{id}/data/{key}) for application state that should follow a person across devices, such as assistant conversations, drafts and interface preferences. Only the owner and admins can read or write it, and nothing in it is audited. It is not a place for annotation data, which belongs in layers where it is shared and queryable.

A key is any string, and apps namespace theirs (igt:assistant:<project>:…​). A value is any JSON document up to [user_data] max_value_mb (5 MB by default). The store can be listed by key prefix or by a glob over the whole key (igt:assistant:*:meta:*), with or without values.

Every entry has a version that each write raises by one. A write that passes the version it was made from (?version=, or the version option of user_data.put / userData.put) is refused with 409 when another write landed in between. version=0 writes only an entry that does not exist yet.

2.2.3. Guidelines

A project may carry a list of guidelines (/projects/{id}/guidelines): short Markdown documents recording the conventions its annotators have agreed on. Each has a title, a body, and a pinned flag, which tells an assistant to read that guideline on every turn. Readers may read them and writers may edit them.

A PATCH that passes updated-at, the value the caller last read, is refused with 409 if someone else saved in between. Writes appear in the project’s audit log, but guidelines have no time travel.

2.3. Clients

Plaid’s functionality is exposed for development as a REST API. This allows you to use Plaid from any programming language with an HTTP client library.

Additionally, we provide official clients for two programming languages, in plaid-client-py/ (Python) and plaid-client-js/ (JavaScript). These clients provide an API for interacting with Plaid that is idiomatic for each programming language and frees you from concern about low-level details of HTTP requests. Here is an example of how to use the Python client:

from plaid_client import PlaidClient

client = PlaidClient("http://localhost:8080", "<SECRET_TOKEN>")

projects = client.projects.list()
print("Available projects:", projects)

first_project_id = projects[0]["id"]
first_project = client.projects.get(first_project_id)
print("First project:", first_project)

documents = client.projects.list_documents(first_project_id)
first_document_id = documents[0]["id"]
first_document = client.documents.get(first_document_id, include_body=True)
print("First document:", first_document)

The same, in JavaScript:

import { PlaidClient } from "@larc-iu/plaid-client"

const client = new PlaidClient("http://localhost:8080", "<SECRET_TOKEN>")

const projects = await client.projects.list()
console.log("Available projects:", projects)

const firstProjectId = projects[0].id
const firstProject = await client.projects.get(firstProjectId)
console.log("First project:", firstProject)

const documents = await client.projects.listDocuments(firstProjectId)
const firstDocumentId = documents[0].id
const firstDocument = await client.documents.get(firstDocumentId, true)
console.log("First document:", firstDocument)

2.3.1. Static pages

Plaid can serve a folder of your own web pages, for example a small frontend for one project. Set static_resources_path under [server] in data/config.toml to the folder and restart. Every file in it is served at its path from the root, without authentication, so put nothing private there.

Plaid also serves the JavaScript client at /client/plaid-client.js, the same version as the server, whether or not this folder is set. A page imports it as a module, so it needs no copy of its own:

<script type="module">
  import { PlaidClient } from "/client/plaid-client.js";
  const client = await PlaidClient.login(location.origin, "user@example.com", "password");
  console.log(await client.projects.list());
</script>

Its type declarations are at /client/plaid-client.d.ts.

2.3.2. Pagination

Collection endpoints (documents, users, API tokens, audit, and similar lists) are paginated. Over raw HTTP they return an envelope of the form {"entries": […​], "next-cursor": "…​"}, where each page holds at most 100 entries by default (1000 maximum, set via ?limit=). To fetch the next page, pass the opaque next-cursor value back verbatim as ?cursor= (it is null on the final page).

The official Python and JavaScript clients handle this for you: their .list()-style methods (e.g. projects.list(), projects.list_documents(…​) / listDocuments(…​)) transparently follow the cursors and return the full flat list. Per-page variants (e.g. list_documents_page / listDocumentsPage) are available when you want to drive pagination yourself.

2.4. Audit Log

Every write to the database, apart from comments, each user’s private data, research telemetry and the record of which services have connected, is recorded in an append-only audit log which provides an account of who performed every change. The log is kept indefinitely, and it is what document time travel reads from (see Document Time Travel). It can be queried like so:

for entry in c.documents.audit("35424d64-a077-4a29-8006-5a0c3b76aedb"):
    time = entry["time"]
    who = entry["user"]["display_name"]
    label = entry.get("message") or "; ".join([o["description"] for o in entry["ops"]])
    print(f"{who}, {time}: {label} ({len(entry['ops'])} writes)")

# Output:
# Luke G, 2025-07-05T09:13:39.614Z: Create document "Document 1" in project 0f0f0574-ae5a-4060-814c-c5bbdce14d67 (1 writes)
# Luke G, 2025-07-09T20:27:59.611Z: Merge morphemes (4 writes)

Each entry names the user who made it, and the named API token if one signed the write. It also records what kind of credential signed it, as credential: login for a password sign-in, named-token for a named API token, and delegated for a token a service was handed to act for someone (see Acting on the requester’s behalf). This tells a script run on a person’s token apart from that person’s own edits.

2.4.1. Logical operations

Many actions are made of several writes: "merge two morphemes" deletes tokens, creates one, and moves spans. To keep the log readable, a client can group such a run into one operation with a label. The audit log then shows the run as one entry whose message is the label and whose ops are the individual writes:

with c.operation("Merge morphemes"):
    c.tokens.delete(left_id)
    c.tokens.delete(right_id)
    c.tokens.create(...)
await c.withOperation("Merge morphemes", async () => {
  await c.tokens.delete(leftId);
  await c.tokens.delete(rightId);
  await c.tokens.create(...);
});

An operation is not a transaction: if the third of five writes fails, the first two stay. Use a batch inside the operation for anything that must be all-or-nothing (see Atomic Batches). An operation opened inside another joins the outer one, so actions can call each other freely.

A request to a service made inside an operation carries the operation along, and the service’s writes are filed under your entry. Pass no_operation=True (noOperation: true in JavaScript) when the service’s work should be an entry of its own.

Each entry carries time, its first write, and end_time. To see a document as it was after the whole operation, read it as of end_time. Each op in ops has an end_time too, for viewing the state after that one write.

2.4.2. Kinds of operation

An operation may also say what kind of operation it is, so that a program can count operations without reading their labels. The kind is one of a fixed list, and any other value is refused with a 400. An optional ref (at most 1024 characters) says what the operation refers to.

with c.operation("Import ELAN corpus", kind="import", ref="format:elan"):
    ...
Kind What it is ref

assistant-plan

An assistant plan the user approved, being applied.

conv:<conversation>/plan:<plan>/service:<service>

service-run

One run of a service, with every write it made.

service:<id>, or builtin:<name> for a rule an app runs in the browser

import

A file or archive read into a project.

format:<format>, such as format:elan

bulk-edit

One change made across many places at once.

action:<name>, such as action:respell

guess-adoption

A person taking a suggested value as their own.

none

repair

Layer rules an app sets up by itself, with any repair of the stored data they need first (see Layer Constraints), and the UMR app’s repair when a document opens (see Reconciliation Across Apps). It is never counted as anyone’s edit.

none

review

A person accepting machine or contributed work as it stands.

none

job

An assistant job’s work in one document, or its removal of the unlinked entries it made (see Assistant jobs).

job:<job>/doc:<document>/pass:gloss, job:<job>/doc:<document>/pass:harmonize, or job:<job>/pass:entries

2.4.3. Custom messages for single writes

A single write can carry its own message in place of the generated description, as the audit_message / auditMessage argument:

c.spans.set_metadata(span_id, {"approved": True}, audit_message="Approve span {span_id}")
# logged as: Approve span 35424d64-a077-4a29-8006-5a0c3b76aedb

A {…​} placeholder is filled in with any parameter of the request, spelled the way your client spells it. The message is stored as written, so do not put a secret in it.

2.4.4. Filtering the log

The audit endpoints take filters, so a client can ask a narrow question without paging a project’s whole history. op_types / opTypes keeps only the operation types you name, written as in an entry’s ops[i].type (span-layer/create, document/update). kinds keeps only operations of the kinds you name.

# Has anything about the project's span layers changed since I last looked?
changes = c.projects.audit(
    project_id,
    start_time=last_checked,
    op_types=["span-layer/create", "span-layer/delete", "span-layer/update"],
)

An entry of a machine run can hold tens of thousands of writes. A page read with ops_limit / opsLimit keeps only each entry’s oldest operations, and op-count says how many there are in all. To read the rest of one entry, pass its id as entry_id / entryId with a start_time at the last operation you have.

2.4.5. Changes in one entry

audit.changes answers, for one entry, every row of a document or a vocabulary it wrote, as it was just before the entry and just after. A row the entry made has no before, and one it deleted has no after. changed_since / changedSince says the row has been changed since. The id of one operation inside an entry answers for that operation alone, and an audit read’s entry_id / entryId takes it too, as an entry of that one operation, whose part_of / partOf names the entry it is part of.

for row in c.audit.changes(entry_id, document_id=document_id):
    print(row["kind"], row.get("before"), row.get("after"))

2.4.6. Undo of one entry

audit.undo puts back every row one entry wrote as it was just before the entry, in its documents and its vocabularies together, as one new entry of its own. A row the entry made is deleted, one it deleted comes back under its old id, and one it changed is set back. A merge comes undone whole: the merged entries come back, their links point to them again, and every reference the merge rewrote names them again.

What people did since is left alone. A row is left as it is, and listed under skipped with its reason, when someone wrote it since (changed-since), or between the entry’s own writes (changed-between), when the entry made it and something that stays is built on it, such as an annotation on a token or a link to an entry (built-on), or when it needs something that will not be there (needs). A text goes back only when nothing in its document wrote a text or a token since the entry began (text-changed-since). A token goes back onto its offsets only when nothing outside the entry wrote its text since the entry began (text-changed-since), and a text goes back only into a layer that holds no other text of the document now (place-taken). A row is never set back to name, in its metadata, a row that will not be there afterwards (needs). When a document’s layers would no longer hold with the entry’s rows put back, that document’s rows are all left (does-not-fit). An undo is held to each layer’s closed list as it is now, unlike a restore, since it puts single values back into today’s document: a value the list no longer allows is left (value-not-allowed). Changes to a project, its layers, a vocabulary’s name or settings, users and memberships are not undone, and not_undone / notUndone names the operations that made them. A project’s deletion cannot be undone. After a vocabulary is deleted, only an administrator can read its entries' history, so only an administrator can undo what an entry did to them.

plan = c.audit.undo(entry_id, dry_run=True)
print(plan["total"], plan["skipped_count"])
done = c.audit.undo(entry_id)
c.audit.undo(done["operation"])  # and back again

document_id / documentId takes one document’s rows only, its links included, and vocab_layer_id / vocabLayerId takes one vocabulary’s entries and the links that name one of them. An undo needs a maintainer: of a document’s project for its rows, of a vocabulary’s home project for its entries. Rows the caller may not maintain are listed as not-permitted and the rest undone, and rows in a document or vocabulary the caller cannot read are neither counted nor listed. A delegated token undoes only the rows of documents in the projects it reaches. Undoing the same entry again finds nothing left and writes nothing. An undo may be sent with an Idempotency-Key or inside a batch: a dry run there writes nothing either. A dry run and a real undo are different requests, so one key answers only the one it was first sent with. A row the entry made is also kept when anything that stays names its id in its metadata, such as an entry that took a word of it as an example (built-on). An entry of more than 20,000 rows is refused, and the refusal says how many rows it has.

Several entries can be undone together as one, with audit.undo_entries / audit.undoEntries, and a whole job with jobs.undo, which finds the job’s entries itself: the gloss pass and the harmonize pass of every document. Each row goes back to how it was before the first of those entries wrote it, so a word one entry linked to a lexicon entry another one made comes undone with it. The answer is the same summary, with entries naming what was undone, and the limit of 20,000 rows counts them all. The entries must all be of one project the caller can read: entries of two projects are refused, and so is an entry the caller cannot read. A job that is still waiting, running or stopping is refused until it has ended. While a job stands undone, its read carries undone, the undo’s operation, time and user, and only a whole undo of that undo takes it away again: undoing one document’s part of it leaves the job undone.

plan = c.jobs.undo(job_id, dry_run=True)
c.jobs.undo(job_id)

2.4.7. Last edit per document

my_last_edits / myLastEdits answers, in one request, when the calling user last wrote to each document in a project:

mine = c.projects.my_last_edits(project_id)
# {document id: timestamp}

A document the caller never wrote to is absent. Writes of kind repair are not counted.

2.4.8. Document Time Travel

A document read accepts an as-of time and returns the document exactly as it stood at that moment:

# Read old name by providing time in the past
client.documents.get(document_id, as_of="2025-07-08T09:00:00Z")["name"]
#=> "Old Name"

A time in the middle of an atomic batch reads as just before the batch, so you never see a half-applied change. Deleted documents can still be read at times when they existed. Access always follows the user’s current membership, not their membership at that time.

A maintainer can also bring a document back to an earlier state with POST /api/v1/documents/{id}/restore?as-of=…​. Everything deleted since comes back under its original id, everything added since is removed, and the restore is itself an ordinary entry in the log. With dry-run=true nothing is written, and the answer says what would change.

2.4.9. Vocabulary Time Travel

A vocabulary may be linked by several projects, so its history is its own. vocabLayers.get(id, includeItems, asOf) reads a vocabulary as it was, vocabLayers.getItemAt(id, itemId, asOf) reads one entry, deleted or not, and vocabLayers.audit reads the vocabulary’s log. A maintainer of the vocabulary’s project can put one entry back as it was with vocabLayers.restoreItem(id, itemId, asOf), though its links come back only by restoring the documents that held them. Changes to a vocabulary also appear in its project’s log, from the version that gave every vocabulary a project on. A vocabulary read as of a time before that version has no project. A log names a project only to someone who can read it, and says "another project" to anyone else.

2.5. Real-time Messaging

Plaid offers a simple system for real-time communication on a per-project basis. This is intended to support two purposes:

  • Ad hoc client-to-client features which you will implement on top of this communication channel, such as chat between individual annotators.

  • Audit log listening, allowing clients to receive immediate notice whenever a change has been made to any document in the project. (Note that these are sent automatically by Plaid.)

Note
To plug in an external tool that does work on request, such as a parser or a tokenizer, use Services below.

This functionality is exposed in two simple functions in the client. The send_message/sendMessage function allows a client to broadcast a message to all clients in the project:

client.messages.send_message(project_id, {"purpose": "ping", "message": "ping"})

Note that the second positional argument, data, can be any JSON value. A message is not saved and does not appear in the audit log.

On the other end, a client may listen like so. Note that there are two arguments for the message. event_type is "message" for data sent via send_message/sendMessage by another client, and "audit-log" for audit log notifications. Consider an example of listener setup:

def on_event(event_type, event_data):
    if event_type == "message":
        sender = event_data["user"]
        time = event_data["time"]
        contents = event_data["data"]
        print(f"User {sender} sent data `{contents}` at {time}")
    elif event_type == "audit-log":
        user = event_data["user"]
        time = event_data["time"]
        op = event_data["ops"][0]
        op_type = op["type"]
        document_id = op["document"]
        description = op["description"]
        print(f"User {user} performed operation `{op_type}` on document {document_id} at {time}: '{description}'")

client.messages.listen(project_id, on_event)

After the send_message invocation we just saw, this on_event function would produce the following output:

User user1@example.com sent data `{'purpose': 'ping', 'message': 'ping'}` at 2025-07-09T20:14:36.168Z

And suppose that another client executed the following code:

client.documents.update(
    "35424d64-a077-4a29-8006-5a0c3b76aedb",
    name="New Document Name")

The listener’s code above would print this:

User user1@example.com performed operation `document:update` on document 35424d64-a077-4a29-8006-5a0c3b76aedb at 2025-07-09T20:27:59.616Z: 'Update document 35424d64-a077-4a29-8006-5a0c3b76aedb name to "New Document Name"'

2.5.1. Event Reference

GET /api/v1/projects/{id}/listen is an ordinary server-sent event stream, so you can consume it without the official clients. It needs reader access, and each event’s data: line holds one JSON object.

connected

Sent once on connect, with the client-id that heartbeat answers quote.

heartbeat

Sent every 30 seconds. Answer each with POST /api/v1/projects/{id}/heartbeat and the body {"client-id": "…​"}, or the server closes the stream after two missed pings.

message

A payload broadcast by another client. Its data is the value that was sent, unchanged.

audit-log

One database write, sent automatically by Plaid.

The official clients handle connected and heartbeat themselves, so a listen callback sees only the last two. An audit-log event looks like this:

{
  "type": "audit-log",
  "id": "01a04604-ff64-70ae-9362-8036acdca9bf",
  "projects": ["b1f2c4d6-0000-4000-8000-000000000001"],
  "documents": ["35424d64-a077-4a29-8006-5a0c3b76aedb"],
  "user": "user1@example.com",
  "time": "2025-07-09T20:27:59.616Z",
  "ops": [
    {
      "id": "01a04604-ff64-70ae-9362-8036acdca9bf",
      "type": "document:update",
      "project": "b1f2c4d6-0000-4000-8000-000000000001",
      "document": "35424d64-a077-4a29-8006-5a0c3b76aedb",
      "description": "Update document 35424d64-a077-4a29-8006-5a0c3b76aedb name to \"New Document Name\""
    }
  ]
}

Every write sends its own event, so ops always holds one operation, and a batch sends one event per write once it commits. documents lists every document the write changed in the project you listen on, and projects names that project. A write to a vocabulary reaches the listeners of its project, and those of every other project that links it and whose documents it changed, each with an event of its own holding only that project’s documents. The event is a lightweight notice: it spells operation types as span_layer:create where the audit endpoint has span-layer/create, and it carries bare ids. For the full entry, read the audit endpoint and match on id.

2.6. Services

A service is an external program that registers itself on a project and performs work on request. The motivating case is NLP: you might have a Python program that runs a parser, a tokenizer, or a large language model, and you want annotators to be able to invoke it from the editor with the click of a button. A service is just another client of the API, so it can be written in any language and run anywhere that can reach the server, with a token that has writer access to the project.

The server sits between a service and the clients that use it. It keeps track of which services are connected, sends each request to the one service it names, and passes that service’s replies back to the client that asked.

2.6.1. Offering a service

A service calls serve with a handler, which Plaid runs once per request. The handler receives the request’s data and a response helper, with which it reports progress(percent, message) and finishes with complete(result) or error(message).

def handle_request(data, response):
    response.progress(10, "Starting…")
    # whatever work the service does
    result = run_my_model(data["document_id"])
    response.complete({"status": "ok", "spans_created": result})

# `service_id` is the stable identifier clients use to address this service;
# `service_name` and `description` are human-readable. `serve` returns
# immediately with a registration handle; the work happens on incoming requests.
registration = client.messages.serve(
    project_id,
    {"service_id": "my-parser", "service_name": "My Parser",
     "description": "Parses a document with my model"},
    handle_request,
)

# The service stays available as long as the program is running and the
# registration is live. Typically you now block until interrupted:
try:
    while registration.is_running():
        time.sleep(1)
except KeyboardInterrupt:
    registration.stop()   # deregister cleanly

The same in JavaScript:

const registration = client.messages.serve(
  projectId,
  { serviceId: "my-parser", serviceName: "My Parser",
    description: "Parses a document with my model" },
  (data, response) => {
    response.progress(10, "Starting…");
    const result = runMyModel(data.documentId);
    response.complete({ status: "ok", spansCreated: result });
  },
);
// later: registration.stop();

2.6.2. Discovering and requesting

A client asks which services a project has with discover_services / discoverServices. Services seen before but not connected now come back with online: false, and only an online service can take work.

services = client.messages.discover_services(project_id)
# [{"service_id": "my-parser", "service_name": "My Parser",
#   "description": "Parses a document with my model", "extras": {},
#   "online": True, "last_seen_at": "2026-06-12T02:43:17Z"}]

request_service / requestService sends work to a service by its id and waits for the value the service passes to complete. An optional callback receives each progress report:

result = client.messages.request_service(
    project_id,
    "my-parser",
    {"document_id": document_id},
    timeout=60.0,
    on_progress=lambda p: print(f'{p["percent"]}%: {p["message"]}'),
)
print("Service returned:", result)
const result = await client.messages.requestService(
  projectId,
  "my-parser",
  { documentId },
  60000,                                  // timeout in ms (Python uses seconds)
  (p) => console.log(`${p.percent}%: ${p.message}`),
);
console.log("Service returned:", result);

The timeout counts silence, not the length of the run, so a service that keeps reporting progress can work as long as it needs. Giving up on the wait does not end the request, and the result can be picked up later (see Presence and persistence). Such an error is marked pending, unlike one where the service itself failed.

2.6.3. Describing a service

A client that offers a service in a fixed slot, such as "tokenize this document", needs to know which services fit the slot and what arguments each one takes. A service says so in its extras, which come back unchanged from discovery. From these, an app can build a picker, an arguments form and a description without knowing anything about a particular service.

from plaid_client import TASKS, Param, build_extras

extras = build_extras(
    tasks=[TASKS.TOKENIZE],                          # which slots this service fills
    summary="## My Tokenizer\nSplits text into words and sentences…",  # markdown
    parameters=[                                       # user-controllable arguments
        Param.enum("language", "Language",
                   [("english", "English"), ("german", "German")],
                   default="english",
                   description="Model language."),
    ],
)
client.messages.serve(project_id, service_info, handle_request, extras)
tasks

The slots the service fills: tokenize, parse, transcribe, link-vocab, analyze (segment and gloss words), draft-graph (draft a meaning representation graph), translate, detect-speech, assist (a chat assistant), or compare (score one document’s annotation against another’s).

summary

A longer description in Markdown, for what the one-line description cannot say.

parameters

The arguments a user may set for each request. Each has a key, a label and a type (string, number, boolean, enum, multiselect, or field, one of the project’s annotation fields), with an optional default.

A parameter’s value reaches the service under exactly the key it declared. A Python BaseService receives every declared parameter coerced against the schema, as the apps' request forms coerce it. A parameter left out takes its default, a number is held to its min and max, an enum value not offered is the default, and a string left blank takes its default when it has one. Most services write to the project and the client reloads, but detect-speech writes nothing: it returns time regions as segments, which the app offers as proposals. In JavaScript, filterServicesByTask picks the services for a slot, and getParamSchema and buildDefaultValues help build the form.

2.6.4. Holding the document

A service that reads a document and then writes it holds the document’s lock (see Locking) from that read through its last write, for the whole run. No one else can edit the document meanwhile, so the run never writes over an edit made while its model worked, and it needs no check that the document is as it read it. The lock is renewed for as long as the run takes, and the requester’s own app keeps the document read-only until the run ends. A change made to the project rather than to the document still reaches it: renaming, deleting or merging the entries of a vocabulary the document links to, renaming that vocabulary, changing its configuration or deleting it, unlinking it from the project, or deleting a layer. Each moves the document to a new version, and the run’s next write is then refused with a conflict (409).

Load a model before taking the lock, so no one waits on the document while it loads or downloads. Once the requester stops the run and it is not inside its writes (critical(), see Stopping a request), the lock is no longer renewed: a run stuck in a call that never reaches a checkpoint lets the document go within the lock window, and a write it makes afterwards is refused.

In Python, held_run from plaid_client.service takes the lock, puts the client in strict mode for that document (see Strict Mode), and sends the run’s result once the lock is released. The handler sets run.result, or run.refusal to the words of a refusal, and may return straight after. Either is sent after the lock is released.

from plaid_client.service import BaseService, held_run

class Glosser(BaseService):
    def process_request(self, request_data, response_helper):
        document_id = request_data["document_id"]
        with held_run(self.client, response_helper, document_id) as run:
            document = self.client.documents.get(document_id, include_body=True)
            glosses = self.gloss(document)        # the model, however long it takes
            with response_helper.critical():
                with self.client.operation("Glossing"):
                    self.write(glosses)
            run.result = {"document_id": document_id, "status": "success",
                          "words_written": len(glosses)}

locked_for_writes is the same lock without the result. Pass it the response_helper so that a stop lets the lock lapse as held_run does. It takes an optional version, for a request that carries the version the app asked a question at, such as a re-tokenize that deletes work only with the requester’s leave, and refuses a document no longer at it.

2.6.5. Service results

A service reports what it did as counts in its result, and sends no message of its own. The app that asked words what the person is told, in its own terms, and the same counts serve again when the page reloads mid-run and the run is picked up later (see Presence and persistence). A run that fails on some items still finishes with status: "success", and its counts say which items failed. A run with nothing to work on is a result too, with zero counts and a reason code the app words: no-text (a document with no text), no-words (a tokenizer found none) or no-speech (a recording with none). An error is for a run that could not be done. Every result has the document_id and a status, and a stopped run comes back as stopped: true instead. These are the results of each task, in the snake_case a Python service writes, which a JavaScript client reads in camelCase.

tokenize

tokens_created, tokens_deleted and sentences_created, with reason no-text or no-words when there was nothing to tokenize.

parse

mode (full for a document parsed whole, preserve for one whose sentences a person worked on were kept), parsed_sentences, unchanged_sentences (those of the parsed that came out as they are stored, so nothing was written for them) and skipped_sentences, with reason no-text for a document with no text.

transcribe

segments_transcribed, the segments the model heard, segments_added, the ones put in the transcript, and segments_skipped, the ones whose time a segment already held. tokens_created equals segments_added. reason is no-speech for a recording with no speech in it.

detect-speech

segments, the time regions found, each {time_begin, time_end} in seconds with an optional speaker. Nothing is written.

analyze

sentences in the document, sentences_sent, the ones holding a word the run could write, words_written and words_replaced. skipped counts the words left alone: protected (a person’s analysis, with overwriting off), precedent (named in the request’s skip_word_ids) and unaligned (given no analysis by the model). sentences_failed lists {sentence_id, reason} for each sentence the run could not analyze, sentences_not_asked the ids of the sentences never sent because the model stopped answering, and translation_field_missing names a translation field the request asked for that the document lacks.

translate

sentences, sentences_written and sentences_replaced. skipped counts protected sentences (a person’s translation, with overwriting off) and empty ones (no words). sentences_failed lists {sentence_id, reason}, sentences_not_asked the ids of the sentences never sent because the model stopped answering, and gloss_field_missing names a gloss field the request asked for that the document lacks.

draft-graph

sentences in the document and drafted, the sentences given a new graph. unchanged counts the sentences whose new draft is the graph they already have, which writes nothing. skipped counts the sentences with a machine-made graph left alone with overwriting off, kept the ones a person made, contributed to or accepted something in, and linked the ones another sentence’s graph links into. failed and sentences_failed, a list of {sentence, reason} by sentence number, name the sentences the run could not draft ({sentence, code} for a cause the app words: no-link-or-gloss, no word has a vocabulary link or a gloss, and no-link-no-gloss-line, no word has a vocabulary link and the project’s UMR settings name no word or morpheme gloss line), and sentences_not_drafted lists, by number, the ones never asked about because the model stopped answering.

compare

against, the {id, name} of the document scored against, scope, scores (sentence, and comprehensive or null), sentences, sentences_scored and sentences_skipped, the ones the metric could not read.

A reason in a failure list is the service’s own account of one item, and the app shows it as it stands. Other keys may be added beside these.

2.6.6. Stopping a request

A client can ask a service to stop with cancelServiceRequest / cancel_service_request. Nothing is interrupted, though: the service stops at the next point where it checks. Every call to progress() is such a point, so a service that reports its progress can be stopped with no further work, and the request ends with stopped: true rather than as a failure.

Writes should not be cut off halfway, so wrap them in the helper’s critical(), inside which a stop waits. Report the result inside the same block, or a stop that arrives after the writes would throw away a finished result. The helper’s raise_if_cancelled() adds a checkpoint to a long loop that reports no progress. In this BaseService, load_sentences, gloss and write stand for the service’s own work.

from plaid_client.service import BaseService

class Glosser(BaseService):
    def process_request(self, request_data, response_helper):
        sentences = self.load_sentences(request_data["document_id"])
        results = []
        for i, sentence in enumerate(sentences):
            # Already a checkpoint: a stop here ends the request cleanly.
            response_helper.progress(10 + int(70 * i / len(sentences)),
                                     f"Glossing {i + 1}/{len(sentences)}")
            results.append(self.gloss(sentence))

        with response_helper.critical():    # too late to stop: finish, and say so
            with self.client.operation("Glossing"):
                self.write(results)
            response_helper.complete({"glossed": len(results)})

In Python the stop is a ServiceCancelled exception derived from BaseException, so except Exception does not swallow it. In JavaScript, a handler that catches every error must re-throw a ServiceCancelled.

2.6.7. Presence and persistence

A service can take work only while it is connected. There is no queue: a request to a service that is down fails at once. If you need a request to survive a service being briefly unavailable, you would build that yourself, for instance by recording pending work in your own layer and retrying.

The requester’s connection is a different matter, because a request outlives it. If a browser tab is reloaded during a long run, the service carries on, and its result is kept on the server for fifteen minutes. The requester picks it up by request id with attachServiceRequest / attach_service_request, which replays the latest progress and then the result.

A client that wants to be able to come back mints the request id itself (the requestId / request_id option), so it can save the id before the request starts. Sending the same id again rejoins the request rather than starting another, so retrying is safe.

Each service id allows one connection per project, and a second one is refused with 409. A service id on a project belongs to the account that first connected it. Another account connecting it is refused with 403, and the refusal names the account it belongs to. An administrator releases the id with DELETE /api/v1/projects/{id}/services/{service-id}/owner (releaseService / release_service), and the next account to connect it then holds it. Forgetting an offline service does not release its id. Deactivating the account releases every id it holds. The project’s services listing (GET /api/v1/projects/{id}/services) gives the display name of whoever holds each id in claimed-by-name, absent when no account holds it. An administrator also gets the account in claimed-by, and an entry with seen false for each held id whose registration was forgotten, so it can still be released. Both clients reconnect a service whenever its connection drops, so restarting the Plaid server does not mean restarting its services. registration.is_connected() / isConnected() tells whether the server sees the service as online right now.

Note
Service requests are not recorded in the audit log, but the writes a service makes are, under the token it runs with. Giving a service its own named API token makes its edits easy to pick out.

2.6.8. Acting on the requester’s behalf

A service normally works with its own credentials, so every edit it makes is its own. That suits a tokenizer or a parser, whose output does not depend on who asked. It does not suit an assistant that edits whatever a user tells it to, which should be able to do only what that user could do.

A service that declares delegation in its extras (BaseService(…​, delegation=True)) acts on the requester’s behalf instead. For each request the server hands it a short-lived token for the person who asked, which BaseService offers as request_data["requester_client"]. Everything done with that client is checked against the requester’s permissions and recorded as theirs, so readers may use such a service too.

The token reaches only the projects the request is about: the one it was made on, and any others it names with project_ids / projectIds. Even there, it cannot do what only maintainers may, such as changing members, layers or configuration. A delegating service acts for people other than the one running it only in projects that person maintains, or anywhere if they are an admin.

The assistants (plaid-agent in the source tree) are delegating services. Their edits are proposed as plans, which the user approves before they are applied under the user’s own account. Whoever runs an assistant can state where its model runs and where users' messages go with --disclosure "…​", and the Assistant panel shows that text.

2.7. Assistant jobs

An assistant job is unattended work by an assistant over many documents of a project, such as glossing every blank word. A writer of the project starts one, naming the app, the documents in order (all of them if left out) and instructions for the model, at most 2,000 characters. A job may also carry a setup, a JSON object of at most 4,000 characters that the app sends when the job starts. The server keeps it as sent, returns it on every read of the job and never changes it, and reads none of its keys. Each app decides what goes in it, such as what the job may fill. Both clients re-case the keys inside it as they re-case every key, so a key of one lowercase word is the only kind that reads the same in every language. A separate program, the job worker, does the work, one job at a time across the server, in the order the jobs were started. Its edits are recorded under the name of the person who started the job, with the operation kind job.

A job is waiting until the worker takes it, then running, and ends done, cancelled or failed. A project has at most one job that has not ended, and starting a second is refused with 409 and job-active. Stopping a job that is running makes it stopping, and the worker ends it after the work in hand. Jobs themselves are not recorded in the audit log.

Route Who may

POST /api/v1/projects/{id}/jobs (start)

a writer or maintainer of the project

GET /api/v1/projects/{id}/jobs, …​/jobs/active, GET /api/v1/jobs/{id}

a reader of the project

POST /api/v1/jobs/{id}/cancel (stop)

the person who started the job while they write the project, a maintainer, an admin

GET /api/v1/jobs/worker

anyone signed in

POST /api/v1/jobs/claim, GET /api/v1/jobs/model-busy, and a job’s heartbeat, documents/{id}, decisions and finish

an admin

a job’s landed

an admin, or the job’s own token

A delegated token can neither start nor stop a job.

2.7.1. The job worker

The worker runs under an administrator’s named API token. It asks for work with jobs.claim, which answers the next job or nothing, and a token for the person who started the job. While it works it sends jobs.heartbeat every 30 seconds, which answers the job’s state and a fresh token. A job whose worker has sent nothing for two minutes is taken by the next worker that asks, and the old worker’s next call is refused with job-taken. Once a job has ended, its document rows and decisions are refused with 409 and job-finished, except for a job core failed whose report the worker has not yet sent.

The worker writes into documents only with the token of the person who started the job. That token reaches only the job’s project and does only what that person may there. It works only while the job is running or stopping on the worker it was issued to. Once the job ends, or another worker takes it, the token is refused with 401 and job-ended. Once they can no longer write in the project, the job fails, and no further token is issued.

The worker sends each write of a job under Idempotency-Keys of a seed of its own (<seed>.<n>). When an answer is lost, or the worker stopped while a write was on its way, POST /api/v1/jobs/{id}/landed with refs, the seeds, answers keys, the keys of every write the job’s starter made in its project under them. The server keeps each operation’s key in the audit log, so for a write it records in History the answer holds at any time. A write it does not record there, such as a comment, is answered for the 24 hours it keeps the answer for replay, in whatever project it was made.

2.7.2. Model pools

A job waits while people use the model it draws on. Every service and assistant that uses the same model key is started with the same --model-pool NAME, and so is the worker. Before each call to the model the worker asks GET /api/v1/jobs/model-busy?pool=NAME how many requests to those services are running, and waits until none has run for a minute.

3. Layer Types

Each project contains a configuration of layers which define a schema for all documents in the project. Each layer holds a single kind of annotation, and each project may have any number of each kind of layer. For instance, you might have two span layers: one for POS tags, and another for lemmas.

3.1. An Example

Suppose we’re working on a project where all we are doing is POS-tagging. The configuration of the project’s layers (in a simplified JSON representation) would look something like this:

{
  id: "1cce50df",
  name: "Example Project",
  textLayers: [
    {
      id: "6283144f",
      name: "Text",
      tokenLayers: [
        {
          id: "d1cc124f",
          name: "Words",
          spanLayers: [
            {
              id: "ad0f5f2c",
              name: "POS tags"
            }
          ]
        }
      ]
    }
  ]
}

This layer structure prescribes the structure of individual documents. Consider a document where we have POS tagged the sentence "Fido barks":

{
  id: "01d01a27",
  name: "Document 1",
  project: "1cce50df",
  textLayers: [
    {
      id: "6283144f",
      name: "Text",
      text: { id: "9cfafcc6", document: "01d01a27", body: "Fido barks" },
      tokenLayers: [
        {
          id: "d1cc124f",
          name: "Words",
          tokens: [
            { id: "54383a26", begin: 0, end: 4 },
            { id: "a8758db2", begin: 5, end: 10 }
          ],
          spanLayers: [
            {
              id: "ad0f5f2c",
              name: "POS tags",
              spans: [
                { id: "4ed828ea", value: "NOUN", tokens: [ "54383a26" ] },
                { id: "b4ef8082", value: "VERB", tokens: [ "a8758db2" ] }
              ]
            }
          ]
        }
      ]
    }
  ]
}

Notice the following:

  • Each layer has a corresponding kind of data in the document: the text layer has a text, the token layer has tokens, and the span layer has spans.

  • The layers are dependent on each other: the text layer is a dependent of the project, the token layer is a dependent of the text layer, and the span layer is a dependent of the token layer. This is a reflection of conceptual dependencies: tokens are defined as atomized substrings of a text, and spans are defined as groupings of one or more tokens.

  • Each individual entity—whether it is a layer or some data within that layer—has a unique ID

  • Entities refer to others with these IDs—for instance, each span’s tokens value has a list of tokens which constitute that span.

We will continue discussing this example in more detail below.

3.2. Projects and Documents

A project is the root of a layer configuration and has a name.

{
  id: "1cce50df",
  name: "Example Project",
  textLayers: [/* ... */]
}

A project has many documents, and each has a name and a unique ID:

{ id: "01d01a27", name: "Document 1", project: "1cce50df" }

Document names are unique in a project, ignoring case: "Story", "story" and "STORY" are one name (full Unicode case folding, as Python’s str.casefold does it, so "Straße" and "STRASSE" are one too, while dotless "kır" and "kir" are two). A create, rename or copy to a name another document of the project holds is refused with 422, and the error and the answer’s name-taken give the name as that document spells it. A create or copy with free-name (freeName in the JavaScript client, free_name in Python) takes the first free name instead: the name itself, else "Name (2)", "Name (3)" and on. Both answer with the name the document was given. A restore keeps the document’s current name when the name it had then belongs to another document now, and its summary says so under name-kept: the name kept and the other document’s name. A vocabulary link that a merge moved to another entry also stays as it is now, and the summary counts it under kept.

3.2.1. Reading part of a document

Reading a document with include_body / includeBody returns every layer in the project. When several apps share a project, each may care about only a slice of it, so you can pass layers to read only the layers you name:

doc = client.documents.get(
    document_id,
    include_body=True,
    layers=[text_layer_id, word_layer_id, pos_layer_id],
)
const doc = await client.documents.get(
  documentId, true, undefined,
  [textLayerId, wordLayerId, posLayerId],
);

A named layer comes with its contents, and the layers above it come along empty to keep the usual nesting. So naming just a POS span layer gives you its spans inside an empty token layer and an empty text layer, and you add those layers to get the tokens and the text. An id that is not a layer of the document’s project is an error.

3.2.2. Recordings

A document may carry one audio or video recording, which the apps use for transcription. Uploading or deleting it appears in the audit log, but the file itself is not kept there, so a deleted recording cannot be brought back. An upload is refused with 409 when the document already has a recording, and with 415 when the file is not audio or video that a browser can play.

A document with a recording has media-url (mediaUrl in JavaScript, media_url in Python), the address to fetch it from. Beside it, media-duration (mediaDuration, media_duration) is the recording’s length in seconds, read from the file when it is uploaded. The length is read for WAV, MP4, M4A, M4V, MOV, FLAC and MP3. A recording in another format, such as WebM or Ogg, has no media-duration.

3.3. Texts

For each text layer, each document may have at most one text, which consists of a single string. This string holds all the text which is to be analyzed in dependent layers. A text object looks something like this:

{ id: "9cfafcc6", document: "01d01a27", body: "Fido barks", digest: "4d7e1a…" }

digest is the SHA-256 of the body’s UTF-8 bytes, in lowercase hex. It names the body’s current state, and edits use it to check that they are working on the body they think they are (see Changing a text’s body).

3.3.1. Composed text

Plaid stores all text in Unicode Normalization Form C (NFC), so an á sent as a plus a combining acute accent is stored as the single character á. This applies to a text’s body and to every other string you write, such as span values, metadata and names.

A body can therefore get shorter when it is saved. Token offsets always refer to the body as stored, which a read or a body edit returns. An app that creates tokens in the same batch as a body can compute their offsets on the stored body with composeText(body, cuts) (compose_text in Python), sending the same cuts as the text’s tokenEdges.

There is one exception. Where a token begins or ends between a letter and a mark, such as a tone mark analysed as its own morpheme, the two are left uncomposed so that each token keeps its own character.

3.3.2. Changing a text’s body

PATCH /texts/{id} changes a text’s body, and the tokens over the text follow the change. You may send the whole new body, and Plaid works out what changed. An editor that knows where the cursor was sends edits instead, which say exactly where each change was made:

{ type: "insert",  index: 4, value: "s" }             // type "s" at 4
{ type: "delete",  index: 9, value: 1 }               // delete 1 code point at 9
{ type: "replace", index: 0, length: 3, value: "dog" } // put "dog" in place of 3 code points at 0

Indices count code points, and each edit applies to the body as the edits before it left it. An edit of another shape, a null among them, or one that reaches past the body is refused with 400, and nothing is written, whether the edits come as edits or as a list under body. An empty edits list writes nothing: the answer is the text as stored, and the document’s version stays as it was. The message names the edit in JSON, as both clients name it when they refuse it first: the keys type, index, length, value and side in that order, then any others sorted, with every number as JavaScript prints it (1.0 as 1). Send base, the digest of the body the edits were made on, and the edits apply only if the stored body is still that body. Otherwise the answer is 409 with text-changed: true, and nothing is written.

PATCH /texts/9cfafcc6
{ edits: [ { type: "insert", index: 10, value: "!" } ], base: "4d7e1a…" }

Tokens follow the text by fixed rules, the same for every save. Letters typed inside a word or at its end become part of the word, and a word is deleted when all of its letters are. A word made only of punctuation and symbols, such as a token over ( or ., takes no text typed against it, so the letters typed there are the same words as where the punctuation has no token: in ab (cd) with a token over (, x typed after ( makes xcd, and x typed after a closing . is in no word. This holds for words only: a row made only of punctuation, such as a pause …​ in a text with no words, takes text as any row does. An emoji and the zero morph ∅ count as letters here. So does a character a word layer’s ignored-tokens rule (ignoredTokens in the plaid namespace) calls a letter: under the punctuation rule a character it lists as letter-like, such as an apostrophe a project spells words with, and under an explicit list every punctuation character but the listed ones. The rule is an object {type, whitelist, blacklist}. Under the type unicodePunctuation a word made only of punctuation is ignored, and each entry of whitelist one character long is a letter. Under the type blacklist the entries of blacklist are the words ignored, and those one character long are the only characters that part words. The list of the other type is kept and not read. Punctuation and symbols are read from the Unicode 16 table the apps read.

A character the word layer’s rule leaves out of a word stays out of a word it is typed against. Under the punctuation rule that is a punctuation mark the built-in tokenizer parts words at (see New words for typed text), and under an explicit list a listed character. Typed at a word’s start or end, the characters next to the word join it up to the first such character or whitespace, and the rest belongs to no word: after kilu ama, , sipa leaves the word ama, and s, makes it amas. Typed inside a word, it is part of the word, and a sentence or a row takes it as it takes any text. A word layer with no rule leaves nothing out.

A replacement, or any change that deletes text and types new text in the same place, counts as a deletion followed by an insertion there. The text it shares with the old text at its start and at its end is no change. Where it replaces text over two or more words, that shared text is no change only back to a word edge, never inside a word, so no word keeps letters of another’s: ab a replaced whole by a deletes both words, and the new a belongs to no word. One word typed over keeps what it shares: cat replaced by cats keeps the word, now cats. A word all of whose letters it replaces is deleted with its annotations, even when the new text spells the same word again. A word it replaces only part of keeps its other letters and takes the letters typed against them, and the rest of the new text belongs to no word. In the cat sat, cat replaced by dog deletes the word cat, and dog belongs to no word, while at s replaced by ow r leaves the words cow and rat. Tokens under a word, such as morphemes, and tokens beside the words on a layer that allows overlap, such as UMR nodes, are deleted the same way when none of their letters is left. A word left starting or ending on whitespace it did not start or end on moves onto its letters, and the tokens under it are cut to it, so a morpheme that began on that whitespace moves with the word. A token under a word that is deleted is deleted with it, a morpheme of only whitespace too. A sentence or a row (see below) holds the text typed over part of it. One whose exact text is replaced, with nothing more, keeps its place and its annotations on the new text, and a row its times. One that a replacement covers along with more text is deleted with its annotations, as a deletion of it would be. In one two. three four. pasted over as uno dos. tres cuatro., the two share only the final ., so the first sentence lies inside the replacement and is deleted, and the second, which the replacement only cuts, takes the new text, where the . is in no word. A save that leaves text in the body and no token on a partitioning layer makes one new token of that layer over the whole body, with no metadata, when the layer held a token before the save. A layer that never held one gets that token at the end of the request when a token stands on a layer under it then, such as the words "Tokenize new text" makes in a text whose sentences were never made. A batch that makes the partition itself after the save gets none. The partition then covers the text and the words are inside it. An app can then split it. A save that would leave a token outside every token of its parent layer is refused with a 409 naming the two layers, and nothing is stored. A token that was already outside every parent before the save does not count, and a parent layer that holds no token is judged at the end of the request.

A next-line character (U+0085) is whitespace to all of these rules, as a line feed is: it parts two words, ends a line, and counts as whitespace at a sentence’s or a row’s edge.

A space typed inside a word does not split it, unless the word layer’s config sets splitOnSpace to true in the plaid namespace. Only a word splits: in a text whose word layer holds no token yet, a space typed inside a row splits nothing. Where two words meet with no space between them, an insert at the boundary may say which word its text joins with side: "before" or side: "after".

Text typed between two tokens of a partitioning layer, such as two sentences, belongs to one of them by where it was typed:

  • Right after the first sentence’s last character, before any whitespace, it joins the first sentence.

  • After whitespace, anywhere in the whitespace between the two, it joins the first sentence that begins at or after the place it was typed, which then begins at the typed text. In The dog ran. The cat sat., Then ` typed just before the second `The gives the sentence Then The cat sat.. Where a sentence holding only whitespace begins at or after that place, before the second sentence, the text joins it and it keeps its place before the second. Each insert goes there whether it is typed alone or with others in one save.

  • A deletion or a replacement in that whitespace counts as a change between the two: an insert before it in the same save stays with the first sentence.

  • Where no whitespace parts the two and the first ends on punctuation, as in 你好。我很好。, it joins the second sentence. Between two letters it joins the first, as text typed between two words does. Punctuation here is as for the words above, so after a character the project calls a letter the text joins the first sentence.

  • Text holding a line break, or with a line break between it and the second sentence’s first letter, leaves the boundary where it was. A line typed there can then be split off as a sentence of its own without moving the second sentence or what is on it.

  • At the start of the text it joins the first sentence, and a line typed there becomes a sentence of its own. A line of only whitespace does not, such as Enter typed after a paragraph indent of ideographic or no-break spaces, which counts as whitespace as an indent of ASCII spaces does. At the end it joins the last.

An insert’s side decides over these where it applies.

A row, a token of a layer that forbids overlap other than the words and the layers under them, such as a time-alignment segment, keeps to the sentences this way:

  • Text typed inside a row stays in it, wherever the sentence boundary near it goes, except in the whitespace at a row’s edge, as the next point says.

  • A row over whole sentences, with or without the whitespace at its edges, follows them. An edge at a sentence’s start or end, or at its first or last letter, goes where that sentence’s start or end goes, and keeps the whitespace it held there and no other. Text typed at that edge, or in the whitespace between the edge and the row’s letters, counts as typed at the sentence boundary: it is in the row when that sentence takes it. In Hi. The end. with a row over each sentence and its whitespace, So typed between the two spaces begins the second sentence and is in the second row.

  • A row over part of a sentence takes the text typed right at an edge it shares with that sentence when the sentence takes it, and no other text of the sentence. This includes text the sentence takes at the start or end of the text and a line typed after it.

  • Any other edge of a row stays where it was.

  • A row keeps every letter it held that the save leaves. A space typed inside a word that splits it (splitOnSpace) moves no row.

  • Rows that did not overlap before a save do not overlap after it.

A whole body save takes its change as the character difference between the two bodies, read by the same rules. Where the difference can stand at more than one place for the same new body, the server does not choose the place that spares the tokens: deleting cat from the cat cow reads as at c deleted, which cuts both words. The reading can differ from what was typed, so an editor sends edits. Edits sent as the body (a list of ops) are applied as sent: text inserted where two tokens meet is in neither, and a partition’s gap is closed by the token before it.

The answer carries the new digest and a reshape, which lists the tokens whose extent changed and the ids of everything the change deleted. An editor can update its own copy of the document from it instead of reading the document again.

3.3.3. New words for typed text

A save, in any of its forms, and a new text may name a word layer of the text as tokenize-layer. The text the save types then gets words on that layer, made after the old tokens are placed by the rules above, so a new word never lands where an old one now stands. A stretch no token of the layer covers is cut into words at whitespace and at punctuation, inside each sentence (each token of a partitioning layer). Punctuation here is the built-in tokenizer’s break class, read from the same Unicode 16 table, under the layer’s ignoredTokens rule in the plaid namespace: a character the rule calls letter-like is part of a word, and under an explicit list only the listed characters part words. A word holding no typed character is not made, and neither is one the rule ignores or one holding a letter of a script written without spaces between words, such as Han, kana or Thai. The new words come in the answer’s reshape.tokens with their layer and text. A new text’s words are judged as a save’s are, at the end of the request: under a partitioning layer that holds no token they get one token of it over the whole body, unless the request makes the partition itself, and a word left in no token of its parent layer is refused (409) with nothing stored.

PATCH /texts/9cfafcc6
{ edits: [ { type: "insert", index: 11, value: " Rex runs." } ], base: "b31c09…", "tokenize-layer": "5b0a…" }
{ id: "9cfafcc6", body: "Fido barks! Rex runs.", digest: "3d1b3c…",
  reshape: { tokens: [ { id: "e0d3…", begin: 12, end: 15, layer: "5b0a…", text: "9cfafcc6", precedence: null, metadata: {} },
                       { id: "f0d4…", begin: 16, end: 20, layer: "5b0a…", text: "9cfafcc6", precedence: null, metadata: {} } ],
             spans: [], "vocab-links": [],
             deleted: { tokens: [], spans: [], relations: [], "vocab-links": [] } } }

3.4. Tokens

For each token layer, each document may have many tokens, which are defined as substrings of a text:

{ id: "54383a26", text: "9cfafcc6", begin: 0, end: 4 }
{ id: "a8758db2", text: "9cfafcc6", begin: 5, end: 10 }

Note the following:

  1. begin and end must form valid substring indices for the given text. They are counted in Unicode code points, not UTF-16 code units or bytes, so an emoji is one position.

  2. By default, zero-length tokens where begin == end are valid.

  3. By default, tokens may overlap.

  4. A token’s place in the linear order of the document is given by its begin. Among tokens with the same begin, the optional precedence value decides, and a token with lower precedence comes earlier. A read of a document does not return its tokens in this order: sort them by begin and then precedence to read them in order.

Token layers are unique in that their invariants are configurable. In addition to the default configuration, it is possible to constrain them further: for example, a token layer may require that no tokens be overlapping in extent of begin and end. For full details, see Token Layer Constraints below.

Tokens are intended to serve as the basic units for further linguistic analysis using spans and relations.

3.5. Spans

For each span layer, each document may have many spans, which are groupings of one or more tokens which have a single value:

{ id: "4ed828ea", value: "NOUN", tokens: [ "54383a26" ] }
{ id: "b4ef8082", value: "VERB", tokens: [ "a8758db2" ] }

There are no restrictions on spans, other than that they must hold at least one token, and that they all must belong to the span layer’s parent token layer.

3.6. Relations

For each relation layer, each document may have many relations, which are directed edges between two spans with a label. Both spans must belong to the relation layer’s parent span layer. For example, if we wanted to extend the example above with a syntactic dependency relation between "Fido" and "barks" expressing that "Fido" is the subject, we could have a relation like this:

{ id: "2f6080ff", source: "b4ef8082", target: "4ed828ea", value: "nsubj" }

3.7. Vocabs

The four basic layer types (text, token, span, and relation) belong to one project each. The fifth layer type, the vocab layer or vocabulary, also belongs to one project, its home, and other projects may use it too. As its name suggests, this layer is intended for recording occurrences of lexical entries.

The vocab layer itself has a name, and project, the id of its project:

{ id: "2b75b0f9", name: "English", project: "7a1c09d2" }

The vocab layer has vocab items, which represent lexical entries, each with a canonical form:

{ id: "da8d4549", form: "Fido" }
{ id: "b5c6e64c", form: "bark" }

Finally, vocab links are used to indicate occurrences of lexical entries. Recall the tokens from before:

// "Fido"
{ id: "54383a26", text: "9cfafcc6", begin: 0, end: 4 }
// "barks"
{ id: "a8758db2", text: "9cfafcc6", begin: 5, end: 10 }

We can create links between them and the above vocab items with vocab links like so:

{ vocabItem: "da8d4549", tokens: [ "54383a26" ] }
{ vocabItem: "b5c6e64c", tokens: [ "a8758db2" ] }

Notice that multiple tokens may be specified, allowing for multi-word and non-contiguous lexical items.

Deleting a vocab item deletes every link to it. To merge entries, vocab_items.merge(survivor_id, loser_ids) / vocabItems.merge(survivorId, loserIds) moves every link of the losers to the survivor, except a link on tokens the survivor is already linked to, which is deleted, and deletes the losers, in one operation. Metadata elsewhere that names a loser by its id is rewritten to name the survivor.

3.7.1. A vocabulary’s project

A vocabulary is made inside a project, vocabLayers.create(projectId, name), by a maintainer of that project. Its rights are that project’s roles:

  • Maintainers rename, move and delete the vocabulary, change its configuration, and rename, merge, delete and restore its entries.

  • Writers add entries and change anything stored on an entry: its fields, type, senses and examples.

  • Readers read it.

Another project may link the vocabulary with projects.linkVocab(projectId, vocabId), which needs a maintainer of that project who also holds a role in the vocabulary’s project. That project borrows it: its members read it and link their documents' tokens to its entries. Rights over the entries follow a person’s role in the vocabulary’s project, whatever project they work in, so a member of the borrowing project who holds no role there changes no entry.

A vocabulary cannot be unlinked from its own project. A borrowing project’s maintainers unlink it with projects.unlinkVocab(projectId, vocabId), which removes its links in that project’s documents. The maintainers of the vocabulary’s project may also end any project’s borrowing the same way, and GET /api/v1/vocab-layers/{id}/borrowers lists the projects that borrow it, by name where they may read them. projects.vocabUnlinking(projectId, vocabId) (GET /api/v1/projects/{id}/vocabs/{vocabId}/unlinking) answers how many links the unlink would remove, and writes nothing. An unlink is listed in the borrowing project’s Activity, and in the vocabulary’s History and its project’s Activity.

vocabLayers.move(id, projectId) makes another project the vocabulary’s project, for a person who maintains both. The old project keeps it linked, as a borrower.

Deleting a vocabulary is refused while another project borrows it, for every user, admins included. Deleting a project deletes its vocabularies with it, and is refused while another project borrows one of them. What a delete would take is a read of its own, which answers the same refusals and writes nothing: vocabLayers.deletion(id) (GET /api/v1/vocab-layers/{id}/deletion) the number of links, projects.deletion(id) (GET /api/v1/projects/{id}/deletion) the vocabularies. These deletes and the unlink take no query parameter but the audit labels, and refuse any other with 400.

3.8. Metadata and Config

It is often desirable to enrich an entity with additional information—for instance, you might want to record some information about the annotator’s confidence in whether a certain span value is correct. Additionally, you might want to do the same with a layer in order to e.g. specify what values are acceptable for spans in a given layer. To accommodate this, Plaid allows arbitrary data to be stored in the config attribute for layer types (project, text layer, token layer, span layer, relation layer, vocab layer) and in the metadata attribute for data types (document, text, token, span, relation, vocab item, vocab link).

You might use a config to store legal tags for a given layer:

{ id: "...", name: "POS Tags", config: { tags: ["NOUN", "VERB", /* ... */] } }

As for metadata, one common use is recording whether a human or a machine produced an annotation. Plaid standardizes this as the Provenance convention:

// Human-made POS tag: NO provenance keys (absence = human)
{
  id: "...",
  tokens: [/*...*/],
  value: "NOUN",
  metadata: {}
}
// Machine-made POS tag, not yet human-verified
{
  id: "...",
  tokens: [/*...*/],
  value: "NOUN",
  metadata: {
    prov: "inferred",
    provSource: "service:my-tagger",
    provProb: 0.8412,                       // flat -> queryable/sortable
    provDetail: {                           // open map for the rest
      model: "my-tagger==2.1",
      valueProbs: { NOUN: 0.8412, PROPN: 0.0966, ADJ: 0.0014 }
    }
  }
}

3.8.1. Editing metadata

set_metadata / setMetadata replaces an entity’s whole metadata. To change part of it, patch_metadata / patchMetadata takes a list of ops, so two people editing different keys of one object do not overwrite each other. Each op sets or deletes the value at a path of keys:

client.documents.patch_metadata(doc_id, [
    {"op": "set", "path": ["corefud", "entities", "c10"], "value": "bird"},
    {"op": "delete", "path": ["corefud", "entities", "c8"]},
])

3.8.2. Editing config

Config is organized by namespace, one per app, and set_config / setConfig writes one key under one namespace without touching the others. A settings page that writes back what it loaded could undo another maintainer’s save, so a write may pass the value it read as expected. It is then refused with 409 if the stored value has changed:

tagsets = project["config"]["igt"]["tagsets"]
client.projects.set_config(project_id, "igt", "tagsets", {**tagsets, "Case": {"mode": "suggest", "values": []}},
                           expected=tagsets)
Note
Config and metadata values are schema-less, meaning that keys and values are unconstrained. If you’d like to see schema support for either of these, please feel free to open an issue.

4. Data Integrity

In collaborative annotation projects, it is crucial to take steps to ensure that data never reaches an invalid state. Plaid provides a few different means for maintaining data integrity, so that you may have confidence that your data will never become corrupt.

4.1. Core Data Integrity Constraints

In the previous section, we noted the constraints which Plaid enforces on each data type. Plaid guarantees that the database will never violate these, no matter what, by ensuring that invalid entities are never created, and often by deleting structures which are indirectly rendered invalid by another change. Consider these examples:

  • If a relation’s source span is deleted, then Plaid deletes the relation as well, because a relation must have a span on either end in order to remain valid.

  • If a few characters are deleted in a text, then all token indexes are updated to maintain validity: tokens containing those characters will shrink or get deleted (if they turn into zero-length tokens), and not containing those characters which are anchored to subsequent text will have their indices decremented by the number of deleted characters.

  • If a span’s only token is deleted, then the span will deleted, along with any dependent relations.

These invariants have been incorporated into Plaid because of their broad desirability in linguistic annotation. However, some invariants will vary by annotation framework. For example, it is quite common to want a span layer’s spans to be in one-to-one correspondence with tokens in the parent token layer.

For token layers, you can ask Plaid to enforce some commonly needed additional invariants for you, server-side: these are described in Token Layer Constraints below. An app can also declare rules of its own on the layers it uses, such as one head per word or one sentence per relation, which Plaid enforces on every write: see Layer Constraints. For everything else, Plaid provides three further mechanisms—strict mode, locking, and atomic batches—which help you coordinate writes from the client so that you can enforce your own data integrity constraints without needing to write any code outside of your client.

4.2. Token Layer Constraints

The constraints in the previous section are always on. Token layers additionally support a set of opt-in, server-side constraints which capture some of the most common token-level invariants in linguistic annotation. Because they are enforced by the server on every write, there is no need for extra measures on the client side in order to maintain them. They are configured per token layer at creation time, and are immutable thereafter.

4.2.1. Overlap Modes

Every token layer has an overlap mode, chosen at creation, which governs how its tokens may sit relative to one another:

  • any (the default): no restrictions. Tokens may overlap, leave gaps, and be zero-width. This is the behavior described in the Tokens section.

  • non-overlapping: no two tokens in the same document may overlap (share any character). Gaps between tokens are still allowed—for instance, the whitespace between words.

  • partitioning: the tokens must form a gap-free, non-overlapping, zero-width-free cover of the whole text. Partitioning is allowed only on a root (parentless) token layer; see Token Layer Hierarchy.

Note

A partitioning layer is, at every moment, in exactly one of two valid states for a given document: empty (no tokens at all—the document is simply not yet tokenized at this layer, which is the state every layer starts in), or a complete partition (the gap-free, overlap-free, zero-width-free cover described above). Plaid never lets it reach a partial state in between: every operation either moves it between empty and complete, or is rejected.

In particular, there is no automatic establishment—a newly created partitioning layer is empty until you bulk-create its partition, and bulk-delete (which must remove the whole partition at once) returns it to empty. A text save that deletes the whole partition and leaves text in the body makes one token over the whole body instead, as does one that makes a token under a partition never made (see Changing a text’s body). One that leaves the body empty leaves the layer empty. The editing operations below each preserve the complete-partition state: a split divides one token, a merge joins two adjacent ones, and a boundary shift only moves a shared boundary within the two tokens it separates (a shift that would collapse, invert, or overshoot a token is rejected).

You set the mode when creating the layer:

# A non-overlapping "words" layer
client.token_layers.create(
    text_layer_id, "Words", overlap_mode="non-overlapping")

Because a partitioning layer must always tile its extent, the ordinary per-token create, delete, and extent-update operations are rejected on it—each would necessarily leave a gap or an overlap. Instead, you build and edit the partition with four operations:

  • bulk-create establishes a whole partition at once.

  • split divides one token into two at an offset.

  • merge combines two adjacent tokens into one.

  • shift-boundary moves the boundary between two tokens, growing one and shrinking its neighbor so that the cover is preserved.

These editing operations are available on any and non-overlapping layers too, where they serve as convenient shortcuts. In every case, dependent spans and vocab-links are preserved: a split keeps them on the original (left) token, and a merge reparents the right token’s spans and vocab-links onto the surviving left token. A split may pass keep: "right" to keep the original token on the right half instead, which suits new text typed in front of a token, such as a transcription inserted before a sentence.

# Establish the partition "dogs run" -> "dogs", "run"
result = client.tokens.bulk_create([
    {"token_layer_id": words, "text": text_id, "begin": 0, "end": 4},
    {"token_layer_id": words, "text": text_id, "begin": 5, "end": 8},
])
dogs_id, run_id = result["ids"]

# Split "dogs" into "dog" + "s"
new_id = client.tokens.split(dogs_id, 3)["id"]

# Merge the two halves back together
client.tokens.merge(dogs_id, new_id)

A relation layer whose relations must stay inside one sentence can declare this as a rule (see Layer Constraints), and a split then deletes any of its relations that would cross the new boundary.

4.2.2. Token Layer Hierarchy

Although all token layers must depend on a text layer, a token layer may also declare another token layer as its parent token layer. As with the parent text layer, this must be done at time of layer creation and it is immutable. This models nesting between levels of tokenization, which is commonly needed for most approaches to interlinear glossed text, where morphemes are nested within whole words which are nested within sentences.

Nesting is expressed purely through code-point offsets: a child token belongs to the parent token whose extent contains it. There is no separate parent pointer on each token to keep in sync. These rules apply to nested token layers:

  • The parent layer must belong to the same text layer as the child, so they index into the same text.

  • The parent layer must be non-overlapping or partitioning—its tokens must not overlap, so that each child has exactly one containing parent. (An any parent is rejected.)

  • A nested layer may not itself be partitioning.

  • Every token in the nested layer must be contained within some parent token. A token that escapes its parent, or that spans across two parents, is rejected.

  • Tokens on a nested layer may not be zero-width—a zero-width token sitting on a parent boundary would belong to two parents at once. (For morphemes that have no clean surface segmentation, see Non-concatenative morphology below.)

Hierarchical containment and overlap mode provide complementary guarantees. Hierarchical containment is used to restrict the ranges in which tokens for that layer may occur: that is to say, only within the extant tokens in the parent layer. On the other hand, the overlap mode dictates how a layer’s own tokens may occur within that range relative to each other.

Here is the canonical three-level IGT hierarchy for the text "dogs run", assuming tl is a text layer and text_id is its text:

# Three token layers: sentences contain words contain morphemes. Only the root
# (sentence) layer may be partitioning. Words are non-overlapping; morphemes are
# `any` so that fused forms (overlapping morphemes) are representable too -- see
# "Non-concatenative morphology" below.
sentences = client.token_layers.create(
    tl, "Sentences", overlap_mode="partitioning")["id"]
words = client.token_layers.create(
    tl, "Words", overlap_mode="non-overlapping",
    parent_token_layer_id=sentences)["id"]
morphemes = client.token_layers.create(
    tl, "Morphemes", overlap_mode="any",
    parent_token_layer_id=words)["id"]

# One sentence covering the whole text
client.tokens.bulk_create([
    {"token_layer_id": sentences, "text": text_id, "begin": 0, "end": 8}
])

# Two words, with a gap at index 4 for the space
client.tokens.bulk_create([
    {"token_layer_id": words, "text": text_id, "begin": 0, "end": 4},
    {"token_layer_id": words, "text": text_id, "begin": 5, "end": 8},
])

# Morphemes within each word: "dog|s", then "run"
client.tokens.bulk_create([
    {"token_layer_id": morphemes, "text": text_id, "begin": 0, "end": 3},
    {"token_layer_id": morphemes, "text": text_id, "begin": 3, "end": 4},
    {"token_layer_id": morphemes, "text": text_id, "begin": 5, "end": 8},
])
Non-concatenative morphology

The morphemes in the example could be seen as purely concatenative: each is a contiguous slice of its word, and together they cover it without gaps. Plenty of morphology does not decompose so cleanly. The French contraction aux, for instance, is analyzed as the preposition à plus the article les, yet neither morpheme corresponds to a contiguous slice of the surface string "aux".

Because the morpheme layer above is any, it accommodates this directly—this is exactly why a morpheme layer is usually any rather than non-overlapping. Represent a fused form with overlapping morphemes that share the fused span, ordered with precedence:

# "aux" is the word token [0,3]; the morpheme layer's overlap-mode is `any`
client.tokens.bulk_create([
    # à
    {"token_layer_id": morphemes, "text": text_id,
     "begin": 0, "end": 3, "precedence": 0},
    # les
    {"token_layer_id": morphemes, "text": text_id,
     "begin": 0, "end": 3, "precedence": 1},
])

Both morphemes span the whole surface form, so the overlap between them is itself the signal that the correspondence is non-concatenative: an agglutinative word yields disjoint, contiguous morphemes, while a fused word yields morphemes that share a span. precedence records their linear order (à before les).

Zero-width morphemes are not an alternative here: tokens on a nested layer may not be zero-width, for the disambiguation reason given above. Anchor the fused morphemes to the whole word’s span, as shown.

4.2.3. Cascading Edits

Plaid’s general philosophy is to make destructive, cascading changes where necessary to preserve its invariants—recall that deleting a token deletes the spans which depended on it. Editing a parent token follows the same philosophy: structural operations cascade down the hierarchy so that nesting always remains valid.

  • Deleting a parent token deletes every token nested within it (and, transitively, their spans, relations, and vocab-links). Deleting a word deletes its morphemes, while leaving the word’s siblings and their morphemes untouched.

  • Splitting a parent at an offset also splits any descendant that straddles that offset, so a nested token is never bisected. (A split that lands exactly on an existing child boundary needs no child split; the children simply re-home into the two new parents by offset.)

  • Shifting a boundary (or otherwise resizing a parent) cascades according to the layer’s mode. On a partitioning parent, the neighbor grows to absorb the freed region, and any descendant straddling the moved boundary is split so that its outer half re-homes to the neighbor. On a non-overlapping or any parent, descendants that retain no overlap with the new extent are deleted—including every child when the parent is shrunk to zero width—and descendants straddling the new edge are trimmed to fit.

  • Merging two parent tokens yields a token spanning both, which therefore still contains all of their descendants—nothing is orphaned.

  • Editing the text body cascades here too: as tokens shrink, grow, or disappear along with the characters they cover (per the core constraints above), nested tokens are clipped consistently, so a child never ends up outside its parent.

Because all of these are enforced and cascaded server-side, you can rely on the hierarchy remaining internally consistent no matter how the document is edited, including under concurrent edits by multiple users.

4.3. Strict Mode

Multiple users may edit the same document simultaneously, and in some cases, undesirable conflicts may occur as users fail to take into account each other’s work. Suppose, for example, that one user is editing a sentence’s lemmas, and the other is editing a sentence’s POS tags. If the lemma editor doesn’t know that a certain POS tag has changed, they might make the wrong decision about which lemma to assign. Plaid clients' optional strict mode causes edits to fail when anyone other than the current client has made an edit. Consider this exact scenario in code:

await client1.spans.update(lemmaSpanOneId, "lemmaOne")
await client2.spans.update(posTagSpanTwoId, "posTagTwo")
// Works fine
await client1.spans.update(lemmaSpanTwoId, "lemmaTwo")

When client 1 executes the second lemma span’s value, unless they happened to have loaded the document anew after client 2’s change, they will not be aware of the new POS tag for the second word.

Strict mode causes requests to fail when anyone other than the client in strict mode, the same user in another client included, has edited the document since that client last read or wrote it. If client 1 had initiated strict mode at the beginning, then the second request would have failed:

await client1.documents.get(documentId) // strict mode checks against the version last read
client1.enterStrictMode(documentId)
await client1.spans.update(lemmaSpanOneId, "lemmaOne")
await client2.spans.update(posTagSpanTwoId, "posTagTwo")
// Fails with HTTP 409, since client 2 made a change
await client1.spans.update(lemmaSpanTwoId, "lemmaTwo")
// Exit strict mode when desired
client1.exitStrictMode()

This failure gives client 1 the opportunity to reload the document only when it is necessary, allowing them to reconsider the current state of the document before making changes.

The same 409 answers a versioned write whose target is gone: a gloss on a word, or a relation from a span, that another user deleted after client 1 read the document. A versioned write that names nothing at all, such as an empty bulk create, is a 400.

4.4. Locking

Sometimes a more heavyweight solution is needed. A lock gives a user exclusive permission to write to a document, preventing all other users from writing to it. Locks have a 60 second expiration timer by default, and they may be released early or renewed by either explicit renewal or any write to the locked document. Consider:

await client2.documents.checkLock(documentId);
// => HTTP 204
const { lockId } = await client1.documents.acquireLock(documentId);
// -> { lockId: "0b6f…", userId: "client1", expiresAt: 1752260966446 }
await client2.documents.checkLock(documentId);
// -> { userId: "client1", expiresAt: 1752260966446 }
await client1.documents.renewLock(documentId, lockId);
// -> { lockId: "0b6f…", userId: "client1", expiresAt: 1752260996446 }
await client1.documents.releaseLock(documentId, lockId);
// -> HTTP 204
await client2.documents.checkLock(documentId);
// -> HTTP 204

A lock belongs to the acquire that took it, which answers with a lockId. Only that id renews or releases the lock, and every other acquire is refused with HTTP 423, even one by the same user. Writes need no lock id: the holder’s writes pass and renew the lock, and anyone else’s are refused with 423.

A lock covers writes to the document itself. A change made at the level of a project or vocabulary, such as merging a vocabulary entry the document links to, does not wait for it. A holder that must not write over such a change can work in strict mode as well.

Locks are useful for situations where a concurrent edit by another user could yield an invalid state with respect to data integrity constraints beyond what is enforced in Plaid’s core. They should be used only where necessary in order to minimize the risk of invariant violations stemming from concurrent modifications.

Both clients offer a block that takes the lock, renews it while the work runs, and releases it at the end, even on an error:

await client.documents.locked(documentId, async () => {
  // delete + recreate this document's tokens
});
with client.documents.locked(document_id):
    rebuild_tokens(client, document_id)  # delete + recreate this document's tokens

If the lock is lost partway through, the block stops the work rather than letting it write.

4.5. Atomic Batches

Finally, you may also submit multiple requests in batches. Batches are atomic meaning that we guarantee that either they will all succeed or all fail. This is a very useful guarantee whenever you have sophisticated data integrity requirements that must be orchestrated using more than one request. One batch request holds at most 1,000 operations, and the clients send a larger batch as several requests, each atomic on its own.

A batch is a separate object from the client. Writes made on a batch are queued and sent together, while writes made on the client go out at once.

Here is an example of how to submit a batch using the JavaScript client:

try {
    const results = await client.batched(async (b) => {
        b.documents.update(documentIdOne, "New Name for Doc 1");
        b.documents.update(documentIdTwo, "New Name for Doc 2");
    });
    console.log("Batch success!")
    for (const response of results) {
        console.log(response);
    }
} catch (e) {
    console.error(`Batch failed: ${e}`)
}

Or in Python, where the results land on the batch because a context manager cannot return a value:

try:
    with client.batched() as b:
        b.documents.update(document_id_one, "New Name for Doc 1")
        b.documents.update(document_id_two, "New Name for Doc 2")
    print("Batch success!")
    for response in b.results:
        print(response)
except Exception as e:
    print(f"Batch failed: {e}")

A write in a batch can use the id of something an earlier write in the same batch creates. b.ref() stands for the id of the write queued last, and b.ref(n) for that of write n, counting from 0:

with client.batched() as b:
    b.vocab_items.create(vocab_id, "dog")
    b.vocab_links.create(b.ref(), [token_id])

The server runs a batch as one transaction, and no other write can proceed meanwhile, so keep a batch to the work that has to be atomic. A few things cannot go in a batch, such as uploads, and the clients refuse them before sending. A lock call made on a batch is sent at once.

4.5.1. Bulk updates

When many writes are all of one kind, the bulk endpoints are much faster than a batch of single writes. bulk_create and bulk_delete exist for tokens, spans, relations, vocab links and vocab items, and bulk_update for tokens, spans, relations and vocab items:

client.spans.bulk_update([
    {"id": span_one, "value": "NOUN", "metadata": [{"op": "set", "path": ["prov"], "value": "inferred"}]},
    {"id": span_two, "metadata": [{"op": "set", "path": ["provConfirmed"], "value": True}]},
])

A bulk delete skips ids that are already gone. A bulk update with an id that does not exist is refused whole.

4.6. Retrying a write

A write’s answer can be lost after the server stored it, for instance when the connection drops or the laptop goes to sleep. Sending the write again must then not store it twice. Both clients take care of this, using the two tools below.

4.6.1. Ids you mint

Every create of project data takes an optional id, so the client can choose a new row’s id before sending it. A create sent again with the same id is refused with 409 and "error": "id-taken", even if the row was deleted meanwhile, so a late retry never brings back a deleted row. The id must be a UUIDv7, which both clients mint (uuidv7() in JavaScript, uuid7() in Python):

const id = uuidv7();
await client.spans.create(layerId, [tokenId], "NOUN", undefined, undefined, { id });
span_id = uuid7()
client.spans.create(layer_id, [token_id], "NOUN", id=span_id)

4.6.2. The Idempotency-Key header

Most writes may carry an Idempotency-Key header. If the same user sends the same request again under the same key within 24 hours, the server answers with the stored answer and writes nothing, marking it Idempotent-Replayed: true. The same key on a different request is refused with 422.

Both clients send a key on every write that takes one, and send a write again under the same key when no answer arrives. wasReplayed(answer) (was_replayed in Python) tells whether an answer was a replay of an earlier send.

4.7. Trust

For all constraints which you might attempt to enforce using the three mechanisms described above, there is, of course, nothing stopping a malicious user from circumventing them and submitting changes which invalidate your formalism-specific data integrity constraints. For example, if you have a document where you want every token to carry exactly one span, a malicious user could simply circumvent your UI code entirely and craft a malicious request to e.g. delete the span on a token, leaving it with none. Since there is no server-side validation for this constraint, the server will happily execute it and advance into a database state that does not violate any core data integrity constraints but does violate your formalism-specific constraints.

This is a fundamental limitation of Plaid which was deliberately adopted in order to support frontend-only development. You therefore must only grant write privileges only to users who you trust not to circumvent client-side validation guardrails. Fortunately, we think this is not an onerous imposition in most real-world circumstances.

Note
The token layer constraints described in Token Layer Constraints and the rules described in Layer Constraints are the exception to this caveat. Because they are enforced on the server, they cannot be circumvented by a malicious client, and they hold no matter how a request is crafted.

5. Layer Interoperability

Plaid’s goal is to provide infrastructure that will allow you to point multiple apps at the same project, with data being shared across apps as much as possible. For example, someone might gloss their texts in an interlinear (IGT) editor and, on the very same documents, build a dependency treebank in a Universal Dependencies (UD) editor. Done well, this is close to magical: no import/export is needed to work on the same data across two different apps. Done poorly, the two apps will quietly corrupt each other’s data. In this chapter, we consider this very scenario as an example, and describe a discipline to be followed by app developers to ensure that apps play as well as they can with each other.

5.1. Structural Sharing

For most linguistic annotation applications built on top of Plaid, the layers can be divided into two categories. First are substrate layers: these are the primary text layer and some token layers, including the token layer for sentences, words, and maybe morphemes. There are many different apps that would be interested in working with these.

Then there are annotation layers: these hold information that probably only exactly one app cares about, such as a Universal Dependencies part-of-speech tag. These annotations typically do not need to be shared between apps.

Now, recall from Metadata and Config that each app writes its configuration under its own namespace. This configuration can help two different apps, such as one used for UD and another used for IGT, to track only the annotations each cares about: the UD app would use configuration cues to know where to look for its tag and dependency layers, and similarly, the IGT app can use configuration to find the gloss layers it uses, which can happily sit side by side on the same tokens. For annotations, each app is oblivious to the other, because each app only ever looks at the layers it created.

The substrate is the part that the apps do share, but even here they will only agree up to a point. A sentence is a sentence in both apps, and the same is usually true of a word. Below the word, though, the apps may have different needs, and one and the same word may be divided in two different ways. In the UD app, the layer beneath the word holds CoNLL-U syntactic words, which sometimes disagree with the CoNLL-U tokens above them: the English token teacher’s, for example, is a multiword token holding the two syntactic words teacher and 's. In the IGT app, the layer that is closest in purpose to UD’s syntactic word layer holds morphemes, which split on all morphology, so that this same word, teacher’s, is instead divided into the three morphemes teach, -er, and 's. It is therefore not always possible to share all token layers across apps.

A reasonable rule of thumb, then, is to share the substrate as far down as the apps agree, and to let them branch below that point. In our example, the two apps would share the text, the sentence layer, and the word layer, and then, beneath the word layer, the UD app would add its syntactic-word layer while the IGT app adds its morpheme layer. (Recall from Token Layer Hierarchy that a single token layer may have more than one child layer, so both of these can sit side by side under the shared word layer.) The apps agree all the way down to the word, and only then go their separate ways. If two apps genuinely do mean the same thing by a layer, so that their morpheme segmentation really is identical, then they are free to share it as well; but this should always be a deliberate decision rather than a silent assumption.

5.2. Roles

Suppose that a third app comes along that has no knowledge of the UD or IGT apps, and that it wants to find out what the extant layers mean. While there is no perfect solution to this, Plaid provides a small shared vocabulary of roles recorded in each layer’s configuration under the plaid namespace. A layer carries at most one role, and the inventory is deliberately small:

Role Layer What the layer holds

baseline

text layer

The primary text being annotated, over which all the other layers are built.

sentence

token layer

Sentence segmentation, normally a partitioning layer at the root.

word

token layer

The orthographic word: a whitespace-and-punctuation tokenization (CoNLL-U’s token).

syntactic-word

token layer

Grammatical words beneath the orthographic word (CoNLL-U’s word, e.g. the pieces of a multiword token like teacher’s).

morpheme

token layer

Morpheme segmentation beneath the word.

time-alignment

token layer

Tokens aligned to a media timeline, for audio or video.

Plaid itself acts on one of these roles only: a space typed inside a word, and a request that splits a token, both deal out the tokens of a syntactic-word layer under it. These names are rather blessed conventional names for communicating the meaning of a layer that apps are expected to honor when present. The app that first sets the project up might record the roles of its sentence and word layers like this:

await client.tokenLayers.setConfig(sentenceLayerId, "plaid", "role", "sentence");
await client.tokenLayers.setConfig(wordLayerId, "plaid", "role", "word");

The plaid namespace is reserved for conventions that every app agrees on, which makes it the right home for these shared role tags. An app’s own private configuration, such as its vocabularies, its colors, and the meanings of its fields, belongs instead under that app’s own namespace, where no other app will mistake it for shared information. Keeping these two kinds of configuration apart is what allows several apps to coexist on one project.

We also advise app developers to consider the absence of a role tag to be a meaningful signal: if a layer is not tagged with a role, then as a matter of convention, no other app should read or modify it.

5.2.1. Text direction

One further shared convention lives in a document’s metadata, under the same reserved namespace:

await client.documents.patchMetadata(docId, [
  { op: "set", path: ["plaid", "textDirection"], value: "rtl" },
]);

The value is "ltr" or "rtl", and without it a client takes the direction from the document’s own text. Set it where the text cannot speak for itself, such as a right-to-left language written in a Latin transliteration. A client that walks metadata generically, such as an exporter, should skip the plaid namespace.

5.3. Provenance

Annotation projects often mix human work with machine output, and it matters which is which: machine output is cheap to replace, while human judgment must never be silently destroyed. Some projects also take in work that is a person’s but not yet checked, such as a student’s glosses. Plaid records these distinctions in a few metadata keys, which put every annotation in one of four states.

prov records where a value came from: no key for a trusted person (a verifier), inferred for a machine, and contributed for a person whose work is reviewed (a contributor). provConfirmed: true records that a verifier has vouched for the value. provSource names who produced it, such as service:my-tagger or user:<userId>.

State Metadata Meaning

human

no provenance keys

A verifier made it.

machine

prov: "inferred", provSource: "<producer>"

An algorithm or service made it, and nobody has vouched for it yet.

contributed

prov: "contributed", provSource: "user:<userId>"

A contributor made it, and no verifier has vouched for it yet.

verified

either of the above, plus provConfirmed: true

A verifier confirmed or edited it.

Who counts as a contributor is the project’s decision, recorded under the plaid config namespace:

"config": { "plaid": { "review": { "users": ["ann@example.org"], "roles": ["writer"] } } }

users names individual people and roles names whole project roles, and an absent review means nobody is reviewed. This is separate from permissions, so a trusted collaborator can be a writer without review and a student can be a maintainer with it. Both clients offer isReviewed / is_reviewed for reading it.

5.3.1. Prediction details

A machine writer may also record how confident it was. provProb is a single number between 0 and 1, the probability of the value it chose, and since it is flat a query can sort by it to put the least confident output first. It should be calibrated on held-out data, so that values given 70% are right about 70% of the time. A writer that cannot calibrate leaves provProb out. Everything else, such as the top few alternatives, goes in the open map provDetail. The alternatives a producer weighed for a value go in provDetail.valueProbs, a map from each value to its probability, and on a gloss provDetail.valueLinks can name the lexicon entry each alternative stands for, as a list of {value, item, prob} (a list, since two senses may share a gloss). A morpheme’s alternative forms go in provDetail.formProbs the same way. Plaid IGT lists these alternatives under Alt+↓ on the cell and in the lexicon popover, and offers each alternative together with its entry as one choice. Plaid UD lists a tag’s or a relation’s alternatives first in its selector, with their percentages.

The apps draw a heavier, darker underline under a machine value whose provProb is below the project’s confidence threshold, and its tooltip gives the percentage. A maintainer sets the threshold on Plaid IGT’s Services settings page, and it is stored as a number between 0 and 1:

"config": { "plaid": { "confidence": { "threshold": 0.5 } } }

Without it the threshold is 0.5. Both clients read it with readConfidenceThreshold / read_confidence_threshold and test a value with isDoubtful / is_doubtful.

A machine writer records what made a prediction in provDetail.model, such as stanza==1.11.0, and every one records the version of its own code in provDetail.version. provDetail describes the original prediction and is kept after a person edits the value. provProb is the producer’s probability for the value the entity holds, however a person entered it. When a person picks or types another value that the producer weighed, the app writes that value’s probability from provDetail.valueProbs (provDetail.formProbs for a morpheme’s form), and when the producer never weighed the new value the app removes provProb. So once provConfirmed is set, do not present provProb as the machine’s confidence in the value.

5.3.2. The write contract

Every machine writer, whether a service, a built-in rule or an importer, is expected to follow three rules:

  1. A machine writer may freely create new material and freely replace machine-made, unverified material, its own or another machine’s.

  2. A machine writer must never modify or delete human-made, contributed or verified material unless the user asks it to for that run. For services, this is a boolean overwrite parameter, off by default.

  3. A verifier’s edit of a machine-made or contributed annotation verifies it, and so does an explicit accept. A contributor’s edit of any annotation marks it contributed.

Rule 3 lets an editor show proposals apart from checked work, with the marking fading as a verifier works through the material. Both clients offer helpers: stampInferred(source) / stamp_inferred(source) builds the keys to add to everything a service creates, and isProtected(metadata) / is_protected(metadata) checks rule 2. On the reading side, provState / prov_state tells which of the four states an entity is in.

Word and sentence tokens are not stamped. The run’s entry in the audit log says what made them.

5.4. Reconciliation Across Apps

In Data Integrity, we discussed various ways of maintaining app-specific data model invariants within a certain app. Here, we consider the more difficult matter of one app updating a shared substrate layer in a way that invalidates a different app’s data model invariants.

Suppose that a sentence has been annotated for UD, and that later on, someone opens the same sentence in the IGT app and splits the sentence. From the IGT app’s perspective, this is a totally unproblematic change to make. However, on the UD side, it is a strict requirement that all dependency relations must never cross sentence boundaries, and this change in the IGT app at the shared sentence tokenization layer has caused an invariant violation in the UD app.

Plaid cannot know about this invariant by itself, because it is application-specific. The UD app can tell it, though, by declaring the rule on its relation layer (see Layer Constraints), and then the split deletes the crossing relations itself. Any invariant that no declared rule expresses is left to the UD app’s reader.

The discipline that resolves this is easy to state, and in the interest of smooth interoperability, we encourage all app developers to adhere to it:

Opening a document writes nothing. An app reads what another app left as its own model says it should be, and the first real edit there stores that reading.

The IGT and UD apps work this way. A word made in the IGT app has no UD word under it, so the UD app shows it as the one UD word over that word, and the first value, lemma or arc a person gives it makes that word in the same batch as the edit. An invariant the app cannot read its way past is reported to the person, who decides what to do.

Writing nothing on open keeps a reader’s visit out of the audit log, lets several people open a document at once without their pages racing to repair it, and leaves the stored data as the other app wrote it until someone edits it. A reader that computes the answer also stays right when the document changes on screen, for instance in response to a real-time update delivered through Real-time Messaging.

There are two exceptions. The first is setting a project up for an app. When a person sets a project made in another app up for UD, the set-up makes the UD word of every word that has none, after it makes UD’s layers, since Grew search, Rewrite and the project’s word counts read stored words. A maintainer can finish a set-up that stopped before it made them all, from the project’s setup page or its Documents tab. A word another app adds after that is read as a stand-in until it is edited.

The second is the UMR app. When a person who can edit a document opens it in the UMR app, the app repairs what another app’s edits left of its graphs before the document can be edited, in one operation of kind repair. A node whose word another app split is put on the half that holds its letters, a variable whose sentence number no longer matches its sentence is renumbered, and a node left outside every sentence is removed. History names what the repair changed.

5.5. Layer Constraints

An app can declare rules on the token, span and relation layers it uses, and Plaid enforces them on every write, whoever makes it. A rule the UD app declares on its dependency layer holds for a split made in the IGT app, a service, or a script alike.

Type Layer Rule

max-in-degree

relation

A span is the target of at most max relations of the layer.

acyclic

relation

The layer’s relations in a document form no cycle. With self-loops: true a relation from a span to itself is allowed. Relations whose value is in except-values are not counted.

same-ancestor

relation

No relation runs from one token of token-layer to another, such as across sentences.

single-span

span

A token is in at most one span of the layer.

value-set

span or relation

A value is in the list values. With delimiters the value is split on each of those characters and every part must be listed, or with parts: "first" only the first. An empty value always passes.

coextensive

token

A token has exactly the extent of a token of its parent layer.

single-link

token

A token is the only token of at most one vocabulary link.

5.5.1. Declaring

Each app declares its rules under its own namespace, so two apps sharing a layer do not overwrite each other’s lists. Declaring needs maintainer privileges, and a list the layer’s data already breaks is refused with 422.

client.relation_layers.set_constraints(deps_layer_id, "ud", [
    {"type": "max-in-degree", "max": 1},
    {"type": "same-ancestor", "token_layer": sentence_layer_id},
])

check_constraints lists what a list would break, without writing anything: the first 100 violations, or as many as limit asks, and violation_count counts them all. repair_constraints fixes what can be fixed and lists the rest, and a writer may run it on a single document by passing document. An app uses this when it sets up a project, or a setting changes the rules it wants, on layers whose data does not hold them yet.

5.5.2. Rule violations

A write that breaks a rule on what it wrote itself, such as a second head for a word, is refused with 422 and stores nothing. The answer lists each violation, and violationsOf(err) (violations_of in Python) reads them. Rules are checked once at the end of a batch, so a batch may pass through a broken state on the way.

A write that breaks a rule on rows it did not write, by changing what they rest on, is not refused. Instead Plaid applies the rule’s remedy: a sentence split deletes the dependency relations left crossing it, and a word merge deletes the analyses left without a word of their extent. max-in-degree, acyclic and value-set have no remedy, so a write that would break them elsewhere is refused.

5.5.3. Values a list does not govern

Three kinds of write may keep a value outside a value-set list: an unverified machine value, a write made by an import, and a restore or copy of what was stored before. Any later edit of such a value is checked against the list. Imported relations are likewise exempt from acyclic and max-in-degree, so a file that has a cycle can still be read in.

5.5.4. Values a list would refuse

Before declaring a value-set list, an app can ask which stored values of a span or relation layer it would refuse. refused_values (refusedValues in JavaScript) takes the list as a declaration does, with values, delimiters and parts, and writes nothing. It reads the stored values the way a declaration of the list would, with the exemptions above, so what it names is exactly what a declaration would be refused for. Anyone who may read the layer may ask.

client.span_layers.refused_values(pos_layer_id, {"values": ["N", "V", "ADJ"]}, limit=5)
# {'count': 12, 'value_count': 2,
#  'values': [{'value': 'NOUN', 'parts': ['NOUN'], 'count': 9},
#             {'value': 'X', 'parts': ['X'], 'count': 3}]}

count is how many stored values the list would refuse, value_count how many distinct ones, and values the most frequent of them with the parts the list does not hold, 20 unless limit (at most 100) says otherwise.

5.6. References in Metadata

An app sometimes needs to point from one of its rows to another row of the same document, in a way no relation can express. Metadata is the place for this, and Plaid keeps such a reference pointing at the right row when rows are restored or a document is copied. Store the reference as the bare id, as a whole string value.

6. Querying

Plaid offers a rich query system for powering searches across one or even multiple projects. Here, we consider a few examples to give you a feel for it. For full details, see the query language reference. The examples below use the JavaScript client and the small "Fido barks" project from An Example.

6.1. A First Query

A simple query specifies what to find, and lists the conditions a match must satisfy in where:

const posLayerId = /* ... some UUID ...*/;
// every span on the pos layer whose value is NOUN
const { results } = await client.query({
  find: ["?s"],
  where: [["span", "?s", { layer: posLayerId, value: "NOUN" }]],
});
// results: one row per match, each carrying the ids you asked to find
//   here, the single span over "Fido"

Read that where line as a sentence: "a span, call it ?s, on the pos layer, whose value is NOUN." The ?s is a variable — any name you invent, written with a ? — and every match comes back as a row holding one id per name in find.

6.2. Relationship Clauses

Here is a more interesting query, where we search for a noun immediately followed by a verb:

await client.query({
  find: ["?noun", "?verb"],
  where: [
    ["span", "?noun", { layer: posLayerId, value: "NOUN" }],
    ["span", "?verb", { layer: posLayerId, value: "VERB" }],
    ["covers", "?noun", "?t1"],   // the noun span covers a token ?t1
    ["covers", "?verb", "?t2"],   // the verb span covers a token ?t2
    ["precedes", "?t1", "?t2"],   // and ?t1 falls immediately before ?t2
  ],
});
// in "Fido barks": one match — the NOUN over "Fido" and the VERB over "barks"

When multiple clauses are provided to where, all must be satisfied in order to yield a match. In the latter three clauses, we use relationship clauses, which are used to enforce structural relations between entities. covers requires that a span have a token as one of its constituent tokens, and precedes requires that one token come immediately before another in linear order. These are two of several relationship clauses; the reference catalogs the rest.

6.3. Return Types

By default a match is a tuple of ids. Different return types are available, identified by return:

// a single number — the total match count
await client.query({ find: ["?s"], where: [["span", "?s", { layer: posLayerId }]],
                     return: "count" });
// -> { return: "count", count: 2, truncated: false }   // two pos spans

// or full entities, exactly as a GET would return them
await client.query({ find: ["?s"], where: [["span", "?s", { layer: posLayerId, value: "NOUN" }]],
                     return: "entities" });
// -> results[0][0] is { id, layer, document, value: "NOUN", tokens: [...] }

6.4. Visibility

Normal permissions restrictions apply to queries: a query submitted by a user is only executed over data that user has permission to read. In order to further restrict visibility to e.g. a single project, the scope parameter may be provided in a query:

await client.query({
  find: ["?s"],
  where: [["span", "?s", { layer: posLayerId, value: "NOUN" }]],
  scope: { projectIds: ["<a project's id>"] },
});

Queries always read the current state of your data: there is no time-travel parameter (one passed as asOf is refused). For a document’s history, see the Audit Log.

6.5. Further Reading

See the query language reference for comprehensive information about how queries work.

7. Configuration

Plaid is configured through a single TOML file. On first launch Plaid writes config.toml into the data directory with every setting at its default. To change something, edit that file and restart the server.

To load a config file from another location, pass --config on the command line or set the PLAID_CONFIG environment variable:

$ java -jar plaid.jar --config /etc/plaid/config.toml
# or
$ PLAID_CONFIG=/etc/plaid/config.toml java -jar plaid.jar

8. Deployment

This section is for whoever runs Plaid for other people: a course, a research group, a documentation project. Plaid is one process with one SQLite database, so a small machine that stays on is enough.

8.1. Requirements

  • A machine that stays on, reachable by your users, running Linux (the jar itself runs anywhere Java does, but these notes are for Linux).

  • JDK 21 or later.

  • plaid-<version>.jar from the releases page, saved as plaid.jar.

  • A hostname, if you want HTTPS (and you do, for anything that leaves your desk).

Everything Plaid stores lives in one directory, data/, in the directory Plaid is started from: the database, uploaded recordings, nightly backups, the configuration file and the signing secret. Keep that directory private to the account that runs Plaid.

8.2. First launch

Create a user for the service and a home for the jar, then run it once by hand:

$ sudo useradd --system --home /srv/plaid --create-home plaid
$ sudo -u plaid mkdir -p /srv/plaid && sudo cp plaid.jar /srv/plaid/
$ cd /srv/plaid && sudo -u plaid java -jar plaid.jar

On first launch Plaid writes data/config.toml and asks for the administrator’s email address and password, which the administrator signs in with. It then starts listening on port 8080. Stop it with Ctrl+C once you have seen it start, and let a service manager run it from here on.

8.3. Running it as a service

A systemd unit keeps Plaid running across reboots and collects its logs:

# /etc/systemd/system/plaid.service
[Unit]
Description=Plaid annotation server
After=network.target

[Service]
User=plaid
WorkingDirectory=/srv/plaid
ExecStart=/usr/bin/java -jar /srv/plaid/plaid.jar
Restart=on-failure

[Install]
WantedBy=multi-user.target
$ sudo systemctl enable --now plaid
$ sudo journalctl -u plaid -f      # follow the log

Leave [logging] file unset so the log goes to journald, which keeps and rotates it.

8.4. HTTPS

Plaid speaks plain HTTP, so put a reverse proxy in front of it to provide HTTPS. Serve the apps and the API through the same hostname, and no CORS configuration is needed. Caddy does this in a few lines and fetches its certificates itself:

plaid.example.edu {
    reverse_proxy localhost:8080
    request_body {
        max_size 220MB
    }
}

Any proxy must allow request bodies as large as [media] max_file_size_mb, since recordings are uploaded whole, and must pass the server’s event streams through without buffering. Caddy does both with the configuration above, while nginx needs client_max_body_size raised and proxy_buffering off on the /api/ location.

The apps' pages carry a Content-Security-Policy header, set by [server] content_security_policy in config.toml. A proxy passes it on unchanged unless told otherwise. A policy the proxy adds of its own applies on top of it.

8.5. Backups

Every night at [backup] time (03:00 by default) Plaid writes a snapshot of the database to data/backups/ and keeps the most recent [backup] retention of them. Each backup is a zip holding one file, plaid.db, the database. A snapshot that fails SQLite’s quick check is not kept: the log says so at ERROR, and the backups from earlier nights stay as they are. An admin can also take a backup by hand from the admin pages.

The backups protect against a bad edit but not against a lost machine. Copy the newest backup somewhere else on a schedule, together with data/media/, data/config.toml and data/jwt-secret.txt, which the zip does not hold.

8.6. Restoring from a backup

A restore needs a backup zip and your off-box copy of data/media/, config.toml and jwt-secret.txt. Without jwt-secret.txt, Plaid makes a new signing secret, so everyone is signed out and every named API token stops working. Without config.toml, Plaid starts with every setting at its default, the port among them.

  1. Stop the service: sudo systemctl stop plaid.

  2. Move the old data/ directory aside, if there is one, rather than deleting it.

  3. Unzip the newest plaid-backup-<stamp>.zip (from the old data/backups/ or your off-box copy) into an empty directory. It holds one file, plaid.db.

  4. Check the file, in the directory it was unzipped into. For a sound database this prints ok, and if it prints anything else, use the next newest backup.

    $ python3 -c "import sqlite3; print(sqlite3.connect('file:plaid.db?mode=ro', uri=True).execute('pragma integrity_check').fetchone()[0])"
  5. Make a new data/ directory and move plaid.db into it, with no plaid.db-wal or plaid.db-shm beside it.

  6. Put back data/media/, data/config.toml and data/jwt-secret.txt from your off-box copy, and make the secret readable by its owner only: chmod 600 data/jwt-secret.txt.

  7. Give everything in data/ to the service’s user: sudo chown -R plaid:plaid data.

  8. Start the service: sudo systemctl start plaid. A newer jar runs its migrations on startup.

  9. Check curl http://localhost:8080/health, with the port from config.toml. It answers "ok": true with the version. Then open a document with a recording in each app and play the recording.

8.7. Upgrading

Take a backup and copy it off the machine, replace plaid.jar, and restart the service. Database migrations run on startup, and the bundled apps are part of the jar, so they upgrade with it. Then reload every open browser tab of Plaid, so that no page of the old version goes on talking to the new one. Do the same after going back to an older version.

An upgraded database cannot be used by the version before it. To go back to that version, restore the backup taken before the upgrade, then start it.

8.7.1. Interrupted migrations

The server starts only once every migration has run. A start stopped while it migrates the database, by a crash, a power cut or a restart, leaves the migration unfinished. The next start runs it again when the migration is written to be resumed, as the one that gives every vocabulary a project is, and says so in its log. For any other migration the server refuses to start, and its log names the migration and the two ways on: restore the backup taken before upgrading and start again, or remove the mark the stopped start left, with python3 -c "import sqlite3; c = sqlite3.connect('data/plaid.db'); c.execute('DELETE FROM schema_migrations WHERE id = -1'); c.commit()", and start again. The server also refuses to start on a database file it cannot write. Do not run the older version on a database whose upgrade stopped partway. Restore the backup instead.

8.7.2. Vocabularies without a project

The version that gives every vocabulary a project refuses to start while a vocabulary exists that no project links. The script that gives each one a project is plaid-core/scripts/vocab_home_orphans.py in the Plaid repository. It runs from a checkout of the repository at the new version’s tag, with Python 3.11 or later and its requests package. It acts as an admin through a named API token kept in a file: an admin mints one on the profile page of any app (see Tokens and sign-out) and pastes it into the file. Before upgrading, with the old version still running:

  1. From the top of the checkout, run the script with the server’s address and the token’s file. It lists every vocabulary no project links and writes nothing.

    $ PYTHONPATH=plaid-client-py/src python3 plaid-core/scripts/vocab_home_orphans.py https://plaid.example.edu token.txt
  2. If it lists any, run the same line with --apply at the end. Each such vocabulary gets a project of its own, named after it, whose maintainers are the vocabulary’s maintainers. Then run it once more without --apply: it must list none.

  3. Take the backup and copy it off the machine.

  4. Optionally, try the new version on a copy of that backup, as Trying an upgrade on a copy says.

  5. Replace the jar and restart.

A deactivated account is not made a maintainer. A vocabulary whose maintainers are all deactivated gets a project maintained only by the admin who ran the script, and the script says so.

The upgrade gives each vocabulary its project and logs every choice. A vocabulary that several projects link goes to the one whose maintainers include the most of its maintainers, then the one with the most links into it, then the one with the smallest id. A maintainer of the vocabulary who does not maintain that project is made a maintainer of it. If the choice is wrong, move the vocabulary afterwards. A migration that rebuilds the audit log keeps the server from answering for a few minutes and cannot be undone, which is why the copy goes off the machine first. The database file does not shrink after such a rebuild: to return the space, stop the service, run python3 -c "import sqlite3; sqlite3.connect('data/plaid.db').execute('VACUUM')", and start it again.

8.7.3. Trying an upgrade on a copy

A copy of the database is started as a server of its own, in a new directory outside the live server’s and on a port of its own. Never start it beside the live database. A server deletes every recording in the media/ directory next to its database that its database does not know, so a copy started in the live data/ directory deletes the recordings uploaded since the backup. Without a port of its own, it also takes the live server’s.

$ sudo mkdir -p /srv/plaid-trial/data
$ sudo unzip /srv/plaid/data/backups/plaid-backup-<stamp>.zip -d /srv/plaid-trial/data
$ printf '[server]\nport = 8090\n' | sudo tee /srv/plaid-trial/data/config.toml
$ sudo cp plaid-<version>.jar /srv/plaid-trial/
$ sudo chown -R plaid:plaid /srv/plaid-trial
$ cd /srv/plaid-trial && sudo -u plaid java -jar plaid-<version>.jar

Read its log as it starts and migrates, stop it with Ctrl+C, and delete /srv/plaid-trial.

8.8. Bringing in a class

The administrator signs in first, at plaid.example.edu/igt/, and creates the instructor’s account on the Users tab of the admin area. The instructor creates the project and, in its Settings > Access, makes invitation links: each grants a role on the project, is good for a set number of uses and days, and lets whoever opens it choose their own password. One link with fifteen uses is a class. A student who already has an account signs in from the link and joins with the same role.

A student who forgets a password gets a password-reset link from the administrator, made in Settings > Access. The IGT guide’s Sharing and Permissions section covers the roles.

8.9. Reading the log

Plaid writes one line per request, plus a line for anything that went wrong:

POST /api/v1/documents 201 34ms user=ada@example.edu ip=10.0.0.14
GET /api/v1/projects/019ecd83-7617-7501/audit 403 2ms user=sam@example.edu ip=10.0.0.31
POST /api/v1/login 401 220ms user=- ip=203.0.113.9

user= names the account that made the request, and a request signed with a named API token also carries token=<id>. The same lines can be read on the Logs tab of the IGT app’s admin area, which needs no log file.

8.10. Settings worth a look

  • [auth] jwt_ttl_seconds: how long a login lasts. The default is thirty days, and a shared lab machine may want a day. Every token lifetime setting must be a positive whole number of seconds, or the server refuses to start.

  • [media] max_file_size_mb: the largest recording a document may carry. The default is 200 MB and the ceiling is 2047. Raise the proxy’s limit with it. The server holds an upload in memory whole, so give Java about the limit’s size for every upload under way at once (java -Xmx2g -jar plaid.jar). The server warns at startup when the limit is more than half its memory.

  • [user_data] max_value_mb: the largest private data value, which is how long an assistant conversation can grow before it is full. The default is 5 MB. [server] max_json_body_mb must be at least 1 more, or the server refuses to start, so raise the two together.

  • [api] expose_openapi: turn off if the API description should not be public.

  • [backup] directory: point it at a mounted volume or a synced folder to get backups off the machine for free.

9. Development

For information on how to work on Plaid itself (not an app which uses Plaid), see the development guide.