New iteration of the ActivityPub delivery implementation and interface.
Advantages over previous interface:
* When sending a ByteString body, the sender is explicitly passed as a
parameter instead of JSON-parsing it out of the ByteString
* Clear 3 operations provided: Send, Resend and Forward
* Support for per-actor keys
* Actor-type-specific functions (e.g. deliverRemoteDB_D) removed
* Only the most high-level API is exposed to Activity handler code, making
handler code more concise and clear
Also added in this patch:
* Foundation for per-actor key support
* 1 key per actor allowed in DB
* Disabled C2S and S2S handlers now un-exported for clarity
* Audience and capability parsing automatically done for all C2S handlers
* Audience and activity composition automatically done for Vervis.Client
builder functions
Caveats:
* Actor documents still don't link to their per-actor keys; that should be the
last piece to complete per-actor key support
* No moderation and anti-spam tools yet
* Delivery API doesn't yet have good integration of persistence layer, e.g.
activity is separately encoded into bytestring for DB and for HTTP; this will
be improved in the next iteration
* Periodic delivery now done in 3 separate steps, running sequentially; it
simplifies the code, but may be changed for efficiency/robustness in the next
iterations
* Periodic delivery collects per-actor keys in a
1-DB-transaction-for-each-delivery fashion, rather than grabbing them in the
big Esqueleto query (or keeping the signed output in the DB; this isn't done
currently to allow for smooth actor key renewal)
* No support yet in the API for delivery where the actor key has already been
fetched, rather than doing a DB transaction to grab it; such support would be
just an optimization, so it's low-priority, but will be added in later
iterations
This is such a huge patch, it's probably impossible to tell what it does by
looking at the code. One thing is clear: It changes *everything* :P so here's
an overview:
- There are now 5 types of actors, each having its own top-level route
- So projects, repos, etc. are no longer "under" sharers
- Actor routes are now based on their KeyHashid, there are no "idents" anymore,
i.e. URLs look random and don't contain user or repo names
- No sharers anymore; people and groups are distinct entities not sharing a
common namespace or anything like that
- Project has been renamed to Deck and it simply means a ticket tracker; repos
are no longer "under" projects
- In addition to Person, Group, Repo and Deck, there's a new actor type Loom,
which is a patch tracker; i.e. Repo actors don't manage MRs anymore
- All C2S and S2S is temporarily disabled, because huge changes to the whole
code are required and I'll do them gradually in the next patches
- Since form-based actions are implemented using C2S, they're disabled as well,
so Vervis is now essentially read-only
- Some views have been temporarily removed, e.g. repo history and commit view
- A huge set of DB migrations has been added to adapt the DB to these changes;
I haven't tested them yet on a read DB so there may be errors there; I'll fix
them in the next patches if I find any (probably going to test on the main
instance where Vervis itself is hosted...)
- Some modules got tech upgrades, e.g. LocalActor became a higher-kinded type
and a similar pattern is probably relevant for several other types
- There's an 'Actor' entity in the DB schema now, and all 5 actor types use it
for common things like inbox and outbox
- Although inbox and outbox are used only by Actor, so essentially could be
removed, I haven't removed them; that's because I wonder if at some point
users can have a tree of inboxes much like in email; I don't have an excuse
for Outbox, but anyway, leaving them as is for now
- Workflows, roles and collaborators are partially removed/unused until I
figure out a sane federated way to provide these features
- Since repo routes don't contain a "sharer" anymore, SSH URIs are now simpler,
they already look like user@host/repo regardless of who "controls" that repo
To be honest, this is a huge patch that changes tons of stuff and probably
should have been broken up into small changes. But I already had the codebase
not building, so... just did all of this at once :P
Basically this patch does the following:
- DB migrations for ticket dependency related tables, e.g. allowing a remote
author and a remote child
- Allowing S2S handlers to provide an async continued processing function,
which is executed and the result then added to the debug page
- Most UI and functionality related to ticket deps is disabled, new
implementation being added gradually via ActivityPub
- Improvements to AP tools, e.g. allow to specify multiple hosts for approved
forwarding when sending out an activity, and allow to specify audience of
software-authored activities using a convenient human-friendly structure
- Implementation of S2S sharerOfferDepF which creates a dependency under a
sharer-hosted ticket/patch and sends back an Accept
Delivery of an activity into local inboxes is being done using custom local
functions. Each C2S or S2S handler has its own specific variant for this.
As part of the ongoing refactoring and evolution of the federation code, I
implemented a general-purpose local delivery function: It takes a
LocalRecipientSet and simply delivers to everyone, no handler-specific
assumptions or limitations.
To limit the recipient set according to handler specific rules, just
filter/adapt/edit it before passing to the delivery function.
The function isn't exported yet, but the existing 'deliverLocal' that delivers
only to actors and to author's followers is now implemented via the new
general-purpose function. I hope that's a step towards doing all the local
delivery using this one function, simplifying the complicated federation
code.
This patch contains migrations that require that there are no follow records.
If you have any, the migration will (hopefully) fail and you'll need to
manually delete any follow records you have. In the next patch I'll try to add
automatic following on the pseudo-client side by running both e.g. createNoteC
and followC in the same POST request handler.
Currently it's a paged Collection where the items are merely URIs. This could
be changed to have actual Commit objects as items; for that we need to examine
the whole thing with the LogEntry type and the Patch type and have an
AP-friendly log item representation, but without commit diffs.
FedURIs, until now, have been requiring HTTPS, and no port number, and DNS
internet domain names. This works just fine on the forge fediverse, but it
makes local dev builds much less useful.
This patch introduces URI types that have a type tag specifying one of 2 modes:
- `Dev`: Works with URIs like `http://localhost:3000/s/fr33`
- `Fed`: Works with URIs like `https://dev.community/s/fr33`
This should allow even to run multiple federating instances for development,
without needing TLS or reverse proxies or editing the hosts files or anything
like that.
This patch also disables the ability to specify deps when creating a ticket,
because those deps won't be in the ticket object anymore. Instead of coding a
workaround and getting complications later, I just disabled that thing. It
wasn't really being used by anyone anyway.
highligher2 doesn't have a JSON syntax and the JS lexer seems to be failing,
not sure exactly why yet. To have an alternative, I'm adding a Skylighting
option.
- Fork and async are no longer class methods, which simplifies things a lot and
allows for many more trivial instances, much like with MonadHandler. Fork and
async are still available, but instead of unnecessarily being class methods,
they are now provided as follows: You can fork and async a worker (no more
fork/async for handler, because I never actually need that, and not sure
there's ever a need for that in general), and you can do that from any
MonadSite. So, you can fork or async a worker from a Handler, from a Worker,
from a ReaderT on top of them e.g. inside runDB, and so on.
- Following the simplification, new MonadSite instances are provided, so far
just the ones in actual use in the code. ReaderT, ExceptT and lazy RWST. More
can be added easily. Oh, and WidgetFor got an instance too.
In particular, this change means there's no usage of `forkHandler` anymore, at
all. I wonder if it ever makes a difference to `forkWorker` versus
`forkHandler`. Like, does it cause memory leaks or anything. I guess could
check why `forkResource` etc. is good for in `forkHandler` implementation. I
suppose if needed, I could fix possible memory leaks in `forkWorker`.
I added a migration that creates an ugly fake OutboxItem for messages that
don't have one. I'll try to turn it into a real one. And then very possibly
remove the whole ugly migration, replacing it with addFielfRefRequiredEmpty,
which should work for empty instances.
Worker is enough and seems much simpler. forkHandler does stuff with
forkResourceT and more stuff that I don't exactly understand and which may
involve more resource allocation. I guess forkWorker would generally be the
preferred approach, and there are bugs with delivery leading to sudden
CPU/memory peaks forcing me to kill the process. Maybe not related, just
mentioning it ^_^
I wrote a function handleOutboxNote that's supposed to do the whole outbox POST
handler process. There's an outbox item table in the DB now, I adapted things
in various source files. Ticket comment federation work is still in progress.
