~ knotless Search/
↑↓ move↵ open⇧↵ all results

grain.social/grain-support-agent

grain's support agent: triages mentions and reports, investigates bugs, proposes fixes

1 collaborator
0
0

Keep anything about a child-safety report from models

Open
Open chadtmiller.com wants to merge into main from grain.social/grain-support-agent:child-safety
1 reviewer 1 asked for changes

The support agent now follows grain-mcp's child-safety rule: what a child-safety report is about never reaches a model, and only a person acts on it.

The test

childSafety(label, reasonType) in apps/server/src/child-safety.ts is grain's labels::child_safety: the label is csae, or the reason type is one that labels::label_for files under csae. That means every tools.ozone.report.defs#reasonChildSafety* and no other reason type. A report is also in scope when grain-serve's social.grain.admin.getReviewSubject gives its account childSafety anything but false. That flag is set by a child-safety report on the account or its records, in any state, or a csae label, negated or not. grain-mcp's clear() makes the same call. If the lookup fails or the item has no account, the report is treated as child safety. Accounts found child safety are cached for the life of the process. Clear accounts are cached for 30s. Lookup errors are never cached.

Guards

  • Intake (reports.ts): the report is stored whole for the dashboard and goes straight to needs_review. report (the brief) holds only its ids, its date and grain-mcp's HUMAN line.
  • Workers (main.ts): the triage worker checks before Clef, and the investigate worker checks before a session starts. Resumed investigations, turns and fixes are checked as well.
  • investigate(), resumeInvestigation(), conversation turns and every support-mcp tool (itemFor) check the item again at the moment of use. They refuse with HUMAN. When an account has become child safety since intake, the item is marked from then on.
  • Prompts read the item through modelText(). Briefs now get an account's earlier reports from getReviewSubject, run through shownReport().
  • SQL: report and classification are denied by the authorizer and hidden from list_tables and describe_table. Tool descriptions and prompts were updated to match.
  • Moderation: act(item, m, by). The "model" path refuses csae, any child-safety item or account, and any open child-safety report on the subject. The dashboard's confirm button uses "person" and still works.
  • Dashboard: messages and investigate return 409 with HUMAN. The thread shows HUMAN as an event.
  • Existing items: a migration marks queued items whose own label or reason type is child safety, and drops any brief written for them.

Tests

test/child-safety.test.ts uses fixtures only: a csae-labeled classifier report on a photo, a child-safety reason type, innocuous reports on accounts with a prior csae report or a csae label, a failed lookup, a clean account, and an account that becomes child safety later. They check that no prompt, brief or modelText contains the reason, subject URI, image link, handle or gallery title. They also check that nothing reached a model host, that investigate throws before opening a session, that act as model refuses while act as person works, and that labelNames leaves out csae. grain-db.test.ts covers the SQL denial. npm test -w server: 37/37 pass; tsc --noEmit is clean.

🤖 Generated with Claude Code

Asked for changes

This pull keeps child-safety reports away from models. The test is in child-safety.ts. It's checked at intake, in the workers, in investigate and its resume, in conversation turns, in every support-mcp tool and in the dashboard routes. Model-initiated moderation is refused, and SQL access to report and classification is closed. Briefs now get an account's earlier reports from getReviewSubject. The guards are thorough. Unknown or failed lookups count as child safety, the migration's LIKE patterns match what the poller writes for both classifier and person reports, and the tests cover the paths the body describes.

One gap stops it delivering its main promise. The SQL tool can still read the label table. Any investigation or conversation session, including one on a clean item, can run SELECT uri, did FROM label WHERE val = 'csae' and get the account DIDs and record URIs of every csae-labeled subject in grain. moderationPrompt already leaves labels out for a child-safety account because "they name its records", and the README says a model never sees "its subject's record". The SQL path undoes both.

The authorizer only allows or denies whole tables and columns, so it can't filter csae rows. Close label along with report and classification, since the brief already gets labels from getReviewSubject and shows them only when it's safe to. Then update the "tables that matter here" bullet and the tool descriptions to match, and add a test to grain-db.test.ts. If takedown stores free-text reasons, check it the same way.

apps/server/src/grain-db.tsline 30
]);
/** Reports and what the classifiers said: a model reads them only as the brief shows them. */
export const MODERATION_TABLES = new Set(["report", "classification"]);
review.knotless.dev

label should be here too. Its rows include every csae label with the uri of the labeled record or account. Today any session's sql_query can read SELECT uri FROM label WHERE val = 'csae' and get exactly the subjects this pull withholds from briefs. The authorizer can't filter rows, so closing the table is the reliable fix. Briefs already get labels from getReviewSubject, via the account object in moderationPrompt, which leaves them out for child-safety accounts.

apps/server/src/investigate.tsline 108
You can also query grain-serve's index, the appview's database, read-only with the grain-db tools. grain/crates/grain-serve/src/index/schema.rs describes every table. The tables that matter here:
- label: labels grain gave out on the account or its records (uri is the bare DID for an account, an at:// URI for a record; neg = 1 is a retraction).
review.knotless.dev

This bullet still sends the agent to the label table, which leaks csae subjects (see the comment on grain-db.ts). Once label is closed, remove this bullet and rely on the labels in the getReviewSubject block above.

Changes requested

1 review asking for changes