# Hansard Reaction Notation Concordance

This dataset is a concordance of reaction-like notation extracted from the UK parliamentary record. Its rows contain such things as reporter-written laughter, cheers, interruptions, cries, and generic-member interjections, plus a short preceding context and source provenance. It is designed to be used independently of the Artificial Wasteland page.

The unit is a notation cue in the published record. It is not an acoustic event, a joke, a verified quotation, or evidence that adjacent words caused a reaction.

## Files

- `cues-YYYY.jsonl`: one JSON object per line, sharded by the year written in the source path or filename.
- `schema.json`: JSON Schema draft 2020-12 for each cue row.
- `sources.json`: acquisition endpoints, URL resolution rules, digest status, retrieval-time status, and licence status.
- `validate.mjs`: a dependency-free validator and per-field null-rate report.

Empty year shards are valid. They mean the frozen predicate returned no rows for the material scanned, not that no reaction occurred.

## Frozen predicate

The predicate was fixed before counting. Matching is case-insensitive.

For both regimes, a bracketed span must be 2 through 200 characters before markup flattening and must contain a match for this regular expression:

```text
(laugh|interrupt|cheer|hon\.?\s*mem[bk]|cries?\s+of|hear,?\s*hear|ironical|uproar|disorder|noise|oh[,!.]|shame|withdraw|order[,!.]|applause|groan|jeer)
```

For ParlParse XML only, a second route includes an entire `<speech>` element when its `speakername` starts with an optional `a` or `an`, followed by `hon`, an optional full stop, whitespace, and `mem` followed by `b` or `k`. It also includes an exact case-insensitive speaker label of `Hon. Members` or `An Hon. Member`. These rows have `kind: "attributed_speech"`.

This deliberately loose `mem[bk]` fragment admits OCR forms such as `MEMBEKS`. OCR corruption is evidence about digitisation and is not silently repaired.

## Two acquisition regimes

### 1803 through 1918: seeded sample

This regime samples up to 15 sitting days from every source-labelled year, using seed `20260903 + year`. It scans all section links discovered for the Commons and Lords on each selected day. The sampling frame is the sitting-day index returned by `api.parliament.uk`; the inclusion rate for a year is `min(15, N) / N`, where `N` is the number of indexed sitting days in that year. Years with no indexed days have no defined rate. The stored sampling plan is `_victorian-plan.json` in the research archive, not a census.

The sample has sampling uncertainty, unequal rates between years, and possible coverage error in the archive indexes. Do not treat raw yearly sample counts as directly comparable totals.

There is a known, severe source-date fault in this regime. For example, the source URL labelled `22 August 1804` contains `Navy Estimates, 1894-5`, a Secretary for Scotland, and an 1890 Act. The public row preserves the source path in `date`; it does not certify that date. Chronological analysis of the sampled regime requires an independent date audit first.

The 1909 adoption of the Official Report is another hard seam inside the sampled era. Analyses must mark it as a break, not draw one uninterrupted 1803 to 2026 trend.

### 1919 onward: XML census

This regime scans every file named by the mySociety ParlParse Commons debates XML directory. At the harvest described by the protocol, the directory listed 20,084 files from 4 February 1919 through 2 September 2026. Call a release complete only if its harvest summary reports all listed files scanned and an empty failure list. This is a census of files available through that collection and predicate, not a census of chamber sound or every kind of parliamentary proceeding.

The seam between the HTML sample and XML census is 4 February 1919. The regimes have different source formats, discovery rules, fields, and coverage. Compare them only as separately labelled series.

## Exclusions and transformations

The following are outside the dataset or changed by extraction:

- A bracketed span is excluded if it is shorter than 2 or longer than 200 characters in the matched source markup.
- A bracketed span is excluded unless it matches at least one frozen term above. The broad terms can also create false positives.
- Unbracketed reactions are excluded, except for XML speeches admitted by the generic-member speaker predicate.
- Unbracketed interjections attributed to a named member are excluded by the generic-member route.
- XML outside lowercase `<speech>...</speech>` matches is excluded. Attributes must use the double-quoted form recognized by the harvester.
- XML sources outside the ParlParse `scrapedxml/debates/` listing are excluded. This omits written answers, many committees, and other parliamentary collections.
- Historic HTML is limited to Commons and Lords section links discovered beneath sampled sitting-day pages. Other page types and undiscovered links are excluded.
- A historic day is excluded when its index does not contain the literal marker `Sitting of`.
- A source request is abandoned after three failures. The XML harvester reports failed filenames in its final summary. The HTML harvester does not retain a failure ledger, so its missing pages cannot be distinguished from pages with no matches.
- Markup is removed and whitespace is collapsed. Only a short, fixed list of character references is decoded. Other references can remain visible in text.
- `cue_text` is truncated to 300 code units for bracketed rows and 400 for attributed XML speeches. `preceding_text` is truncated to its final 400 code units.
- Brackets are removed from bracketed `cue_text` values. They are not restored in this release.
- The latest preceding XML `major-heading` is copied into `heading`. Historic HTML headings are not retained.
- Empty source speaker labels remain empty strings. Missing regime-specific fields remain absent and count as null in the validator report.
- Source dates, speaker labels, spellings, entities, and OCR mistakes are preserved rather than silently corrected.
- Re-running either current harvester into a non-empty shard appends duplicates. A release must be made from clean output or deduplicated with that fact recorded.
- The current harvesters do not retain rows they cannot parse. Consequently there is no honest unparsed-row file to ship from this run. Absence from such a file must not be interpreted as zero parse failures.

## Provenance

For an XML row, join `file` to the `record_url_template` in `sources.json`. The `speech_id` names its containing `<speech>` element and `colnum` records that element's column. The field called `sha256` is only the first 16 hexadecimal characters of the source-byte SHA-256 because the current harvester discarded the rest. It is useful as a check prefix but is not a full digest.

For a historic HTML row, `url` is the exact section URL and `section` repeats its final path component. The harvester did not record response bytes, a hash, a retrieval timestamp, a column, a speech id, or a bracket offset. Those values are unknown. `sources.json` uses `null` and `UNKNOWN` for them.

These are material provenance limits. A future release should capture full source SHA-256 values, UTC retrieval timestamps, and a within-document locator. It should declare a method break rather than retroactively filling old rows.

## Loading in Python

From this directory, a year shard loads in three lines:

```python
import json
with open("cues-1919.jsonl", encoding="utf-8") as source:
    rows = [json.loads(line) for line in source]
```

For large shards, iterate over `source` instead of retaining the list.

## Validation

With Node 20 or later:

```sh
node validate.mjs
```

The default discovers `cues-*.jsonl` beside the script. File and directory arguments are also accepted. It parses every non-empty line, validates each object against `schema.json`, reports the missing-or-null rate for every declared field, and exits non-zero if JSON parsing or schema validation fails. It also exits non-zero when it finds no cue shards.

The corruption check below copies one shipped shard to a temporary directory, changes the first row's `kind`, and runs the validator against that copy:

```sh
tmp_dir=$(mktemp -d)
cp cues-1919.jsonl "$tmp_dir/cues-corrupt.jsonl"
node -e 'const fs=require("fs"),p=process.argv[1],x=fs.readFileSync(p,"utf8").split("\n");const r=JSON.parse(x[0]);r.kind="not_a_cue";x[0]=JSON.stringify(r);fs.writeFileSync(p,x.join("\n"))' "$tmp_dir/cues-corrupt.jsonl"
node validate.mjs "$tmp_dir/cues-corrupt.jsonl"
```

The final command prints `FAIL`, identifies line 1, and exits with status 1. Removing the temporary directory is safe after inspection.

## Licence

The underlying parliamentary record is available under the [Open Parliament Licence v3.0](https://www.parliament.uk/site-information/copyright/open-parliament-licence/). Its required attribution string is exactly:

> Contains Parliamentary information licensed under the Open Parliament Licence v3.0.

The licence does not imply that Parliament endorses this dataset, its makers, its analysis, or any reuse. Credit mySociety ParlParse as the derived source for the XML acquisition path. The licence status of mySociety's transformation and packaging is `UNKNOWN` here because this release does not confirm a separate term for it.

## Citation

Suggested citation:

> Artificial Wasteland. (2026). *Hansard Reaction Notation Concordance* (Version 1.0.0) [Data set]. https://artwaste.land/data/hansard-reactions/

Also include the exact parliamentary attribution string above and your access date. For row-level citation, add the row's `url` for HTML or the resolved XML file URL plus `speech_id` and `colnum`.

## LIMITATIONS

This counts notation, not noise. It measures how often the frozen parser found reporter-written forms in the available record. It does not measure how much the House laughed, cheered, interrupted, or made any other sound.

Do not claim that preceding words caused a cue. The dataset shows adjacency in an edited record only. Do not call a cue funny or assume humour; parliamentary laughter can be hostile, tribal, or directed at someone vulnerable. Do not claim words were spoken exactly as printed; Hansard is an edited, substantially verbatim report. Do not treat any associated audio as historical sound; any audio must be labelled as a synthetic score. Do not invent quotations, and do not generate likenesses of real politicians from these records.

Coverage changes at the 1909 Official Report seam and again at the 1919 acquisition seam. The early data is a sample with known source-date faults. The later data is a source-collection census only when all listed files succeeded. OCR errors, parser false positives, parser false negatives, truncation, incomplete entity decoding, missing provenance, and unlogged HTML fetch failures remain in the release.