Create a Tweede Kamer Connector

A Tweede Kamer connector ingests Dutch parliamentary open data from the Tweede Kamer OData v5 portal (gegevensmagazijn) into the Zeta Alpha platform. This guide shows you how to create and configure one.

Info: The portal is public and unauthenticated, so this connector needs no credentials. Everything specific to your setup lives in the content source configuration below.

Prerequisites

Before you begin, ensure you have:

1. Access to the Zeta Alpha Platform UI
2. A tenant created
3. An index created whose document_fields_configuration declares the fields listed under Indexed metadata

Step 1: Choose a corpus

The portal exposes two things worth indexing, and a content source crawls exactly one of them. Create one content source per corpus so each gets its own schedule, sync cursor and job history.

• documents: the Document entities, moties (motions), brieven (letters), verslagen (reports), and the other PDFs and office files. Each becomes one searchable document, enriched with the metadata of the Zaak (the parliamentary case it belongs to) behind it, including how that case was decided.
• meetings: the vergaderverslagen (meeting reports). Each sitting becomes one plain-text document rendered from its transcript, with speaker attribution and vote tallies preserved. A sitting is a Vergadering, of the full chamber (Plenair) or of a committee (Commissie).

Both corpora write the same metadata field names, so they facet together in one index.

Step 2: Create the basic configuration

• corpus: (string, required) "documents" or "meetings", as above.
• odata_base_url: (string, optional) Base URL of the portal. Defaults to "https://gegevensmagazijn.tweedekamer.nl/OData/v5".
• from_date: (string, optional) Oldest publication date (documents) or sitting date (meetings) to crawl, e.g. "2016-01-01". Without it the whole archive is crawled.
• until_date: (string, optional) Newest date to crawl, exclusive. Combine with from_date for a window.
• documents_configuration: (object, optional) Documents only. Filters that apply to the documents corpus: content_types (array of strings, e.g. ["application/pdf"]) and soorten (array of strings). "Soorten" is Dutch for kinds, the plural of the portal's Soort field, which holds the document kind: Motie (motion), Brief regering (government letter), Bijlage (attachment), Schriftelijke vragen (written questions), Amendement (amendment to a bill), Stenogram (verbatim record of a debate), and 100 more. Values are matched exactly and must be written in Dutch as the portal spells them; list them all with Document?$apply=groupby((Soort),aggregate($count as n)). Either filter unset crawls everything supported. The meetings corpus has no filters of its own.
• request_delay_seconds: (number, optional) Pause between requests to the portal, default 0, meaning no pause. The portal publishes no rate limit and does not throttle a single sequential crawler; if it ever starts returning 429 or 503 the connector backs off and honours Retry-After on its own, and you can set a pause here as well. What does need setting is fetch_concurrency to 1: meeting transcripts are downloaded per sitting and the platform runs ten fetch batches at once by default.

Note: There is deliberately no "crawl only N documents" option. A full sync reconciles deletions by removing everything it did not see, so a deliberately shortened crawl would delete the rest of the corpus. Validate a configuration on a scratch index, or with the connector's files run mode, which performs the same run against the portal but writes the result to a directory instead of an index.

Example Configuration

{
    "name": "Tweede Kamer documents",
    "description": "Dutch parliamentary documents from the open data portal",
    "is_indexable": true,
    "connector": "tweede_kamer",
    "connector_configuration": {
        "tweede_kamer": {
            "corpus": "documents",
            "documents_configuration": {
                "soorten": [
                    "Motie"
                ]
            },
            "from_date": "2016-01-01",
            "allow_access_rights": [
                {
                    "name": "public"
                }
            ]
        }
    }
}

Step 3: Configure access rights

The portal's data is public, so there is no per-document ACL to read: the platform stamps the content source's configured access rights onto every document it ingests.

• allow_access_rights: (array of objects, optional) Users with any of these rights can access the documents. If not set, no user can retrieve them. Set [{"name": "public"}] to make the corpus visible to everyone with tenant access.
• deny_access_rights: (array of objects, optional) Users with these rights cannot access the documents.

Note: Access rights are stamped when a document is written, so changing them applies only to documents ingested afterwards. A full sync does not re-stamp the rest: it re-fetches only documents that changed at the source, and leaves the others untouched with their old rights. Re-applying new rights across an existing corpus needs those documents re-ingested deliberately; ask the platform team rather than assuming a full sync did it.

Step 4: Leave field mappings empty

The connector ships a default field map that turns the portal's field names into the index fields listed under Indexed metadata, so leave field_mappings empty and you get those names.

Set it only to index under different names. A mapping you configure replaces the defaults for the whole dotted root it touches, so one DCMI.title entry suppresses every other DCMI.* default. A mapping also keeps only the fields it names, so anything you leave out is dropped rather than passed through: map every field you want, not just the ones you are renaming.

Step 5: Create the content source

1. Navigate to your tenant and click View next to your target index
2. Click View under Content Sources for the index
3. Click Create Content Source
4. Paste your JSON configuration
5. Click Submit

Step 6: Schedule the sync

This connector runs on scheduled tasks rather than a single crawl schedule. Configure both per content source:

• content_delta picks up everything changed at the portal since the last run, including documents the portal has removed. Suggested: every 6 hours for documents, hourly for meetings (a live debate's transcript is republished every 20-60 minutes).
• content_full re-enumerates the whole corpus, re-fetches only what changed, and reconciles deletions. Suggested: weekly.

Run content_full once before enabling the delta schedule: it performs the initial load and seeds the delta cursor.

Indexed metadata

Both corpora emit the same vocabulary. Every field must be declared in the index's document_fields_configuration or ingestion rejects the batch.

Field	Contents
DCMI.title, DCMI.abstract, DCMI.creator, DCMI.created, DCMI.modified, DCMI.identifier, DCMI.language, DCMI.source	Standard bibliographic metadata. DCMI.creator is the signatories (documents) or the speakers (meetings)
tk_entiteit	Document or Vergadering, which separates the two corpora in one index
document_type	Always document. Not a reserved field, so it has to be declared like the rest
tk_soort	Kind: the document kind (Motie, Bijlage, …), or the sitting kind, Plenair (full chamber) or Commissie (committee)
tk_vergaderjaar, tk_kamer, tk_volgnummer, tk_aanhangselnummer	Vergaderjaar is the parliamentary year, September to September (2023-2024); kamer the chamber; volgnummer the sequence number within a dossier; aanhangselnummer the number in the appendix to the proceedings
tk_actor_namen, tk_fracties, tk_actor_functies, tk_actor_relaties	Who signed the document, or who spoke at the sitting. A fractie is a party group, the members of one party sitting together; functies are their roles and relaties how they relate to the document (ondertekenaar = signatory)
tk_zaak_nummers, tk_zaak_soorten, tk_zaak_onderwerpen, tk_citeertitel	The Zaak, the parliamentary case behind the document: its number, kind, subject (onderwerp) and citeertitel, the short citation title of the act or bill it concerns. tk_zaak_nummers is the join key: filtering on one case number returns every document of that case
tk_afgedaan, tk_kabinetsappreciatie, tk_besluit_soorten, tk_besluit_teksten	Case outcome. Afgedaan means settled, so whether the case is closed; kabinetsappreciatie is the cabinet's stated position on a motion; the besluit fields are the decision taken, including the vote result
tk_kamerstukdossier_nummers, tk_document_guid	Kamerstukdossier numbers, the numbered paper series the document is filed under, and the source entity GUID
tk_verslag_soort, tk_activiteit_nummers, tk_commissies, tk_commissie_afkortingen	Meetings only. A verslag is the transcript, and its soort says which version was indexed: Eindpublicatie (corrected final), Tussenpublicatie (interim) or Voorpublicatie (pre-publication). Then the agenda activity numbers, and the committees involved with their abbreviations

Note: About half the document corpus has no case link at the source (bijlagen/attachments, stenogrammen/verbatim records, convocaties/meeting notices), so those documents having no tk_zaak_* fields is expected, not a gap. tk_afgedaan, tk_besluit_soorten and tk_actor_relaties are always empty on meeting documents.

Crawling behaviour

• Documents are indexed by reference: the connector hands the platform the portal's file URL and the pipeline downloads and extracts the file.
• Meetings are indexed as text. The portal serves transcripts as VLOS XML, which the pipeline cannot extract, so the connector parses it here into a readable transcript: a header, one section per agenda item, speaker turns as Name (party): text, votes rendered as a tally, and a closing speaker roster.
• A sitting's transcript is republished many times while a debate runs, and once more when the corrected version lands. All of them update the same document, and the corrected final replaces the interim text in place.
• Document links stay stable across portal changes. A document's identity is derived from the portal's own entity GUID rather than from its URL, so a future endpoint change re-points the links without re-issuing the corpus.