Skip to main content

Create a SharePoint Connector

A SharePoint connector enables you to ingest data from your SharePoint instance (and OneDrive) into the Zeta Alpha platform. This guide shows you how to create and configure a SharePoint connector for your data ingestion workflows, including options that impact crawling performance.

The connector ingests document library files and OneNote notebooks (each notebook section becomes its own document) by default. When enabled, it also ingests SharePoint lists (each list item becomes its own document) and SharePoint site pages (each page becomes its own document).

Info: This guide presents an example configuration for a SharePoint connector. For a complete set of configuration options, see the SharePoint Connector Configuration Reference.

Prerequisites​

Before you begin, ensure you have:

  1. Access to the Zeta Alpha Platform UI
  2. A tenant created
  3. An index created
  4. Microsoft App (SharePoint/OneDrive) credentials (refer to the tutorial Configure Microsoft App Access for detailed instructions)
  5. To render OneNote notebooks (onenote.mode: visual), the delegated Notes.Read.All Microsoft Graph permission on that same app with admin consent granted, and the username and password of an account that can open the notebooks, since the OneNote API accepts user-delegated tokens only (see Required permissions and authentication below and Optional: OneNote notebooks in the Microsoft app access tutorial)
  6. To ingest SharePoint lists, the Sites.FullControl.All application permission on the Office 365 SharePoint Online API (not Microsoft Graph), and certificate or username/password credentials: reading item access rights goes through the SharePoint REST API, which rejects client-secret tokens (see Required permissions and credentials below and SharePoint API permissions in the Microsoft app access tutorial)

Step 1: Create the SharePoint Basic Configuration​

To create a SharePoint connector, define a configuration file with the following basic fields:

  • is_document_owner: (boolean) Indicates whether this connector "owns" the crawled documents. When set to true, other connectors cannot crawl the same documents.
  • schedule: (string, optional) The schedule to crawl the SharePoint instance (cron format).
  • content_source_name: (string) The name that identifies the content source in the index.
  • certificate_credentials: (object, optional) The application credentials for certificate-based authentication. This is the recommended method — it supports all connector features including incremental permission sync, private keys never leave your infrastructure, and certificates can be rotated without updating Azure AD:
    • client_id: The client ID of your SharePoint application
    • tenant_id: The tenant ID of your SharePoint application
    • certificate_private_key: The PEM-encoded private key of the certificate
    • certificate_thumbprint: (optional) The SHA-1 thumbprint of the certificate. Not required when certificate_public_key is provided, since MSAL computes it automatically. Only needed if you omit the public certificate.
    • certificate_public_key: (optional) The PEM-encoded public certificate. Not required when certificate_thumbprint is provided. Required when using Subject Name/Issuer (SNI) authentication.
  • access_credentials: (object, optional) The application credentials using a client secret. Discouraged — client secret authentication does not support incremental permission detection, requiring a full access rights crawl instead which is slower and uses more API requests:
    • client_id: The client ID of your SharePoint application
    • client_secret: The client secret of your SharePoint application
    • tenant_id: The tenant ID of your SharePoint application
  • public_access_credentials: (object, optional) The user credentials for username/password authentication. Highly discouraged — requires storing user passwords and the account must not have MFA enabled:
    • username: The username for SharePoint access
    • password: The password for SharePoint access
    • client_secret: The client secret of your SharePoint application
    • tenant_id: The tenant ID of your SharePoint application
  • certificate_credentials: (object, optional) The application credentials for certificate-based authentication. This is more secure than client secrets and recommended for production environments:
    • client_id: The client ID of your SharePoint application
    • tenant_id: The tenant ID of your SharePoint application
    • certificate_private_key: The PEM-encoded private key of the certificate
    • certificate_thumbprint: (optional) The SHA-1 thumbprint of the certificate. Not required when certificate_public_key is provided, since MSAL computes it automatically. Only needed if you omit the public certificate.
    • certificate_public_key: (optional) The PEM-encoded public certificate. Not required when certificate_thumbprint is provided. Required when using Subject Name/Issuer (SNI) authentication.
  • logo_url: (string, optional) The URL of a logo to display on document cards when no image is extracted from the pipeline.
  • custom_metadata: (object, optional) Static key-value pairs added to every ingested document. See Content Source Custom Metadata.

Note: You must provide exactly one of certificate_credentials (certificate-based, recommended), access_credentials (client secret, discouraged), or public_access_credentials (username/password, highly discouraged).

Example Configuration​

Here is an example of a basic SharePoint connector configuration with certificate-based authentication (recommended):

{
    "name": "my_sharepoint_connector",
    "description": "My SharePoint connector for product data",
    "is_indexable": true,
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            "is_document_owner": true,
            "content_source_name": "SharePoint Files",
            "certificate_credentials": {
                "client_id": "my_client_id",
                "tenant_id": "my_tenant_id",
                "certificate_private_key": "-----BEGIN PRIVATE KEY-----\nMIIEv...\n-----END PRIVATE KEY-----",
                "certificate_thumbprint": "AB12CD34EF56...",
                "certificate_public_key": "-----BEGIN CERTIFICATE-----\nMIIC/j...\n-----END CERTIFICATE-----"
            },
            "logo_url": "https://mycompany.com/logo.png"
        }
    }
}

Alternatively, you can use client secret authentication with access_credentials, but this is discouraged because it does not support incremental permission detection. Permission updates require a full access rights crawl, which is slower and consumes more API requests:

{
    "name": "my_sharepoint_connector",
    "description": "My SharePoint connector for product data",
    "is_indexable": true,
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            "is_document_owner": true,
            "content_source_name": "SharePoint Files",
            "access_credentials": {
                "client_id": "my_client_id",
                "client_secret": "my_client_secret",
                "tenant_id": "my_tenant_id"
            },
            "logo_url": "https://mycompany.com/logo.png"
        }
    }
}

You can also use username/password authentication with public_access_credentials, but this is highly discouraged:

{
    "name": "my_sharepoint_connector",
    "description": "My SharePoint connector for product data",
    "is_indexable": true,
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            "is_document_owner": true,
            "content_source_name": "SharePoint Files",
            "public_access_credentials": {
                "username": "user@domain.com",
                "password": "user_password",
                "client_secret": "my_client_secret",
                "tenant_id": "my_tenant_id"
            },
            "logo_url": "https://mycompany.com/logo.png"
        }
    }
}

Step 2: Add Field Mapping Configuration​

When crawling SharePoint, the connector extracts document metadata and content as described in the SharePoint Connector Configuration Reference. You can map these SharePoint fields to your index fields using the field_mappings configuration.

Example Field Mappings​

The following example shows field mappings for the default index fields:

{
    ...
    "connector_configuration": {
        "sharepoint": {
        ...
        "field_mappings": [
                {
                    "content_source_field_name": "name",
                    "index_field_name": "DCMI.title"
                },
                {
                    "content_source_field_name": "last_modified_date_time",
                    "index_field_name": "DCMI.modified"
                },
                {
                    "content_source_field_name": "created_date_time",
                    "index_field_name": "DCMI.created"
                },
                {
                    "content_source_field_name": "author",
                    "index_field_name": "DCMI.creator",
                    "inner_field_mappings": [
                        {
                            "content_source_field_name": "display_name",
                            "index_field_name": "full_name"
                        },
                        {
                            "content_source_field_name": "display_name",
                            "index_field_name": "Ontology_ID"
                        }
                    ]
                },
                {
                    "content_source_field_name": "content_source_name",
                    "index_field_name": "DCMI.source"
                }
            ],
            ...
        }
    ...
    }
}

Step 3: Specify What to Crawl​

By default the connector crawls the files and OneNote notebooks of every site that its credentials can reach. Narrow the scope to the content you need: a smaller scope crawls faster and makes fewer API requests.

Available Configuration Options​

The options have two layers:

  • Scope options choose the sites. They apply to every content type.
  • Content type options choose what is crawled from those sites. File options are flat fields directly on sharepoint. OneNote, lists and pages each have a nested block under content_configuration, with its own filters. So the flat path_inclusion_regex_patterns filters files only, onenote.path_inclusion_regex_patterns filters notebook sections, and pages.path_inclusion_regex_patterns filters site pages.

The content_configuration blocks take effect only on a source that runs on scheduled_tasks (see Scheduling with Task Types). A source on the single schedule cron ignores all four blocks: it crawls files with the flat file options.

A PATCH that sends connector_configuration but leaves out the drives or onenote block keeps the stored block. If it leaves out the lists or pages block, that block resets to its default, which turns the content type off. To change a block, or to keep lists or pages on, send the block.

Scope: every content type​

  • site_paths: (array of objects, optional) The sites to crawl. Without it, the connector crawls every site that the crawl identity can see. Each object contains:
    • collection_hostname: The hostname of the SharePoint instance, such as contoso.sharepoint.com
    • site_relative_path: The path of the site, such as sites/research
  • include_sub_sites: (boolean, optional, default false) Without site_paths, also crawl the sub-sites of each site.
  • site_inclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a site's name as Microsoft Graph returns it, often the site's title, such as Research Team. A sub-site's name follows its parent's: Research Team/Lab. Only matching sites are crawled.
  • site_exclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a site's name, as for the inclusion patterns; matching sites are skipped. Exclusion takes precedence over inclusion.
  • allow_access_rights: (array of objects, optional) Access rights added to every document of this source. Each object contains:
    • name: The user's email, or the group's object id
    • type: The access-right type. Use a type that the connector records for files, so that the right matches the same users: sharepoint_user with an email, or msft_group with a Microsoft Entra group object id.
  • deny_access_rights: (array of objects, optional) Access rights denied on every document of this source. Structure is the same as allow_access_rights.

Files and notebooks can come from other places than the sites: one_drive_users and drive_ids replace the sites for files, and one_drive_users and onenote.drive_ids for notebooks. Lists and pages always come from the sites.

Files: flat fields on sharepoint​

Document library files are crawled by default. Their options are flat fields directly on sharepoint. The only file option in content_configuration is the switch that turns files off.

  • content_configuration.drives.enabled: (boolean, optional, default true) Crawl document library files. Set to false to crawl only the other content types that the source enables; they are unaffected. Files already indexed are not deleted, see Turning document libraries off.
  • one_drive_users: (array of strings, optional) Email addresses of users whose OneDrive files and notebooks are crawled, in place of the sites' files and notebooks. drive_ids takes precedence over it for files, and onenote.drive_ids for notebooks. The drive patterns apply to these users' files.
  • drive_ids: (array of strings, optional) Crawl the files of exactly these drives, in place of one_drive_users and the drives of the sites crawled. The site options, the drive patterns and include_one_drives do not apply to them; the path patterns do. Drive IDs can be obtained from the Microsoft Graph API.
  • include_one_drives: (boolean, optional, default false) Also crawl the drives named OneDrive in the sites crawled. Does not apply to one_drive_users or drive_ids, which name their drives directly.
  • drive_inclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a drive's name, such as Documents; only files in matching drives are crawled.
  • drive_exclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a drive's name; files in matching drives are skipped. Exclusion takes precedence over inclusion.
  • path_inclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a file's path in its drive; only matching files are crawled. Microsoft Graph returns the folder part percent-encoded and the file name as is, so the file Q1 report.docx in the folder Annual Reports has the path /Annual%20Reports/Q1 report.docx.
  • path_exclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a file's path, as for the inclusion patterns; matching files are skipped. Exclusion takes precedence over inclusion.
  • download_content_as_pdf: (boolean, optional) Convert Office documents to PDF before indexing them. Defaults to the connector deployment's own setting.
  • extract_sensitivity_labels: (boolean, optional, default false) Read each file's Microsoft Purview sensitivity label into confidentiality_level.
  • decrypt_protected_files: (boolean, optional, default false) Decrypt MIP-protected files so their content is indexed. Requires MIP decryption on the connector deployment.

OneNote: the content_configuration.onenote block​

OneNote is crawled by default: each section's .one file is indexed as text. A configuration without the block gets one: enabled, text, with the flat file configurations include_one_drives, drive_ids and drive and path patterns copied in. After that, the block's own drive and path fields choose the notebooks, and the flat ones apply to files only. See OneNote notebooks.

  • enabled: (boolean, optional, default true) Crawl OneNote at all. Set to false to leave every .one file out.
  • mode: (string, optional, default text) text indexes each section's .one file. visual renders each section of the notebooks in scope as one HTML document through the OneNote API instead; it requires public_access_credentials as the source's only credentials and the delegated Notes.Read.All permission, since the OneNote API rejects certificate and client-secret credentials. include_resources applies to visual only.
  • include_one_drives: (boolean, optional, default false) Also crawl the notebooks stored in drives named OneDrive in the sites crawled. Does not apply to one_drive_users or onenote.drive_ids.
  • drive_ids: (array of strings, optional) Crawl exactly the notebooks stored in these drives, in place of one_drive_users and the notebooks of the sites crawled. Independent of the flat drive_ids.
  • drive_inclusion_regex_patterns: (array of strings, optional) Regular expressions matched against the name of the drive holding a notebook; only notebooks in matching drives are crawled.
  • drive_exclusion_regex_patterns: (array of strings, optional) Regular expressions matched against the name of the drive holding a notebook; notebooks in matching drives are skipped. Exclusion takes precedence over inclusion.
  • path_inclusion_regex_patterns: (array of strings, optional) Regular expressions matched against each section's .one path in its drive, with no percent-encoding, such as /Notebooks/Team Notebook/General.one, in both modes. Only matching sections are crawled. A notebook's folder followed by /, such as ^/Notebooks/Team Notebook/, matches every section in it.
  • path_exclusion_regex_patterns: (array of strings, optional) Regular expressions matched against each section's .one path, as for the inclusion patterns; matching sections are skipped. Exclusion takes precedence over inclusion.
  • include_resources: (boolean, optional, default false) Inline the images and file attachments a section's pages embed into the section's own content. See Embedded images and attachments.

Lists: the content_configuration.lists block​

Lists are off by default. They come from the sites in scope: one_drive_users, drive_ids and the flat file filters do not apply. See SharePoint lists.

  • enabled: (boolean, optional, default false) Ingest the SharePoint lists of the sites crawled, one text/html document per list item. Requires certificate or username/password credentials. Reading each list's permissions takes the Sites.FullControl.All SharePoint permission.
  • include_hidden: (boolean, optional, default false) Also ingest lists hidden from the site contents page. This includes some lists SharePoint creates for itself, such as CSPViolationReportList, so pair it with name_inclusion_regex_patterns.
  • name_inclusion_regex_patterns: (array of strings, optional) Regular expressions matched against list display names; only matching lists are ingested.
  • name_exclusion_regex_patterns: (array of strings, optional) Regular expressions matched against list display names; matching lists are skipped. Exclusion takes precedence over inclusion.

Pages: the content_configuration.pages block​

Site pages are off by default. They come from the sites in scope: one_drive_users, drive_ids and the flat file filters do not apply. See SharePoint Site Pages.

  • enabled: (boolean, optional, default false) Ingest the published site pages and news posts of the sites crawled, each page as its own HTML document.
  • include_resources: (boolean, optional, default false) Inline the images and files uploaded into a site page into the page's own content. See Images and files on a page.
  • path_inclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a page's path /SitePages/{file name}; only matching pages are crawled. See Choosing which pages are crawled.
  • path_exclusion_regex_patterns: (array of strings, optional) Regular expressions matched against a page's path /SitePages/{file name}; matching pages are skipped. Exclusion takes precedence over inclusion.

Example Configurations​

Example 1: Crawl the files and notebooks of every site

{
    "name": "my_sharepoint_connector",
    "description": "My SharePoint connector for product data",
    "is_indexable": true,
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            "is_document_owner": true,
            "content_source_name": "SharePoint Files",
            "certificate_credentials": { "...": "..." },
            "include_sub_sites": true,
            "include_one_drives": true,
            "field_mappings": [
                ...
            ]
        }
    }
}

Example 2: Crawl the OneDrive of specific users

{
    "name": "my_sharepoint_connector",
    "description": "My SharePoint connector for product data",
    "is_indexable": true,
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            "is_document_owner": true,
            "content_source_name": "SharePoint Files",
            "certificate_credentials": { "...": "..." },
            "one_drive_users": [
                "zeta@zeta-alpha.com",
                "alpha@zeta-alpha.com"
            ],
            "field_mappings": [
                ...
            ]
        }
    }
}

Example 3: Filter each content type

The scope and file filters are flat fields. The OneNote, lists and pages filters are in their blocks, which need scheduled_tasks:

{
    "name": "my_sharepoint_connector",
    "description": "My SharePoint connector for product data",
    "is_indexable": true,
    "connector": "sharepoint",
    "scheduled_tasks": [
        { "task_type": "content_delta", "schedule": "*/15 * * * *" },
        { "task_type": "access_rights_delta", "schedule": "*/15 * * * *" },
        { "task_type": "content_full", "schedule": "0 4 * * 0" },
        { "task_type": "access_rights_full", "schedule": "0 5 * * 0" }
    ],
    "connector_configuration": {
        "sharepoint": {
            "is_document_owner": true,
            "content_source_name": "SharePoint",
            "certificate_credentials": { "...": "..." },
            "include_sub_sites": true,
            "site_exclusion_regex_patterns": ["Zeta Private Team"],
            "drive_inclusion_regex_patterns": ["^ZetaDrive$"],
            "path_inclusion_regex_patterns": ["^/ZetaFiles/.*\\.docx$", "^/ZetaFiles/.*\\.xlsx$"],
            "content_configuration": {
                "onenote": { "path_exclusion_regex_patterns": ["/Personal Notebook/"] },
                "lists": { "enabled": true, "name_exclusion_regex_patterns": ["^Scratch"] },
                "pages": { "enabled": true, "path_exclusion_regex_patterns": ["^/SitePages/Archive-"] }
            },
            "field_mappings": [
                ...
            ]
        }
    }
}

Here files come only from ZetaDrive, but notebooks are not limited to it: the flat drive and path filters do not apply to OneNote.

Step 4: Create the SharePoint Connector​

To create your SharePoint connector in the Zeta Alpha Platform UI:

  1. Navigate to your tenant and click View next to your target index
  2. Click View under Content Sources for the index
  3. Click Create Content Source
  4. Paste your JSON configuration
  5. Click Submit

Create SharePoint Connector

Crawling Behavior​

The first time the connector runs, it crawls all information specified in the connector configuration.

After the initial crawl, only new, deleted, and modified documents will be crawled for the specified sites, drives, and paths. This incremental approach avoids crawling unnecessary data and improves performance.

Turning document libraries off​

Document library files are crawled by default. A source that should ingest only lists, or only OneNote, declines them:

{
    "connector_configuration": {
        "sharepoint": {
            "...": "...",
            "content_configuration": {
                "drives": { "enabled": false },
                "lists": { "enabled": true }
            }
        }
    }
}

No file document or file permission is then emitted, .one section files aside. List and OneNote crawling are unaffected: in text mode the connector still lists the drives the onenote block selects for their .one section files, and in visual mode it follows the document libraries holding the rendered notebooks, since that is where their edits and permission changes surface.

Declining files stops crawling them; it does not remove the files already indexed. The next full crawl leaves every one of them unseen. If they are more than the delete fraction of the source's documents (0.2 unless max_delete_fraction_override is set), the platform's delete guard refuses the deletion, so they stay in the index with the permissions they last had. To remove them deliberately, raise max_delete_fraction_override on the content source, run one full crawl, then unset it.

Incremental Permission Sync​

In addition to content changes, the connector also detects permission-only changes (e.g., when a user is granted or revoked access to a document without modifying the document itself). This uses the SharePoint REST getchanges API to track role assignment additions and removals.

Note: Incremental permission detection requires delegated authentication (certificate or ROPC credentials). Client secret authentication (app-only tokens) does not support the SharePoint getchanges API and will skip incremental permission updates. Use the full access rights crawl as a workaround (see below).

Full Access Rights Crawl​

You can configure the connector to perform a full access rights refresh without re-downloading content. This iterates all documents and emits updated permissions, then advances the access rights change token while leaving content tokens untouched.

This is useful for:

  • Bulk-refreshing permissions at any point in time
  • Working around the client secret limitation for incremental permission detection
  • Recovering from a missed permission change window

To enable, set full_access_rights_crawl to true in the connector configuration:

{
    "name": "my_sharepoint_connector",
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            ...
            "full_access_rights_crawl": true
        }
    }
}

Since-Date Crawl​

You can reseed delta crawling from a specific date by setting since_crawl_date (in YYYY-MM-DD format). When set, the connector ignores the stored delta tokens for that run and re-detects content and access-right changes modified on or after that date, then refreshes the tokens so subsequent runs resume incrementally.

This is useful for:

  • Recovering from a missed change window
  • Backfilling changes after expanding the crawl scope (e.g., adding new site_paths or drive_ids)
  • Catching up changes without paying for a full re-crawl

since_crawl_date is ignored when full_crawl is true.

{
    "name": "my_sharepoint_connector",
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            ...
            "since_crawl_date": "2024-01-15"
        }
    }
}

Crawl Mode Priority​

When the connector runs, it selects the crawl mode in the following order:

  1. Full crawl — if full_crawl is set to true
  2. Since-date crawl — if since_crawl_date is set
  3. Full crawl — if this is the first run (no delta tokens stored)
  4. Full access rights crawl — if full_access_rights_crawl is set to true
  5. Update crawl — incremental mode (default)

Authentication and Feature Compatibility​

FeatureCertificateROPCClient Secret
Full crawl✅✅✅
Incremental content changes✅✅✅
Incremental permission changes✅✅❌
Full access rights crawl✅✅✅
OneNote rendering❌✅❌
SharePoint lists✅✅❌

OneNote rendering (onenote.mode: visual) is the one feature that inverts the recommendation: its Graph API takes user-delegated tokens only, so it needs ROPC (public_access_credentials) and rejects both application credential types. The default text mode runs on any credentials. See Required permissions and authentication.

Lists rule out the client secret for a different reason: item access rights are read over the SharePoint REST API, which accepts app-only tokens only when they are proven with a certificate. A client-secret source gets 401 Unsupported app only token on every list item, whatever permissions are consented. See Required permissions and credentials.

Scheduling with Task Types​

A SharePoint source syncs along two dimensions — document content and access rights — and each runs as either a full pass or a delta pass. These map to four task types you can schedule independently:

  • content_full — crawl the configured scope in full: ingest new and changed documents, refresh metadata, and remove documents the source no longer has.
  • content_delta — ingest only the documents that changed since the last content run (incremental crawl).
  • access_rights_full — re-apply permissions to every document in scope, without re-downloading content (the Full Access Rights Crawl above).
  • access_rights_delta — apply only the permission changes since the last access-rights run (the Incremental Permission Sync above).

See Task types for the authoritative definitions. SharePoint does not accept the enhancement_* task types — those belong to enhancement connectors.

Schedule each task on its own cadence. Set scheduled_tasks, a top-level list on the content source (alongside name and connector, not inside connector_configuration), with one { "task_type": ..., "schedule": "<cron>" } entry per task type:

{
    "name": "my_sharepoint_connector",
    "connector": "sharepoint",
    "scheduled_tasks": [
        { "task_type": "content_delta", "schedule": "*/15 * * * *" },
        { "task_type": "access_rights_delta", "schedule": "*/15 * * * *" },
        { "task_type": "content_full", "schedule": "0 4 * * 0" }
    ],
    "connector_configuration": {
        "sharepoint": {
            "...": "..."
        }
    }
}

Each task_type may appear at most once. Editing the list reconciles the schedules; an empty list clears them. A source runs on either the single schedule cron or scheduled_tasks — use scheduled_tasks (and leave schedule unset) when you want independent per-task cadences and full-crawl reconciliation. Per-task scheduling is configured through the content-source API or the platform-admin content-source editor (create / edit). See How a content source runs for the general scheduling model.

Delta runs capture only forward changes. Unlike the single-schedule crawl, which performs a full crawl on its first run (see Crawl Mode Priority), an explicitly scheduled content_delta does not backfill: its first run on a source with no prior delta state records the current position and ingests nothing, and from then on it ingests only what changed since the previous content run. content_full is the authoritative pass — it re-crawls the scope, refreshes metadata, and removes documents the source has dropped. If the connector's delta state expires on the SharePoint side, the next delta run re-anchors to the current position and the following content_full reconciles any skipped window. A periodic content_full is therefore what guarantees the index converges on the source.

Recommended lifecycle.

  1. Seed the delta first. Run content_delta once before the initial full crawl. It is cheap — it only records the starting position — and arming the delta cursor first means edits made during the initial full crawl (which can take hours on a large source) are still picked up by the next delta run, closing the change-gap.
  2. Run content_full once to load and reconcile the full corpus.
  3. Steady state. Schedule content_delta and access_rights_delta frequently (for example every 15 minutes) for freshness, and content_full on a slower cadence (for example weekly, off-peak) to reconcile — refreshing stale metadata and removing documents the source no longer has.

On a source that renders OneNote, the content_full cadence decides how soon a new notebook is rendered when it sits in a document library the source does not read yet: a delta run finds new notebooks only in the libraries it already reads. See Change detection and deletions for what that means in practice, and Running OneNote as its own sync job for giving OneNote its own schedule.

On a source that renders OneNote, schedule access_rights_full as well, and size its cadence to the revocation latency you can accept. access_rights_delta applies a permission change made on a section itself, but a change on a notebook or a section group, which its sections inherit, reaches them with the next access_rights_full, or when a section is next edited.

On a source with lists enabled, schedule access_rights_full as well, and size its cadence to the revocation latency you can accept: a site-level permission change and a change to a list's Item-level Permissions setting are applied only by the full. See List item access rights.

Fetch tuning (advanced). SharePoint downloads document bodies in parallel batches. Two optional base connector-configuration fields tune this: fetch_batch_size (documents per fetch, which is also the ingestion batch size) and fetch_concurrency (how many batches download at once). Leave them unset to use platform defaults; lower fetch_concurrency when the source shares a Microsoft Graph rate-limit budget with other connectors, or raise it to speed up large crawls.

OneNote Notebooks​

The connector ingests OneNote notebooks stored in SharePoint sites and in OneDrive. By default (content_configuration.onenote.mode: text) a notebook's section files are indexed as text, like document library files, whether or not the source crawls those. With mode: visual, each notebook section is ingested as its own document instead: its pages rendered by Microsoft to HTML and joined in OneNote's order. onenote.enabled: false leaves OneNote out altogether.

Rendering takes two things beyond the connector configuration below: username/password credentials with the delegated Notes.Read.All Graph permission (Required permissions and authentication), and, if sections should have an in-app preview or be visually processed, the html → pdf rule in the source's ingestion workflow (PDF rendering, previews and visual processing).

What is ingested, and what is not​

A notebook exists twice in Microsoft 365:

  • In the document library, a notebook is a folder holding a .onetoc2 table of contents and one .one file per section, with a subfolder per section group. The library shows the notebook as a single item.
  • In the OneNote service, the same notebook is a notebook, section and page tree, and each page can be rendered to HTML.
File in the document librarytext (default)visualenabled: false
A section's .one fileindexed as textreplaced by the rendered section, under the same document idnot ingested
A .one file outside any notebookindexed as textnot ingestednot ingested
The .onetoc2 table of contentsnever ingestednever ingestednever ingested
Anything in the notebook's recycle bin (OneNote_RecycleBin)never ingestednever ingestednever ingested

Because a rendered section keeps the document id of its .one file, switching between text and visual replaces each section's document at the next crawl, a content_delta included, rather than adding a second copy.

Note that a OneNote page means a page inside a notebook section. SharePoint site pages (.aspx) are a separate content type, crawled with the pages block. See SharePoint Site Pages.

Required permissions and authentication​

OneNote is a separate Microsoft 365 service with its own Graph permission, and rendering it (onenote.mode: visual) is the one part of the connector that cannot run on application-level credentials. The default text mode indexes each section's .one file through the drive API and needs no OneNote permission. As Microsoft states in the OneNote API reference, "the Microsoft Graph OneNote API doesn't support app-only authentication." Only user-delegated tokens are accepted; the OneNote service rejects the tokens issued for certificate and client-secret credentials, whatever permissions they carry, with:

40001 The request does not contain a valid authentication token

Azure AD still lists Notes.Read.All under Application permissions, still lets an administrator consent to it, and still issues tokens carrying it. The rejection happens at the OneNote service rather than at sign-in, so nothing in the Azure portal flags the combination as invalid. There is no app-only alternative for OneNote.

A visual OneNote source therefore needs both of the following:

  1. The delegated Notes.Read.All permission on the Azure AD application from Configure Microsoft App Access, added under Microsoft Graph → Delegated permissions and admin-consented, since the username/password flow cannot prompt a user for consent. The tutorial's Optional: OneNote notebooks section walks through it; the permission is only read access to OneNote content.
  2. public_access_credentials on the content source: the username and password of an account that can open the notebooks. This is the only credential type the connector uses to obtain a delegated token, so a source configured with certificate_credentials or access_credentials cannot render OneNote. As with any username/password source, the account must not have MFA enabled.

Because files are best crawled with certificate credentials while visual OneNote requires username/password, run the two as two content sources over the same sites: the file source on certificate_credentials with onenote.enabled: false, and a second source on public_access_credentials that declines files and sets onenote.mode: visual. See Running OneNote as its own sync job: a split OneNote source is worth having regardless, for its own schedule and fetch concurrency.

Important: a source that sets onenote.mode: visual while it authenticates with a certificate or a client secret is rejected when it is saved. So is one that sets public_access_credentials next to either of them, because those take precedence and would crawl with an app-only token. The fix is the credentials, not another permission grant.

Further points to be aware of when planning the grant:

  • Reach follows the crawl account. Delegated Notes.Read.All reads the notebooks that the account can access, so the OneNote crawl is bounded by that account's own SharePoint access; there is no tenant-wide grant to narrow. Give the account access to the sites whose notebooks should be indexed, then restrict the crawl further in the connector configuration with site_paths, site_inclusion_regex_patterns and the path patterns described below. As with every other setting, Zeta Alpha only reads what the connector configuration allows.
  • OneDrive notebooks belonging to other users are reachable only when shared with the crawl account: a delegated token carries no rights over another user's OneNote. Site notebooks are unaffected: they follow the account's site access.
  • Section permissions come from the document library. OneNote exposes no permission API, so each section takes the permissions of its .one file, exactly as that file has them as a document. This reuses the Files.Read.All / Sites.Read.All access the connector already needs for files, with no additional permission. Access is never widened to the site's permissions: a site member who cannot open the notebook does not gain access to its sections through search.

Enabling OneNote alongside files​

A single source can render notebooks and crawl files together, provided it authenticates with public_access_credentials: one source carries one credential set, and visual OneNote needs the delegated one. Set the onenote block's mode on it:

{
    "name": "my_sharepoint_connector",
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            "...": "...",
            "content_configuration": {
                "onenote": { "mode": "visual", "include_resources": true }
            }
        }
    }
}

Choosing which notebooks are crawled​

Both modes take their notebooks from one selector, the most specific the source sets, as files do:

  • onenote.drive_ids takes exactly the notebooks stored in those drives.
  • Otherwise, one_drive_users takes the notebooks in those users' OneDrive: reachable only where those users have shared them with the crawl account, or where the crawl account is the user.
  • Otherwise, site_paths, site_inclusion_regex_patterns, site_exclusion_regex_patterns and include_sub_sites select the sites whose notebooks are read. A source that sets no site_paths reads every site.

The onenote block then filters by what the document library shows: the drive_* patterns against the name of the drive holding the notebook, and the path_* patterns against each section's .one path in that drive. A path pattern matches the path as Microsoft Graph lists it, with no percent-encoding, such as /Notebooks/Team Notebook/General.one, in both modes. A text-mode content_delta matches the same path, since its change rows carry the folder path. A folder followed by /, such as ^/Notebooks/Team Notebook/, selects every section of a notebook or section group. A notebook found in a site's drive named OneDrive is crawled only with onenote.include_one_drives. No file setting applies to a notebook, so a file crawl pinned to the top-level drive_ids or narrowed by the top-level path patterns leaves the notebooks as they are. A source configured before the onenote block existed starts with a copy of those file settings in its block, so its .one files stay where they were. A copied path pattern that matches a folder only in its decoded form, such as one with a literal space, never matched that folder for files, and now applies to that folder's .one files: an inclusion pattern selects them, and an exclusion pattern drops them. Pages cannot be filtered: a section is the smallest unit.

"content_configuration": {
    "onenote": {
        "mode": "visual",
        "path_exclusion_regex_patterns": ["^/Archive/", "/Personal Notebook/"]
    }
}

In visual mode no .one file is indexed as text, so a section the filters exclude is not indexed at all, and neither is a .one file outside any notebook. A notebook whose document library folder the connector cannot reach from its OneNote link is skipped with a warning in the run's logs. Once the connector can reach it, the next content_full renders it.

Embedded images and attachments​

A rendered section references its images and attachments as OneNote URLs that only the crawling application can open, so they are unusable to anyone reading the document later. With onenote.include_resources enabled, each of those resources is downloaded and embedded directly in the section's content, so the section carries its own media. Images are taken at full resolution when Microsoft provides one.

Resources are fetched individually and never break a section: one that fails to download, or that would push the section over the index's maximum file size, is left as its original link and the section is still ingested. Embedded video is an external link with no file behind it and is left untouched. Enabling this option increases the crawl's download volume and the stored size of each section. Changing it re-renders every section at the next crawl, a content_delta included.

PDF rendering, previews and visual processing​

A rendered section is ingested as HTML, and its text is extracted from that HTML: searching, retrieval and citation work with no extra configuration. Two things need a PDF rendering of the section, which an HTML document only gets if the ingestion workflow is configured to produce one:

  • The in-app page preview and thumbnail, both rendered from the PDF.
  • Visual processing. The Visual Processor agent takes a document's pdf representation as its input, so on a section without one it has nothing to look at: the section's diagrams, screenshots and layout are not described, and only its text reaches the index. Text alone already carries what Microsoft's own image OCR returns, since a page's alt text is folded into the extracted text; visual processing is what adds a reading of the section as it looks.

This is opt-in per content source. To enable both for a OneNote source, add the html → pdf rule to the pdftotext task of the workflow assigned to it:

{
  "name": "pdftotext",
  "local_settings": {
    "fields_conversion_map": {
      "pdf": [
        { "input_type": "representations", "input_field": "content", "output_type": "pdf" },
        { "input_type": "representations", "input_field": "html", "output_type": "pdf" }
      ],
      "text": [
        { "input_type": "representations", "input_field": "pdf", "output_type": "text" },
        { "input_type": "representations", "input_field": "markdown", "output_type": "text" },
        { "input_type": "representations", "input_field": "text", "output_type": "text" },
        { "input_type": "representations", "input_field": "html", "output_type": "text" },
        { "input_type": "representations", "input_field": "content", "output_type": "text" }
      ],
      "nested_text": [{ "input_type": "representations", "input_field": "nested_content", "output_type": "text" }]
    }
  }
}

Give the whole map, not just the pdf entry: a fields_conversion_map in a task replaces the deployment's default map rather than merging with it, so any output field you leave out stops being produced for that workflow.

Both pdf rules are listed because one workflow serves documents of different types: the content rule renders Office files and images, the html rule renders HTML documents such as OneNote sections. Keep pdf declared before text, and keep every processor that consumes the PDF (image_extractor, thumbnail_maker and agent_processor) after pdftotext in the workflow's steps, so the rendered PDF exists by the time they run. For visual processing, pdf must also appear in the agent_processor task's agent_input_representations.

Text extraction is unaffected by the rule: an HTML document's text always comes from its HTML, never from the rendered PDF, so enabling this cannot change what is already searchable. Rendering each section does add work to ingestion, and visual processing adds an agent call per section on top, so enable it on the sources whose sections users read in the app or whose visual content matters, rather than on every workflow. Without the rule, a section is still fully searchable and its document link opens the section in OneNote.

Change detection and deletions​

OneNote provides no change feed, at any level, and its timestamps cannot be trusted: an open Microsoft Graph regression reports a page's lastModifiedDateTime unchanged after edits. The connector watches the SharePoint document library instead. Every page edit rewrites its section's .one file, and the library's change feed reports that rewrite, so a delta run re-renders the section with all its pages. Deleting a section moves its .one file into the notebook's recycle bin, which the library's change feed reports as well, so a delta run removes the section. Sections inside a section group behave the same.

Event in OneNoteApplied by content_deltaApplied by content_full
Page created, edited or deleted✅✅
Section created, renamed or deleted✅✅

Changes to whole notebooks:

  • A new notebook is rendered by the next content_delta when its document library already holds a notebook the source keeps. In any other library, one the source crawls for files included, it waits for the next content_full.
  • A deleted notebook is removed from the index by the next content_delta in visual mode. In text mode its .one documents leave at the next content_full.
  • Renaming a notebook or a section group rewrites no section file, so section titles keep the old name until the section is next edited.

The same holds for the source's scope: after changing which sites, users or notebooks the source covers, run content_full. Until it runs, only the sections edited since are picked up, and a notebook the new settings exclude stays in the index. Switching between text and visual, or toggling include_resources, needs no full crawl: the next content_delta relists every section in scope and replaces its document. On a visual source that delta renders every section, so it takes as long as a full crawl of the notebooks.

Running OneNote as its own sync job​

Credentials pull OneNote onto its own content source: rendering needs username/password credentials while files are best crawled with a certificate (Required permissions and authentication), and a source carries exactly one credential set. A separate source also gets its own schedule and its own fetch concurrency, which OneNote needs: it serves at most five concurrent requests per user, while the platform runs ten fetches at once by default. Give a second content source onenote.mode: visual, decline its files, and set fetch_concurrency to 4:

{
    "name": "my_sharepoint_onenote",
    "description": "OneNote notebooks",
    "is_indexable": true,
    "connector": "sharepoint",
    "scheduled_tasks": [
        { "task_type": "content_delta", "schedule": "*/15 * * * *" },
        { "task_type": "access_rights_delta", "schedule": "*/15 * * * *" },
        { "task_type": "content_full", "schedule": "0 3 * * *" },
        { "task_type": "access_rights_full", "schedule": "0 5 * * *" }
    ],
    "connector_configuration": {
        "sharepoint": {
            "is_document_owner": true,
            "content_source_name": "SharePoint OneNote",
            "fetch_concurrency": 4,
            "content_configuration": {
                "drives": { "enabled": false },
                "onenote": { "mode": "visual", "include_resources": true }
            },
            "public_access_credentials": {
                "client_id": "my_client_id",
                "tenant_id": "my_tenant_id",
                "username": "onenote-crawler@contoso.com",
                "password": "my_password"
            },
            "site_paths": [
                { "collection_hostname": "contoso.sharepoint.com", "site_relative_path": "sites/research" }
            ]
        }
    }
}

Here new notebooks are rendered nightly and the sections' access rights are refreshed nightly, while the file source can keep its weekly full crawl. The task types and the scheduled_tasks mechanics are the same as for files: see Scheduling with Task Types, including the recommendation to seed content_delta once before the first full crawl.

Points to keep in mind when splitting the source in two:

  • Turn OneNote off on the files source, with content_configuration.onenote.enabled: false, and run it on scheduled_tasks, where the block applies. A rendered section has the same document id as its .one file, so a files source still indexing .one text would write the same documents: only the first source to write a document owns it, and the other source's jobs end incomplete, or failed when every document they write collides. With OneNote off there, sections and files are distinct documents and both sources can be document owners. Give the OneNote source its own content_source_name.
  • .one files outside any notebook are indexed by neither source: visual mode renders notebook sections only, and the files source crawls no OneNote.
  • Keep both sources on the same crawl scope settings you intend to cover; the OneNote source discovers notebooks through its own site_paths / site patterns. The two sources cover the same sites only if the OneNote source's crawl account can open them: its reach is the account's access, not the application's.
  • Keep the files source on certificate_credentials. Only the OneNote source needs username/password credentials, so splitting confines the ROPC account to the notebooks.
  • A OneNote-only source keeps full access rights on its sections, from the same place a combined source takes them: each section's .one file (see Required permissions and authentication). A section takes them when it is fetched, and access_rights_full reads each .one file's permissions again. access_rights_delta follows the document library's permission change log and re-reads a section whose own permissions changed. A change made on a notebook or a section group reaches its sections only with access_rights_full, so schedule both. What such a source never emits is access rights for library files, since it ingests none.

Other OneNote behaviour worth knowing​

  • Section size. OneNote does not report a section's size in advance, so the index's maximum file size is applied while its pages download and again after rendering. An oversized section is handled like an oversized file. When resource embedding is on, the limit also applies to the section's combined size: a section that would exceed it keeps its original resource links instead.
  • Metadata. A section's document title is its logical path, {site name or OneDrive user}/{notebook}/{section} (with the section group segments inserted when the section is in one), the same convention as a file, whose title is its site, drive and path. The document link opens the section in OneNote on the web. Created and modified timestamps and the author come from the section.
  • Password-protected and empty sections are ingested with their title only: Microsoft lists a password-protected section but returns its pages as empty, with no error, so the connector cannot tell it apart from an empty section. Protecting or emptying a section that was already ingested removes its earlier content at the next crawl.
  • Throttling. The OneNote service allows 120 requests per minute and 400 per hour per application and user, and, unlike the rest of Graph, its rate-limit responses carry no Retry-After hint. The connector backs off and retries on its own; a sustained throttle slows a crawl but does not drop sections. Running OneNote as its own source keeps that budget separate from the file crawl.

SharePoint Lists​

With content_configuration.lists.enabled, the connector ingests the SharePoint lists of the sites crawled: task trackers, issue logs, contact lists, custom lists. Each list item becomes its own document, so a row is searchable, retrievable and citable on its own.

Enable it on an existing source, or on a dedicated one:

{
    "name": "my_sharepoint_connector",
    "connector": "sharepoint",
    "connector_configuration": {
        "sharepoint": {
            "...": "...",
            "content_configuration": {
                "lists": { "enabled": true, "name_exclusion_regex_patterns": ["^Scratch"] }
            }
        }
    }
}

Unlike OneNote, lists need no credential split: certificate credentials cover files and lists in one source. List ingestion is also independent of the OneNote flags, so a source can combine any of the content types its credentials allow.

What becomes a document​

  • Every item of every in-scope list is rendered as a self-contained text/html document: a heading with the list and item title, then one line per column with the column's label and value.
  • Visible columns are rendered; hidden columns and column types with no text form (image thumbnails, geolocation) are omitted. Person, lookup, managed-metadata and hyperlink columns render as their display text. At most 12 lookup and person columns per list are expanded, a Microsoft Graph limit; further ones are omitted.
  • The columns SharePoint maintains itself (Created, Modified, Author, Editor, Content Type and Attachments) are left out of the text. The first three are already document fields (created date, last updated date, authors), so a line for them repeats a value the document already carries. Columns that compute their value from a formula are content, not bookkeeping, and do render.
  • The item's title is its Title column, or the first non-empty text column, or the item's numeric id. The document's path is {site}/Lists/{list}/{title}, and its link opens the item in SharePoint.
  • Column values are also carried in the column_values field, keyed by internal column name, so field_mappings can promote them to index fields. Every column is there, including the ones kept out of the text: column_values.Editor maps like any other. See the configuration reference.
  • Attachments ride inside the item's document as downloadable links: file names are searchable text (also carried in attachment_names, promoted to an index field the same way as column_values), while the file contents are embedded for download but not indexed. An attachment that would exceed the index's maximum file size is listed by name only. A failed attachment download fails the whole item, so it is retried on the next run. Attachments are listed with their size and version at enumeration, so an attachment replaced in place is re-indexed on the next full content pass even though the item itself did not change.

Document libraries are not lists to this connector; files are always crawled through the file settings. Lists that SharePoint marks as system (galleries, the user information list, workflow history) and the Web Template Extensions, Access Requests and item-reference lists are never ingested. Hidden lists are ingested only with include_hidden.

Required permissions and credentials​

Enumerating lists and reading item content use the Microsoft Graph permissions the connector already has. List access rights are the extra requirement: they are read from the SharePoint REST role-assignment endpoint, whose authorization comes from the Office 365 SharePoint Online API permissions of the app registration, not the Graph ones. Two things must hold:

  1. A Full Control grant. Enumerating role assignments requires SharePoint's Enumerate Permissions right, which only Full Control includes. Add the Office 365 SharePoint Online → Sites.FullControl.All application permission and grant admin consent; the Microsoft app access tutorial walks through it. The SharePoint Sites.Read.All permission that suffices for incremental permission sync is not enough here: with a lower grant the read of a list's permissions returns 403 System.UnauthorizedAccessException, and every item of the list carries only the source's allow_access_rights (see below).
  2. Certificate or username/password credentials. SharePoint REST rejects app-only tokens that were not proven with a certificate, before any authorization check, so no permission grant changes it. A source on access_credentials (client secret) gets 401 Unsupported app only token on every list item; the fix is the credential type, not another permission.

For a username/password source the Full Control requirement translates to two grants, both mandatory: the delegated Office 365 SharePoint Online → AllSites.FullControl permission, admin-consented, and Full Control for the crawl account itself on every site in scope (site Owners group or site collection administrator). Effective rights are the intersection of the two, so a Members or Visitors account still gets the 403. Sites the account cannot open are skipped, as everywhere with delegated credentials.

With Sites.Selected instead of tenant-wide permissions, the per-site grant must carry the fullcontrol role for list permissions to be read; with read, list items carry only the source's allow_access_rights. See step 19 of the tutorial.

If Full Control cannot be granted at all, only the permission read is affected: a list's role assignments resolve empty and each item carries the source's allow_access_rights, skipped when that leaves none. The connector still reads each list's settings and its attachments over SharePoint REST, so it needs a read-level Office 365 SharePoint Online grant (Sites.Read.All app-only, AllSites.Read delegated). With no grant on that API the settings read returns 401/403, every item fails to fetch and the list crawl produces no documents. See List item access rights below.

Choosing which lists are crawled​

  • site_paths, site_inclusion_regex_patterns, site_exclusion_regex_patterns and include_sub_sites select the sites whose lists are read. Neither one_drive_users nor drive_ids narrows them.
  • name_inclusion_regex_patterns and name_exclusion_regex_patterns then filter on the list's display name. Exclusion takes precedence, and a non-empty inclusion set gates lists exactly as path patterns gate files: a list matching no inclusion pattern is skipped.
  • include_hidden adds every list hidden from the site contents page that Microsoft Graph does not mark as system. Some of those are SharePoint's own, such as CSPViolationReportList and SharePointHomeCacheList, so set name_inclusion_regex_patterns to the hidden lists you want alongside it. System lists and the Web Template Extensions, Access Requests and item-reference lists stay excluded regardless.
  • drive_ids does not constrain lists; it targets the file crawl only. path_inclusion_regex_patterns and path_exclusion_regex_patterns apply to files only, not to list items.

Lists follow the site scope and nothing else, so a source that pins its file crawl with drive_ids or one_drive_users but sets no site_paths and no site patterns enumerates the lists of every site the crawl identity can see, which on a tenant-wide grant is the whole tenant. Set site_paths or the site regex patterns whenever list crawling is on.

List item access rights​

Every item carries its list's permissions: the list's role assignments are read once per list and apply to all of its items, with no per-item read. An item with unique permissions (broken inheritance) is covered by its list's permissions too, which may be broader or narrower than its own.

A refused (401/403) read of a list's permissions is the expected state of a deployment without Full Control, so it degrades: the list's permissions resolve empty and each item carries the source's allow_access_rights, and is skipped entirely when that leaves no access rights, so an item is never ingested with open access. A transient failure of the read fails the item, so it is retried rather than ingested against an empty list ACL.

Each user or group that a role assignment grants read is recorded with the same access right it has on files: a Microsoft Entra or Microsoft 365 group as msft_group with its object id, a user as sharepoint_user with their email, and a SharePoint group as sharepoint_site_group with its group id. List items therefore match the same user and group mappings as files. A principal with no such access right, such as Everyone, Everyone except external users or a user without an email, is recorded under its SharePoint login name with the type sharepoint_principal and is not matched to platform users, so an item granted only to it is found through the source's allow_access_rights.

Lists whose Item-level Permissions restrict reads to "items that were created by the user" are honoured: an item carries only the users and groups whose permission level includes Override List Behaviors (Design and Full Control, so a site's Owners but not its Members, who hold Edit). The item's creator is not added, so on such a list other users find none of the items they created. Changing this setting alters no role assignment, so SharePoint's change log does not report it and it reaches the items on the next access_rights_full.

Both access-rights task types cover list items. access_rights_full re-resolves every in-scope item's rights without re-downloading content. access_rights_delta follows each list's SharePoint change log and re-reads a list whose own permissions changed.

SharePoint records a role assignment against the object it was made on, and an item that inherits holds none of its own, so changing a list's, a library's or a site's permissions names no items in the change log. Removing a permission from the parent is not recorded against items with unique permissions either, so a container-level change reaches no item by name in either direction. Applying one means re-reading everything under it:

  • For a list, access_rights_delta re-reads the list itself and re-emits every item. Every item's rights are the list's, read once per run, so the pass costs the item listing rather than a call per item.
  • For document library files, re-reading costs one permissions call per item. The delta logs that it saw the change, naming the drive, and access_rights_full is what applies it.

A permission change at the site level is always applied by access_rights_full, because SharePoint's list change log does not report it at all.

Change detection for lists​

Lists have a real change feed, so freshness works like files, not like OneNote:

Event in the listApplied incrementallyApplied by a full pass
Item created or editedcontent_deltacontent_full
Item deletedcontent_deltacontent_full
Attachment added or removedcontent_deltacontent_full
Attachment file replaced in placenot detectedcontent_full
Permission change on a single itemno effect: items carry their list's permissionsno effect
Permission change on the whole listaccess_rights_deltaaccess_rights_full
Item-level Permissions setting changednot detectedaccess_rights_full

A list that is new to the connector (created, newly in scope, or list crawling just enabled) is listed in full on the next content_delta, and a list whose change feed has expired on the SharePoint side is listed in full again on the run that notices. Such a listing adds and updates items only: an item deleted while the feed was expired is removed by the next content_full. The access-rights change log works the same way: a list whose token has aged past SharePoint's retention has its items' permissions re-read and its token reset on that run.

An item's identity is stable within its list: edits update the same document. An item moved to another list, or a list deleted and recreated by a migration, changes identity; the affected items are re-ingested as new documents and the old ones are removed by the next content_full.

SharePoint Site Pages​

The connector can ingest the modern site pages of the sites it crawls: articles, home pages and news posts. Each page becomes its own HTML document.

Enable it with the pages block of content_configuration on the content source:

{
    "connector_configuration": {
        "sharepoint": {
            "...": "...",
            "content_configuration": {
                "pages": { "enabled": true }
            }
        }
    }
}

Which pages are ingested​

The connector reads each page from the Microsoft Graph pages API and composes its HTML from the page's web parts. It does not ingest the page's .aspx file in the Site Pages library.

Only published pages are ingested, at the version the page's readers see; a draft never reaches the index. A page that was never published is not crawled. On a source with username/password credentials, a page with a saved draft is fetched again only after it is published, and a page first found with a saved draft is ingested at that point.

News posts are ingested with ordinary pages. The crawled record's promotion_kind is newsPost for a news post and page for an ordinary page. Map it with field mappings to use it as a search facet.

Classic pages, such as wiki pages and the pages of classic publishing sites, are not ingested, because the Microsoft Graph pages API serves modern pages only. Page templates are not ingested either. A page whose web parts hold no readable content is skipped.

What a page's document contains​

The document follows the page's layout: the banner, then each section in order, with its column widths and background emphasis. It contains:

  • The text, links and images that SharePoint extracted from each web part, third-party web parts included.
  • One block per repeated element, such as a hero tile or a quick link, so that a tile's image, heading, text and link stay together.
  • The shape of the web parts you write content into: image, image gallery, hero, editorial card, quick links, people, markdown, code snippet, embed, YouTube, button, call to action and link. An image keeps its caption and alternative text. Quick links keep their lines or cards. An editorial card keeps its kicker and description. A markdown or code snippet web part contributes its body. If one of these web parts has an unexpected shape, the connector treats it as any other web part.
  • The file's name beside each link to a file on the site's SharePoint host, except in live-query web parts.
  • For a live-query web part (news, highlighted content, events, site activity), the snapshot of results that SharePoint saved with the page. The connector does not run the query again, so the snapshot can be out of date. Results that are files or pages are indexed as their own documents when the source crawls them.
  • The heading of a web part, when the page shows one, such as the title of a news, list or quick links web part.

It leaves out:

  • The rows of a list view. SharePoint stores only the list and the view that the web part shows, so the document holds the list's name as a link to the list.
  • The name and description of each web part type, such as "Hero" or "Show an image on your page".
  • Links with no readable label, so that no raw URL appears in the text.

Page metadata follows the file convention: the document title is {site name}/SitePages/{page title}, the link is the page URL, the timestamps are the page's own, and the author is the page's creator.

As with OneNote, page previews and visual processing need the html → pdf rule in the source's ingestion workflow. See PDF rendering, previews and visual processing.

Choosing which pages are crawled​

Pages belong to a site, so the site settings (site_paths, site_inclusion_regex_patterns, site_exclusion_regex_patterns) decide which pages are in scope. In each site, the pages block's own path_inclusion_regex_patterns and path_exclusion_regex_patterns filter pages by the path /SitePages/{file name}. The top-level path patterns filter files, not site pages, so the two filters can differ:

"content_configuration": {
    "pages": { "enabled": true, "path_exclusion_regex_patterns": ["/SitePages/Archive-"] }
}

drive_ids and one_drive_users scope the file crawl only. A source that sets either still takes its pages from the site settings. With no site settings, that is every site the crawl identity can see.

Images and files on a page​

With include_resources on, the connector inlines the page's assets: the pictures and files that an author adds to the page as a copy. SharePoint keeps them in an assets folder for that page alone, with the page's permissions. They are never indexed as documents of their own.

The connector does not read an asset's own permissions. If someone narrows them later, anyone who can read the page can still see the asset in the indexed page.

  • If site owners narrow the permissions of page assets, keep include_resources off.
  • If an asset was narrowed after it was inlined, remove it from the page or turn include_resources off, then publish the page again. The connector fetches a page again only when the page changes, so a full crawl does not remove the inlined copy.

A picture that is not inlined appears as a link with its alternative text. These are not inlined:

  • Every picture, with include_resources off.
  • Pictures that an author added by sharing the original instead of a copy, because they keep their own permissions.
  • Assets of a page renamed after its assets folder was created, because SharePoint keeps the folder's original name.
  • Template artwork, such as the hero tiles of a communication site's default home page.
  • Pictures on other hosts, such as Microsoft's public content CDN.

Files that a page links to are never inlined. The file crawl indexes them under their own access rights.

The index's maximum file size applies to each resource and to the whole page. A page over the limit is ingested with links in place of its inlined resources, and skipped if it is still over.

Access rights​

A page carries the permissions of its .aspx item in the Site Pages library, recorded as for a file, and needs no permission or credential beyond the file crawl's. If they cannot be read, the run reports a fetch failure for the page; they are never widened to the site's.

access_rights_delta reads the Site Pages library's change log, which needs certificate or username/password credentials (see Incremental Permission Sync). access_rights_full reads every page again. A permission change on the Site Pages library itself makes the delta re-read every page of the site.

Change detection for pages​

content_delta reads the change feed of each site's Site Pages library. It fetches pages edited, renamed, moved or restored, and removes pages deleted, unpublished or moved out of the path patterns. If the feed position has expired, the next content_delta lists the site's pages again, and the next content_full removes pages deleted in the gap.

Running pages as their own sync job​

To give pages their own schedule (see Scheduling with Task Types), turn the other content types off on a separate source:

"content_configuration": {
    "drives": { "enabled": false },
    "onenote": { "enabled": false },
    "pages": { "enabled": true }
}

Such a source does not index the files its pages link to, so run a files source over the same sites next to it.