Data sources

A data source keeps a knowledge base in sync with a system your company already uses: a SharePoint site or a OneDrive, a Windows file server, an Exchange shared mailbox, Microsoft Teams channels or a website. Instead of uploading files by hand, you connect the source once. Genie imports what you select, then picks up new, changed and deleted content on a schedule, so people always get answers from the current version. Admins and Owners of the workspace add and manage data sources.

Add a data source

A data source belongs to one knowledge base and syncs into it. Create the knowledge base first (see Knowledge bases), then:

1. Open Knowledge in the sidebar and open the knowledge base, for example Human resources.
2. Choose the Data sources tab at the top of the knowledge base. Its settings open on Data Sources, in the Storage & Data group. (The Settings tab next to it opens the same settings.)
3. Choose Add Data Source.
4. Select Type: choose the system to connect.
5. Credentials: give the source a Name and fill in the connection details described in the sections below. Choose Test Connection, then Next. If you skip the test, Next runs it for you.
6. Resources: tick the sites, libraries, folders, mailboxes, shares or channels to sync, then choose Next. A website has no Resources step: it always starts from the addresses you entered.
7. Strategy: choose the Sync schedule and the other options (see Schedules and sync history), then choose Create & Sync Now.

The first sync starts right away. The data source appears as a card with its type, status, schedule and the result of the last sync. To follow a sync, open the ⋯ menu on the card and choose Logs.

Give each data source a clear name, such as "Contoso HR SharePoint". Every document it imports gets a tag with that name, so people can filter the knowledge base by source.

Besides the sources on this page, the type list also offers Confluence, Notion, GitLab, Seafile and RSS / Atom feeds, which follow the same steps. Power BI is described in Databases and Power BI.

SharePoint and OneDrive

SharePoint / OneDrive syncs the document libraries of SharePoint Online sites, or one person's OneDrive for Business.

What your Microsoft 365 administrator prepares

Genie reads SharePoint as an application registered in your Microsoft Entra ID, not as a person. A Microsoft 365 administrator who can grant admin consent (for example a Global Administrator) sets it up once:

1. In the Microsoft Entra admin center, open App registrations → New registration. Name it, for example Genie connector, choose Accounts in this organizational directory only (single tenant), leave the redirect URI empty and register.
2. On the app's Overview page, copy the Directory (tenant) ID and the Application (client) ID.
3. Under Certificates & secrets, create a client secret and copy its Value right away. It is shown only once, and the Secret ID is not the secret. Note the expiry date: when the secret expires, every sync stops until you enter a new one in Genie.
4. Under API permissions → Add a permission → Microsoft Graph → Application permissions, add one of:
   • Sites.Read.All and Files.Read.All: the app can read every site and every OneDrive. This is the simplest choice, and it is required to sync a OneDrive.
   • Sites.Selected: the app can read only the sites you grant it. After admin consent, grant read access site by site, for example with PnP PowerShell: Grant-PnPAzureADAppSitePermission -AppId <client-id> -DisplayName "Genie connector" -Site https://contoso.sharepoint.com/sites/HR -Permissions Read.
5. If Genie should keep SharePoint's permissions on the synced files (see Keep the source's permissions), also add GroupMember.Read.All and give the app full control of the sites: Sites.FullControl.All, or -Permissions FullControl instead of Read with Sites.Selected. With read rights only, Microsoft may return only part of a file's permissions, and the people left out cannot see the file in Genie.
6. Choose Grant admin consent. Every permission must show as granted.

The administrator then gives you the tenant ID, the client ID and the secret value through a safe channel. Only the global Microsoft 365 cloud is supported, not national clouds such as GCC High or DoD.

Connect a SharePoint site

Example: Minh Le connects the SharePoint site "Contoso HR" to the Human resources knowledge base.

1. In Human resources, choose Data sources → Add Data Source → SharePoint / OneDrive.
2. Fill in Directory (tenant) ID (or a domain such as contoso.onmicrosoft.com), Application (client) ID and Client secret value. If another SharePoint, Exchange or Teams data source in the workspace already uses the same app, an App registration list appears: choose Reuse the app of "…" to copy its credentials without typing the secret again.
3. Under Sync scope, keep Source on SharePoint sites and enter https://contoso.sharepoint.com/sites/HR under Site URLs. Enter one site per line; a link to a page or library inside the site also works. To sync one person's OneDrive instead, choose A user’s OneDrive and enter their sign-in address under User.
4. Choose Next. Genie checks the app's credentials.
5. Under Resources, tick the whole site, single document libraries or single folders. A site brings all its libraries, including libraries added later; tick single libraries to leave out ones such as Site Assets. A folder brings everything below it.
6. Under Strategy, choose the schedule. If HR files must stay limited to the people who can open them in SharePoint, turn on Mirror source permissions. Choose Create & Sync Now.

Screenshot coming soonAdding the SharePoint site Contoso HR to the Human resources knowledge basewebsite-docs/public/screenshots/en/data-source-sharepoint.png

Adding the SharePoint site Contoso HR to the Human resources knowledge base

Synced files land in folders named after the site, the library and the folder path in SharePoint. A file renamed or moved inside the selection is updated, not duplicated. Files Genie cannot read (archives, programs, OneNote notebooks) are skipped, and files above the knowledge base's upload limit are listed as failed in the sync history.

If the connection test fails

Message	What to do
AADSTS7000215	The secret is wrong. Copy the secret's Value, not its Secret ID.
AADSTS7000222	The secret has expired. Create a new one in Entra ID, then edit the data source and choose Replace under the credentials.
AADSTS700016 or AADSTS90002	The client ID or the tenant ID is wrong.
The SharePoint site is not found	Check the URL. With Sites.Selected, grant the app access to that site.
Microsoft Graph denied access (HTTP 403)	Admin consent is missing, or the site is not granted under Sites.Selected.
The OneDrive of a user is not found	The user does not exist, or has never opened OneDrive, so it does not exist yet.

Windows file server

Windows file server (SMB) syncs folders of a Windows file server, a NAS or a Samba share into a knowledge base. Contoso syncs \\files.contoso.test\Operations into the Operations knowledge base.

Before you start

1. A read-only service account. In Active Directory, create an account such as CONTOSO\svc-genie with a password that does not expire. Give it read access to the folders to sync, both on the share and in the NTFS permissions. Read access is enough, also for Genie to read the NTFS permissions.
2. A network path. Genie connects to the file server itself, over SMB (port 445; with Kerberos, also a domain controller on port 88). Because Genie is hosted for you, ask your Genie provider to connect it to your network and to allow files.contoso.test. A system administrator can also allow the server on SSRF protection allowlist (see Network and antivirus). Until both are done, the connection test fails.
3. For NTFS permissions only: the Active Directory sign-in must be set up. Genie uses that directory to turn the Windows accounts and groups on each file into people in Genie.

Connect the file server

1. In the Operations knowledge base, choose Data sources → Add Data Source → Windows file server (SMB).
2. Under File server, enter the Server files.contoso.test (only the server, without a share or folder) and keep Port 445.
3. Under Sign-in, keep NTLM. Choose Kerberos only for a server that refuses NTLM, then fill in Realm and KDC (domain controller).
4. Under Share names, enter Operations. This is optional: leave it empty to list every share the account can see, and fill it in when the account may not list shares.
5. Enter the Service account CONTOSO\svc-genie and its Password. Domain is optional when the account name includes it.
6. Optionally, under Files to import, narrow what is imported: Only paths matching, Skip paths matching (for example **/Archive/**), File types and Largest file (50 MB by default). Skip hidden and system files is on by default.
7. Choose Next. Genie signs in and shows What the file server holds: folders, files, how many files and megabytes will be imported, and what the filters leave out.
8. Under Resources, tick the shares or folders, then complete Strategy as for any data source.

Each sync lists the selected folders and reads only the files whose size or modification time changed. A changed file keeps its document and gets a new version. A file that is open and locked (in Excel, for example) is retried at the next sync. If a selected share or folder no longer exists, the sync stops before it deletes anything; a folder the service account may no longer list keeps its files and is reported in the sync history.

Limits: DFS namespaces are not followed, so enter the server that actually holds the share. One data source covers at most 200,000 files and 64 folder levels. Share-level permissions are not read: keep the share open to the people concerned and control access with NTFS permissions.

Exchange shared mailbox

Exchange Online mailbox syncs the mail of shared mailboxes, such as an HR requests inbox hr-requests@contoso.test, into a knowledge base. Each email becomes a document, and each attachment its own document next to it.

Restrict the app before the first sync

The Microsoft Graph permission Genie needs, Mail.Read, lets the app read every mailbox in your organisation, including those of managers and HR. Limit it to the shared mailboxes you want to index before you enter the credentials in Genie.

Mailbox permissions cannot be carried over: everyone who can read the knowledge base can read every synced email and attachment. Give each mailbox its own knowledge base, for example "HR inbox", shared only with the team that handles the mailbox.

What your Microsoft 365 administrator prepares

1. Register an app in Entra ID as for SharePoint, or reuse that app.
2. Add the application permission Mail.Read and grant admin consent. Mail.ReadBasic.All is not enough: it leaves out message bodies and attachments.
3. An Exchange administrator restricts the app to the chosen mailboxes with an Exchange Online Application Access Policy (New-ApplicationAccessPolicy) or with RBAC for Applications. Genie writes the commands for you: in the data source dialog, enter the mailboxes and choose Show the PowerShell commands. A new policy can take up to an hour to apply.

Connect a mailbox

1. Choose Add Data Source → Exchange Online mailbox.
2. Enter the app's credentials, or reuse an app as for SharePoint.
3. Under Shared mailboxes, enter the addresses, one per line (up to 20).
4. Under Filters, choose what to take: Received after, Only from sender domains, Skip sender domains and Subject keywords. Skip automatic mail (out-of-office replies, delivery reports, newsletters) and Trim quoted replies are on by default.
5. Under Attachments, keep Sync attachments on, and adjust Attachment types and Largest attachment (20 MB by default). Include inline images is off, so signature logos are skipped.
6. Choose Test Connection. Mailboxes reached lists each mailbox's folders with their message counts, and names any mailbox Genie cannot read (not found, or excluded by the access policy).
7. Under Resources, choose the folders. Each mailbox's Inbox is selected by default; a folder brings its subfolders, and a whole mailbox brings every folder except Deleted Items, Junk Email, Drafts, Outbox and a few system folders.
8. Under Strategy, keep Incremental and keep Sync deletions (remove knowledge when deleted at source) on, so mail that is deleted or moved away also leaves the knowledge base.

Changing a filter later applies at the next sync: mail that no longer matches is removed and mail that now matches is added. Drafts, encrypted (S/MIME) mail, attached Outlook items and cloud attachments (OneDrive or SharePoint links) are not imported.

To stop indexing a mailbox, remove it from Shared mailboxes and save; its mail leaves the knowledge base at the next sync. Only then remove it from the access policy.

Microsoft Teams files

Microsoft Teams syncs the files of Teams channels, the ones people see on a channel's Files tab, and optionally the channel conversations.

Your Microsoft 365 administrator adds these application permissions to the Genie app (the SharePoint or Exchange app can be reused) and grants admin consent:

Permission	Needed for
Team.ReadBasic.All and Channel.ReadBasic.All	Listing teams and channels, private and shared channels included
Files.Read.All (or Sites.Read.All)	Channel files
ChannelMessage.Read.All	Channel conversations. Microsoft must approve your app for this permission; until then the files still sync
GroupMember.Read.All and ChannelMember.Read.All	Only to keep the channel's permissions (see Keep the source's permissions)

To connect Teams:

1. Choose Add Data Source → Microsoft Teams, and enter or reuse the app's credentials.
2. Under What to sync, keep Channel files. Add Channel messages if conversations should be searchable too, and choose how far back the first sync reads under Message history.
3. Under Resources, tick teams or single channels. A team brings all its channels, including channels created later. Private channels are marked with a lock.
4. Complete Strategy, optionally with Mirror source permissions.

Channel files sync like a SharePoint folder. Each conversation becomes one document with its replies. Chats outside channels (one-to-one, group and meeting chats) are not synced.

Website

Website crawls a website into a knowledge base: public product pages, a supplier portal or your intranet. It starts from sitemaps and start pages, stays inside the scope you set and honours the site's robots.txt.

To add a website:

1. Choose Add Data Source → Website.
2. Under Where to start, enter Sitemap URLs (for example https://www.contoso.test/sitemap.xml) and/or Start pages, one per line. Every in-scope page of a sitemap is read; on start pages, Genie also follows links up to the link depth.
3. Adjust the options in the table below. For a site that needs a fixed sign-in header, add it under Request headers.
4. Choose Next. Genie shows What the crawler can reach: the robots.txt rules, how many pages each sitemap lists and how many are in scope, and the links found on each start page.
5. Choose the schedule (every 6 hours by default) and Create & Sync Now.

Setting	Default	What it does
Pages to crawl	The whole site	The whole site stays on the start points' host. Only below the start pages keeps to their folder, for example /products/.
Link depth	2	How many clicks away from a start page to go. 0 reads only the start pages.
Maximum pages	500	Pages and documents per crawl, closest first. Pages beyond the limit leave the knowledge base.
Only URLs matching, Skip URLs matching	Empty	One pattern per line, such as /news/* or /tag/*. Exclusions win.
Ignore query strings	Off	Treats /page?a=1 and /page?a=2 as one page.
Respect robots.txt	On	Skips pages the site closes to crawlers. Turn it off only for a site your company owns.
Delay between requests, Parallel requests	1 second, 2	Keeps the load on the site low.
Download linked documents	Off	PDF, Word, Excel and PowerPoint files linked from crawled pages become documents too.
Largest document	20 MB	Larger documents are skipped.

To crawl an intranet such as https://intranet.contoso.test/, the host must first be allowed by a system administrator (see Network and antivirus), and your Genie provider must make it reachable. Pages that show their text only through JavaScript, and pages behind a sign-in form, cannot be crawled.

A changed page replaces its document, which keeps its tags and gets a new version. A page that returns "not found" is removed at once; pages that are no longer reached are removed only after a crawl that completed without errors.

Schedules and sync history

Sync options

The Strategy step of every data source has these options. To change them later, open the card's ⋯ menu and choose Edit.

Setting	What it does
Sync schedule	Every 30 min, Every hour, Every 6 hours (the default), Every 12 hours or Daily.
Sync mode	Incremental (the default) imports only what changed since the last sync. Full reads everything again at every run; keep Incremental.
Conflict strategy	Leave it on Overwrite, the default.
Sync deletions (remove knowledge when deleted at source)	On by default: content deleted in the source, or moved out of the selection, leaves the knowledge base at the next sync. Off: it stays until you delete it.
Mirror source permissions	Shown for sources that can keep their permissions. See Keep the source's permissions.

Run, pause and delete

The ⋯ menu on a data source card has:

• Sync Now: syncs right away, also when the data source is paused. Saving changes with Edit does not start a sync, so choose Sync Now afterwards to apply them.
• Logs: the sync history, below.
• Webhook: faster updates, see Faster updates with webhooks.
• Pause and Resume: stop and restart the scheduled syncs.
• Delete: removes the data source. Documents it already synced stay in the knowledge base; delete them there if you no longer need them.

Only one sync of a data source runs at a time.

Sync history

Logs opens Sync History: the number of Runs, how many succeeded or failed, and the Items processed, then each run with its start and end time and how many documents were added, updated, deleted, skipped or failed. Open a run to see the Failed documents and the reason for each.

Run status	Meaning
Success	Everything in the selection was processed.
Partial	Some documents failed (too large, unreadable, rejected by the virus scan); the rest was imported. Failed documents are retried at the next sync.
Failed	The sync could not run, for example because the credentials were refused or a selected folder no longer exists.
Canceled	Another sync of the same data source was still running. Nothing needs fixing.

The card shows the data source's status: Connected, Paused or Error, with the error message. While a data source shows Error, its scheduled syncs are skipped. Fix the cause (an expired secret, a removed folder, a withdrawn permission), then choose Sync Now. When the sync succeeds, the status returns to Connected.

Deleted and changed content

• With Sync deletions (remove knowledge when deleted at source) on, a file, email or page deleted in the source, or moved out of the selection, is removed from the knowledge base at the next sync.
• Genie does not delete on incomplete information: if a selected site, library, folder, share or mailbox cannot be read or no longer exists, the sync fails and deletes nothing. Update the selection, then sync again.
• A file changed in the source replaces its document at the next sync. For file servers, mailboxes, websites and Teams conversations, the document keeps its tags and citations and gets a new version.

Faster updates with webhooks

A schedule checks the source at fixed times, so a change can take up to the schedule's interval to reach Genie. A webhook lets the source tell Genie as soon as something changes, and the knowledge base is up to date within about a minute.

• Microsoft 365 change notifications are available for SharePoint / OneDrive and Exchange data sources. Microsoft 365 tells Genie when a file or an email changes.
• A webhook URL can be called by any other system, for example your intranet's content system right after it publishes a page.

To set it up:

1. Open the ⋯ menu on the data source card and choose Webhook.
2. Turn on Enable the webhook.
3. For SharePoint / OneDrive or Exchange, turn on Sync when Microsoft 365 reports a change. Genie subscribes to the changes with the app's existing permissions and renews the subscription by itself; the panel lists each subscription with its expiry date and any error from Microsoft Graph. This works only when your Genie can be reached from the internet over HTTPS: ask your Genie provider whether yours can.
4. For another system, choose the Authentication: HMAC signature (recommended) or Token header (for tools that can only send a fixed header). Optionally set Wait before syncing (60 seconds by default; calls within that time lead to one sync) and Allowed addresses (the IP addresses the calls may come from).
5. Choose Save. Copy the Secret shown once and give it, with the Webhook URL, to the team that sets up the calling system; Example call shows what the call looks like. Rotate secret replaces the secret, and the old one stops working at once.

A call runs a normal incremental sync of the whole data source. A paused data source ignores calls. The webhook is not affected by the workspace's network restriction, so use Allowed addresses to limit who may call it. Activity shows the Last call, Last sync queued and Syncs queued.

Keep the source's permissions

By default, everyone who can read the knowledge base can read every synced document, whoever may open it in the source. For SharePoint / OneDrive, Microsoft Teams, the Windows file server, Confluence and Power BI, turn on Mirror source permissions in the Strategy step: each document then stays visible only to the people and groups who may open it in the source, matched to Genie accounts by their Microsoft Entra ID or email and checked again at every sync. People without a Genie account are left out and listed in the sync history, and the card shows Permissions mirrored. Mailbox permissions cannot be carried over, so keep each shared mailbox in its own knowledge base. For SharePoint groups, extra rules you set yourself and what each person then sees, see Document permissions.

Good to know

• Admins and Owners add, edit, sync and delete data sources. The person who created a knowledge base sees its data sources and their sync history but cannot change them.
• Data sources exist only for document knowledge bases, not FAQ knowledge bases, and only once the knowledge base has been created.
• Saved credentials are never shown again. To change them, edit the data source and choose Replace. A reused app is copied, not linked: after you renew a client secret, update every data source that uses it.
• Synced documents skip approval unless the knowledge base has Also require approval for synced documents turned on (see Knowledge bases).
• Files imported by data sources are scanned for viruses only when a system administrator turns this on (see Network and antivirus).
• The knowledge base's upload size limit also applies to synced files. To change it, ask your Genie provider.
• Microsoft 365 sources work with the global Microsoft 365 cloud only.

Related pages

• Knowledge bases
• Document permissions
• Databases and Power BI
• Network and antivirus
• Members and sign-in
• Ask your documents (user guide)