Data Loaders

Loaders are the data ingestion agents in Katara 1.0 that pull information from external sources into the system for further processing. These agents retrieve content from various platforms, capturing relevant data while ignoring non-essential elements like images and code blocks.

Each loader can be scheduled for auto-refresh and has the ability to write to the DocDB, ensuring the data is constantly up-to-date. Loaders can also index all URLs or files they retrieve, allowing them to track changes in the source, identify new or missing content, and prompt the user with handling options.