No project description provided
Project description
WordLift Python SDK
A Python toolkit for orchestrating WordLift imports: fetch URLs from sitemaps, Google Sheets, or explicit lists, filter out already imported pages, enqueue search console jobs, push RDF graphs, and call the WordLift APIs to import web pages.
Features
- URL sources: XML sitemaps (with optional regex filtering), Google Sheets (
urlcolumn), or Python lists. - Change detection: skips URLs that are already imported unless
OVERWRITEis enabled; re-imports whenlastmodis newer. - Web page imports: sends URLs to WordLift with embedding requests, output types, retry logic, and pluggable callbacks.
- Search Console refresh: triggers analytics imports when top queries are stale.
- Graph templates: renders
.ttl.liquidtemplates underdata/templateswith account data and uploads the resulting RDF graphs. - Extensible: override protocols via
WORDLIFT_OVERRIDE_DIRwithout changing the library code.
Installation
pip install wordlift-sdk
# or
poetry add wordlift-sdk
Requires Python 3.10–3.13.
Configuration
Settings are read in order: config/default.py (or a custom path you pass to ConfigurationProvider.create), environment variables, then (when available) Google Colab userdata.
Common options:
WORDLIFT_KEY(required): WordLift API key.API_URL: WordLift API base URL, defaults tohttps://api.wordlift.io.SITEMAP_URL: XML sitemap to crawl;SITEMAP_URL_PATTERNoptional regex to filter URLs.SHEETS_URL,SHEETS_NAME,SHEETS_SERVICE_ACCOUNT: use a Google Sheet as source; service account points to credentials file.URLS: list of URLs (e.g.,["https://example.com/a", "https://example.com/b"]).OVERWRITE: re-import URLs even if already present (defaultFalse).WEB_PAGE_IMPORT_WRITE_STRATEGY: WordLift write strategy (defaultcreateOrUpdateModel).EMBEDDING_PROPERTIES: list of schema properties to embed.WEB_PAGE_TYPES: output schema types, defaults to["http://schema.org/Article"].GOOGLE_SEARCH_CONSOLE: enable/disable Search Console handler (defaultTrue).CONCURRENCY: max concurrent handlers, defaults tomin(cpu_count(), 4).WORDLIFT_OVERRIDE_DIR: folder containing protocol overrides (defaultapp/overrides).
Example config/default.py:
WORDLIFT_KEY = "your-api-key"
SITEMAP_URL = "https://example.com/sitemap.xml"
SITEMAP_URL_PATTERN = r"^https://example.com/article/.*$"
GOOGLE_SEARCH_CONSOLE = True
WEB_PAGE_TYPES = ["http://schema.org/Article"]
EMBEDDING_PROPERTIES = [
"http://schema.org/headline",
"http://schema.org/abstract",
"http://schema.org/text",
]
Running the import workflow
import asyncio
from wordlift_sdk import run_kg_import_workflow
if __name__ == "__main__":
asyncio.run(run_kg_import_workflow())
The workflow:
- Renders and uploads RDF graphs from
data/templates/*.ttl.liquidusing account info. - Builds the configured URL source and filters out unchanged URLs (unless
OVERWRITE). - Sends each URL to WordLift for import with retries and optional Search Console refresh.
You can build components yourself when you need more control:
import asyncio
from wordlift_sdk.container.application_container import ApplicationContainer
async def main():
container = ApplicationContainer()
workflow = await container.create_kg_import_workflow()
await workflow.run()
asyncio.run(main())
Custom callbacks and overrides
Override the web page import callback by placing web_page_import_protocol.py with a WebPageImportProtocol class under WORDLIFT_OVERRIDE_DIR (default app/overrides). The callback receives a WebPageImportResponse and can push to graph_queue or entity_patch_queue.
Templates
Add .ttl.liquid files under data/templates. Templates render with account fields available (e.g., {{ account.dataset_uri }}) and are uploaded before URL handling begins.
Testing
poetry install --with dev
poetry run pytest
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file wordlift_sdk-2.7.1.tar.gz.
File metadata
- Download URL: wordlift_sdk-2.7.1.tar.gz
- Upload date:
- Size: 154.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4d4affdc9c166707f6e4d887a050034446f9c624aa943c02131d7a60237df8ea
|
|
| MD5 |
cccb7f0b5786571496c0c7169130d3a5
|
|
| BLAKE2b-256 |
3eac9c720b51cff8c3d70d56e2c55574017825510778fbc37cdc7c850fb06c45
|
File details
Details for the file wordlift_sdk-2.7.1-py3-none-any.whl.
File metadata
- Download URL: wordlift_sdk-2.7.1-py3-none-any.whl
- Upload date:
- Size: 183.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
bba5576e12f57999033e2a7c2da8802909f72b1e0689d03c1ef2d75d84195525
|
|
| MD5 |
53632e5e1ebba0d33608ec78857c3083
|
|
| BLAKE2b-256 |
f605f9ef517453de75489f69d5eb789f5baa4fcd0ec6d4dbe1c2a9d31d8b1654
|
