MLflow adapter for CrateDB

» Documentation | Changelog | PyPI | Issues | Source code | License | CrateDB | Community Forum

About

MLflow is an open source AI engineering platform for managing the whole ML lifecycle for agents, LLMs, and ML models, including experimentation, reproducibility, and deployment.

CrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. CrateDB is based on Lucene and Elasticsearch, but compatible with PostgreSQL.

Details

MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

The MLflow adapter for CrateDB is an adapter for MLflow to use CrateDB as a storage database for its various subsystems.

Features

The MLflow Experiment Tracking subsystem is an API and UI for logging and recording parameters, code versions, metrics, and output files when running your machine learning code, and for later visualizing the results by querying experiments across code, data, and config.

The MLflow Model Registry is a centralized model store, set of APIs and a UI designed to collaboratively manage the full lifecycle of a machine learning model, including lineage, versioning, aliasing, metadata tagging, and annotation support.

The MLflow Dataset Tracking module is a comprehensive solution for dataset management throughout the ML model development workflow. It enables you to track, version, and manage datasets used in training, validation, and evaluation, providing complete lineage from raw data to model predictions.

What's inside

The source code of the mlflow-cratedb package, which implements the MLflow adapter for CrateDB. It works with both CrateDB and CrateDB Cloud.

The source code is effectively a few monkey patches that amalgamate MLflow with the necessary changes to support CrateDB. The patches are curated until the adapter can eventually be upstreamed into MLflow mainline as another storage database type.

Usage

General information about what you can do with MLflow, and how, can be found on the Machine Learning with CrateDB and examples about MLflow and CrateDB.

Install

For installation per PyPI package, OCI image, and usage information, please refer to the handbook document, and the documentation about container usage.

The repository also includes a ready-to-run Compose project that provisions MLflow, CrateDB, and RustFS, see MLflow and CrateDB with Docker Compose.

To set up a development environment on your workstation, please refer to the development sandbox documentation. When you see the software tests succeed, you should be ready to start hacking.

Project Information

Resources

• Source code
• Documentation
• Python Package Index (PyPI)

Contributing

The MLflow adapter for CrateDB is an open-source project, and is managed on GitHub. Feel free to use the adapter as provided or else modify / extend it as appropriate for your own applications.

Any kind of contribution, feedback, or patch, is much welcome. Create an issue or submit a patch if you think we should include a new feature, or to report or fix a bug.

Acknowledgements

Kudos to the authors of all the many software components this library is inheriting from and building upon, with special thanks to Siddharth Murching, Corey Zumar, Harutaka Kawamura, Ben Wilson, and all other contributors for conceiving and maintaining MLflow.

Andreas Nigg for contributing the tracking_merlion.py and tracking_pycaret.py ML experiment programs, using Merlion and PyCaret.

License

The project is licensed under the terms of the Apache License 2.0, like MLflow and CrateDB, see LICENSE.