daks lib utilities

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for mhferjani from gravatar.com mhferjani

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License (MIT)
  • Author: Mohamed Hedi FERJANI
  • Tags dask , python
  • Requires: Python >=3.10
  • Provides-Extra: dev , doc
Classifiers
Report project as malware

Project description

DASK utility library

main PyPI version Python DownloadsThis library provides custom Dask classes for connect and execute computation on DASK cluster.

  • Scan and parse Dask config file (cfg parser)
  • Extension of the default Dask Client
  • Custom functions to retrieve information from Dask cluster
  • Test utilities: generate local cluster and initialize client
  • Decorators: Singleton decorator
  • Packaging utility to create a zip dask tasks dependencies
  • Queue utility to deal with futures (Future) for different Dask clients

Requirements

  • Python >= 3.10 (not yet tested with Python =2.7 and =3.4)

Installation

To install this library:

poetry add dask-utils

or

uv add dask-utils

or directly with pip

Dependencies

  • dask[distribute]

Configuration

Default config file [useful for development purposes]

By default, dask-utils come with a build-in config file named client.cfg. client.cfg contains a minimum of config options which client used to perform connection to Dask scheduler and submitting computation on cluster.

[client]
host = localhost
port = 8786
timeout=10
name=client_name

[worker]
memory=1e5
cpu=2
process=1

Example:

from  dask_utils import Client
#Init client and connect to localhost scheduler on 8786 port
c= Client()
#sublit compuation on localhost dask
c.submit(f,*args)

Note : field names in worker section can be changed if --resources "XX=YY" params used when launched worker respect the same convention name

Custom config file [Recommended for production usage]

You can overwrite the default config file client.cfg by creating your custom file config into your project. In this case, you should define DASK_UTILS_ROOT_CONFIG variable env for the runtime.

Examples:

export DASK_UTILS_ROOT_CONFIG=/etc/mlbox/client.cfg

from  dask_utils import Client
#Init client and connect to scheduler defined into DASK_UTILS_ROOT_CONFIG 
c= Client()
c.submit(f,*args)

Note : In development time, you can overwrite options defined in config file (defined in DASK_UTILS_ROOT_CONFIG) by passing options directly as parameters to Client class.

from  dask_utils import Client
#overwrite config
my_custom_address='192.168.2.2:52879'
c= Client(address=my_custom_address)
c.submit(f,*args)

or

my_custom_resources:{'cpu':1,'memory':'2G'}
c.submit(f,*args,resources=my_custom_resources)

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for mhferjani from gravatar.com mhferjani

Unverified details

These details have not been verified by PyPI
Meta
  • License: MIT License (MIT)
  • Author: Mohamed Hedi FERJANI
  • Tags dask , python
  • Requires: Python >=3.10
  • Provides-Extra: dev , doc
Classifiers

Release history Release notifications | RSS feed

This version

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_utils-0.1.0.tar.gz (75.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

dask_utils-0.1.0-py3-none-any.whl (14.7 kB view details)

Uploaded Python 3

File details

Details for the file dask_utils-0.1.0.tar.gz.

File metadata

  • Download URL: dask_utils-0.1.0.tar.gz
  • Upload date:
  • Size: 75.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for dask_utils-0.1.0.tar.gz
Algorithm Hash digest
SHA256 120455654af39a1ab684fd746114c68e6d1161aa0f4710b5929ce3709f547ce5
MD5 b103f9134d2c9ea3a2924d7b855bd9e6
BLAKE2b-256 b0e93b1946e488716419ed8d286e66c3a3f400eabcf65985fa6f9d0acdd06c4a

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_utils-0.1.0.tar.gz:

Publisher: publish.yml on mhferjani/dask-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_utils-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: dask_utils-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 14.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for dask_utils-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 5f8f550d82241aaf7d4d4891e81ab38567248d8f16fa3a5962d1c354a64e4548
MD5 74e28af9cd439ac86b68afeae3e04451
BLAKE2b-256 dece65c5f25eb4ba0e71ad4061943671a4c438fc57a3833fe92ade8c23944758

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_utils-0.1.0-py3-none-any.whl:

Publisher: publish.yml on mhferjani/dask-utils

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page