Downloading…

Helper functions to download XML datasets and pretrained models

This module is the xcube downloading counterpart of fastai’s External data. Specifically, untar_data is repleaced with untar_xxx.

To download any of the datasets or pretrained weights, simply run untar_xxx by passing any dataset name mentioned above like so:

path = untar_xxx(XURLs.MIMIC3_L2R)
path.ls()

>> (#1) [Path('/home/deb/.xcube/data/mimic3/l2r')]

To download model pretrained weights:

path = untar_xxx(XURLs.)
path.ls()

>> (#2) []

source

xcube_cfg

 xcube_cfg ()

Config object for xcube’s config.ini

This is a basic Config file that consists of data, model, storage and archive. All future downloads occur at the paths defined in the config file based on the type of download. For example, all future xcube datasets are downloaded to the data while all pretrained model weights are download to model unless the default download location is updated.

cfg = xcube_cfg()
cfg.data, cfg.path('archive')
('data', Path('/home/deb/.xcube/archive'))

source

xcube_path

 xcube_path (folder:str)

Local path to folder in Config

xcube_path('archive')
Path('/home/deb/.xcube/archive')

source

XURLs

 XURLs ()

Global cosntants for datasets and model URLs.

The default local path is at ~/.xcube/archive/ but this can be updated by passing a different c_key. Note: c_key should be one of 'archive', 'data', 'model', 'storage'.

url = XURLs.MIMIC3_L2R
local_path = XURLs.path(url)
test_eq(local_path.parent, xcube_path('archive'))
local_path
Path('/home/deb/.xcube/archive/mimic3_l2r.tgz')
local_path = XURLs.path(url, c_key='model')
test_eq(local_path.parent, xcube_path('model'))
local_path
Path('/home/deb/.xcube/models/mimic3_l2r.tgz')

source

untar_xxx

 untar_xxx (url:str, archive:pathlib.Path=None, data:pathlib.Path=None,
            c_key:str='data', force_download:bool=False,
            base:str='~/.xcube')

Download url using FastDownload.get

Type Default Details
url str File to download
archive Path None Optional override for Config’s archive key
data Path None Optional override for Config’s data key
c_key str data Key in Config where to extract file
force_download bool False Setting to True will overwrite any existing copy of data
base str ~/.xcube Directory containing config file and base of relative paths
Returns Path Path to extracted file(s)

untar_xxx is a thin wrapper for FastDownload.get. It downloads and extracts url, by default to subdirectories of ~/.xcube, and returns the path to the extracted data. Setting the force_download flag to ‘True’ will overwrite any existing copy of the data already present. For an explanation of the c_key parameter, see XURLs.

p = untar_xxx(XURLs.MIMIC3_L2R)
p
Path('/home/deb/.xcube/data/mimic3_l2r')
list(p.glob('**/*.csv'))
[Path('/home/deb/.xcube/data/mimic3_l2r/code_descriptions.csv'),
 Path('/home/deb/.xcube/data/mimic3_l2r/mimic3-9k.csv')]