dlt.config¶
Options¶
parse¶
-
dlt.config.
parse
(verbose=False)¶ Parses Command Line Arguments using the built-in settings (and any added extra).
Parameters: verbose (bool, optional) – Print the parsed settings (default False). Comes with built-in commonly used settings. To add extra settings use
add_extras()
.For a comprehensive list of available settings, their categories and use of configuration files please see the Workflow Quickstart example.
add_extras¶
-
dlt.config.
add_extras
(extras)¶ Adds extra options for the parser, in addition to the built-in ones.
- Args:
- extras (list or dict, optional): Extra command line arguments to parse.
- If a list is given, a new category extras is added with the listed arguments. If a dict is given, the keys contain the category names and the elements are lists of arguments (default None).
- Note:
The extras parameter must have one of the following structures:
# List of arguments as dicts (this will add extras to ) extras = [dict(flag='--arg1', type=float), dict(flag='--arg2', type=int), dict(flag='--other_arg', type=int)] # OR dictionary with category names in keys # and lists of dicts for as values for each category extras = {'my_category': [dict(flag='--arg1', type=float), dict(flag='--arg2', type=int)] 'other_category': [dict(flag='--other_arg', type=int)]}
The keys accepted in the argument dictionaries are the ones used as arguments in the argparse package add_argument function.
- DuplStdOut
- Warning:
- The parser takes strings as inputs, so passing ‘False’ at the command for a bool argument, it will be converted to True. Instead of using type=bool, use type=dlt.util.str2bool.
make_subsets¶
-
dlt.config.
make_subsets
(subsets)¶ Splits command line argument categories into subsets.
The subset names are appended at the end of each of the categories options after an underscore. For example the dataset category can be split into training and validation subsets by passing the following as subsets:
subsets = {dataset=['training', 'validation']}
This will cause:
- `--data` to split into `--data_training` and `--data_validation`. - `--load_all` to split into `--load_all_training` and `--load_all_validation` - etc ...
Parameters: subsets (dict, optional) – Dictionary containing parameter categories as keys and its subsets as a list of strings (default None).
Data¶
torchvision_dataset¶
-
dlt.config.
torchvision_dataset
(transform=None, target_transform=None, train=True, subset=None)¶ Creates a dataset from torchvision, configured using Command Line Arguments.
Parameters: - transform (callable, optional) – A function that transforms an image (default None).
- target_transform (callable, optional) – A function that transforms a label (default None).
- train (bool, optional) – Training set or validation - if applicable (default True).
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- dataset: –data, –torchvision_dataset.
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.Warning
Unlike the torchvision datasets, this function returns a dataset that uses NumPy Arrays instead of a PIL Images.
directory_dataset¶
-
dlt.config.
directory_dataset
(load_fn=<function imread>, preprocess=None, subset=None)¶ Creates a
dlt.util.DirectoryDataset
, configured using Command Line Arguments.Parameters: - load_fn (callable, optional) – Function that loads the data files
(default
dlt.hdr.imread()
). - preprocess (callable, optional) – A function that takes a single data point from the dataset to preprocess on the fly (default None).
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- dataset: –data, –load_all, –extensions.
- dataloader: –num_threads.
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.- load_fn (callable, optional) – Function that loads the data files
(default
loader¶
-
dlt.config.
loader
(dataset, preprocess=None, subset=None, worker_init_fn=None)¶ Creates a torch DataLoader using the dataset, configured using Command Line Arguments.
Parameters: - dataset (Dataset) – A torch compatible dataset.
- preprocess (callable, optional) – A function that takes a single data point from the dataset to preprocess on the fly (default None).
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- dataloader: –batch_size, –num_threads, –pin_memory, –shuffle, –drop_last.
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.
Models¶
model_checkpointer¶
-
dlt.config.
model_checkpointer
(model, preprocess=None, subset=None)¶ Returns the model checkpointer. Configurable using command line arguments.
The function also loads any previous checkpoints if present.
Parameters: - model (nn.Module) – The network for the checkpointer.
- preprocess (callable, optional) – Callable to change the loaded state dict before assigning it to the network.
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- general: –experiment_name, –save_path.
- model: –overwrite_model_chkp, –timestamp_model_chkp, –count_model_chkp.
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.
Optimization¶
optimizer¶
-
dlt.config.
optimizer
(model, extra_params=None, subset=None)¶ Returns the optimizer for the given model.
Parameters: - model (nn.Module) – The network for the optimizer.
- extra_params (generator, optional) – Extra parameters to pass to the optimizer.
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- optimizer: –optimizer, –lr, –momentum,
- –dampening, –beta1, –beta2, –weight_decay.
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.
optimizer_checkpointer¶
-
dlt.config.
optimizer_checkpointer
(optimizer, subset=None)¶ Returns the optimizer checkpointer. Configurable using command line arguments.
The function also loads any previous checkpoints if present.
Parameters: - optimizer (torch.optim.Optimizer) – The optimizer.
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- general: –experiment_name, –save_path.
- model: –overwrite_optimizer_chkp, –timestamp_optimizer_chkp,
- –count_optimizer_chkp.
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.
scheduler¶
-
dlt.config.
scheduler
(optimizer, subset=None)¶ Returns a scheduler callable closure which accepts one argument.
Configurable using command line arguments.
Parameters: - optimizer (torch.optim.Optimizer) – The optimizer for the scheduler.
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- scheduler: –lr_schedule, –lr_step_size, –lr_patience,
- –lr_cooldown, –lr_ratio, –lr_min,
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.
Training¶
trainer_checkpointer¶
-
dlt.config.
trainer_checkpointer
(trainer, subset=None)¶ Returns the trainer checkpointer. Configurable using command line arguments.
The function also loads any previous checkpoints if present.
Parameters: - trainer (BaseTrainer) – The trainer for the checkpointer.
- subset (string, optional) – Specifies the subset of the relevant categories, if any of them was split (default, None).
Relevant Command Line Arguments:
- general: –experiment_name, –save_path.
- trainer: –overwrite_trainer_chkp, –timestamp_trainer_chkp, –count_trainer_chkp.
Note
Settings are automatically acquired from a call to
dlt.config.parse()
from the built-in ones. Ifdlt.config.parse()
was not called in the main script, this function will call it.