attelo.harness package

attelo experimental harness helpers

The modules here are meant to help with building your own test harnesses around attelo. They provide opinionated support for experiment layout and interfacing with attelo

Submodules

attelo.harness.config module

Configuring the harness

class attelo.harness.config.ClusterStage

Bases: enum.Enum

What stage of cluster usage we are at

This is used when you want to distribute the evaluation across multiple nodes of a cluster.

The idea is that you would run the harness in separate stages:

  • a single “start” stage, then
  • in parallel * nodes running “main” stages for some folds * a node running a “combined_model” stage
  • finally, a single “end” stage
combined_models = <ClusterStage.combined_models: 3>
end = <ClusterStage.end: 4>
main = <ClusterStage.main: 2>
start = <ClusterStage.start: 1>
class attelo.harness.config.DataConfig

Bases: attelo.harness.config.DataConfig

Data tables read during harness evaluation

This class may be folded into HarnessConfig eventually

class attelo.harness.config.EvaluationConfig

Bases: attelo.harness.config.EvaluationConfig

Combination of learners, decoders and decoder settings for an attelo evaluation

The settings can really be of type that has a ‘key’ field; but you should have a way of extracting at least a DecodingMode from it

Parameters:
  • learner (Keyed learnercfg) – Some sort of keyed learner configuration. This is usually of type LearnerConfig but there are cases where you have fancier objects in place
  • parser (Keyed (learnercfg -> Parser)) – A (keyed) function that builds a parser from whatever learner configuration you used in learner
  • settings (Keyed (???)) –
classmethod simple_key(learner, decoder)

generate a short unique name for a learner/decoder combo

class attelo.harness.config.Keyed

Bases: attelo.harness.config.Keyed

A keyed object is just any object that is attached with a short unique (mnemonic) identifier.

Keys often appear in filenames so it’s best to avoid whitespace, fancy characters, and for portability reasons, anything non-ASCII.

class attelo.harness.config.LearnerConfig

Bases: attelo.util.Team

Combination of an attachment and a relation learner variant

class attelo.harness.config.RuntimeConfig

Bases: attelo.harness.config.RuntimeConfig

Harness runtime options.

These are mostly relevant to when using the harness on a cluster.

Parameters:
  • mode (string ('resume' or 'jumpstart') or None) –
    • jumpstart: copy model and fold files from a previous evaluation
    • resume: pick an evaluation up from where it left off
  • folds ([int] or None) – Which folds to run the harness on. None to run on all folds
  • n_jobs (int (-1 or natural)) – Number of parallel jobs to run (-1 for max cores). See joblib doc for details
  • stage (ClusterStage or None) – Which evaluation stage to run
classmethod empty()

Empty configuration

attelo.harness.evaluate module

attelo.harness.example module

attelo.harness.graph module

attelo.harness.interface module

Basic interface that all test harnesses should respect

class attelo.harness.interface.Harness(dataset, testset)

Bases: object

Test harness configuration.

Among other things, this is about defining conventions for filepaths.

Notes

You should have a method that calls load. It should be invoked once before running the harness. A natural idiom may be to implement a single run function that does this.

combined_dir_path()

Return path to directory where combined/global models should be stored

This would be for all training data, ie. without paying attention to folds

Returns:
Return type:filepath
config_files

Files needed to reproduce the configuration behind a particular set of scores.

Will be copied into the provenance section of the report.

Some harnesses have parameter files that should be saved in case there is any need to reproduce results much futher into the future. Specifying them here gives you some extra insurance in case you neglect to put them under version control.

create_folds(mpack)

Generate the folds dictionary for the given multipack, optionally caching them to disk

In some harness configurations, it may make sense to have a fixed set of folds rather than generating them on the fly

Returns:fold_dict – dictionary from document names to fold
Return type:dict(string, int)
decode_output_path(econf, fold)

Return path to output graph for given fold and config

detailed_evaluations

Set of evaluations for which we would like detailed reporting

eval_dir

Directory to store evaluation results.

Basically anything that should be considered as important for long-term archiving and reproducibility

evaluations

List of evaluations to use on the training data

fold_dir_path(fold)

Return path to working directory for a given fold

Parameters:fold (int) –
Returns:
Return type:filepath
fold_file

Path to the fold allocation dictionary

graph_docs

List of document names for which we would like to generate graphs

load(runcfg, eval_dir, scratch_dir)
Parameters:
  • eval_dir (filepath) – Directory to store evaluation results, basically anything that should be considered as important for long-term archiving and reproducibility
  • scratch_dir (filepath) – Directory for relatively emphemeral intermediary results. One would be more inclined to delete scratch than eval
  • runcfg (RuntimeConfig or None) – Runtime configuration. None for default options

See also

See()

metrics

Selection of metrics to compute in reports.

model_paths(rconf, fold, parser)

Return attelo model paths in dictionary form

Parameters:
Returns:
Return type:

Dictionary from attelo parser cache keys to paths
mpack_paths(test_data, stripped=False)

Return a dict of paths needed to read a datapack.

Usual keys are: * edu_input * pairings * features * vocab

Parameters:
  • test_data (bool) – If True, it’s test data we wanted.
  • stripped (bool, defaults to False) – If True, return path for a “stripped” version of the data (faster loading, but only useful for scoring).
Returns:

res – Paths to files that enable to read a datapack.
Return type:

dict
report_digits

Number of digits to display floats in reports.

report_dir_path(test_data, fold=None, is_tmp=True)

Path to a directory containing reports.

Parameters:
  • test_data (bool) – If True, the report is about the test set, otherwise the (usually, training) dataset.
  • fold (int, optional) – Number of the fold under scrutiny ; if None, all folds.
  • is_tmp (bool, defaults to True) – If True, only return the path to a provisional report in progress.
runcfg

Runtime configuration settings for the harness

scratch_dir

Directory for relatively emphemeral intermediary results.

One would be more inclined to delete scratch than eval

test_evaluation

The test evaluation for this harness, or None if it’s unset

exception attelo.harness.interface.HarnessException

Bases: exceptions.Exception

Things that go wrong in the test harness itself.

attelo.harness.parse module

attelo.harness.report module

attelo.harness.util module

Miscellaneous utility functions

attelo.harness.util.call(args, **kwargs)

Execute a command and die prettily if it fails

Symlink from source to link_name, removing any prexisting file at link_name

attelo.harness.util.makedirs(path, **kwargs)

Create a directory and its parents if it does not already exist

attelo.harness.util.md5sum_dir(path, blocksize=65536)

Read a dir and return its md5 sum

attelo.harness.util.md5sum_file(path, blocksize=65536)

Read a file and return its md5 sum

attelo.harness.util.subdirs(parent)

Return all subdirectories within the parent dir (with combined path, ie. parent/subdir)

attelo.harness.util.timestamp()

Current date/time to minute resolution in an ISO format.