Python `WDL` package ¶

Contents

Python WDL package
- Tree
- Expr
- Env
- Type
- Value
- Error
- Lint
- runtime
Python codelabs

miniwdl is a developer toolkit and local runner for the bioinformatics-focused Workflow Description Language (WDL). This documentation covers the Python3 WDL package facilitating parsing & static analysis of WDL documents. Simply import WDL once miniwdl has been installed.

GitHub repo for installation and further background
Codelabs on using this package

WDL.load(uri: str, path: Optional[List[str]] = None, check_quant: bool = True, read_source: Optional[Callable[[str, List[str], Optional[WDL.Tree.Document]], Awaitable[ReadSourceResult]]] = None, import_max_depth: int = 10) → WDL.Tree.Document[source]¶

Parse a WDL document given filename/URI, recursively descend into imported documents, then typecheck the tasks and workflow.

Parameters:

path – local filesystem directories to search for imports, in addition to the current working directory
check_quant – set to False to relax static typechecking of the optional (?) and nonempty (+) type quantifiers. This is discouraged, but may be useful for older WDL workflows which assume less-rigorous static validation of these annotations.
read_source – async routine to read the WDL source code from filename/URI; see read_source_default() below for details
import_max_depth – to prevent recursive import infinite loops, fail when there are too many import nesting levels (default 10)

Raises:

WDL.Error.SyntaxError – when the document is syntactically invalid under the WDL grammar
WDL.Error.ValidationError – when the document is syntactically OK, but fails typechecking or other static validity checks
WDL.Error.MultipleValidationErrors – when multiple validation errors are detected in one pass, listed in the exceptions attribute
WDL.Error.ImportError – when an imported sub-document can’t be loaded; the __cause__ attribute has the specific error

class WDL.ReadSourceResult[source]¶: The NamedTuple to be returned by the read_source routine. Its source_text: str field provides the WDL source code, and the abspath: str field is the absolute filename/URI from which the source was read (e.g. after resolving a relative path).

WDL.values_from_json(values_json: Dict[str, Any], available: WDL.Env.Bindings[typing.Union[WDL.Tree.Decl, WDL.Type.Base]][Union[WDL.Tree.Decl, WDL.Type.Base]], required: Optional[WDL.Env.Bindings[typing.Union[WDL.Tree.Decl, WDL.Type.Base]][Union[WDL.Tree.Decl, WDL.Type.Base]]] = None, namespace: str = '') → WDL.Env.Bindings[WDL.Value.Base][WDL.Value.Base][source]¶

Given a dict parsed from Cromwell-style JSON and the available input (or output) declarations of a task or workflow, create a WDL.Env.Bindings[Value.Base].

Parameters:	required – raise an error if any of these required inputs aren’t present namespace – expect each key to start with this namespace prefixed to the input/output names (e.g. the workflow name)

WDL.values_to_json(values_env: WDL.Env.Bindings[WDL.Value.Base][WDL.Value.Base], namespace: str = '') → Dict[str, Any][source]¶

Convert a WDL.Env.Bindings[WDL.Value.Base] to a dict which json.dumps to Cromwell-style JSON.

Parameters:	namespace – prefix this namespace to each key (e.g. workflow name)

WDL.load_async(uri: str, path: Optional[List[str]] = None, check_quant: bool = True, read_source: Optional[Callable[[str, List[str], Optional[WDL.Tree.Document]], Awaitable[ReadSourceResult]]] = None, import_max_depth: int = 10) → WDL.Tree.Document[source]¶: Async version of load(), with all the same arguments

WDL.read_source_default(uri: str, path: List[str], importer: Optional[WDL.Tree.Document]) → WDL.ReadSourceResult[source]¶

Default async routine for the read_source parameter to load() and load_async(), which they use to read the desired WDL document and its imports. This default routine handles local files only, supplying the search path logic to resolve relative filenames; it fails with network URIs.

Parameters:	uri – Filename/URI to read, as provided to `load()` or the WDL import statement; may be relative path – Local directories to search for relative imports importer – The document importing the one here requested, if any; the `importer.pos.uri` and `importer.pos.abspath` fields may be relevant to resolve relative imports.
Returns:	`ReadSourceResult(source_text="...", abspath="...")`

Callers may wish to override read_source with logic to download source code from network URIs, and for local filenames fall back to return await WDL.read_source_default(...).

Note: the synchronous load() merely calls load_async() on the current asyncio.get_event_loop() and awaits the result.

WDL.resolve_file_import(uri: str, path: List[str], importer: Optional[WDL.Tree.Document]) → str[source]¶

Exposes the logic by which read_source_default() resolves uri to the absolute path of an extant file. If uri is already an absolute path, it’s normalized and returned. A relative uri is resolved by first joining it to either, the directory of the importer document (if any), or the process current working directory (otherwise). Failing that, it’s searched in the path directories (in reverse order).

Security-focused applications may wish to override read_source with logic to restrict allowable results of resolve_file_import, to prevent WDL source code from triggering access to arbitrary filesystem paths. No such restrictions are applied by default.

Tree ¶

Abstract syntax tree (AST) for WDL documents, containing tasks and workflows, which contain declarations, calls, and scatter & if sections. The AST is typically constructed by load().

The WDL.Tree.* classes are also exported by the base WDL module, i.e. WDL.Tree.Document can be abbreviated WDL.Document.

class WDL.Tree.StructTypeDef(pos: WDL.Error.SourcePosition, name: str, members: Dict[str, WDL.Type.Base], imported: Optional[Tuple[WDL.Tree.Document, WDL.Tree.StructTypeDef]] = None)[source]¶

Bases: WDL.Error.SourceNode

WDL struct type definition

name = None¶

Type:	str

Name of the struct type (in the current document)

members = None¶

Type:	Dict[str, WDL.Type.Base]

Member names and types

imported = None¶

Type:	Optional[Tuple[Document,StructTypeDef]]

If this struct is imported from another document, references that document and its definition there. The referenced definition might itself be imported from yet another document.

type_id¶

Type:	str

A string canonically describing the member names and their types, excluding the struct type name; useful to unify aliased struct types.

class WDL.Tree.WorkflowNode(workflow_node_id: str, pos: WDL.Error.SourcePosition)[source]¶

Bases: WDL.Error.SourceNode, abc.ABC

Base class for workflow “nodes” including declarations, calls, and scatter/if sections and their bodies.

Each node has a human-readable ID string which is unique within the workflow. It also exposes the set of workflow node IDs upon which it depends. Abstractly, workflow execution can proceed by “visiting” each node once all of its dependencies have been visited, performing some action(s) appropriate to the specific node type (such as evaluating a WDL expression and binding a name in the environment, or executing a task and binding its outputs).

workflow_node_id = None¶

Type:	str

Human-readable node ID unique within the current workflow

scatter_depth = None¶

Type:	int

How many nested scatter sections the node lies within. This information is useful for runtime dependency analysis in workflows with scatters. When scatter sections are nested within conditional sections or vice versa, this counts the scatters only.

workflow_node_dependencies¶

Type:	Set[str]

Set of workflow node IDs on which this node depends. Available once workflow has been typechecked.

class WDL.Tree.Decl(pos: WDL.Error.SourcePosition, type: WDL.Type.Base, name: str, expr: Optional[WDL.Expr.Base] = None, id_prefix='decl')[source]¶

Bases: WDL.Tree.WorkflowNode

A value declaration within a task or workflow.

Within a task, the declarations can be viewed as “workflow nodes” insofar as they must be evaluated in an order consistent with their dependency structure, and ensured acyclic. The “workflow node IDs” of a task’s declarations are unique within the task only, and unrelated to the top-level workflow, if any, in the WDL document.

type = None¶

Type:	WDL.Type.Base

name = None¶

Declared value name

Type:	str

expr = None¶

Type:	Optional[WDL.Expr.Base]

Bound expression, if any

class WDL.Tree.Task(pos: WDL.Error.SourcePosition, name: str, inputs: Optional[List[WDL.Tree.Decl]], postinputs: List[WDL.Tree.Decl], command: WDL.Expr.String, outputs: List[WDL.Tree.Decl], parameter_meta: Dict[str, Any], runtime: Dict[str, WDL.Expr.Base], meta: Dict[str, Any])[source]¶

Bases: WDL.Error.SourceNode

WDL Task

name = None¶

Type:	str

inputs = None¶

Type:	Optional[List[WDL.Tree.Decl]]

Declarations in the input{} task section, if it’s present

postinputs = None¶

Type:	List[WDL.Tree.Decl]

Declarations outside of the input{} task section

command = None¶

Type:	WDL.Expr.String

outputs = None¶

Type:	List[WDL.Tree.Decl]

Output declarations

parameter_meta = None¶

Type:	Dict[str,Any]

parameter_meta{} section as a JSON-like dict

runtime = None¶

Type:	Dict[str,WDL.Expr.Base]

runtime{} section, with keys and corresponding expressions to be evaluated

meta = None¶

Type:	Dict[str,Any]

meta{} section as a JSON-like dict

available_inputs¶

Type:	WDL.Env.Bindings[WDL.Tree.Decl]

Yields the task’s input declarations. This is all declarations in the task’s input{} section, if it’s present. Otherwise, it’s all declarations in the task, excluding outputs. (This dichotomy bridges pre-1.0 and 1.0+ WDL versions.)

Each input is at the top level of the Env, with no namespace.

required_inputs¶

Type:	WDL.Env.Bindings[WDL.Tree.Decl]

Yields the input declarations which are required to call the task (available inputs that are unbound and non-optional).

Each input is at the top level of the Env, with no namespace.

effective_outputs¶

Type:	WDL.Env.Bindings[Type.Base]

Yields each task output with its type, at the top level of the Env with no namespace. (Present for isomorphism with Workflow.effective_outputs)

class WDL.Tree.Call(pos: WDL.Error.SourcePosition, callee_id: List[str], alias: Optional[str], inputs: Dict[str, WDL.Expr.Base])[source]¶

Bases: WDL.Tree.WorkflowNode

A call (within a workflow) to a task or sub-workflow

callee_id = None¶

Type:	List[str]

The called task; either one string naming a task in the current document, or an import namespace and task name.

name = None¶

Type:	string

Call name, defaults to task/workflow name

inputs = None¶

Type:	Dict[str,WDL.Expr.Base]

Call inputs provided

callee = None¶

Type:	Union[WDL.Tree.Task, WDL.Tree.Workflow]

Refers to the Task or imported Workflow object to be called (after AST typechecking)

available_inputs¶

Type:	WDL.Env.Bindings[WDL.Tree.Decl]

Yields the task/workflow inputs which are not supplied in the call inputs:, and thus may be supplied at workflow launch; in namespaces according to the call names.

required_inputs¶

Type:	WDL.Env.Bindings[WDL.Tree.Decl]

Yields the required task/workflow inputs which are not supplied in the call inputs: (incomplete calls), and thus must be supplied at workflow launch; in namespaces according to the call name.

effective_outputs¶

Type:	WDL.Env.Bindings[WDL.Tree.Decl]

Yields the effective outputs of the callee Task or Workflow, in a namespace according to the call name.

class WDL.Tree.Gather(section: WDL.Tree.WorkflowSection, referee: Union[WDL.Tree.Decl, WDL.Tree.Call, WDL.Tree.Gather])[source]¶

Bases: WDL.Tree.WorkflowNode

A Gather node symbolizes the operation to gather an array of declared values or call outputs in a scatter section, or optional values from a conditional section. These operations are implicit in the WDL syntax, but explicating them in the AST facilitates analysis of the workflow’s data types and dependency structure.

Each scatter/conditional section provides Gather nodes to expose the section body’s products to the rest of the workflow. When a WDL.Expr.Ident elsewhere identifies a node inside the section, its referee attribute is the corresponding Gather node, which in turn references the interior node. The interior node might itself be another Gather node, from a nested scatter/conditional section.

section = None¶

Type:	WorkflowSection

The Scatter/Conditional section implying this Gather operation

referee = None¶

Type:	Union[Decl, Call, Gather]

The Decl, Call, or sub-Gather node from which this operation “gathers”

final_referee¶: The Decl or Call node found at the end of the referee chain through any nested Gather nodes

class WDL.Tree.WorkflowSection(body: List[WDL.Tree.WorkflowNode], *args, **kwargs)[source]¶

Bases: WDL.Tree.WorkflowNode

Base class for workflow nodes representing scatter and conditional sections

body = None¶

Type:	List[WorkflowNode]

Section body, potentially including nested sections.

gathers = None¶

Type:	Dict[str, Gather]

Gather nodes exposing the section body’s products to the rest of the workflow. The dict is keyed by workflow_node_id of the interior node, to expedite looking up the corresponding gather node.

The section’s body and gather nodes do not explicitly include the section node among their dependencies. Such dependence is implicit because the body subgraph can be “instantiated” only upon visiting the section node at runtime.

class WDL.Tree.Scatter(pos: WDL.Error.SourcePosition, variable: str, expr: WDL.Expr.Base, body: List[WDL.Tree.WorkflowNode])[source]¶

Bases: WDL.Tree.WorkflowSection

Workflow scatter section

variable = None¶

Type:	string

Scatter variable name

expr = None¶

Type:	WDL.Expr.Base

Expression for the array over which to scatter

class WDL.Tree.Conditional(pos: WDL.Error.SourcePosition, expr: WDL.Expr.Base, body: List[WDL.Tree.WorkflowNode])[source]¶

Bases: WDL.Tree.WorkflowSection

Workflow conditional (if) section

expr = None¶

Tree:	WDL.Expr.Base

Boolean expression

class WDL.Tree.Workflow(pos: WDL.Error.SourcePosition, name: str, inputs: Optional[List[WDL.Tree.Decl]], body: List[WDL.Tree.WorkflowNode], outputs: Optional[List[WDL.Tree.Decl]], parameter_meta: Dict[str, Any], meta: Dict[str, Any], output_idents: Optional[List[List[str]]] = None, output_idents_pos: Optional[WDL.Error.SourcePosition] = None)[source]¶

Bases: WDL.Error.SourceNode

name = None¶

Type:	str

inputs = None¶

Type:	List[WDL.Tree.Decl]

Declarations in the input{} workflow section, if it’s present

body = None¶

Type:	List[Union[WDL.Tree.Decl,WDL.Tree.Call,WDL.Tree.Scatter,WDL.Tree.Conditional]]

Workflow body in between input{} and output{} sections, if any

outputs = None¶

Type:	Optional[List[WDL.Tree.Decl]]

Workflow output declarations, if the output{} section is present

parameter_meta = None¶

Type:	Dict[str,Any]

parameter_meta{} section as a JSON-like dict

meta = None¶

Type:	Dict[str,Any]

meta{} section as a JSON-like dict

complete_calls = None¶: After typechecking, False if the workflow has a call which does not supply all required inputs (and thus cannot be called from another workflow).

available_inputs¶

Type:	WDL.Env.Bindings[WDL.Tree.Decl]

The workflow’s input declarations. This includes:

1. If the input{} workflow section is present, all declarations within that section. Otherwise, all declarations in the top-level workflow body, excluding outputs. (This dichotomy bridges pre-1.0 and 1.0+ WDL versions.) These appear at the top level of the Env, with no namespace.

Available inputs of all calls in the workflow, namespaced by the call names.

required_inputs¶

Type:	WDL.Env.Bindings[Decl]

The subset of available inputs which are required to start the workflow.

effective_outputs¶

Type:	WDL.Env.Bindings[Decl]

If the output{} workflow section is present, yields the names and types therein, at the top level of the Env. Otherwise, yield all the call outputs, namespaced and typed appropriately.

get_node(workflow_node_id: str) → WDL.Tree.WorkflowNode[source]¶: Look up WorkflowNode by workflow_node_id

class WDL.Tree.SourceComment(pos, text)¶

Bases: tuple

Position and text of a comment. The text includes the # and any preceding or trailing spaces/tabs.

pos¶: Alias for field number 0

text¶: Alias for field number 1

class WDL.Tree.DocImport(pos, uri, namespace, aliases, doc)¶

Bases: tuple

Represents one imported document, with position of the import statement, import URI, namespace, struct type aliases, and (after typechecking) the Document object.

aliases¶: Alias for field number 3

doc¶: Alias for field number 4

namespace¶: Alias for field number 2

pos¶: Alias for field number 0

uri¶: Alias for field number 1

class WDL.Tree.Document(source_text: str, pos: WDL.Error.SourcePosition, imports: List[WDL.Tree.DocImport], struct_typedefs: Dict[str, WDL.Tree.StructTypeDef], tasks: List[WDL.Tree.Task], workflow: Optional[WDL.Tree.Workflow], comments: List[WDL.Tree.SourceComment], wdl_version: Optional[str])[source]¶

Bases: WDL.Error.SourceNode

Top-level document, with imports, tasks, and up to one workflow. Typically returned by load().

imports = None¶

Type:	List[DocImport]

Imported documents

struct_typedefs = None¶

Type:	Env.Bindings[WDL.Tree.StructTypeDef]

tasks = None¶

Type:	List[WDL.Tree.Task]

workflow = None¶

Type:	Optional[WDL.Tree.Workflow]

source_text = None¶

Type:	str

Original WDL source code text

source_lines = None¶

Type:	List[str]

Original WDL source code text split by newlines. SourcePosition line numbers are one-based, so line number L corresponds to source_lines[L-1].

source_comments = None¶

Type:	List[Optional[SourceComment]]

Lookup table for source code comments. source_comments has the same length as source_lines, and each entry is the WDL.Tree.SourceComment found on the corresponding source line, or None if the line has no comment.

wdl_version = None¶

Type:	Optional[str]

Declared WDL language version; if absent, then assume draft-2.

typecheck(check_quant: bool = True) → None[source]¶

Typecheck each task in the document, then the workflow, if any.

Documents returned by load() have already been typechecked.

class WDL.Tree.ReadSourceResult(source_text, abspath)¶

Bases: tuple

abspath¶: Alias for field number 1

source_text¶: Alias for field number 0

Expr ¶

WDL expressions composing literal values, arithmetic, comparison, conditionals, string interpolation, arrays & maps, and function applications. These appear on the right-hand side of value declarations and in task command substitutions, task runtime sections, and workflow scatter and conditional sections.

The abstract syntax tree (AST) for any expression is represented by an instance of a Python class deriving from WDL.Expr.Base. Any such node may have other nodes attached “beneath” it. An expression can be evaluated to a Value given a suitable WDL.Env.Bindings[Value.Base].

class WDL.Expr.Base(pos: WDL.Error.SourcePosition)[source]¶

Bases: WDL.Error.SourceNode, abc.ABC

Superclass of all expression AST nodes

type¶

Type:	WDL.Type.Base

WDL type of this expression. Undefined on construction; populated by one invocation of infer_type.

infer_type(self, type_env : Env.Bindings[Type.Base]) → WDL.Expr.Base[source]¶

Infer the expression’s type within the given type environment. Must be invoked exactly once prior to use of other methods.

Parameters:	stdlib – a context-specific standard function library for typechecking check_quant – when `False`, disables static validation of the optional (?) type quantifier when typecheck() is called on this expression, so for example type `T?` can satisfy an expected type `T`. Applies recursively to the type inference and checking of any sub-expressions.
Raises:	WDL.Error.StaticTypeMismatch – when the expression fails to type-check
Returns:	self

typecheck(self, expected : Type.Base) → WDL.Expr.Base[source]¶

Check that this expression’s type is, or can be coerced to, expected.

Raises:	WDL.Error.StaticTypeMismatch –
Returns:	self

eval(env: WDL.Env.Bindings[WDL.Value.Base][WDL.Value.Base], stdlib: Optional[WDL.StdLib.Base] = None) → WDL.Value.Base[source]¶

Evaluate the expression in the given environment

Parameters:	stdlib – a context-specific standard function library implementation

class WDL.Expr.Boolean(pos: WDL.Error.SourcePosition, literal: bool)[source]¶

Bases: WDL.Expr.Base

Boolean literal

value = None¶

Type:	bool

Literal value

class WDL.Expr.Int(pos: WDL.Error.SourcePosition, literal: int)[source]¶

Bases: WDL.Expr.Base

Integer literal

value = None¶

Type:	int

Literal value

class WDL.Expr.Float(pos: WDL.Error.SourcePosition, literal: float)[source]¶

Bases: WDL.Expr.Base

Numeric literal

value = None¶

Type:	float

Literal value

class WDL.Expr.Placeholder(pos: WDL.Error.SourcePosition, options: Dict[str, str], expr: WDL.Expr.Base)[source]¶

Bases: WDL.Expr.Base

Holds an expression interpolated within a string or command

options = None¶

Type:	Dict[str,str]

Placeholder options (sep, true, false, default)

expr = None¶

Type:	WDL.Expr.Base

Expression to be evaluated and substituted

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

infer_type(self, type_env : Env.Bindings[Type.Base]) → WDL.Expr.Base[source]¶

Infer the expression’s type within the given type environment. Must be invoked exactly once prior to use of other methods.

Parameters:	stdlib – a context-specific standard function library for typechecking check_quant – when `False`, disables static validation of the optional (?) type quantifier when typecheck() is called on this expression, so for example type `T?` can satisfy an expected type `T`. Applies recursively to the type inference and checking of any sub-expressions.
Raises:	WDL.Error.StaticTypeMismatch – when the expression fails to type-check
Returns:	self

class WDL.Expr.String(pos: WDL.Error.SourcePosition, parts: List[Union[str, WDL.Expr.Placeholder]], command: bool = False)[source]¶

Bases: WDL.Expr.Base

String literal, possibly interleaved with expression placeholders for interpolation

parts = None¶

Type:	List[Union[str,WDL.Expr.Placeholder]]

The parts list begins and ends with matching single- or double- quote marks. Between these is a sequence of literal strings and/or interleaved placeholder expressions. Escape sequences in the literals have NOT been decoded.

command = None¶

Type:	bool

True if this expression is a task command template, as opposed to a string expression anywhere else. Controls whether backslash escape sequences are evaluated or (for commands) passed through for shell interpretation.

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.Array(pos: WDL.Error.SourcePosition, items: List[WDL.Expr.Base])[source]¶

Bases: WDL.Expr.Base

Array literal

items = None¶

Type:	List[WDL.Expr.Base]

Expression for each item in the array literal

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.Pair(pos: WDL.Error.SourcePosition, left: WDL.Expr.Base, right: WDL.Expr.Base)[source]¶

Bases: WDL.Expr.Base

Pair literal

left = None¶

Type:	WDL.Expr.Base

Left-hand expression in the pair literal

right = None¶

Type:	WDL.Expr.Base

Right-hand expression in the pair literal

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.Map(pos: WDL.Error.SourcePosition, items: List[Tuple[WDL.Expr.Base, WDL.Expr.Base]])[source]¶

Bases: WDL.Expr.Base

Map literal

items = None¶

Type:	List[Tuple[WDL.Expr.Base,WDL.Expr.Base]]

Expressions for the map literal keys and values

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.Struct(pos: WDL.Error.SourcePosition, members: List[Tuple[str, WDL.Expr.Base]])[source]¶

Bases: WDL.Expr.Base

Struct literal

members = None¶

Type:	Dict[str,WDL.Expr.Base]

The struct literal is modelled initially as a bag of keys and values, which can be coerced to a specific struct type during typechecking.

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.IfThenElse(pos: WDL.Error.SourcePosition, condition: WDL.Expr.Base, consequent: WDL.Expr.Base, alternative: WDL.Expr.Base)[source]¶

Bases: WDL.Expr.Base

Ternary conditional expression

condition = None¶

Type:	WDL.Expr.Base

A Boolean expression for the condition

consequent = None¶

Type:	WDL.Expr.Base

Expression evaluated when the condition is true

alternative = None¶

Type:	WDL.Expr.Base

Expression evaluated when the condition is false

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.Ident(pos: WDL.Error.SourcePosition, name: str)[source]¶

Bases: WDL.Expr.Base

An identifier referencing a named value or call output.

Ident nodes are wrapped in Get nodes, as discussed below.

name = None¶

Type:	str

Name, possibly including a dot-separated namespace

referee = None¶: After typechecking within a task or workflow, stores the AST node to which the identifier refers: a WDL.Tree.Decl for value references; a WDL.Tree.Call for call outputs; a WDL.Tree.Scatter for scatter variables; or a WDL.Tree.Gather object representing a value or call output that resides within a scatter or conditional section.

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.Get(pos: WDL.Error.SourcePosition, expr: WDL.Expr.Base, member: Optional[str])[source]¶

Bases: WDL.Expr.Base

AST node representing access to a value by identifier (including namespaced ones), or accessing a member of a pair or struct as .member.

The entaglement of these two cases is inherent in WDL. Consider the syntax leftname.midname.rightname. One interpretation is that leftname is an identifier for a struct value, and .midname.rightname represents a chain of struct member accesses. But another possibility is that leftname is a call, midname is a struct output of that call, and rightname is a member of that struct. These cases can’t be distinguished by the syntax parser alone, but must be resolved during typechecking with reference to the calls and identifiers available in the environment.

The typechecker does conveniently resolve such cases, and to minimize the extent to which it has to restructure the AST in doing so, all identifiers (with or without a namespace) are represented as a Get node wrapping an Ident node. The Get node may specify a member name to access, but may not if the identifier is to be accessed directly. On the other hand, the expression inside a Get node need not be a simple identifier, e.g. arr[1].memb.left is be represented as: Get(Get(Apply("_at", Get(Ident("arr")), 1),"memb"),"left")

expr = None¶

Type:	WDL.Expr.Base

The expression whose value is accessed

member = None¶

Type:	Optional[str]

If the expression is accessing a pair/struct member, then expr.type is WDL.Type.Pair or WDL.Type.StructInstance and this field gives the desired member name (left or right for pairs).

Otherwise the expression accesses expr directly, and member is None.

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

class WDL.Expr.Apply(pos: WDL.Error.SourcePosition, function: str, arguments: List[WDL.Expr.Base])[source]¶

Bases: WDL.Expr.Base

Application of a built-in or standard library function

function_name = None¶

Name of the function applied

Type:	str

arguments = None¶

Type:	List[WDL.Expr.Base]

Expressions for each function argument

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

Env ¶

Environments, for identifier resolution during WDL typechecking and evaluation.

class WDL.Env.Binding(name: str, value: T, info: Any = None)[source]¶

Bases: typing.Generic

An individual, immutable binding of a possibly-namespaced name to a right-hand-side value of type T. T is typically Value.Base (value environments) or Type.Base (type environments). The binding may also reference an additional informational value of arbitrary type.

name¶

Type:	str

Namedspaced names are flat, dot-separated strings.

value¶

Type:	T

info¶

Type:	Any

class WDL.Env.Bindings(binding: Optional[WDL.Env.Binding[T]] = None, next: Optional[WDL.Env.Bindings[T]] = None)[source]¶

Bases: typing.Generic

An environment consisting of an immutable linked-list of WDL.Env.Binding objects. WDL.Env.Bindings() is the empty environment. Bindings[T] is iterable for the individual Binding[T] objects:

env = WDL.Env.Bindings()
env = env.bind("x", 1).bind("y", 42)
print(env["x"])                             # 1
print(",".join(str(b.value) for b in env))  # 1,42

bind(name: str, value: T, info: Any = None) → WDL.Env.Bindings[~T][T][source]¶: Return an environment with a new binding prepended. Any existing binding for the same name is shadowed by the new one. (This should not usually arise in view of the immutability of WDL values.)

resolve_binding(name: str) → WDL.Env.Binding[~T][T][source]¶

Look up a WDL.Env.Binding object by name

Raises:	KeyError – no such binding

resolve(name: str) → T[source]¶

Look up a bound value by name. Equivalently, env[name]

Raises:	KeyError – no such binding

has_binding(name: str) → bool[source]¶: Determine existence of a binding for the name. Equivalently, name in env

map(f: Callable[[WDL.Env.Binding[~T][T]], Optional[WDL.Env.Binding[~S][S]]]) → WDL.Env.Bindings[~S][S][source]¶: Copy the environment with each binding transformed by the given function. If the function returns None then the binding is excluded.

filter(pred: Callable[[WDL.Env.Binding[~T][T]], bool]) → WDL.Env.Bindings[~T][T][source]¶: Copy the environment with only those bindings for which pred returns True

subtract(rhs: WDL.Env.Bindings[~S][S]) → WDL.Env.Bindings[~T][T][source]¶: Copy the environment excluding any binding for which rhs has a binding with the same name

namespaces¶

Type:	Set[str]

Return the environment’s namespaces, all the distinct dot-separated prefixes of the binding names. Each element ends with a dot.

has_namespace(namespace: str) → bool[source]¶: Determine existence of a namespace in the environment

enter_namespace(namespace: str) → WDL.Env.Bindings[~T][T][source]¶: Generate an environment with only those bindings in the given namespace, with the namespace prefix removed from each binding’s name.

wrap_namespace(namespace: str) → WDL.Env.Bindings[~T][T][source]¶: Copy the environment with the given namespace prefixed to each binding name

with_empty_namespace(namespace: str) → WDL.Env.Bindings[~T][T][source]¶: Return an environment with an empty namespace registered, which will appear in namespaces and has_namespace() even if there are no actual bindings with the namespace prefix.

WDL.Env.merge(*args) → WDL.Env.Bindings[~T][T][source]¶: Merge several Bindings[T] environments into one. For efficiency, the largest environment should be supplied as the last argument.

Type ¶

WDL data types

WDL has both atomic types such as Int, Boolean, and String; and parametric types like Array[String] and Map[String,Array[Array[Float]]]. Here, each type is represented by an immutable instance of a Python class inheriting from WDL.Type.Base. Such types are associated with expressions, statically prior to evaluation, as well as with values and identifier bindings after evaluation.

An atomic type like Int is represented by WDL.Type.Int(). Atomic types can be checked either with isinstance(t,WDL.Type.Int), which ignores the possible optional quantifier (thus satisfied by Int or Int?), or with t == WDL.Type.Int(optional=True) to include the quantifier in the comparison.

A parametric type like Array[String] is represented by WDL.Type.Array(WDL.Type.String()). Any kind of array satisfies isinstance(t,WDL.Type.Array), and WDL.Type.Array(WDL.Type.String()) == WDL.Type.Array(WDL.Type.String()), but for example WDL.Type.Array(WDL.Type.String()) != WDL.Type.Array(WDL.Type.Float()).

The type classes include a method indicating if a value of the type can be coerced to some other desired type, according to the following rules:

Int coerces to Float
Boolean, Int, Float, and File coerce to String
String coerces to File, Int, and Float
Array[T] coerces to String provided T does as well
T coerces to T? but the reverse is not true in general*
Array[T]+ coerces to Array[T] but the reverse is not true in general*

(*) The reverse coercions are statically permitted in expressions set up with Expr.infer_type(check_quant=False) although they may fail at runtime. This also enables coercion of T to Array[T]+ (an array of length 1).

class WDL.Type.Base[source]¶

Bases: abc.ABC

The abstract base class for WDL types

Each specific type inherits from this base, e.g.:

assert issubclass(WDL.Type.Int, WDL.Type.Base)
assert isinstance(WDL.Type.Array(WDL.Type.Int()), WDL.Type.Base)

All instances are immutable.

coerces(rhs: WDL.Type.Base, check_quant: bool = True) → bool[source]¶

True if this is the same type as, or can be coerced to, rhs.

Parameters:	check_quant – when `False`, disables static enforcement of the optional (?) type quantifier

optional¶

Type:	bool

True when the type has the optional quantifier, T?

parameters¶

Type:	Iterable[WDL.Type.Base]

The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)

copy(self, optional : Optional[bool] = None) → WDL.Type.Base[source]¶: Create a copy of the type, possibly with a different setting of the optional quantifier.

class WDL.Type.Any(optional: bool = False)[source]¶

Bases: WDL.Type.Base

A symbolic type which coerces to any other type; used to represent e.g. the item type of an empty array literal, or the result of read_json().

coerces(rhs: WDL.Type.Base, check_quant: bool = True) → bool[source]¶

True if this is the same type as, or can be coerced to, rhs.

Parameters:	check_quant – when `False`, disables static enforcement of the optional (?) type quantifier

class WDL.Type.Boolean(optional: bool = False)[source]¶: Bases: WDL.Type.Base

class WDL.Type.Float(optional: bool = False)[source]¶: Bases: WDL.Type.Base

class WDL.Type.Int(optional: bool = False)[source]¶: Bases: WDL.Type.Base

class WDL.Type.File(optional: bool = False)[source]¶: Bases: WDL.Type.Base

class WDL.Type.String(optional: bool = False)[source]¶: Bases: WDL.Type.Base

class WDL.Type.Array(item_type: WDL.Type.Base, optional: bool = False, nonempty: bool = False)[source]¶

Bases: WDL.Type.Base

Array type, parameterized by the type of the constituent items.

item_type = None¶

Type:	WDL.Type.Base

item_type may be Any when not known statically, such as in a literal empty array [].

nonempty¶

Type:	bool

True when the type has the nonempty quantifier, Array[T]+

parameters¶

Type:	Iterable[WDL.Type.Base]

The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)

copy(self, optional : Optional[bool] = None) → WDL.Type.Base[source]¶: Create a copy of the type, possibly with a different setting of the optional quantifier.

class WDL.Type.Map(item_type: Tuple[WDL.Type.Base, WDL.Type.Base], optional: bool = False, literal_keys: Optional[Set[str]] = None)[source]¶

Bases: WDL.Type.Base

Map type, parameterized by the (key,value) item type.

item_type = None¶

Type:	Tuple[WDL.Type.Base,WDL.Type.Base]

The key and value types may be Any when not known statically, such as in a literal empty map {}.

literal_keys = None¶

parameters¶

Type:	Iterable[WDL.Type.Base]

The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)

class WDL.Type.Pair(left_type: WDL.Type.Base, right_type: WDL.Type.Base, optional: bool = False)[source]¶

Bases: WDL.Type.Base

Pair type, parameterized by the left and right item types.

left_type = None¶

Type:	WDL.Type.Base

right_type = None¶

Type:	WDL.Type.Base

parameters¶

Type:	Iterable[WDL.Type.Base]

The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)

class WDL.Type.StructInstance(type_name: str, optional: bool = False)[source]¶

Bases: WDL.Type.Base

Type of an instance of a struct

Not to be confused with struct type definition, WDL.Tree.StructTypeDef. To find the WDL.Tree.StructTypeDef in the current doc: WDL.Tree.Document corresponding to ty: WDL.Type.StructInstance, use doc.struct_typedefs[ty.type_name].

type_name = None¶

Type:	str

The struct type name with which the instance is declared; note that the same struct type can go by different names.

members = None¶

Type:	Dict[str,WDL.Type.Base]

Names and types of the struct members, from the struct type definition (available after typechecking)

type_id¶

Type:	str

A string canonically describing the member names and their types, excluding the struct type name; useful to unify aliased struct types.

parameters¶

Type:	Iterable[WDL.Type.Base]

The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)

WDL.Type.unify(types: List[WDL.Type.Base], check_quant: bool = True, force_string: bool = False) → WDL.Type.Base[source]¶

Given a list of types, compute a type to which they’re all coercible, or WDL.Type.Any if no more-specific inference is possible.

Parameters:	force_string – permit last-resort unification to `String` even if no item is currently a `String`, but all can be coerced

Value ¶

WDL values instantiated at runtime

Each value is represented by an instance of a Python class inheriting from WDL.Value.Base.

class WDL.Value.Base(type: WDL.Type.Base, value: Any, expr: Optional[Expr.Base] = None)[source]¶

Bases: abc.ABC

The abstract base class for WDL values

type = None¶

Type:	WDL.Type.Base

value = None¶: The “raw” Python value

expr = None¶: Reference to the WDL expression that generated this value, if it originated from WDL.Expr.eval

coerce(desired_type: Optional[WDL.Type.Base] = None) → WDL.Value.Base[source]¶

Coerce the value to the desired type and return it. Types should be checked statically on WDL.Expr.Base prior to evaluation.

Raises:	ReferenceError for a null value and non-optional type

expect(desired_type: Optional[WDL.Type.Base] = None) → WDL.Value.Base[source]¶: Alias for coerce

json¶: Return a value representation which can be serialized to JSON using json.dumps (str, int, float, list, dict, or null)

class WDL.Value.Boolean(value: bool, expr: Optional[Expr.Base] = None)[source]¶

Bases: WDL.Value.Base

value has Python type bool

class WDL.Value.Float(value: float, expr: Optional[Expr.Base] = None)[source]¶

Bases: WDL.Value.Base

value has Python type float

class WDL.Value.Int(value: int, expr: Optional[Expr.Base] = None)[source]¶

Bases: WDL.Value.Base

value has Python type int

class WDL.Value.String(value: str, expr: Optional[Expr.Base] = None)[source]¶

Bases: WDL.Value.Base

value has Python type str

class WDL.Value.File(value: str, expr: Optional[Expr.Base] = None)[source]¶

Bases: WDL.Value.String

value has Python type str

class WDL.Value.Array(item_type: WDL.Type.Base, value: List[WDL.Value.Base], expr: Optional[Expr.Base] = None)[source]¶

Bases: WDL.Value.Base

value is a Python list of other WDL.Value.Base instances

class WDL.Value.Map(item_type: Tuple[WDL.Type.Base, WDL.Type.Base], value: List[Tuple[WDL.Value.Base, WDL.Value.Base]], expr: Optional[Expr.Base] = None)[source]¶: Bases: WDL.Value.Base

class WDL.Value.Pair(left_type: WDL.Type.Base, right_type: WDL.Type.Base, value: Tuple[WDL.Value.Base, WDL.Value.Base], expr: Optional[Expr.Base] = None)[source]¶: Bases: WDL.Value.Base

class WDL.Value.Null(expr: Optional[Expr.Base] = None)[source]¶

Bases: WDL.Value.Base

Represents the missing value which optional inputs may take. type and value are both None.

class WDL.Value.Struct(type: Union[WDL.Type.Object, WDL.Type.StructInstance], value: Dict[str, WDL.Value.Base], expr: Optional[Expr.Base] = None)[source]¶: Bases: WDL.Value.Base

WDL.Value.from_json(type: WDL.Type.Base, value: Any) → WDL.Value.Base[source]¶

Instantiate a WDL value of the specified type from a parsed JSON value (str, int, float, list, dict, or null).

If type is WDL.Type.Any(), attempts to infer a WDL type & value from the JSON’s intrinsic types. This isn’t ideal; for example, Files can’t be distinguished from Strings, and JSON lists and dicts with heterogeneous item types may give undefined results.

Raises:	WDL.Error.InputError – if the given value isn’t coercible to the specified type

WDL.Value.rewrite_files(v: WDL.Value.Base, f: Callable[[str], str]) → WDL.Value.Base[source]¶: Produce a deep copy of the given Value with all File names rewritten by the given function (including Files nested inside compound Values).

WDL.Value.rewrite_env_files(env: WDL.Env.Bindings[WDL.Value.Base][WDL.Value.Base], f: Callable[[str], str]) → WDL.Env.Bindings[WDL.Value.Base][WDL.Value.Base][source]¶: Produce a deep copy of the given Value Env with all File names rewritten by the given function.

Error ¶

class WDL.Error.SourcePosition[source]¶

Bases: WDL.Error.SourcePosition

Source position attached to AST nodes and exceptions; NamedTuple of uri the filename/URI passed to WDL.load() or a WDL import statement, which may be relative; abspath the absolute filename/URI; and one-based int positions line end_line column end_column

exception WDL.Error.SyntaxError(pos: WDL.Error.SourcePosition, msg: str, wdl_version: str, declared_wdl_version: Optional[str])[source]¶

Bases: Exception

Failure to lex/parse a WDL document

exception WDL.Error.ImportError(pos: WDL.Error.SourcePosition, import_uri: str, message: Optional[str] = None)[source]¶

Bases: Exception

Failure to open/retrieve an imported WDL document

The __cause__ attribute may hold the inner error object.

class WDL.Error.SourceNode(pos: WDL.Error.SourcePosition)[source]¶

Bases: object

Base class for an AST node, recording the source position

pos = None¶

Type:	SourcePosition

Source position for this AST node

children¶

Type:	Iterable[SourceNode]

Yield all child nodes

exception WDL.Error.ValidationError(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition], message: str)[source]¶

Bases: Exception

Base class for a WDL validation error (when the document loads and parses, but fails typechecking or other static validity tests)

source_text = None¶

Type:	Optional[str]

The complete source text of the WDL document (if available)

node = None¶

Type:	Optional[SourceNode]

pos = None¶

Type:	SourcePosition

exception WDL.Error.InvalidType(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition], message: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.IndeterminateType(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition], message: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.NoSuchTask(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition], name: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.NoSuchFunction(node: WDL.Error.SourceNode, name: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.WrongArity(node: WDL.Error.SourceNode, expected: int)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.NotAnArray(node: WDL.Error.SourceNode)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.NoSuchMember(node: WDL.Error.SourceNode, member: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.StaticTypeMismatch(node: WDL.Error.SourceNode, expected: WDL.Type.Base, actual: WDL.Type.Base, message: str = '')[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.IncompatibleOperand(node: WDL.Error.SourceNode, message: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.UnknownIdentifier(node: WDL.Error.SourceNode)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.NoSuchInput(node: WDL.Error.SourceNode, name: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.UncallableWorkflow(node: WDL.Error.SourceNode, name: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.MultipleDefinitions(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition], message: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.StrayInputDeclaration(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition], message: str)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.CircularDependencies(node: WDL.Error.SourceNode)[source]¶: Bases: WDL.Error.ValidationError

exception WDL.Error.MultipleValidationErrors(*exceptions)[source]¶

Bases: Exception

Propagates several validation/typechecking errors

exceptions = None¶

Type:	List[ValidationError]

exception WDL.Error.RuntimeError[source]¶: Bases: Exception

exception WDL.Error.EvalError(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition], message: str)[source]¶

Bases: WDL.Error.RuntimeError

Error evaluating a WDL expression or declaration

node = None¶

Type:	Optional[SourceNode]

pos = None¶

Type:	SourcePosition

exception WDL.Error.OutOfBounds(node: WDL.Error.SourceNode)[source]¶: Bases: WDL.Error.EvalError

exception WDL.Error.EmptyArray(node: WDL.Error.SourceNode)[source]¶: Bases: WDL.Error.EvalError

exception WDL.Error.NullValue(node: Union[WDL.Error.SourceNode, WDL.Error.SourcePosition])[source]¶: Bases: WDL.Error.EvalError

exception WDL.Error.InputError[source]¶

Bases: WDL.Error.RuntimeError

Error reading an input value/file

Lint ¶

Annotate WDL document AST with hygiene warnings (underlies miniwdl check)

Given a doc: WDL.Document, the lint warnings can be retrieved like so:

lint = WDL.Lint.collect(WDL.Lint.lint(doc, descend_imports=False))
for (pos, lint_class, message, suppressed) in lint:
    assert isinstance(pos, WDL.SourcePosition)
    assert isinstance(lint_class, str) and isinstance(message, str)
    if not suppressed:
        print(json.dumps({
            "uri"        : pos.uri,
            "abspath"    : pos.abspath,
            "line"       : pos.line,
            "end_line"   : pos.end_line,
            "column"     : pos.column,
            "end_column" : pos.end_column,
            "lint"       : lint_class,
            "message"    : message,
        }))

The descend_imports flag controls whether lint warnings are generated for imported documents recursively (true, default), or otherwise only the given document (false).

runtime ¶

The recommended way to run a WDL workflow programmatically is to invoke miniwdl run as a subprocess, capturing its JSON standard output. This leverages its logging, configuration, and flexible input loading features; and avoids conflicting with the runtime’s thread pools and signal handlers. Alternatively, it’s possible to call WDL.runtime.run() directly if needed.

WDL.runtime.run(cfg: WDL.runtime.config.Loader, exe: Union[WDL.Tree.Task, WDL.Tree.Workflow], inputs: WDL.Env.Bindings[WDL.Value.Base][WDL.Value.Base], **run_kwargs) → Tuple[str, WDL.Env.Bindings[WDL.Value.Base][WDL.Value.Base]][source]¶

Run the task or workflow given the inputs environment and configuration, returning the outputs environment. inputs may be parsed from a JSON dict using values_from_json(), which can also validate them; see example below.

Parameters:	run_id – a run identifier used in logs and filenames; defaults to executable name run_dir – directory under which to create a timestamp-named subdirectory for this run (defaults to current working directory). If the final path component is `.` then operate in run_dir directly.

Typical usage:

# Convert JSON-like inputs dict to WDL environment, validating them against exe's available
# and required inputs. The dict keys should NOT be namespaced by the executable name;
# if namespaces are present, then add namespace=exe.name to effectively remove them.
inputs_env = WDL.values_from_json(inputs_dict, exe.available_inputs, exe.required_inputs)
# Load configuration (see below)
cfg = WDL.runtime.config.Loader(logging.getLogger(__name__))
# Run executable
run_subdir, outputs_env = WDL.runtime.run(cfg, exe, inputs_env, run_dir="/tmp")
# Generate JSON-like outputs dict, with keys namespaced by the executable name
outputs_dict = WDL.values_to_json(outputs_env, exe.name)

class WDL.runtime.config.Loader(logger: logging.Logger, filenames: Optional[List[str]] = None, overrides: Optional[Dict[str, Dict[str, str]]] = None)[source]¶

Runtime configuration options, identified by section & key, are sourced in the following priority order:

Supplied overrides dict, {"section": {"key": value}}
Environment variables MINIWDL__SECTION__KEY (uppercased with double-underscores)
Custom configuration file (mutually exclusive):
1. filename given to __init__
2. file named by environment variable MINIWDL_CFG
3. miniwdl.cfg in XDG_CONFIG_HOME or XDG_CONFIG_DIRS, usually ~/.config/miniwdl.cfg
WDL/runtime/config_templates/default.cfg from installed package

If filenames is an empty list, then no custom configuration file is used. Or filenames may be a prioritized list of candidate filenames instead of (a-c) above; the first extant file is used, if any.

log_all()[source]¶: Write a debug log message with all options

log_unused_options()[source]¶: After a run, log warnings about any options that were set, but were never accessed and don’t correspond to any default option.