Python WDL package
miniwdl is a developer toolkit and local runner for
the bioinformatics-focused Workflow Description Language (WDL). This
documentation covers the Python3 WDL package facilitating parsing & static analysis of WDL
documents. Simply import WDL once miniwdl has been installed.
GitHub repo for installation and further background
Codelabs on using this package
- WDL.load(uri: str, path: List[str] | None = None, check_quant: bool = True, read_source: Callable[[str, List[str], Document | None], Awaitable[ReadSourceResult]] | None = None, import_max_depth: int = 10) Document[source]
Parse a WDL document given filename/URI, recursively descend into imported documents, then typecheck the tasks and workflow.
- Parameters:
path – local filesystem directories to search for imports, in addition to the current working directory
check_quant – set to
Falseto relax static typechecking of the optional (?) and nonempty (+) type quantifiers. This is discouraged, but may be useful for older WDL workflows which assume less-rigorous static validation of these annotations.read_source – async routine to read the WDL source code from filename/URI; see
read_source_default()below for detailsimport_max_depth – to prevent recursive import infinite loops, fail when there are too many import nesting levels (default 10)
- Raises:
WDL.Error.SyntaxError – when the document is syntactically invalid under the WDL grammar
WDL.Error.ValidationError – when the document is syntactically OK, but fails typechecking or other static validity checks
WDL.Error.MultipleValidationErrors – when multiple validation errors are detected in one pass, listed in the
exceptionsattributeWDL.Error.ImportError – when an imported sub-document can’t be loaded; the
__cause__attribute has the specific error
- async WDL.load_async(uri: str, path: List[str] | None = None, check_quant: bool = True, read_source: Callable[[str, List[str], Document | None], Awaitable[ReadSourceResult]] | None = None, import_max_depth: int = 10) Document[source]
Async version of
load(), with all the same arguments
- async WDL.read_source_default(uri: str, path: List[str], importer: Document | None) ReadSourceResult[source]
Default async routine for the
read_sourceparameter toload()andload_async(), which they use to read the desired WDL document and its imports. This default routine handles local files only, supplying the search path logic to resolve relative filenames; it fails with network URIs.- Parameters:
uri – Filename/URI to read, as provided to
load()or the WDL import statement; may be relativepath – Local directories to search for relative imports
importer – The document importing the one here requested, if any; the
importer.pos.uriandimporter.pos.abspathfields may be relevant to resolve relative imports.
- Returns:
WDL.ReadSourceResult(source_text="...", abspath="...")
The returned
NamedTupleprovides the WDL source code and the absolute filename/URI from which the source was read (e.g. after resolving a relative path).Callers may wish to override
read_sourcewith logic to download source code from network URIs, and for local filenames fall back toreturn await WDL.read_source_default(...).Note: the synchronous
load()merely callsload_async()on the currentasyncio.get_event_loop()and awaits the result.
- async WDL.resolve_file_import(uri: str, path: List[str], importer: Document | None) str[source]
Exposes the logic by which
read_source_default()resolvesurito the absolute path of an extant file. Ifuriis already an absolute path, it’s normalized and returned. A relativeuriis resolved by first joining it to either, the directory of the importer document (if any), or the process current working directory (otherwise). Failing that, it’s searched in thepathdirectories (in reverse order).Security-focused applications may wish to override
read_sourcewith logic to restrict allowable results ofresolve_file_import, to prevent WDL source code from triggering access to arbitrary filesystem paths. No such restrictions are applied by default.
- WDL.values_from_json(values_json: Dict[str, Any], available: Bindings[Decl] | Bindings[Base], required: Bindings[Decl] | Bindings[Base] | None = None, namespace: str = '') Bindings[Base][source]
Given a dict parsed from Cromwell-style JSON and the available input (or output) declarations of a task or workflow, create a
WDL.Env.Bindings[Value.Base].- Parameters:
required – raise an error if any of these required inputs aren’t present
namespace – expect each key to start with this namespace prefixed to the input/output names (e.g. the workflow name)
- WDL.values_to_json(values_env: Bindings[Base] | Bindings[Decl] | Bindings[Base], namespace: str = '') Dict[str, Any][source]
Convert a
WDL.Env.Bindings[WDL.Value.Base]to a dict whichjson.dumpsto Cromwell-style JSON.- Parameters:
namespace – prefix this namespace to each key (e.g. workflow name)
Tree
Abstract syntax tree (AST) for WDL documents, containing tasks and workflows, which contain
declarations, calls, and scatter & if sections. The AST is typically constructed by
load().
The WDL.Tree.* classes are also exported by the base WDL module, i.e. WDL.Tree.Document
can be abbreviated WDL.Document.
- class WDL.Tree.StructTypeDef(pos: SourcePosition, name: str, members: Dict[str, Base], imported: Tuple[Document, StructTypeDef] | None = None)[source]
Bases:
SourceNodeWDL struct type definition
- name: str
- Type:
str
Name of the struct type (in the current document)
- members: Dict[str, Base]
- Type:
Dict[str, WDL.Type.Base]
Member names and types
- imported: Tuple[Document, StructTypeDef] | None
- Type:
Optional[Tuple[Document,StructTypeDef]]
If this struct is imported from another document, references that document and its definition there. The referenced definition might itself be imported from yet another document.
- property type_id: str
- Type:
str
A string canonically describing the member names and their types, excluding the struct type name; useful to unify aliased struct types.
- class WDL.Tree.WorkflowNode(workflow_node_id: str, pos: SourcePosition)[source]
Bases:
SourceNode,ABCBase class for workflow “nodes” including declarations, calls, and scatter/if sections and their bodies.
Each node has a human-readable ID string which is unique within the workflow. It also exposes the set of workflow node IDs upon which it depends. Abstractly, workflow execution can proceed by “visiting” each node once all of its dependencies have been visited, performing some action(s) appropriate to the specific node type (such as evaluating a WDL expression and binding a name in the environment, or executing a task and binding its outputs).
- workflow_node_id: str
- Type:
str
Human-readable node ID unique within the current workflow
- scatter_depth: int
- Type:
int
How many nested scatter sections the node lies within. This information is useful for runtime dependency analysis in workflows with scatters. When scatter sections are nested within conditional sections or vice versa, this counts the scatters only.
- property workflow_node_dependencies: Set[str]
- Type:
Set[str]
Set of workflow node IDs on which this node depends. Available once workflow has been typechecked.
- class WDL.Tree.Decl(pos: SourcePosition, type: Base, name: str, expr: Base | None = None, id_prefix='decl')[source]
Bases:
WorkflowNodeA value declaration within a task or workflow.
Within a task, the declarations can be viewed as “workflow nodes” insofar as they must be evaluated in an order consistent with their dependency structure, and ensured acyclic. The “workflow node IDs” of a task’s declarations are unique within the task only, and unrelated to the top-level workflow, if any, in the WDL document.
- name: str
Declared value name
- Type:
str
- expr: Base | None
- Type:
Optional[WDL.Expr.Base]
Bound expression, if any
- decor: Dict[str, Any]
- class WDL.Tree.Task(pos: SourcePosition, name: str, inputs: List[Decl] | None, postinputs: List[Decl], command: String, outputs: List[Decl], parameter_meta: Dict[str, Any], runtime: Dict[str, Base], meta: Dict[str, Any])[source]
Bases:
SourceNodeWDL Task
- name: str
- Type:
str
- inputs: List[Decl] | None
- Type:
Optional[List[WDL.Tree.Decl]]
Declarations in the
input{}task section, if it’s present
- postinputs: List[Decl]
- Type:
List[WDL.Tree.Decl]
Declarations outside of the
input{}task section
- outputs: List[Decl]
- Type:
List[WDL.Tree.Decl]
Output declarations
- runtime: Dict[str, Base]
- Type:
Dict[str,WDL.Expr.Base]
runtime{}section, with keys and corresponding expressions to be evaluated
- effective_wdl_version: str
- Type:
str
Effective WDL version of the containing document
- property available_inputs: Bindings[Decl]
- Type:
Yields the task’s input declarations. This is all declarations in the task’s
input{}section, if it’s present. Otherwise, it’s all declarations in the task, excluding outputs. (This dichotomy bridges pre-1.0 and 1.0+ WDL versions.)Each input is at the top level of the Env, with no namespace.
- property required_inputs: Bindings[Decl]
- Type:
Yields the input declarations which are required to call the task (available inputs that are unbound and non-optional).
Each input is at the top level of the Env, with no namespace.
- property effective_outputs: Bindings[Base]
- Type:
Yields each task output with its type, at the top level of the Env with no namespace. (Present for isomorphism with
Workflow.effective_outputs)
- property digest: str
Content digest of the task, for use e.g. as a cache key. The digest is an opaque string of a few dozen alphanumeric characters, sensitive to the task’s source code (with best effort to exclude comments and whitespace).
- class WDL.Tree.Call(pos: SourcePosition, callee_id: List[str], alias: str | None, inputs: Dict[str, Base], after: List[str] | None = None)[source]
Bases:
WorkflowNodeA call (within a workflow) to a task or sub-workflow
- callee_id: List[str]
- Type:
List[str]
The called task; either one string naming a task in the current document, or an import namespace and task name.
- name: str
- Type:
string
Call name, defaults to task/workflow name
- inputs: Dict[str, Base]
- Type:
Dict[str,WDL.Expr.Base]
Call inputs provided
- callee: Task | Workflow | None
- Type:
Union[WDL.Tree.Task, WDL.Tree.Workflow]
Refers to the
Taskor importedWorkflowobject to be called (after AST typechecking)
- after: List[str]
- Type:
string
Call names on which this call depends (even if none of their outputs are used in this call’s inputs)
- property available_inputs: Bindings[Decl]
- Type:
Yields the task/workflow inputs which are not supplied in the call
inputs:, and thus may be supplied at workflow launch; in namespaces according to the call names.
- property required_inputs: Bindings[Decl]
- Type:
Yields the required task/workflow inputs which are not supplied in the call
inputs:(incomplete calls), and thus must be supplied at workflow launch; in namespaces according to the call name.
- class WDL.Tree.Gather(section: WorkflowSection, referee: Decl | Call | Gather)[source]
Bases:
WorkflowNodeA
Gathernode symbolizes the operation to gather an array of declared values or call outputs in a scatter section, or optional values from a conditional section. These operations are implicit in the WDL syntax, but explicating them in the AST facilitates analysis of the workflow’s data types and dependency structure.Each scatter/conditional section provides
Gathernodes to expose the section body’s products to the rest of the workflow. When aWDL.Expr.Identelsewhere identifies a node inside the section, itsrefereeattribute is the correspondingGathernode, which in turn references the interior node. The interior node might itself be anotherGathernode, from a nested scatter/conditional section.- section: WorkflowSection
- Type:
The
Scatter/Conditionalsection implying this Gather operation
- class WDL.Tree.WorkflowSection(body: List[WorkflowNode], *args, **kwargs)[source]
Bases:
WorkflowNodeBase class for workflow nodes representing scatter and conditional sections
- body: List[WorkflowNode]
- Type:
List[WorkflowNode]
Section body, potentially including nested sections.
- gathers: Dict[str, Gather]
- Type:
Dict[str, Gather]
Gathernodes exposing the section body’s products to the rest of the workflow. The dict is keyed byworkflow_node_idof the interior node, to expedite looking up the corresponding gather node.The section’s body and gather nodes do not explicitly include the section node among their dependencies. Such dependence is implicit because the body subgraph can be “instantiated” only upon visiting the section node at runtime.
- class WDL.Tree.Scatter(pos: SourcePosition, variable: str, expr: Base, body: List[WorkflowNode])[source]
Bases:
WorkflowSectionWorkflow scatter section
- variable: str
- Type:
string
Scatter variable name
- class WDL.Tree.Conditional(pos: SourcePosition, expr: Base, body: List[WorkflowNode])[source]
Bases:
WorkflowSectionWorkflow conditional (if) section
- class WDL.Tree.Workflow(pos: SourcePosition, name: str, inputs: List[Decl] | None, body: List[WorkflowNode], outputs: List[Decl] | None, parameter_meta: Dict[str, Any], meta: Dict[str, Any], output_idents: List[List[str]] | None = None, output_idents_pos: SourcePosition | None = None)[source]
Bases:
SourceNode- name: str
- Type:
str
- inputs: List[Decl] | None
- Type:
List[WDL.Tree.Decl]
Declarations in the
input{}workflow section, if it’s present
- body: List[WorkflowNode]
- Type:
List[Union[WDL.Tree.Decl,WDL.Tree.Call,WDL.Tree.Scatter,WDL.Tree.Conditional]]
Workflow body in between
input{}andoutput{}sections, if any
- outputs: List[Decl] | None
- Type:
Optional[List[WDL.Tree.Decl]]
Workflow output declarations, if the
output{}section is present
- complete_calls: bool
After typechecking, False if the workflow has a call which does not supply all required inputs (and thus cannot be called from another workflow).
- effective_wdl_version: str
- Type:
str
Effective WDL version of the containing document
- property available_inputs: Bindings[Decl]
- Type:
The workflow’s input declarations. This includes:
1. If the
input{}workflow section is present, all declarations within that section. Otherwise, all declarations in the top-level workflow body, excluding outputs. (This dichotomy bridges pre-1.0 and 1.0+ WDL versions.) These appear at the top level of the Env, with no namespace.Available inputs of all calls in the workflow, namespaced by the call names.
- property required_inputs: Bindings[Decl]
- Type:
The subset of available inputs which are required to start the workflow.
- property effective_outputs: Bindings[Base]
- Type:
If the
output{}workflow section is present, yields the names and types therein, at the top level of the Env. Otherwise, yield all the call outputs, namespaced and typed appropriately.
- get_node(workflow_node_id: str) WorkflowNode[source]
Look up
WorkflowNodebyworkflow_node_id
- property digest: str
Content digest of the workflow, for use e.g. as a cache key. The digest is an opaque string of a few dozen alphanumeric characters, sensitive to the workflow’s source code (with best effort to exclude comments and whitespace) and the tasks and subworkflows it calls.
- class WDL.Tree.SourceComment(pos, text)
Bases:
tuplePosition and text of a comment. The text includes the
#and any preceding or trailing spaces/tabs.- pos: SourcePosition
Alias for field number 0
- text: str
Alias for field number 1
- class WDL.Tree.DocImport(pos, uri, namespace, aliases, doc)
Bases:
tupleRepresents one imported document, with position of the import statement, import URI, namespace, struct type aliases, and (after typechecking) the
Documentobject.- aliases: List[Tuple[str, str]]
Alias for field number 3
- namespace: str
Alias for field number 2
- pos: SourcePosition
Alias for field number 0
- uri: str
Alias for field number 1
- class WDL.Tree.Document(source_text: str, pos: SourcePosition, imports: List[DocImport], struct_typedefs: Dict[str, StructTypeDef], tasks: List[Task], workflow: Workflow | None, comments: List[SourceComment], wdl_version: str | None)[source]
Bases:
SourceNodeTop-level document, with imports, tasks, and up to one workflow. Typically returned by
load().- struct_typedefs: Bindings[StructTypeDef]
- tasks: List[Task]
- Type:
List[WDL.Tree.Task]
- workflow: Workflow | None
- Type:
Optional[WDL.Tree.Workflow]
- source_text: str
- Type:
str
Original WDL source code text
- source_lines: List[str]
- Type:
List[str]
Original WDL source code text split by newlines.
SourcePositionline numbers are one-based, so line numberLcorresponds tosource_lines[L-1].
- source_comments: List[SourceComment | None]
- Type:
List[Optional[SourceComment]]
Lookup table for source code comments.
source_commentshas the same length assource_lines, and each entry is theWDL.Tree.SourceCommentfound on the corresponding source line, orNoneif the line has no comment.
- wdl_version: str | None
- Type:
Optional[str]
Declared WDL language version, if any
- effective_wdl_version: str
:type”
wdl_version if wdl_version is not None else "draft-2"
Expr
WDL expressions composing literal values, arithmetic, comparison, conditionals, string interpolation, arrays & maps, and function applications. These appear on the right-hand side of value declarations and in task command substitutions, task runtime sections, and workflow scatter and conditional sections.
The abstract syntax tree (AST) for any expression is represented by an instance
of a Python class deriving from WDL.Expr.Base. Any such node may have other
nodes attached “beneath” it. An expression can be evaluated to a Value
given a suitable WDL.Env.Bindings[Value.Base].
- class WDL.Expr.Base(pos: SourcePosition)[source]
Bases:
SourceNode,ABCSuperclass of all expression AST nodes
- property type: Base
- Type:
WDL type of this expression. Undefined on construction; populated by one invocation of
infer_type.
- infer_type(self, type_env: Env.Bindings[Type.Base], stdlib: StdLib.Base) WDL.Expr.Base[source]
Infer the expression’s type within the given type environment. Must be invoked exactly once prior to use of other methods.
- Parameters:
stdlib – a context-specific standard function library for typechecking
check_quant – when
False, disables static validation of the optional (?) type quantifier when typecheck() is called on this expression, so for example typeT?can satisfy an expected typeT. Applies recursively to the type inference and checking of any sub-expressions.
- Raises:
WDL.Error.StaticTypeMismatch – when the expression fails to type-check
- Returns:
self
- typecheck(self, expected: Type.Base) WDL.Expr.Base[source]
Check that this expression’s type is, or can be coerced to,
expected.- Raises:
- Returns:
self
- class WDL.Expr.Boolean(pos: SourcePosition, literal: bool)[source]
Bases:
BaseBoolean literal
- value: bool
- Type:
bool
Literal value
- class WDL.Expr.Int(pos: SourcePosition, literal: int)[source]
Bases:
BaseInteger literal
- value: int
- Type:
int
Literal value
- class WDL.Expr.Float(pos: SourcePosition, literal: float)[source]
Bases:
BaseNumeric literal
- value: float
- Type:
float
Literal value
- class WDL.Expr.Null(pos: SourcePosition)[source]
Bases:
BaseWDL
Noneliteral(called
Nullto avoid conflict with PythonNone)- value: None
- Type:
None
- class WDL.Expr.Placeholder(pos: SourcePosition, options: Dict[str, str], expr: Base)[source]
Bases:
BaseHolds an expression interpolated within a string or command
- options: Dict[str, str]
- Type:
Dict[str,str]
Placeholder options (sep, true, false, default)
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.String(pos: SourcePosition, parts: List[str | Placeholder], command: bool = False)[source]
Bases:
BaseString literal, possibly interleaved with expression placeholders for interpolation
- parts: List[str | Placeholder]
- Type:
List[Union[str,WDL.Expr.Placeholder]]
The parts list begins and ends with the original delimiters (quote marks, braces, or triple angle brackets). Between these is a sequence of literal strings and/or interleaved placeholder expressions. Escape sequences in the literals are NOT decoded until the string literal is evaluated (although the parser will have checked they’re valid).
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.TaskCommand(pos: SourcePosition, parts: List[str | Placeholder])[source]
Bases:
StringSpecialization of
Stringfor task commands, with slightly different evaluation rules: common leading whitespace is removed (dedentation) and escape sequences are passed through for shell interpretation. Also, for historical reasons, thepartsdoes not include the beginning & ending delimiters.- parts: List[str | Placeholder]
- Type:
List[Union[str,WDL.Expr.Placeholder]]
The parts list begins and ends with the original delimiters (quote marks, braces, or triple angle brackets). Between these is a sequence of literal strings and/or interleaved placeholder expressions. Escape sequences in the literals are NOT decoded until the string literal is evaluated (although the parser will have checked they’re valid).
- pos: SourcePosition
- Type:
Source position for this AST node
- class WDL.Expr.MultiLineString(pos: SourcePosition, parts: List[str | Placeholder], command: bool = False)[source]
Bases:
StringSpecialization of
Stringfor WDL 1.2 multi-line string literals, with special evaluation rules: trim whitespace following the<<<and preceding the>>>delimiters and dedent the remaining non-blank lines, while allowing newlines to be escaped.- parts: List[str | Placeholder]
- Type:
List[Union[str,WDL.Expr.Placeholder]]
The parts list begins and ends with the original delimiters (quote marks, braces, or triple angle brackets). Between these is a sequence of literal strings and/or interleaved placeholder expressions. Escape sequences in the literals are NOT decoded until the string literal is evaluated (although the parser will have checked they’re valid).
- pos: SourcePosition
- Type:
Source position for this AST node
- class WDL.Expr.Array(pos: SourcePosition, items: List[Base])[source]
Bases:
BaseArray literal
- items: List[Base]
- Type:
List[WDL.Expr.Base]
Expression for each item in the array literal
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.Pair(pos: SourcePosition, left: Base, right: Base)[source]
Bases:
BasePair literal
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.Map(pos: SourcePosition, items: List[Tuple[Base, Base]])[source]
Bases:
BaseMap literal
- items: List[Tuple[Base, Base]]
- Type:
List[Tuple[WDL.Expr.Base,WDL.Expr.Base]]
Expressions for the map literal keys and values
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.Struct(pos: SourcePosition, members: List[Tuple[str, Base]], struct_type_name: str | None = None)[source]
Bases:
BaseStruct literal
- members: Dict[str, Base]
- Type:
Dict[str,WDL.Expr.Base]
The struct literal is modelled initially as a bag of keys and values, which can be coerced to a specific struct type during typechecking.
- struct_type_name: str | None
- Type:
Optional[str]
In WDL 2.0+ each struct literal may specify the intended struct type name.
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.IfThenElse(pos: SourcePosition, condition: Base, consequent: Base, alternative: Base)[source]
Bases:
BaseTernary conditional expression
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.Ident(pos: SourcePosition, name: str)[source]
Bases:
BaseAn identifier referencing a named value or call output.
Identnodes are wrapped inGetnodes, as discussed below.- name: str
- Type:
str
Name, possibly including a dot-separated namespace
- referee: None | Tree.Decl | Tree.Call | Tree.Scatter | Tree.Gather
After typechecking within a task or workflow, stores the AST node to which the identifier refers: a
WDL.Tree.Declfor value references; aWDL.Tree.Callfor call outputs; aWDL.Tree.Scatterfor scatter variables; or aWDL.Tree.Gatherobject representing a value or call output that resides within a scatter or conditional section.
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.Get(pos: SourcePosition, expr: Base, member: str | None)[source]
Bases:
BaseAST node representing access to a value by identifier (including namespaced ones), or accessing a member of a pair or struct as
.member.The entaglement of these two cases is inherent in WDL. Consider the syntax
leftname.midname.rightname. One interpretation is thatleftnameis an identifier for a struct value, and.midname.rightnamerepresents a chain of struct member accesses. But another possibility is thatleftnameis a call,midnameis a struct output of that call, andrightnameis a member of that struct. These cases can’t be distinguished by the syntax parser alone, but must be resolved during typechecking with reference to the calls and identifiers available in the environment.The typechecker does conveniently resolve such cases, and to minimize the extent to which it has to restructure the AST in doing so, all identifiers (with or without a namespace) are represented as a
Getnode wrapping anIdentnode. TheGetnode may specify a member name to access, but may not if the identifier is to be accessed directly. On the other hand, the expression inside aGetnode need not be a simple identifier, e.g.arr[1].memb.leftis be represented as:Get(Get(Apply("_at", Get(Ident("arr")), 1),"memb"),"left")- member: str | None
- Type:
Optional[str]
If the expression is accessing a pair/struct member, then
expr.typeisWDL.Type.PairorWDL.Type.StructInstanceand this field gives the desired member name (leftorrightfor pairs).Otherwise the expression accesses
exprdirectly, andmemberisNone.
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- class WDL.Expr.Apply(pos: SourcePosition, function: str, arguments: List[Base])[source]
Bases:
BaseApplication of a built-in or standard library function
- function_name: str
Name of the function applied
- Type:
str
- arguments: List[Base]
- Type:
List[WDL.Expr.Base]
Expressions for each function argument
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
Env
Environments, for identifier resolution during WDL typechecking and evaluation.
- class WDL.Env.Binding(name: str, value: T, info: Any | None = None)[source]
Bases:
Generic[T]An individual, immutable binding of a possibly-namespaced name to a right-hand-side value of type
T.Tis typicallyValue.Base(value environments) orType.Base(type environments). The binding may also reference an additional informational value of arbitrary type.- property name: str
- Type:
str
Namedspaced names are flat, dot-separated strings.
- property value: T
- Type:
T
- class WDL.Env.Bindings(binding: WDL.Env.Binding[T] | None = None, next: WDL.Env.Bindings[T] | None = None)[source]
Bases:
Generic[T]An environment consisting of an immutable linked-list of
WDL.Env.Bindingobjects.WDL.Env.Bindings()is the empty environment.Bindings[T]is iterable for the individualBinding[T]objects:env = WDL.Env.Bindings() env = env.bind("x", 1).bind("y", 42) print(env["x"]) # 1 print(",".join(str(b.value) for b in env)) # 1,42
- bind(name: str, value: T, info: Any | None = None) Bindings[T][source]
Return an environment with a new binding prepended. Any existing binding for the same name is shadowed by the new one. (This should not usually arise in view of the immutability of WDL values.)
- resolve_binding(name: str) Binding[T][source]
Look up a
WDL.Env.Bindingobject by name- Raises:
KeyError – no such binding
- resolve(name: str) T[source]
Look up a bound value by name. Equivalently,
env[name]- Raises:
KeyError – no such binding
- get(name: str, default: T | None = None) T | None[source]
Look up a bound value by name, returning the default value or
Noneif there’s no such binding.
- has_binding(name: str) bool[source]
Determine existence of a binding for the name. Equivalently,
name in env
- map(f: Callable[[Binding[T]], Binding[S] | None]) Bindings[S][source]
Copy the environment with each binding transformed by the given function. If the function returns
Nonethen the binding is excluded.
- filter(pred: Callable[[Binding[T]], bool]) Bindings[T][source]
Copy the environment with only those bindings for which
predreturns True
- subtract(rhs: Bindings[S]) Bindings[T][source]
Copy the environment excluding any binding for which
rhshas a binding with the same name
- property namespaces: Set[str]
- Type:
Set[str]
Return the environment’s namespaces, all the distinct dot-separated prefixes of the binding names. Each element ends with a dot.
Type
WDL data types
WDL has both atomic types such as Int, Boolean, and String; and
parametric types like Array[String] and
Map[String,Array[Array[Float]]]. Here, each type is represented by an
immutable instance of a Python class inheriting from WDL.Type.Base. Such
types are associated with expressions, statically prior to evaluation, as well
as with values and identifier bindings after evaluation.
An atomic type like Int is represented by WDL.Type.Int(). Atomic types
can be checked either with isinstance(t,WDL.Type.Int), which ignores the
possible optional quantifier (thus satisfied by Int or Int?), or with
t == WDL.Type.Int(optional=True) to include the quantifier in the
comparison.
A parametric type like Array[String] is represented by
WDL.Type.Array(WDL.Type.String()). Any kind of array satisfies
isinstance(t,WDL.Type.Array), and
WDL.Type.Array(WDL.Type.String()) == WDL.Type.Array(WDL.Type.String()), but
for example
WDL.Type.Array(WDL.Type.String()) != WDL.Type.Array(WDL.Type.Float()).
The type classes include a method indicating if a value of the type can be coerced to some other desired type, according to the following rules:
Intcoerces toFloatBoolean,Int,Float, andFilecoerce toStringStringcoerces toFile,Int, andFloatArray[T]coerces toStringprovidedTdoes as wellTcoerces toT?but the reverse is not true in general*Array[T]+coerces toArray[T]but the reverse is not true in general*
(*) The reverse coercions are statically permitted in expressions set up with
Expr.infer_type(check_quant=False) although they may fail at runtime. This
also enables coercion of T to Array[T]+ (an array of length 1).
- class WDL.Type.Base[source]
Bases:
ABCThe abstract base class for WDL types
Each specific type inherits from this base, e.g.:
assert issubclass(WDL.Type.Int, WDL.Type.Base) assert isinstance(WDL.Type.Array(WDL.Type.Int()), WDL.Type.Base)
All instances are immutable.
- coerces(rhs: Base, check_quant: bool = True) bool[source]
True if this is the same type as, or can be coerced to,
rhs.- Parameters:
check_quant – when
False, disables static enforcement of the optional (?) type quantifier
- check(rhs: Base, check_quant: bool = True) None[source]
Verify this is the same type as, or can be coerced to
rhs. TheTypeErrorexception raised otherwise MAY include a specific error message (but not if the obvious “cannot coerce self to rhs” suffices).- Parameters:
check_quant – when
False, disables static enforcement of the optional (?) type quantifier
- property optional: bool
- Type:
bool
True when the type has the optional quantifier,
T?
- property parameters: Iterable[Base]
- Type:
Iterable[WDL.Type.Base]
The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)
- copy(self, optional: bool | None = None) WDL.Type.Base[source]
Create a copy of the type, possibly with a different setting of the
optionalquantifier.
- equatable(rhs: Base, compound: bool = False) bool[source]
Check if values of the given types may be equated with the WDL == operator (and its negation). This mostly requires they have the same type, with quirks like ignoring quantifiers and Int/Float coercion (allowed at ‘top level’ but not within compound types).
- class WDL.Type.Any(optional: bool = False, null: bool = False)[source]
Bases:
BaseA symbolic type which coerces to any other type; used to represent e.g. the item type of an empty array literal, or the result of read_json().
The
optionalattribute shall be true only for WDLNoneliterals, which coerce to optional types only.- equatable(rhs, compound: bool = False)[source]
Check if values of the given types may be equated with the WDL == operator (and its negation). This mostly requires they have the same type, with quirks like ignoring quantifiers and Int/Float coercion (allowed at ‘top level’ but not within compound types).
- class WDL.Type.Float(optional: bool = False)[source]
Bases:
Base- equatable(rhs, compound: bool = False)[source]
Check if values of the given types may be equated with the WDL == operator (and its negation). This mostly requires they have the same type, with quirks like ignoring quantifiers and Int/Float coercion (allowed at ‘top level’ but not within compound types).
- class WDL.Type.Int(optional: bool = False)[source]
Bases:
Base- equatable(rhs, compound: bool = False)[source]
Check if values of the given types may be equated with the WDL == operator (and its negation). This mostly requires they have the same type, with quirks like ignoring quantifiers and Int/Float coercion (allowed at ‘top level’ but not within compound types).
- class WDL.Type.Array(item_type: Base, optional: bool = False, nonempty: bool = False)[source]
Bases:
BaseArray type, parameterized by the type of the constituent items.
- item_type: Base
- Type:
item_typemay beAnywhen not known statically, such as in a literal empty array[].
- property nonempty: bool
- Type:
bool
True when the type has the nonempty quantifier,
Array[T]+
- property parameters: Iterable[Base]
- Type:
Iterable[WDL.Type.Base]
The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)
- copy(self, optional: bool | None = None) WDL.Type.Base[source]
Create a copy of the type, possibly with a different setting of the
optionalquantifier.
- equatable(rhs, compound: bool = False)[source]
Check if values of the given types may be equated with the WDL == operator (and its negation). This mostly requires they have the same type, with quirks like ignoring quantifiers and Int/Float coercion (allowed at ‘top level’ but not within compound types).
- class WDL.Type.Map(item_type: Tuple[Base, Base], optional: bool = False, literal_keys: Set[str] | None = None)[source]
Bases:
BaseMap type, parameterized by the (key,value) item type.
- item_type: Tuple[Base, Base]
- Type:
Tuple[WDL.Type.Base,WDL.Type.Base]
The key and value types may be
Anywhen not known statically, such as in a literal empty map{}.
- literal_keys: Set[str] | None
- property parameters: Iterable[Base]
- Type:
Iterable[WDL.Type.Base]
The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)
- equatable(rhs, compound: bool = False)[source]
Check if values of the given types may be equated with the WDL == operator (and its negation). This mostly requires they have the same type, with quirks like ignoring quantifiers and Int/Float coercion (allowed at ‘top level’ but not within compound types).
- class WDL.Type.Pair(left_type: Base, right_type: Base, optional: bool = False)[source]
Bases:
BasePair type, parameterized by the left and right item types.
- property parameters: Iterable[Base]
- Type:
Iterable[WDL.Type.Base]
The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)
- class WDL.Type.StructInstance(type_name: str, optional: bool = False)[source]
Bases:
BaseType of an instance of a struct
Not to be confused with struct type definition,
WDL.Tree.StructTypeDef. To find theWDL.Tree.StructTypeDefin the currentdoc: WDL.Tree.Documentcorresponding toty: WDL.Type.StructInstance, usedoc.struct_typedefs[ty.type_name].- type_name: str
- Type:
str
The struct type name with which the instance is declared; note that the same struct type can go by different names.
- members: Dict[str, Base] | None
- Type:
Dict[str,WDL.Type.Base]
Names and types of the struct members, from the struct type definition (available after typechecking)
- property type_id: str
- Type:
str
A string canonically describing the member names and their types, excluding the struct type name; useful to unify aliased struct types.
- property parameters: Iterable[Base]
- Type:
Iterable[WDL.Type.Base]
The type’s parameters, if any (e.g. item type of Array; left & right types of Pair; etc.)
- WDL.Type.unify(types: List[Base], check_quant: bool = True, force_string: bool = False) Base[source]
Given a list of types, compute a type to which they’re all coercible, or
WDL.Type.Anyif no more-specific inference is possible.- Parameters:
force_string – permit last-resort unification to
Stringeven if no item is currently aString, but all can be coerced
Value
WDL values instantiated at runtime
Each value is represented by an instance of a Python class inheriting from
WDL.Value.Base.
- class WDL.Value.Base(type: Base, value: Any, expr: Expr.Base | None = None)[source]
Bases:
ABCThe abstract base class for WDL values
- value: Any
The “raw” Python value
- property expr: Expr.Base | None
Reference to the WDL expression that generated this value, if it originated from
WDL.Expr.eval
- coerce(desired_type: Base | None = None) Base[source]
Coerce the value to the desired type and return it. Types should be checked statically on
WDL.Expr.Baseprior to evaluation.- Raises:
ReferenceError for a null value and non-optional type
- property json: Any
Return a value representation which can be serialized to JSON using
json.dumps
- class WDL.Value.Boolean(value: bool, expr: Expr.Base | None = None)[source]
Bases:
Basevaluehas Python typebool- value: Any
The “raw” Python value
- class WDL.Value.Float(value: float, expr: Expr.Base | None = None)[source]
Bases:
Basevaluehas Python typefloat- value: Any
The “raw” Python value
- class WDL.Value.Int(value: int, expr: Expr.Base | None = None)[source]
Bases:
Basevaluehas Python typeint- value: Any
The “raw” Python value
- class WDL.Value.String(value: str, expr: Expr.Base | None = None, subtype: Base | None = None)[source]
Bases:
Basevaluehas Python typestr- value: Any
The “raw” Python value
- class WDL.Value.File(value: str, expr: Expr.Base | None = None)[source]
Bases:
Stringvaluehas Python typestr- value: Any
The “raw” Python value
- class WDL.Value.Directory(value: str, expr: Expr.Base | None = None)[source]
Bases:
Stringvaluehas Python typestr- value: Any
The “raw” Python value
- class WDL.Value.Array(item_type: Base, value: List[Base], expr: Expr.Base | None = None)[source]
Bases:
Basevalueis a Pythonlistof otherWDL.Value.Baseinstances
- class WDL.Value.Map(item_type: Tuple[Base, Base], value: List[Tuple[Base, Base]], expr: Expr.Base | None = None)[source]
Bases:
Base
- class WDL.Value.Pair(left_type: Base, right_type: Base, value: Tuple[Base, Base], expr: Expr.Base | None = None)[source]
Bases:
Base
- class WDL.Value.Null(expr: Expr.Base | None = None)[source]
Bases:
BaseRepresents the missing value which optional inputs may take.
typeandvalueare both None.- value: Any
The “raw” Python value
- class WDL.Value.Struct(type: Object | StructInstance, value: Dict[str, Base], expr: Expr.Base | None = None, extra: Set[str] | None = None)[source]
Bases:
Base
- WDL.Value.from_json(type: Base, value: Any) Base[source]
Instantiate a WDL value of the specified type from a parsed JSON value (str, int, float, list, dict, or null).
If type is
WDL.Type.Any(), attempts to infer a WDL type & value from the JSON’s intrinsic types. This isn’t ideal; for example, Files can’t be distinguished from Strings, and JSON lists and dicts with heterogeneous item types may give undefined results.- Raises:
WDL.Error.InputError – if the given value isn’t coercible to the specified type
- WDL.Value.rewrite_paths(v: Base, f: Callable[[File | Directory], str | None]) Base[source]
Produce a deep copy of the given Value with all File & Directory paths (including those nested inside compound Values) rewritten by the given function. The function may return None to replace the File/Directory value with None/Null.
- WDL.Value.rewrite_env_paths(env: Bindings[Base], f: Callable[[File | Directory], str | None]) Bindings[Base][source]
Produce a deep copy of the given Value Env with all File & Directory paths rewritten by the given function.
- WDL.Value.rewrite_files(v: Base, f: Callable[[str], str | None]) Base[source]
Produce a deep copy of the given Value with all File names rewritten by the given function (including Files nested inside compound Values).
(deprecated: use
rewrite_pathsto handle Directory values as well)
Error
- class WDL.Error.SourcePosition(uri, abspath, line, column, end_line, end_column)[source]
Bases:
SourcePositionSource position attached to AST nodes and exceptions; NamedTuple of
urithe filename/URI passed toWDL.load()or a WDL import statement, which may be relative;abspaththe absolute filename/URI; and one-based int positionslineend_linecolumnend_column
- exception WDL.Error.SyntaxError(pos: SourcePosition, msg: str, wdl_version: str, declared_wdl_version: str | None)[source]
Bases:
ExceptionFailure to lex/parse a WDL document
- exception WDL.Error.ImportError(pos: SourcePosition, import_uri: str, message: str | None = None)[source]
Bases:
ExceptionFailure to open/retrieve an imported WDL document
The
__cause__attribute may hold the inner error object.
- class WDL.Error.SourceNode(pos: SourcePosition)[source]
Bases:
objectBase class for an AST node, recording the source position
- parent: SourceNode | None = None
- Type:
Optional[SourceNode]
Parent node in the AST, if any
- pos: SourcePosition
- Type:
Source position for this AST node
- property children: Iterable[SourceNode]
- Type:
Iterable[SourceNode]
Yield all child nodes
- exception WDL.Error.ValidationError(node: SourceNode | SourcePosition, message: str)[source]
Bases:
ExceptionBase class for a WDL validation error (when the document loads and parses, but fails typechecking or other static validity tests)
- source_text: str | None = None
- Type:
Optional[str]
The complete source text of the WDL document (if available)
- node: SourceNode | None = None
- Type:
Optional[SourceNode]
- pos: SourcePosition
- Type:
- exception WDL.Error.InvalidType(node: SourceNode | SourcePosition, message: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.IndeterminateType(node: SourceNode | SourcePosition, message: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.NoSuchTask(node: SourceNode | SourcePosition, name: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.NoSuchCall(node: SourceNode | SourcePosition, name: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.NoSuchFunction(node: SourceNode | SourcePosition, name: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.WrongArity(node: SourceNode | SourcePosition, expected: int)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.NotAnArray(node: SourceNode | SourcePosition)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.NoSuchMember(node: SourceNode | SourcePosition, member: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.StaticTypeMismatch(node: SourceNode, expected: Base, actual: Base, message: str = '')[source]
Bases:
ValidationError
- exception WDL.Error.IncompatibleOperand(node: SourceNode, message: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.UnknownIdentifier(node: SourceNode, message: str | None = None)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.NoSuchInput(node: SourceNode, name: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.UncallableWorkflow(node: SourceNode, name: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.MultipleDefinitions(node: SourceNode | SourcePosition, message: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.StrayInputDeclaration(node: SourceNode | SourcePosition, message: str)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.CircularDependencies(node: SourceNode)[source]
Bases:
ValidationError- pos: SourcePosition
- Type:
- exception WDL.Error.MultipleValidationErrors(*exceptions: List[ValidationError | MultipleValidationErrors])[source]
Bases:
ExceptionPropagates several validation/typechecking errors
- exceptions: List[ValidationError]
- Type:
List[ValidationError]
- exception WDL.Error.RuntimeError(*args, more_info: Dict[str, Any] | None = None, **kwargs)[source]
Bases:
Exception- more_info: Dict[str, Any]
Backend-specific information about an error (for example, pointer to a centralized log system)
- exception WDL.Error.EvalError(node: SourceNode | SourcePosition, message: str)[source]
Bases:
RuntimeErrorError evaluating a WDL expression or declaration
- node: SourceNode | None = None
- Type:
Optional[SourceNode]
- pos: SourcePosition
- Type:
- exception WDL.Error.OutOfBounds(node: SourceNode | SourcePosition, message: str)[source]
Bases:
EvalError- pos: SourcePosition
- Type:
- more_info: Dict[str, Any]
Backend-specific information about an error (for example, pointer to a centralized log system)
- exception WDL.Error.EmptyArray(node: SourceNode)[source]
Bases:
EvalError- pos: SourcePosition
- Type:
- more_info: Dict[str, Any]
Backend-specific information about an error (for example, pointer to a centralized log system)
- exception WDL.Error.NullValue(node: SourceNode | SourcePosition)[source]
Bases:
EvalError- pos: SourcePosition
- Type:
- more_info: Dict[str, Any]
Backend-specific information about an error (for example, pointer to a centralized log system)
- exception WDL.Error.InputError(*args, more_info: Dict[str, Any] | None = None, **kwargs)[source]
Bases:
RuntimeErrorError reading an input value/file
- more_info: Dict[str, Any]
Backend-specific information about an error (for example, pointer to a centralized log system)
Lint
Annotate WDL document AST with hygiene warnings (underlies miniwdl check)
Given a doc: WDL.Document, the lint warnings can be retrieved like so:
import WDL
import WDL.Lint
lint = WDL.Lint.collect(WDL.Lint.lint(doc, descend_imports=False))
for (pos, lint_class, message, suppressed) in lint:
assert isinstance(pos, WDL.SourcePosition)
assert isinstance(lint_class, str) and isinstance(message, str)
if not suppressed:
print(json.dumps({
"uri" : pos.uri,
"abspath" : pos.abspath,
"line" : pos.line,
"end_line" : pos.end_line,
"column" : pos.column,
"end_column" : pos.end_column,
"lint" : lint_class,
"message" : message,
}))
The descend_imports flag controls whether lint warnings are generated for imported documents
recursively (true, default), or otherwise only the given document (false).
Zip
Routines for packaging a WDL source file, with all imported source files, into a ZIP file.
New in v1.5.0
- WDL.Zip.build(top_doc: Document, archive: str, logger: Logger, inputs: Dict[str, Any] | None = None, meta: Dict[str, Any] | None = None, archive_format: str = 'zip', additional_files: List[str] | None = None)[source]
Generate zip archive of the WDL document, all its imports, optional default inputs, and a generated manifest JSON.
If imports are drawn from outside the main WDL’s directory (or by URI), they’ll be stored in a special subdirectory and import statements will be rewritten to match.
- class WDL.Zip.UnpackedZip(dir, main_wdl, input_file)
Contextual value of WDL.Zip.unpack(): absolute paths of source directory, main WDL, and default input JSON file (if any). The source directory prefixes the latter paths.
- dir: str
Alias for field number 0
- input_file: str | None
Alias for field number 2
- main_wdl: str
Alias for field number 1
- WDL.Zip.unpack(archive_fn: str) Iterator[UnpackedZip][source]
Open a context with the WDL source archive unpacked into a temp directory, yielding UnpackedZip. The temp directory will be deleted on context exit.
A path to the MANIFEST.json of an already-unpacked source archive may also be used, or a directory containing one. In this case, it is NOT deleted on context exit.
with WDL.Zip.unpack("/path/to/source.zip") as unpacked: doc = WDL.load(unpacked.main_wdl) ...
runtime
The recommended way to run a WDL workflow programmatically is to invoke miniwdl run as a
subprocess, capturing its JSON standard output. This leverages its logging, configuration, and
flexible input loading features; and avoids conflicting with the runtime’s thread pools and signal
handlers. Alternatively, it’s possible to call WDL.runtime.run() directly if needed.
- WDL.runtime.run(cfg: Loader, exe: Task | Workflow, inputs: Bindings[Base], **run_kwargs: Dict[str, Any]) Tuple[str, Bindings[Base]][source]
Run the task or workflow given the inputs environment and configuration, returning the outputs environment.
inputsmay be parsed from a JSON dict usingvalues_from_json(), which can also validate them; see example below.- Parameters:
run_id – a run identifier used in logs and filenames; defaults to executable name
run_dir – directory under which to create a timestamp-named subdirectory for this run (defaults to current working directory). If the final path component is
.then operate in run_dir directly.
Typical usage:
import WDL import WDL.runtime # Convert JSON-like inputs dict to WDL environment, validating them against exe's available # and required inputs. The dict keys should NOT be namespaced by the executable name; # if namespaces are present, then add namespace=exe.name to effectively remove them. inputs_env = WDL.values_from_json(inputs_dict, exe.available_inputs, exe.required_inputs) # Load configuration (see below) cfg = WDL.runtime.config.Loader(logging.getLogger(__name__)) # Run executable run_subdir, outputs_env = WDL.runtime.run(cfg, exe, inputs_env, run_dir="/tmp") # Generate JSON-like outputs dict, with keys namespaced by the executable name outputs_dict = WDL.values_to_json(outputs_env, exe.name)
- class WDL.runtime.config.Loader(logger: Logger, filenames: List[str] | None = None, overrides: Dict[str, Dict[str, str]] | None = None)[source]
Runtime configuration options, identified by section & key, are sourced in the following priority order:
Supplied
overridesdict,{"section": {"key": value}}Environment variables
MINIWDL__SECTION__KEY(uppercased with double-underscores)Custom configuration file (mutually exclusive):
filename given to
__init__file named by environment variable
MINIWDL_CFGminiwdl.cfgin XDG_CONFIG_HOME or XDG_CONFIG_DIRS, usually~/.config/miniwdl.cfg
WDL/runtime/config_templates/default.cfgfrom installed package
If
filenamesis an empty list, then no custom configuration file is used. Orfilenamesmay be a prioritized list of candidate filenames instead of (a-c) above; the first extant file is used, if any.