API documentation

The following documentation is based on the source code of version 23.2 of the executor package. The following modules are available:

The executor module

Core functionality of the executor package.

If you’re looking for an easy way to run external commands from Python take a look at the execute() function. When you need more flexibility consider using the underlying ExternalCommand class directly instead.

execute() versus ExternalCommand

In executor 1.x the execute() function was the only interface for external command execution. This had several drawbacks:

  • The documentation for the execute() function was getting way too complex given all of the supported options and combinations.
  • There was no way to execute asynchronous external commands (running in the background) without sidestepping the complete executor module and going straight for subprocess.Popen (with all of the verbosity that you get for free with subprocess :-).
  • There was no way to prepare an external command without starting it immediately, making it impossible to prepare a batch of external commands before starting them (whether synchronously or asynchronously).

To solve these problems executor 2.x introduced the ExternalCommand class. This explains why execute() is now a trivial wrapper around ExternalCommand: It’s main purpose is to be an easy to use shortcut that preserves compatibility with the old interface.

Classes and functions

executor.DEFAULT_ENCODING = 'UTF-8'

The default encoding of the standard input, output and error streams (a string).

executor.DEFAULT_WORKING_DIRECTORY = '.'

The default working directory for external commands (a string). Defaults to the working directory of the current process using os.curdir.

executor.DEFAULT_SHELL = 'bash'

The default shell used to evaluate shell expressions (a string).

This variable isn’t based on the $SHELL environment variable because:

  1. Shells like sh, dash, bash and zsh all have their own subtly incompatible semantics.
  2. People regularly use shells like fish as their default login shell :-).

At an interactive prompt this is no problem (advanced users have obviously learned to context switch) but when you’re writing source code the last thing you want to worry about is which shell is going to evaluate your commands! The executor package expects this shell to support the following features:

  • The -c option to evaluate a shell command provided as a command line argument.
  • The - argument to instruct the shell to read shell commands from its standard input stream and evaluate those.

Apart from these two things nothing else is expected from the default shell so you’re free to customize it if you really want to write your shell commands in fish or zsh syntax :-).

executor.COMMAND_NOT_FOUND_CODES = (2,)

Numeric error codes returned when a command isn’t available on the system (a tuple of integers).

executor.COMMAND_NOT_FOUND_STATUS = 127

The exit status used by shells when a command is not found (an integer).

executor.execute(*command, **options)[source]

Execute an external command and make sure it succeeded.

Parameters:
  • command – All positional arguments are passed on to the constructor of ExternalCommand.
  • options – All keyword arguments are passed on to the constructor of ExternalCommand.
Returns:

Refer to execute_prepared().
Raises:

ExternalCommandFailed when the command exits with a nonzero exit code (and check is True).

If asynchronous is True then execute() will automatically start the external command for you using start() (but it won’t wait for it to end). If you want to create an ExternalCommand object instance without immediately starting the external command then you can use ExternalCommand directly.

Some examples

By default the status code of the external command is returned as a boolean:

>>> from executor import execute
>>> execute('true')
True

However when an external command exits with a nonzero status code an exception is raised, this is intended to “make it easy to do the right thing” (never forget to check the status code of an external command without having to write a lot of repetitive code):

>>> execute('false')
Traceback (most recent call last):
  File "executor/__init__.py", line 124, in execute
    cmd.start()
  File "executor/__init__.py", line 516, in start
    self.wait()
  File "executor/__init__.py", line 541, in wait
    self.check_errors()
  File "executor/__init__.py", line 568, in check_errors
    raise ExternalCommandFailed(self)
executor.ExternalCommandFailed: External command failed with exit code 1! (command: false)

What’s also useful to know is that exceptions raised by execute() expose command and returncode attributes. If you know a command is likely to exit with a nonzero status code and you want execute() to simply return a boolean you can do this instead:

>>> execute('false', check=False)
False
executor.execute_prepared(command)[source]

The logic behind execute() and remote().

Parameters:command – An ExternalCommand object (or an object created from a subclass with a compatible interface like for example RemoteCommand).
Returns:The return value of this function depends on several options:
Raises:See execute() and remote().
class executor.ExternalCommand(*command, **options)[source]

Programmer friendly subprocess.Popen wrapper.

The ExternalCommand class wraps subprocess.Popen to make it easier to do the right thing (the simplicity of os.system() with the robustness of subprocess.Popen) and to provide additional features (e.g. asynchronous command execution that preserves the ability to provide input and capture output).

Process manipulation
ExternalCommand inherits from ControllableProcess which means that all of the process manipulation supported by ControllableProcess is also supported by ExternalCommand objects.
Context manager

ExternalCommand objects can be used as context managers by using the with statement:

  • When the scope of the with statement starts the start() method is called (if the external command isn’t already running).
  • When the scope of the with statement ends terminate() is called if the command is still running. The load_output() and cleanup() functions are used to cleanup after the external command. If an exception isn’t already being raised check_errors() is called to make sure the external command succeeded.
Event callbacks

The start_event, retry_event and finish_event properties can be set to callbacks (callable values like functions) to subscribe to the corresponding events. The callback receives a single positional argument which is the ExternalCommand object.

The start_event and finish_event properties were originally created for use in command pools, for example to report to the operator when specific commands are started and when they finish. The event handling is performed inside the ExternalCommand class though, so you’re free to repurpose these events outside the context of command pools.

Here’s an overview of the ExternalCommand class:

Superclass: ControllableProcess
Special methods: __enter__(), __exit__(), __init__(), __iter__() and __str__()
Public methods: async_fdel(), async_fget(), async_fset(), check_errors(), check_retry_allowed(), cleanup(), format_error_message(), get_decoded_output(), invoke_event_callback(), kill_helper(), load_output(), prefix_shell_command(), reduce_shell_command(), reset(), start(), start_once(), terminate_helper() and wait()
Properties: async, asynchronous, buffer_size, buffered, callback, capture, capture_stderr, check, command, command_line, decoded_stderr, decoded_stdout, dependencies, directory, encoded_input, encoding, environment, error_message, error_type, failed, fakeroot, finish_event, group_by, have_superuser_privileges, input, ionice, ionice_command, is_finished, is_finished_with_retries, is_running, is_terminated, merge_streams, output, really_silent, result, retry, retry_allowed, retry_count, retry_event, retry_limit, returncode, shell, silent, start_event, stderr, stderr_file, stdin, stdout, stdout_file, subprocess, succeeded, sudo, sudo_command, tty, uid, user, virtual_environment and was_started

You can set the values of the asynchronous, buffer_size, buffered, callback, capture, capture_stderr, check, command, dependencies, directory, encoding, environment, error_message, error_type, fakeroot, finish_event, group_by, input, ionice, merge_streams, really_silent, retry, retry_allowed, retry_count, retry_event, retry_limit, returncode, shell, silent, start_event, stderr_file, stdout_file, subprocess, sudo, tty, uid, user, virtual_environment and was_started properties by passing keyword arguments to the class initializer.

__init__(*command, **options)[source]

Initialize an ExternalCommand object.

Parameters:

The external command is not started until you call start() or wait().

asynchronous[source]

Enable asynchronous command execution.

If this option is True (not the default) preparations are made to execute the external command asynchronously (in the background). This has several consequences:

  • Calling start() will start the external command but will not block until the external command is finished, instead you are responsible for calling wait() at some later point in time.

  • When input is set its value will be written to a temporary file and the standard input stream of the external command is connected to read from the temporary file.

    By using a temporary file the external command can consume its input as fast or slow as it pleases without needing a separate thread or process to “feed” the external command.

  • When capture is True the standard output of the external command is redirected to a temporary file whose contents are read once the external command has finished.

    By using a temporary file the external command can produce output as fast or slow as it pleases without needing a thread or subprocess on our side to consume the output in real time.

See also

async (backwards compatible alias)

Note

The asynchronous property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

buffer_size[source]

Control the size of the stdin/stdout/stderr pipe buffers.

The value of buffer_size becomes the bufsize argument that’s passed to subprocess.Popen by start().

If asynchronous is True and buffered is False the value of buffer_size defaults to 0 which means unbuffered, in all other cases its value defaults to -1 which means to use the system default buffer size.

Note

The buffer_size property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

buffered[source]

Control whether command output is buffered to temporary files.

When asynchronous is True and the standard output and/or error streams are being captured, temporary files will be used to collect the output. This enables the use of the output, stdout and stderr properties to easily get the full output of the command in a single string.

You can set buffered to False to disable the use of temporary files, in this case subprocess.PIPE is passed to subprocess.Popen. Once is_running is True you can use the stdin, stdout and/or stderr properties to communicate with the command.

This enables runtime processing of the standard input, output and error streams and makes it possible to run commands that never return but keep producing output (for example xscreensaver-command -watch). Here’s an example that sets buffered to False and uses the magic method __iter__() to iterate over the lines of output in realtime:

# Run external commands when xscreensaver changes state.

import os
from executor import execute

known_states = set(['BLANK', 'LOCK', 'UNBLANK'])

while True:
    options = dict(asynchronous=True, capture=True, buffered=False)
    with execute('xscreensaver-command', '-watch', **options) as command:
        for line in command:
            tokens = line.split()
            if tokens and tokens[0] in known_states:
                value = os.environ.get('XSCREENSAVER_%s_COMMAND' % tokens[0])
                if value:
                    execute(value)

Some sanity checks and error handling have been omitted from the example above, in order to keep it simple, but I did test it and it should actually work (at least it did for me):

$ export XSCREENSAVER_BLANK_COMMAND='echo $(date) - Screen is now blanked'
$ export XSCREENSAVER_LOCK_COMMAND='echo $(date) - Screen is now locked'
$ export XSCREENSAVER_UNBLANK_COMMAND='echo $(date) - Screen is now unblanked'
$ python xscreensaver-monitor.py
Sat Jan 20 16:03:07 CET 2018 - Screen is now blanked
Sat Jan 20 16:03:15 CET 2018 - Screen is now unblanked
Sat Jan 20 16:03:20 CET 2018 - Screen is now locked
Sat Jan 20 16:03:34 CET 2018 - Screen is now unblanked

Note

The buffered property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

callback[source]

Optional callback used to generate the value of result.

The callback and result properties were created for use in command pools, where it can be useful to define how to process (parse) a command’s output when the command is constructed.

Note

The callback property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

capture[source]

Enable capturing of the standard output stream.

If this option is True (not the default) the standard output of the external command is captured and made available to the caller via stdout and output.

The standard error stream will not be captured, use capture_stderr for that. You can also silence the standard error stream using the silent option.

If callback is set capture defaults to True (but you can still set capture to False if that is what you want).

Note

The capture property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

capture_stderr[source]

Enable capturing of the standard error stream.

If this option is True (not the default) the standard error stream of the external command is captured and made available to the caller via stderr.

Note

The capture_stderr property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

check[source]

Enable automatic status code checking.

If this option is True (the default) and the external command exits with a nonzero status code ExternalCommandFailed will be raised by start() (when asynchronous isn’t set) or wait() (when asynchronous is set).

Note

The check property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command[source]

A list of strings with the command to execute.

Note

In executor version 14.0 it became valid to set input and shell without providing command (in older versions it was required to set command regardless of the other options).

Note

The command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The command line of the external command.

The command line used to actually run the external command requested by the user (a list of strings). The command line is constructed based on command according to the following rules:

  • If shell is True the external command is run using bash -c '...' (assuming you haven’t changed DEFAULT_SHELL) which means constructs like semicolons, ampersands and pipes can be used (and all the usual caveats apply :-).
  • If virtual_environment is set the command is converted to a shell command line and prefixed by the applicable source ... command.
  • If uid or user is set the sudo -u command will be prefixed to the command line generated here.
  • If fakeroot or sudo is set the respective command name is prefixed to the command line generated here (sudo is only prefixed when the current process doesn’t already have super user privileges).
  • If ionice is set the appropriate command is prefixed to the command line generated here.
decoded_stdout

The value of stdout decoded using encoding.

This is a unicode() object (in Python 2) or a str object (in Python 3).

decoded_stderr

The value of stderr decoded using encoding.

This is a unicode() object (in Python 2) or a str object (in Python 3).

dependencies[source]

The dependencies of the command (a list of ExternalCommand objects).

The dependencies property enables low level concurrency control in command pools by imposing a specific order of execution:

Note

The dependencies property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

directory[source]

The working directory for the external command.

A string, defaults to DEFAULT_WORKING_DIRECTORY.

Note

The directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

encoded_input

The value of input encoded using encoding.

This is a str object (in Python 2) or a bytes object (in Python 3).

encoding[source]

The character encoding of standard input and standard output.

A string, defaults to DEFAULT_ENCODING. This option is used to encode input and to decode output.

Note

The encoding property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

environment[source]

A dictionary of environment variables for the external command.

You only need to specify environment variables that differ from those of the current process (that is to say the environment variables of the current process are merged with the variables that you specify here).

Note

The environment property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

error_message[source]

A string describing how the external command failed or None.

Note

The error_message property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

error_type[source]

An appropriate exception class or None (when no error occurred).

CommandNotFound if the external command exits with return code COMMAND_NOT_FOUND_STATUS or ExternalCommandFailed if the external command exits with any other nonzero return code.

Note

The error_type property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

failed

Whether the external command has failed.

  • True if returncode is a nonzero number or error_type is set (e.g. because the external command doesn’t exist).
  • False if returncode is zero.
  • None when the external command hasn’t been started or is still running.
fakeroot[source]

Run the external command under fakeroot.

If this option is True (not the default) and the current process doesn’t have superuser privileges the external command is run with fakeroot. If the fakeroot program is not installed the external command will fail.

Note

The fakeroot property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

finish_event[source]

Optional callback that’s called just after the command finishes (see event callbacks).

Note

The finish_event property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

group_by[source]

Identifier that’s used to group the external command (any hashable value).

The group_by property enables high level concurrency control in command pools by making it easy to control which commands are allowed to run concurrently and which are required to run serially:

  • Command pools will never start more than one command within a group of commands that share the same value of group_by (for values that aren’t None).
  • If group_by is None it has no effect and concurrency is controlled by dependencies and concurrency.

Note

The group_by property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

have_superuser_privileges

Whether the parent Python process is running under superuser privileges.

True if running with superuser privileges, False otherwise. Used by command_line to decide whether sudo needs to be used.

input[source]

The input to feed to the external command on the standard input stream.

When you provide a unicode() object (in Python 2) or a str object (in Python 3) as input it will be encoded using encoding. To avoid the automatic conversion you can simply pass a str object (in Python 2) or a bytes object (in Python 3). This conversion logic is implemented in the encoded_input attribute.

When input is set to True a pipe will be created to communicate with the external command in real time. See also the buffered and stdin properties.

Defaults to None.

Note

The input property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ionice[source]

The I/O scheduling class for the external command (a string or None).

When this property is set then ionice will be used to set the I/O scheduling class for the external command. This can be useful to reduce the impact of heavy disk operations on the rest of the system.

Raises:Any exceptions raised by validate_ionice_class().

Note

The ionice property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ionice_command

The ionice command based on ionice (a list of strings).

is_finished

Whether the external command has finished execution (excluding retries).

True once the external command has been started and has since finished (excluding retries), False when the external command hasn’t been started yet or is still running.

is_finished_with_retries

Whether the external command has finished execution (including retries).

True once the external command has been started and has since finished (including retries), False when the external command hasn’t been started yet, is still running or can be retried.

is_running

True if the process is currently running, False otherwise.

is_terminated

Whether the external command has been terminated (a boolean).

True if the external command was terminated using SIGTERM (e.g. by terminate()), False otherwise.

merge_streams[source]

Whether to merge the standard output and error streams.

A boolean, defaults to False. If this option is enabled stdout will contain the external command’s output on both streams.

Note

The merge_streams property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

output

The value of stdout decoded using encoding.

This is a unicode() object (in Python 2) or a str object (in Python 3).

This is only available when capture is True. If capture is not True then output will be None.

After decoding any leading and trailing whitespace is stripped and if the resulting string doesn’t contain any remaining newlines then the string with leading and trailing whitespace stripped will be returned, otherwise the decoded string is returned unchanged:

>>> from executor import ExternalCommand
>>> cmd = ExternalCommand('echo naïve', capture=True)
>>> cmd.start()
>>> cmd.output
u'na\xefve'
>>> cmd.stdout
'na\xc3\xafve\n'

This is intended to make simple things easy (output makes it easy to deal with external commands that output a single line) while providing an escape hatch when the default assumptions don’t hold (you can always use stdout to get the raw output).

See also the __iter__() magic method which makes it very easy to iterate over the lines of output produced by the command.

really_silent[source]

Whether output is really silenced or actually captured (a boolean).

When the silent option was originally added to executor it was implemented by redirecting the output streams to os.devnull, similar to how command &> /dev/null works in Bash.

Since I made that decision I’ve regretted it many times because I ran into situations where check and silent were both set and ExternalCommandFailed was raised but I had no way to determine what had gone wrong.

This is why the really_silent property was introduced in executor release 19.0:

This change was made after much consideration because it is backwards incompatible and not only in a theoretical sense: Imagine a daemon process spewing megabytes of log output on its standard error stream.

As an escape hatch to restore backwards compatibility you can set really_silent to True to override the computed value.

Note

The really_silent property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

result

The result of calling the value given by callback.

If the command hasn’t been started yet start() is called. When the command hasn’t finished yet func:wait() is called. If callback isn’t set None is returned.

retry[source]

Whether the external command should be retried when it fails (a boolean, defaults to False).

Warning

Retrying of failing commands is an experimental feature that was introduced with the release of executor 20.0. Please refer to the 20.0 release notes for details.

Note

The retry property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

retry_allowed[source]

True if the external command can be retried, False otherwise.

The value of this property is computed by checking if the following conditions hold:

Note that when the retry_event callback returns False to cancel the retrying of a failed command, the computed value of retry_allowed is overridden by assigning retry_allowed the value False.

Note

The retry_allowed property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

retry_count[source]

The number of times that the command was retried (an integer number, defaults to 0).

The value of retry_count is automatically incremented by start_once() when it notices that was_started is True before start_once() has started the command.

Note

The retry_count property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

retry_event[source]

Optional callback that’s called when a command is retried (see event callbacks).

The callback can return False to abort retrying.

Note

The retry_event property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

retry_limit[source]

The maximum number of times to retry the command when it fails (an integer, defaults to 2).

Given the default value of two, when retry is True the command will be run at most three times (the initial run and two retries). The value 0 means the command will be retried until it succeeds.

Note

The retry_limit property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

returncode[source]

The return code of the external command (an integer) or None.

This will be None until the external command has finished.

Note

The returncode property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

shell[source]

Whether to evaluate the external command as a shell command.

A boolean, the default depends on the value of command:

When shell is True the external command is evaluated by the shell given by DEFAULT_SHELL, otherwise the external command is run without shell evaluation.

Note

The shell property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

silent[source]

Whether the external command’s output should be silenced (a boolean).

If this is True (not the default) any output of the external command is silenced by redirecting the output streams to os.devnull (if really_silent is True) or by capturing the output (if really_silent is False).

You can enable capture and silent together to capture the standard output stream while silencing the standard error stream.

Note

The silent property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

start_event[source]

Optional callback that’s called just before the command is started (see event callbacks).

Note

The start_event property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

stderr

The standard error stream of the external command.

This property is only available when capture_stderr is True. In all other cases the value of stderr will be None.

When buffered is True (the default) this is a str object (in Python 2) or a bytes object (in Python 3).

If you set buffered to False then stderr will be a pipe that’s connected to the standard error stream of the command (for as long as is_running is True).

stderr_file[source]

Capture the standard error stream to the given file handle.

When this property is set to a writable file object the standard error stream of the external command is redirected to the given file. The default value of this property is None.

This can be useful to (semi) permanently store command output or to run commands whose output is hidden but can be followed using tail -f if the need arises. By setting stdout_file and stderr_file to the same file object the output from both streams can be merged and redirected to the same file. This accomplishes roughly the same thing as setting merge_streams but leaves the caller in control of the file.

If this property isn’t set but capture is True the external command’s output is captured to a temporary file that’s automatically cleaned up after the external command is finished and its output has been cached (read into memory).

Note

The stderr_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

stdin

The standard input stream of the external command.

If you set input to True and buffered to False then stdin will be a pipe that’s connected to the standard input stream of the command (for as long as is_running is True).

In all other cases the value of stdin will be None.

stdout

The standard output stream of the external command.

This property is only available when capture is True. In all other cases the value of stdout will be None.

When buffered is True (the default) this is a str object (in Python 2) or a bytes object (in Python 3).

If you set buffered to False then stdout will be a pipe that’s connected to the standard output stream of the command (for as long as is_running is True).

stdout_file[source]

Capture the standard output stream to the given file handle.

When this property is set to a writable file object the standard output stream of the external command is redirected to the given file. The default value of this property is None.

This can be useful to (semi) permanently store command output or to run commands whose output is hidden but can be followed using tail -f if the need arises. By setting stdout_file and stderr_file to the same file object the output from both streams can be merged and redirected to the same file. This accomplishes roughly the same thing as setting merge_streams but leaves the caller in control of the file.

If this property isn’t set but capture is True the external command’s output is captured to a temporary file that’s automatically cleaned up after the external command is finished and its output has been cached (read into memory).

Note

The stdout_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

subprocess[source]

A subprocess.Popen object or None.

The value of this property is set by start() and it’s cleared by wait() (through cleanup()) as soon as the external command has finished. This enables garbage collection of the resources associated with the subprocess.Popen object which helps to avoid IOError: [Errno 24] Too many open files errors.

Note

The subprocess property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

succeeded

Whether the external command succeeded.

  • True if returncode is zero.
  • False if returncode is a nonzero number or error_type is set (e.g. because the external command doesn’t exist).
  • None when the external command hasn’t been started or is still running.
sudo[source]

Whether sudo should be used to gain superuser privileges.

If this option is True (not the default) and the current process doesn’t have superuser privileges the external command is run with sudo to ensure that the external command runs with superuser privileges.

The use of this option assumes that the sudo command is available.

Note

The sudo property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

sudo_command

The sudo command used to change privileges (a list of strings).

This option looks at the sudo, uid and user properties to decide whether command should be run using sudo or not. If it should then a prefix for command is constructed from sudo, uid, user and/or environment and returned. Some examples:

>>> from executor import ExternalCommand
>>> ExternalCommand('true', sudo=True).sudo_command
['sudo']
>>> ExternalCommand('true', uid=1000).sudo_command
['sudo', '-u', '#1000']
>>> ExternalCommand('true', user='peter').sudo_command
['sudo', '-u', 'peter']
>>> ExternalCommand('true', user='peter', environment=dict(gotcha='this is tricky')).sudo_command
['sudo', '-u', 'peter', 'gotcha=this is tricky']
tty[source]

Whether the command will be attached to the controlling terminal (a boolean).

By default tty is True when:

If any of these conditions don’t hold tty defaults to False. When tty is False the standard input stream of the external command will be connected to os.devnull.

Note

The tty property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

uid[source]

The user ID of the system user that’s used to run the command.

If this option is set to an integer number (it defaults to None) the external command is prefixed with sudo -u #UID to run the command as a different user than the current user.

The use of this option assumes that the sudo command is available.

Note

The uid property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

user[source]

The name of the system user that’s used to run the command.

If this option is set to a string (it defaults to None) the external command is prefixed with sudo -u USER to run the command as a different user than the current user.

The use of this option assumes that the sudo command is available.

Note

The user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

virtual_environment[source]

The Python virtual environment to activate before running the command.

If this option is set to the directory of a Python virtual environment (a string) then the external command will be prefixed by a source shell command that evaluates the bin/activate script in the Python virtual environment before executing the user defined external command.

Note

The virtual_environment property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

was_started[source]

Whether the external command has already been started.

True once start() has been called to start executing the external command, False when start() hasn’t been called yet.

Note

The was_started property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

async_fdel()[source]

Delete the value of asynchronous.

async_fget()[source]

Get the value of asynchronous.

async_fset(value)[source]

Set the value of asynchronous.

check_retry_allowed()[source]

Check if retrying is allowed by invoking the retry_event callback.

Returns:True if retry_allowed is True and the retry_event callback didn’t return False, otherwise False.
format_error_message(message, *args, **kw)[source]

Add the command’s captured standard output and/or error to an error message.

Refer to compact() for details on argument handling. The get_decoded_output() method is used to try to decode the output without raising exceptions.

get_decoded_output(name)[source]

Try to decode the output captured on standard output or error.

Parameters:name – One of the strings ‘stdout’ or ‘stderr’.
Returns:A Unicode string, byte string or None.
prefix_shell_command(preamble, command)[source]

Prefix a shell command to a command line.

Parameters:
  • preamble – Any shell command (a string).
  • command – The command line (a string, tuple or list).
Returns:

The command line to run the combined commands through a shell (a list of strings).

This function uses reduce_shell_command() to convert command into a string and then prefixes the preamble to the command, delimited by &&.

reduce_shell_command(command)[source]

Reduce a command line to a shell command.

Parameters:command – The command line (a string, tuple or list).
Returns:The shell command (a string).

If the given command is a DEFAULT_SHELL invocation that uses the -c option it is reduced to the argument of the -c option. All other command lines are simply quoted and returned.

This method is used in various places where a command needs to be transformed into a shell command so that a command like cd or source can be prefixed to the command line.

start()[source]

Start execution of the external command (including retry support).

Raises:

This method instantiates a subprocess.Popen object based on the defaults defined by ExternalCommand and the overrides configured by the caller. What happens then depends on asynchronous:

  • If asynchronous is set start() starts the external command but doesn’t wait for it to end (use wait() for that).
  • If asynchronous isn’t set the communicate() method on the subprocess object is called to synchronously execute the external command.
start_once(check=None, **kw)[source]

Start execution of the external command (excluding retry support).

Parameters:

The code in this internal method used to be the second half of start() but was extracted into a separate method so that it can be called more than once, which made it possible to add retry support.

wait(check=None, **kw)[source]

Wait for the external command to finish.

Parameters:
  • check – Override the value of check for the duration of this call to wait(). Defaults to None which means check is not overridden.
  • kw – Any keyword arguments are passed on to wait_for_process().
Raises:

ExternalCommandFailed when check is True, asynchronous is True and the external command exits with a nonzero status code.

The wait() function is only useful when asynchronous is True, it performs the following steps:

  1. If was_started is False the start() method is called.
  2. If is_running is True the wait_for_process() method is called to wait for the child process to end.
  3. If subprocess isn’t None the cleanup() method is called to wait for the external command to end, load its output into memory and release the resources associated with the subprocess object.
  4. Finally check_errors() is called (in case the caller didn’t disable check).
terminate_helper()[source]

Gracefully terminate the process.

Raises:Any exceptions raised by the subprocess module.

This method sets check to False, the idea being that if you consciously terminate a command you don’t need to be bothered with an exception telling you that you succeeded :-).

kill_helper()[source]

Forcefully kill the process.

Raises:Any exceptions raised by the subprocess module.

This method sets check to False, the idea being that if you consciously kill a command you don’t need to be bothered with an exception telling you that you succeeded :-).

load_output()[source]

Load output captured from the standard output/error streams.

Reads the contents of the temporary file(s) created by start() (when asynchronous and capture are both set) into memory so that the output doesn’t get lost when the temporary file is cleaned up by cleanup().

cleanup()[source]

Clean up after the external command has ended.

This internal method is called by methods like start() and wait() to clean up the following temporary resources:

  • The temporary file(s) used to to buffer the external command’s input, stdout and stderr (only when asynchronous is True).
  • File handles to the previously mentioned temporary files and os.devnull (used to implement the silent option).
  • The reference to the subprocess.Popen object stored in subprocess. By destroying this reference as soon as possible we enable the object to be garbage collected and its related resources to be released.
reset()[source]

Reset internal state created by start().

check_errors(check=None)[source]

Raise an exception if the external command failed.

This raises error_type when check is set and the external command failed.

Parameters:check – Override the value of check for the duration of this call. Defaults to None which means check is not overridden.
Raises:error_type when check is set and error_type is not None.

This internal method is used by start() and wait() to make sure that failing external commands don’t go unnoticed.

invoke_event_callback(name)[source]

Invoke one of the event callbacks.

Parameters:name – The name of the callback (a string).
Returns:The return value of the callback.
__enter__()[source]

Start the external command if it hasn’t already been started.

Returns:The ExternalCommand object.

When you use an ExternalCommand as a context manager in the with statement, the command is automatically started when entering the context and terminated when leaving the context.

If the proces hasn’t already been started yet asynchronous is automatically set to True (if it’s not already True), otherwise the command will have finished execution by the time the body of the with statement is executed (which isn’t really all that useful :-).

__exit__(exc_type=None, exc_value=None, traceback=None)[source]

Automatically terminate and clean up after the external command.

Terminates the external command if it is still running (using terminate()), cleans up (using cleanup()) and checks for errors (using check_errors(), only if an exception is not already being handled).

__iter__()[source]

Iterate over the lines of text in the captured output.

Returns:An iterator of Unicode strings (unicode() objects in Python 2 or str objects in Python 3).

If capture is True this will iterate over the lines in stdout, alternatively if capture_stderr is True this will iterate over the lines in stderr instead. In both cases the output will be decoded using encoding.

If buffered is True the captured output will be split into a list of strings which is then iterated. If it is False then the following idiom is used to create an iterator instead:

iter(handle.readline, b'')

To understand why this is useful, consider the following:

  • Iteration over file-like objects in Python uses a hidden read-ahead buffer for efficiency. Refer to the file.next() documentation for details.
  • Pipes are file-like objects so if you iterate over them but the command doesn’t emit a lot of output, the iteration may not produce any lines until the hidden read-ahead buffer is full. This makes realtime processing of command output harder than it should be.
__str__()[source]

Render a human friendly string representation of the external command.

This special method calls quote() on command_line which is used by executor to enable lazy formatting of log messages containing the quoted command line.

async

An alias for the asynchronous property.

In release 21.0 the async property was renamed to asynchronous because Python 3.7 introduces the async keyword, invalidating its use as an identifier. This alias ensures backwards compatibility with callers that are still using the old async name. If you’re wondering which of the two properties you should use:

  • If Python >= 3.7 compatibility is important to you then your only choice is asynchronous.
  • If you’re working in a Python < 3.7 code base you can pick either one of the properties, it really doesn’t matter.
  • Of course most Python 2 code bases will eventually need to be upgraded to Python 3 and the future proof choice is asynchronous.
class executor.CachedStream(command, kind)[source]

Manages a temporary file with input for / output from an external command.

__init__(command, kind)[source]

Initialize a CachedStream object.

Parameters:
  • command – The ExternalCommand object that’s using the CachedStream object.
  • kind – A simple (alphanumeric) string with the name of the stream.
prepare_temporary_file()[source]

Prepare the stream’s temporary file.

prepare_input()[source]

Initialize the input stream.

Returns:A value that can be passed to the constructor of subprocess.Popen as the stdin argument.
prepare_output(file, capture)[source]

Initialize an (asynchronous) output stream.

Parameters:
  • file – A file handle or None.
  • capture – True if capturing is enabled, False otherwise.
Returns:

A value that can be passed to the constructor of subprocess.Popen as the stdout and/or stderr argument.
redirect(obj)[source]

Capture the stream in a file provided by the caller.

Parameters:obj – A file-like object that has an associated file descriptor.
load()[source]

Load the stream’s contents from the temporary file.

Returns:The output of the stream (a string) or None when the stream was never initialized.
finalize(output=None)[source]

Load or override the stream’s contents and cleanup the temporary file.

Parameters:output – Override the stream’s contents (defaults to None which means the contents are loaded from the temporary file instead).
cleanup()[source]

Cleanup temporary resources.

reset()[source]

Reset internal state.

executor.quote(*args)[source]

Quote a string or a sequence of strings to be used as command line argument(s).

This function is a simple wrapper around pipes.quote() which adds support for quoting sequences of strings (lists and tuples). For example the following calls are all equivalent:

>>> from executor import quote
>>> quote('echo', 'argument with spaces')
"echo 'argument with spaces'"
>>> quote(['echo', 'argument with spaces'])
"echo 'argument with spaces'"
>>> quote(('echo', 'argument with spaces'))
"echo 'argument with spaces'"
Parameters:args – One or more strings, tuples and/or lists of strings to be quoted.
Returns:A string containing quoted command line arguments.
executor.which(program, mode=1, path=None)[source]

Find the pathname(s) of a program on the executable search path ($PATH).

Parameters:program – The name of the program (a string).
Returns:A list of pathnames (strings) with found programs.

Some examples:

>>> from executor import which
>>> which('python')
['/home/peter/.virtualenvs/executor/bin/python', '/usr/bin/python']
>>> which('vim')
['/usr/bin/vim']
>>> which('non-existing-program')
[]
executor.get_search_path(path=None)[source]

Get the executable search path ($PATH).

Parameters:path – Override the value of $PATH (a string or None).
Returns:A list of strings with pathnames of directories.

The executable search path is constructed as follows:

  1. The search path is taken from the environment variable $PATH.
  2. If $PATH isn’t defined the value of os.defpath is used.
  3. The search path is split on os.pathsep to get a list.
  4. On Windows the current directory is prepended to the list.
  5. Duplicate directories are removed from the list.
executor.get_path_extensions(extensions=None)[source]

Get the executable search path extensions ($PATHEXT).

Returns:A list of strings with unique path extensions (on Windows) or a list containing an empty string (on other platforms).
executor.is_executable(filename, mode=1)[source]

Check whether the given file is executable.

Parameters:filename – A relative or absolute pathname (a string).
Returns:True if the file is executable, False otherwise.
executor.validate_ionice_class(value)[source]

Ensure that the given value is a valid I/O scheduling class for ionice.

Parameters:value – The value to validate (a string).
Returns:The validated value (one of the strings ‘1’, ‘2’, ‘3’, ‘idle’, ‘best-effort’, or ‘realtime’).
Raises:ValueError when the given value isn’t one of the strings mentioned above.

The strings ‘idle’, ‘best-effort’ and ‘realtime’ are preferred for readability but not supported in minimalistic environments like busybox which only support the values ‘1’, ‘2’ and ‘3’ (refer to #16). It’s up to the caller to choose the correct value, no translation is done.

exception executor.ExternalCommandFailed(command, **options)[source]

Raised when an external command exits with a nonzero status code.

This exception is raised by execute(), start() and wait() when an external command exits with a nonzero status code.

Here’s an overview of the ExternalCommandFailed class:

Superclasses: PropertyManager and Exception
Special methods: __init__()
Properties: command, error_message, pool and returncode

When you initialize a ExternalCommandFailed object you are required to provide a value for the command property. You can set the values of the command, error_message and pool properties by passing keyword arguments to the class initializer.

__init__(command, **options)[source]

Initialize an ExternalCommandFailed object.

Parameters:
command[source]

The ExternalCommand object that triggered the exception.

pool[source]

The CommandPool object that triggered the exception.

This property will be None when the exception wasn’t raised from a command pool.

returncode

Shortcut for the external command’s returncode.

error_message[source]

An error message explaining what went wrong (a string).

Defaults to error_message but can be overridden using the keyword argument of the same name to __init__().

exception executor.CommandNotFound(command, **options)[source]

Raised when an external command is not available on the system.

This exception is raised by execute(), start() and wait() when an external command can’t be started because the command isn’t available.

It inherits from ExternalCommandFailed to enable uniform error handling but it also inherits from OSError for backwards compatibility (see errno and strerror).

Here’s an overview of the CommandNotFound class:

Superclasses: ExternalCommandFailed and OSError
Properties: errno and strerror
errno

The numeric error code ENOENT from errno (an integer).

strerror

The text corresponding to errno (a string).

The executor.chroot module

Simple command execution in chroot environments.

The executor.chroot module defines the ChangeRootCommand class which makes it easy to run commands inside chroots. If this doesn’t mean anything to you, here’s a primer:

  • The word ‘chroot’ is an abbreviation of the phrase ‘change root’ which is Unix functionality that changes the ‘root’ directory (/) of a running process, so ‘change root’ describes an action.
  • Even though the phrase ‘change root’ describes an action the word ‘chroot’ is also used to refer to the directory which serves as the new root directory (English is flexible like that :-). This is why it makes sense to say that “you’re entering the chroot”.

Warning

This is low level functionality. This module performs absolutely no chroot initialization, for example /etc/resolv.conf may be incorrect and there won’t be any bind mounts available in the chroot (unless you’ve prepared them yourself).

If you need your chroot to be initialized for you then consider using the executor.schroot module instead. It takes a bit of time to set up schroot but it provides a more high level experience than chroot.

executor.chroot.CHROOT_PROGRAM_NAME = 'chroot'

The name of the chroot program (a string).

class executor.chroot.ChangeRootCommand(*args, **options)[source]

ChangeRootCommand objects use the chroot program to execute commands inside chroots.

Here’s an overview of the ChangeRootCommand class:

Superclass: ExternalCommand
Special methods: __init__()
Properties: chroot, chroot_command, chroot_directory, chroot_group, chroot_user, command_line, directory and have_superuser_privileges

When you initialize a ChangeRootCommand object you are required to provide a value for the chroot property. You can set the values of the chroot, chroot_command, chroot_directory, chroot_group and chroot_user properties by passing keyword arguments to the class initializer.

__init__(*args, **options)[source]

Initialize a ChangeRootCommand object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the ExternalCommand class.
  • options – Any keyword arguments are passed on to the initializer of the ExternalCommand class.

If the keyword argument chroot isn’t given but positional arguments are provided, the first positional argument is used to set the chroot property.

The command is not started until you call start() or wait().

chroot[source]

The pathname of the root directory of the chroot (a string).

Note

The chroot property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

chroot_command[source]

The command used to run the chroot program.

This is a list of strings, by default the list contains just CHROOT_PROGRAM_NAME. The chroot, chroot_group and chroot_user properties also influence the chroot command line used.

Note

The chroot_command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_directory[source]

The working directory _inside the chroot (a string, defaults to None).

Note

The chroot_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_group[source]

The name or ID of the system group that runs the command (a string or number, defaults to ‘root’).

Note

The chroot_group property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_user[source]

The name or ID of the system user that runs the command (a string or number, defaults to ‘root’).

Note

The chroot_user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The complete chroot command including the command to run inside the chroot.

This is a list of strings with the chroot command line to enter the requested chroot and execute command.

directory

Set the working directory inside the chroot.

When you set this property you change chroot_directory, however reading back the property you’ll just get DEFAULT_WORKING_DIRECTORY. This is because the superclass ExternalCommand uses directory as the working directory for the chroot command, and directories inside chroots aren’t guaranteed to exist on the host system.

have_superuser_privileges

Whether the command inside the chroot will be running under superuser privileges.

This is True when chroot_user is the number 0 or the string ‘root’, False otherwise.

This overrides ExternalCommand.have_superuser_privileges to ensure that sudo isn’t used inside the chroot unless it’s really necessary. This is important because not all chroot environments have sudo installed and failing due to a lack of sudo when we’re already running as root is just stupid :-).

The executor.cli module

Usage: executor [OPTIONS] COMMAND …

Easy subprocess management on the command line based on the Python package with the same name. The “executor” program runs external commands with support for timeouts, dynamic startup delay (fudge factor) and exclusive locking.

You can think of “executor” as a combination of the “flock” and “timelimit” programs with some additional niceties (namely the dynamic startup delay and integrated system logging on UNIX platforms).

Supported options:

Option Description
-t, --timeout=LIMIT Set the time after which the given command will be aborted. By default LIMIT is counted in seconds. You can also use one of the suffixes “s” (seconds), “m” (minutes), “h” (hours) or “d” (days).
-f, --fudge-factor=LIMIT This option controls the dynamic startup delay (fudge factor) which is useful when you want a periodic task to run once per given interval but the exact time is not important. Refer to the --timeout option for acceptable values of LIMIT, this number specifies the maximum amount of time to sleep before running the command (the minimum is zero, otherwise you could just include the command “sleep N && …” in your command line :-).
-e, --exclusive Use an interprocess lock file to guarantee that executor will never run the external command concurrently. Refer to the --lock-timeout option to customize blocking / non-blocking behavior. To customize the name of the lock file you can use the --lock-file option.
-T, --lock-timeout=LIMIT By default executor tries to claim the lock and if it fails it will exit with a nonzero exit code. This option can be used to enable blocking behavior. Refer to the --timeout option for acceptable values of LIMIT.
-l, --lock-file=NAME Customize the name of the lock file. By default this is the base name of the external command, so if you’re running something generic like “bash” or “python” you might want to change this :-).
-v, --verbose Increase logging verbosity (can be repeated).
-q, --quiet Decrease logging verbosity (can be repeated).
-h, --help Show this message and exit.
executor.cli.LOCKS_DIRECTORY = '/var/lock'

The pathname of the preferred directory for lock files (a string).

Refer to get_lock_path() for more details.

executor.cli.INTERRUPT_FILE = 'executor-fudge-factor-interrupt'

The base name of the file used to interrupt the fudge factor (a string).

executor.cli.main()[source]

Command line interface for the executor program.

executor.cli.apply_fudge_factor(fudge_factor)[source]

Apply the requested scheduling fudge factor.

Parameters:fudge_factor – The maximum number of seconds to sleep (a number).

Previous implementations of the fudge factor interrupt used UNIX signals (specifically SIGUSR1) but the use of this signal turned out to be sensitive to awkward race conditions and it wasn’t very cross platform, so now the creation of a regular file is used to interrupt the fudge factor.

executor.cli.get_lock_path(lock_name)[source]

Get a pathname that can be used for an interprocess lock.

Parameters:lock_name – The base name for the lock file (a string).
Returns:An absolute pathname (a string).
executor.cli.run_command(arguments, timeout=None)[source]

Run the specified command (with an optional timeout).

Parameters:
  • arguments – The command line for the external command (a list of strings).
  • timeout – The optional command timeout (a number or None).
Raises:

CommandTimedOut if the command times out.
exception executor.cli.CommandTimedOut(command, timeout)[source]

Raised when a command exceeds the given timeout.

Here’s an overview of the CommandTimedOut class:

Superclass: ExternalCommandFailed
Special methods: __init__()
__init__(command, timeout)[source]

Initialize a CommandTimedOut object.

Parameters:
  • command – The command that timed out (an ExternalCommand object).
  • timeout – The timeout that was exceeded (a number).

The executor.concurrent module

Support for concurrent external command execution.

The executor.concurrent module defines the CommandPool class which makes it easy to prepare a large number of external commands, group them together in a pool, start executing a configurable number of external commands simultaneously and wait for all external commands to finish. For fine grained concurrency control please refer to the dependencies and group_by properties of the ExternalCommand class.

class executor.concurrent.CommandPool(concurrency=None, **options)[source]

Execute multiple external commands concurrently.

After constructing a CommandPool instance you add commands to it using add() and when you’re ready to run the commands you call run().

Here’s an overview of the CommandPool class:

Superclass: PropertyManager
Special methods: __init__()
Public methods: add(), collect(), get_spinner(), run(), spawn() and terminate()
Properties: concurrency, delay_checks, is_finished, logger, logs_directory, num_commands, num_failed, num_finished, num_running, results, running_groups, spinner and unexpected_failures

You can set the values of the concurrency, delay_checks, logger, logs_directory and spinner properties by passing keyword arguments to the class initializer.

__init__(concurrency=None, **options)[source]

Initialize a CommandPool object.

Parameters:
concurrency[source]

The number of external commands that the pool is allowed to run simultaneously.

This is a positive integer number. It defaults to the return value of multiprocessing.cpu_count() (which may not make much sense if your commands are I/O bound instead of CPU bound).

Setting concurrency to one is a supported use case intended to make it easier for users of the executor.concurrent module to reuse the code they’ve built on top of command pools (if only for debugging, but there are lots of use cases :-).

Note

The concurrency property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

delay_checks[source]

Whether to postpone raising an exception until all commands have run (a boolean).

If this option is True (not the default) and a command with check set to True fails the command pool’s execution is not aborted, instead all commands will be allowed to run. After all commands have finished a CommandPoolFailed exception will be raised that tells you which command(s) failed.

Note

The delay_checks property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

logger[source]

The logging.Logger object to use.

If you are using Python’s logging module and you find it confusing that command pool execution is logged under the executor.concurrent name space instead of the name space of the application or library using executor you can set this attribute to inject a custom (and more appropriate) logger.

Note

The logger property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

logs_directory[source]

The pathname of a directory where captured output is stored (a string).

If this property is set to the pathname of a directory (before any external commands have been started) the merged output of each external command is captured and stored in a log file in this directory. The directory will be created if it doesn’t exist yet.

Output will start appearing in the log files before the external commands are finished, this enables tail -f to inspect the progress of commands that are still running and emitting output.

Note

The logs_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

is_finished

True if all commands in the pool have finished (including retries), False otherwise.

num_commands

The number of commands in the pool (an integer).

num_finished

The number of commands in the pool that have already finished, including retries (an integer).

num_failed

The number of commands in the pool that failed (an integer).

num_running

The number of currently running commands in the pool (an integer).

running_groups

A set of running command groups.

The value of running_groups is a set with the group_by values of all currently running commands (None is never included in the set).

results

A mapping of identifiers to external command objects.

This is a dictionary with external command identifiers as keys (refer to add()) and ExternalCommand objects as values. The ExternalCommand objects provide access to the return codes and/or output of the finished commands.

spinner[source]

Control if and how an animated spinner is shown when the command pool is active.

The following values are supported:

  • The default value None means “auto detect”, which means the spinner is shown only when the process is connected to a terminal.
  • The value True unconditionally enables the spinner.
  • The value False unconditionally disables the spinner.
  • A Spinner object can be provided by the caller, giving them the chance to configure how the spinner behaves.

Note

The spinner property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

unexpected_failures

A list of ExternalCommand objects that failed unexpectedly.

The resulting list includes only commands where check and failed are both True.

add(command, identifier=None, log_file=None)[source]

Add an external command to the pool of commands.

Parameters:
  • command – The external command to add to the pool (an ExternalCommand object).
  • identifier – A unique identifier for the external command (any value). When this parameter is not provided the identifier is set to the number of commands in the pool plus one (i.e. the first command gets id 1).
  • log_file – Override the default log file name for the command (the identifier with .log appended) in case logs_directory is set.

When a command is added to a command pool the following options are changed automatically:

  • The asynchronous property is set to True. If you want the commands to execute with a concurrency of one then you should set concurrency to one.
  • The tty property is set to False when concurrency is higher than one because interaction with multiple concurrent subprocesses in a single terminal is prone to serious miscommunication (when multiple subprocesses present an interactive prompt at the same time and the user tries to answer one of the prompts it will be impossible to tell which of the subprocesses will receive the user’s reply).
get_spinner(timer)[source]

Get a Spinner to be used by run().

run()[source]

Keep spawning commands and collecting results until all commands have run.

Returns:The value of results.
Raises:Any exceptions raised by collect().

This method calls spawn() and collect() in a loop until all commands registered using add() have run and finished. If collect() raises an exception any running commands are terminated before the exception is propagated to the caller.

If you’re writing code where you want to own the main loop then consider calling spawn() and collect() directly instead of using run().

When concurrency is set to one, specific care is taken to make sure that the callbacks configured by start_event and finish_event are called in the expected (intuitive) order.

spawn()[source]

Spawn additional external commands up to the concurrency level.

Returns:The number of external commands that were spawned by this invocation of spawn() (an integer).

The commands to start are picked according to three criteria:

  1. The command’s was_started property is False.
  2. The command’s group_by value is not present in running_groups.
  3. The is_finished_with_retries properties of all of the command’s dependencies are True.
collect()[source]

Collect the exit codes and output of finished commands.

Returns:

The number of external commands that were collected by this invocation of collect() (an integer).
Raises:
If delay_checks is True:

After all external commands have started and finished, if any commands that have check set to True failed CommandPoolFailed is raised.

If delay_checks is False:

The exceptions ExternalCommandFailed, RemoteCommandFailed and RemoteConnectFailed can be raised if a command in the pool that has check set to True fails. The pool attribute of the exception will be set to the pool.

Warning

If an exception is raised, commands that are still running will not be terminated! If this concerns you then consider calling terminate() from a finally block (this is what run() does).

terminate()[source]

Terminate any commands that are currently running.

Returns:The number of commands that were terminated (an integer).

If terminate() successfully terminates commands, you then call collect() and the check property of a terminated command is True you will get an exception because terminated commands (by definition) report a nonzero returncode.

exception executor.concurrent.CommandPoolFailed(pool)[source]

Raised by collect() when not all commands succeeded.

This exception is only raised when delay_checks is True.

__init__(pool)[source]

Initialize a CommandPoolFailed object.

Parameters:pool – The CommandPool object that triggered the exception.
commands

A shortcut for unexpected_failures.

error_message

An error message that explains which commands failed unexpectedly (a string).

The executor.contexts module

Dependency injection for command execution contexts.

The contexts module defines the LocalContext, RemoteContext and SecureChangeRootContext classes. All of these classes support the same API for executing external commands, they are simple wrappers for ExternalCommand, RemoteCommand and SecureChangeRootCommand.

This allows you to script interaction with external commands in Python and perform that interaction on your local system, on remote systems over SSH or inside chroots using the exact same Python code. Dependency injection on steroids anyone? :-)

Here’s a simple example:

from executor.contexts import LocalContext, RemoteContext
from humanfriendly import format_timespan

def details_about_system(context):
    return "\n".join([
        "Information about %s:" % context,
        " - Host name: %s" % context.capture('hostname', '--fqdn'),
        " - Uptime: %s" % format_timespan(float(context.capture('cat', '/proc/uptime').split()[0])),
    ])

print(details_about_system(LocalContext()))

# Information about local system (peter-macbook):
#  - Host name: peter-macbook
#  - Uptime: 1 week, 3 days and 10 hours

print(details_about_system(RemoteContext('file-server')))

# Information about remote system (file-server):
#  - Host name: file-server
#  - Uptime: 18 weeks, 3 days and 4 hours

Whether this functionality looks exciting or horrible I’ll leave up to your judgment. I created it because I’m always building “tools that help me build tools” and this functionality enables me to very rapidly prototype system integration tools developed using Python:

During development:
I write code on my workstation which I prefer because of the “rich editing environment” but I run the code against a remote system over SSH (a backup server, database server, hypervisor, mail server, etc.).
In production:
I change one line of code to inject a LocalContext object instead of a RemoteContext object, I install the executor package and the code I wrote on the remote system and I’m done!
executor.contexts.MIRROR_TO_DISTRIB_MAPPING = {u'http://archive.ubuntu.com/ubuntu': u'ubuntu', u'http://deb.debian.org/debian': u'debian'}

Mapping of canonical package mirror URLs to “distributor ID” strings.

Each key in this dictionary is the canonical package mirror URL of a Debian based Linux distribution and each value is the corresponding distributor ID. The following canonical mirror URLs are currently supported:

Mirror URL Value
http://deb.debian.org/debian debian
http://archive.ubuntu.com/ubuntu/ ubuntu

For more details refer to the AbstractContext.apt_sources_info property.

executor.contexts.create_context(**options)[source]

Create an execution context.

Parameters:options – Any keyword arguments are passed on to the context’s initializer.
Returns:A LocalContext, SecureChangeRootContext or RemoteContext object.

This function provides an easy to use shortcut for constructing context objects:

class executor.contexts.AbstractContext(*args, **options)[source]

Abstract base class for shared logic of all context classes.

Here’s an overview of the AbstractContext class:

Superclass: PropertyManager
Special methods: __enter__(), __exit__() and __init__()
Public methods: atomic_write(), capture(), cleanup(), execute(), exists(), find_chroots(), find_program(), get_options(), glob(), is_directory(), is_executable(), is_file(), is_readable(), is_writable(), list_entries(), merge_options(), prepare(), prepare_command(), prepare_interactive_shell(), read_file(), start_interactive_shell(), test() and write_file()
Properties: apt_sources_info, command_type, cpu_count, distribution_codename, distributor_id, have_ionice, have_superuser_privileges, lsb_release_variables, options and parent

When you initialize a AbstractContext object you are required to provide a value for the command_type property. You can set the values of the command_type, options and parent properties by passing keyword arguments to the class initializer.

__init__(*args, **options)[source]

Initialize an AbstractContext object.

Parameters:
  • args – Any positional arguments are passed on to the initializer of the PropertyManager class (for future extensibility).
  • options –

    The keyword arguments are handled as follows:

    • Keyword arguments whose name matches a property of the context object are used to set that property (by passing them to the initializer of the PropertyManager class).
    • Any other keyword arguments are collected into the options dictionary.
apt_sources_info[source]

A tuple with two strings (the distributor ID and distribution codename).

The values of the distributor_id and distribution_codename properties are determined by one of the following three methods (in decreasing order of preference):

  1. If lsb_release_variables is available it’s used.
  2. If the lsb_release program is available it’s used.
  3. Finally /etc/apt/sources.list is parsed for hints.

The apt_sources_info property concerns the third step which works as follows:

  • The deb directives in /etc/apt/sources.list are parsed to determine the primary package mirror URL (it’s fine if this file doesn’t exist, no error will be reported).
  • The MIRROR_TO_DISTRIB_MAPPING dictionary is used to look up the distributor ID corresponding to the package mirror URL that was found in /etc/apt/sources.list.
  • If the mirror URL is successfully translated to a distributor ID, the third token in the deb directive is taken to be the distribution codename.

The apt_sources_info property was added in response to issue #17 where it was reported that official Debian Docker images don’t contain the /etc/lsb-release file nor the lsb_release program.

This is only used as a last resort because of its specificness to Debian based Linux distributions and because I have concerns about how robust this new functionality will turn out to be.

Note

The apt_sources_info property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

command_type[source]

The type of command objects created by this context (ExternalCommand or a subclass).

Note

The command_type property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named command_type (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

cpu_count

The number of CPUs in the system (an integer).

Note

This is an abstract property that must be implemented by subclasses.

distribution_codename[source]

The code name of the system’s distribution (a lowercased string like precise or trusty).

How this property is computed depends on the execution context:

  1. When the file /etc/lsb-release exists and defines the variable DISTRIB_CODENAME then this is the preferred source (for details see lsb_release_variables).
  2. When lsb_release is installed the output of the command lsb_release --short --codename is used.
  3. Finally apt_sources_info is used if possible.

The returned string is guaranteed to be lowercased, in order to enable reliable string comparison.

Note

The distribution_codename property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

distributor_id[source]

The distributor ID of the system (a lowercased string like debian or ubuntu).

How this property is computed depends on the execution context:

  1. When the file /etc/lsb-release exists and defines the variable DISTRIB_ID then this is the preferred source (for details see lsb_release_variables).
  2. When lsb_release is installed the output of the command lsb_release --short --id is used.
  3. Finally apt_sources_info is used if possible.

The returned string is guaranteed to be lowercased, in order to enable reliable string comparison.

Note

The distributor_id property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

have_ionice[source]

True when ionice is installed, False otherwise.

Note

The have_ionice property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

have_superuser_privileges

True if the context has superuser privileges, False otherwise.

lsb_release_variables[source]

The contents of /etc/lsb-release as a dictionary.

The values of distributor_id and distribution_codename are based on the information provided by lsb_release_variables. If /etc/lsb-release doesn’t exist or can’t be parsed a debug message is logged and an empty dictionary is returned. Here’s an example:

>>> from executor.contexts import LocalContext
>>> context = LocalContext()
>>> context.lsb_release_variables
{'DISTRIB_CODENAME': 'bionic',
 'DISTRIB_DESCRIPTION': 'Ubuntu 18.04.1 LTS',
 'DISTRIB_ID': 'Ubuntu',
 'DISTRIB_RELEASE': '18.04'}

The lsb_release_variables property was added in response to issue #10 where it was reported that the lsb_release program wasn’t available in vanilla Ubuntu 18.04 Docker images.

Note

The lsb_release_variables property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

options[source]

The options that are passed to commands created by the context (a dictionary).

Note

The options property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

parent[source]

The parent context (a context object or None).

The parent property (and the code in prepare_command() that uses the parent property) enables the use of “nested contexts”.

For example find_chroots() creates SecureChangeRootContext objects whose parent is set to the context that found the chroots. Because of this the SecureChangeRootContext objects can be used to create commands without knowing or caring whether the chroots reside on the local system or on a remote system accessed via SSH.

Warning

Support for parent contexts was introduced in executor version 15 and for now this feature is considered experimental and subject to change. While I’m definitely convinced of the usefulness of nested contexts I’m not happy with the current implementation at all. The most important reason for this is that it’s very surprising (and not in a good way) that a context with a parent will create commands with the parent’s command_type instead of the expected type.

Note

The parent property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

__enter__()[source]

Initialize a new “undo stack” (refer to cleanup()).

__exit__(exc_type=None, exc_value=None, traceback=None)[source]

Execute any commands on the “undo stack” (refer to cleanup()).

atomic_write(**kwds)[source]

Create or update the contents of a file atomically.

Parameters:filename – The pathname of the file to create/update (a string).
Returns:A context manager (see the with keyword) that returns a single string which is the pathname of the temporary file where the contents should be written to initially.

If an exception is raised from the with block and the temporary file exists, an attempt will be made to remove it but failure to do so will be silenced instead of propagated (to avoid obscuring the original exception).

The temporary file is created in the same directory as the real file, but a dot is prefixed to the name (making it a hidden file) and the suffix ‘.tmp-‘ followed by a random integer number is used.

capture(*command, **options)[source]

Execute an external command in the current context and capture its output.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The value of ExternalCommand.output.
cleanup(*args, **kw)[source]

Register an action to be performed before the context ends.

Parameters:
  • args – The external command to execute or callable to invoke.
  • kw – Options to the command or keyword arguments to the callable.
Raises:

ValueError when cleanup() is called outside a with statement.

This method registers the intent to perform an action just before the context ends. To actually perform the action(s) you need to use (the subclass of) the AbstractContext object as a context manager using the with statement.

The last action that is registered is the first one to be performed. This gives the equivalent functionality of a deeply nested try / finally structure without actually having to write such ugly code :-).

The handling of arguments in cleanup() depends on the type of the first positional argument:

  • If the first positional argument is a string, the positional arguments and keyword arguments are passed on to the initializer of the command_type class to execute an external command just before the context ends.
  • If the first positional argument is a callable, it is called with any remaining positional arguments and keyword arguments before the context ends.

Warning

If a cleanup command fails and raises an exception no further cleanup commands are executed. If you don’t care if a specific cleanup command reports an error, set its check property to False.

execute(*command, **options)[source]

Execute an external command in the current context.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The command_type object.

Note

After constructing a command_type object this method calls start() on the command before returning it to the caller, so by the time the caller gets the command object a synchronous command will have already ended. Asynchronous commands don’t have this limitation of course.

exists(pathname)[source]

Check whether the given pathname exists.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname exists, False otherwise.

This is a shortcut for the test -e ... command.

find_chroots(namespace='chroot')[source]

Find the chroots available in the current context.

Parameters:namespace – The chroot namespace to look for (a string, defaults to DEFAULT_NAMESPACE). Refer to the schroot documentation for more information about chroot namespaces.
Returns:A generator of SecureChangeRootContext objects whose parent is set to the context where the chroots were found.
Raises:ExternalCommandFailed (or a subclass) when the schroot program isn’t installed or the schroot --list command fails.
find_program(program_name, *args)[source]

Find the absolute pathname(s) of one or more programs.

Parameters:program_name – Each of the positional arguments is expected to be a string containing the name of a program to search for in the $PATH. At least one is required.
Returns:A list of strings with absolute pathnames.

This method is a simple wrapper around which.

get_options()[source]

Get the options that are passed to commands created by the context.

Returns:A dictionary of command options.

By default this method simply returns the options dictionary, however the purpose of get_options() is to enable subclasses to customize the options passed to commands on the fly.

glob(pattern)[source]

Find matches for a given filename pattern.

Parameters:pattern – A filename pattern (a string).
Returns:A list of strings with matches.

Some implementation notes:

  • This method emulates filename globbing as supported by system shells like Bash and ZSH. It works by forking a Python interpreter and using that to call the glob.glob() function. This approach is of course rather heavyweight.
  • Initially this method used Bash for filename matching (similar to this StackOverflow answer) but I found it impossible to make this work well for patterns containing whitespace.
  • I took the whitespace issue as a sign that I was heading down the wrong path (trying to add robustness to a fragile solution) and so the new implementation was born (which prioritizes robustness over performance).
is_directory(pathname)[source]

Check whether the given pathname points to an existing directory.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname points to an existing directory, False otherwise.

This is a shortcut for the test -d ... command.

is_executable(pathname)[source]

Check whether the given pathname points to an executable file.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname points to an executable file, False otherwise.

This is a shortcut for the test -x ... command.

is_file(pathname)[source]

Check whether the given pathname points to an existing file.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname points to an existing file, False otherwise.

This is a shortcut for the test -f ... command.

is_readable(pathname)[source]

Check whether the given pathname exists and is readable.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname exists and is readable, False otherwise.

This is a shortcut for the test -r ... command.

is_writable(pathname)[source]

Check whether the given pathname exists and is writable.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname exists and is writable, False otherwise.

This is a shortcut for the test -w ... command.

list_entries(directory)[source]

List the entries in a directory.

Parameters:directory – The pathname of the directory (a string).
Returns:A list of strings with the names of the directory entries.

This method uses find -mindepth 1 -maxdepth 1 -print0 to list directory entries instead of going for the more obvious choice ls -A1 because find enables more reliable parsing of command output (with regards to whitespace).

merge_options(overrides)[source]

Merge default options and overrides into a single dictionary.

Parameters:overrides – A dictionary with any keyword arguments given to execute() or start_interactive_shell().
Returns:The dictionary with overrides, but any keyword arguments given to the initializer of AbstractContext that are not set in the overrides are set to the value of the initializer argument.

The ionice option is automatically unset when have_ionice is False, regardless of whether the option was set from defaults or overrides.

prepare(*command, **options)[source]

Prepare to execute an external command in the current context.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The command_type object.

Note

After constructing a command_type object this method doesn’t call start() which means you control if and when the command is started. This can be useful to prepare a large batch of commands and execute them concurrently using a CommandPool.

prepare_command(command, options)[source]

Create a command_type object based on options.

Parameters:
  • command – A tuple of strings (the positional arguments to the initializer of the command_type class).
  • options – A dictionary (the keyword arguments to the initializer of the command_type class).
Returns:

A command_type object that hasn’t been started yet.
prepare_interactive_shell(options)[source]

Create a command_type object that starts an interactive shell.

Parameters:options – A dictionary (the keyword arguments to the initializer of the command_type class).
Returns:A command_type object that hasn’t been started yet.
read_file(filename, **options)[source]

Read the contents of a file.

Parameters:
  • filename – The pathname of the file to read (a string).
  • options – Optional keyword arguments to execute().
Returns:

The contents of the file (a byte string).

This method uses cat to read the contents of files so that options like sudo are respected (regardless of whether we’re dealing with a LocalContext or RemoteContext).

start_interactive_shell(**options)[source]

Start an interactive shell in the current context.

Parameters:options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:The command_type object.

Note

After constructing a command_type object this method calls start() on the command before returning it to the caller, so by the time the caller gets the command object a synchronous command will have already ended. Asynchronous commands don’t have this limitation of course.

test(*command, **options)[source]

Execute an external command in the current context and get its status.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The value of ExternalCommand.succeeded.

This method automatically sets check to False and silent to True.

write_file(filename, contents, **options)[source]

Change the contents of a file.

Parameters:
  • filename – The pathname of the file to write (a string).
  • contents – The contents to write to the file (a byte string).
  • options – Optional keyword arguments to execute().

This method uses a combination of cat and output redirection to change the contents of files so that options like sudo are respected (regardless of whether we’re dealing with a LocalContext or RemoteContext). Due to the use of cat this method will create files that don’t exist yet, assuming the directory containing the file already exists and the context provides permission to write to the directory.

class executor.contexts.LocalContext(*args, **options)[source]

Context for executing commands on the local system.

Here’s an overview of the LocalContext class:

Superclass: AbstractContext
Special methods: __str__()
Public methods: glob()
Properties: command_type and cpu_count
command_type

The type of command objects created by this context (ExternalCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed using multiprocessing.cpu_count().

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

glob(pattern)[source]

Find matches for a given filename pattern.

Parameters:pattern – A filename pattern (a string).
Returns:A list of strings with matches.

This method overrides AbstractContext.glob() to call glob.glob() directly instead of forking a new Python interpreter.

This optimization is skipped when options contains sudo, uid or user to avoid reporting wrong matches due to insufficient filesystem permissions.

__str__()[source]

Render a human friendly string representation of the context.

class executor.contexts.ChangeRootContext(*args, **options)[source]

Context for executing commands in change roots using chroot.

Here’s an overview of the ChangeRootContext class:

Superclass: AbstractContext
Special methods: __init__() and __str__()
Public methods: get_options()
Properties: chroot, command_type and cpu_count

When you initialize a ChangeRootContext object you are required to provide a value for the chroot property. You can set the value of the chroot property by passing a keyword argument to the class initializer.

__init__(*args, **options)[source]

Initialize a ChangeRootContext object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the AbstractContext class (for future extensibility).
  • options – Any keyword arguments are passed on to the initializer of the AbstractContext class.

If the keyword argument chroot isn’t given but positional arguments are provided, the first positional argument is used to set the chroot property.

chroot[source]

The pathname of the root directory of the chroot (a string).

Note

The chroot property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

command_type

The type of command objects created by this context (ChangeRootCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed using multiprocessing.cpu_count().

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

get_options()[source]

The options including chroot.

__str__()[source]

Render a human friendly string representation of the context.

class executor.contexts.SecureChangeRootContext(*args, **options)[source]

Context for executing commands in change roots using schroot.

Here’s an overview of the SecureChangeRootContext class:

Superclass: AbstractContext
Special methods: __init__() and __str__()
Public methods: get_options()
Properties: chroot_name, command_type and cpu_count

When you initialize a SecureChangeRootContext object you are required to provide a value for the chroot_name property. You can set the value of the chroot_name property by passing a keyword argument to the class initializer.

__init__(*args, **options)[source]

Initialize a SecureChangeRootContext object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the AbstractContext class (for future extensibility).
  • options – Any keyword arguments are passed on to the initializer of the AbstractContext class.

If the keyword argument chroot_name isn’t given but positional arguments are provided, the first positional argument is used to set the chroot_name property.

chroot_name[source]

The name of a chroot managed by schroot (a string).

Note

The chroot_name property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot_name (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

command_type

The type of command objects created by this context (SecureChangeRootCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed using multiprocessing.cpu_count().

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

get_options()[source]

The options including chroot_name.

__str__()[source]

Render a human friendly string representation of the context.

class executor.contexts.RemoteContext(*args, **options)[source]

Context for executing commands on a remote system over SSH.

Here’s an overview of the RemoteContext class:

Superclasses: RemoteAccount and AbstractContext
Special methods: __str__()
Public methods: get_options()
Properties: command_type and cpu_count
command_type

The type of command objects created by this context (RemoteCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed by executing the remote command nproc. If that command fails cpu_count falls back to the command grep -ci '^processor\s*:' /proc/cpuinfo.

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

get_options()[source]

The options including the SSH alias and remote user.

__str__()[source]

Render a human friendly string representation of the context.

executor.contexts.normalize_mirror_url(value)[source]

Normalize a package mirror URL to enable string equality comparison.

The executor.process module

Portable process control functionality for the executor package.

The executor.process module defines the ControllableProcess abstract base class which enables process control features like waiting for a process to end, gracefully terminating it and forcefully killing it. The process control functionality in ControllableProcess is separated from the command execution functionality in ExternalCommand to make it possible to re-use the process control functionality in other Python packages, see for example the proc.core.Process class.

executor.process.DEFAULT_TIMEOUT = 10

The default timeout used to wait for process termination (number of seconds).

class executor.process.ControllableProcess(**kw)[source]

Abstract, portable process control functionality.

By defining a subclass of ControllableProcess and implementing the pid, command_line and is_running properties and the terminate_helper() and kill_helper() methods you get the wait_for_process(), terminate() and kill() methods for free. This decoupling has enabled me to share a lot of code between two Python projects of mine with similar goals but very different requirements:

  1. The executor package builds on top of the subprocess module in the Python standard library and strives to be as cross platform as possible. This means things like UNIX signals are not an option (although signals exist on Windows they are hardly usable). The package mostly deals with subprocess.Popen objects internally (to hide platform specific details as much as possible).
  2. The proc package exposes process information available in the Linux process information pseudo-file system available at /proc. The package mostly deals with process IDs internally. Because this is completely specialized to a UNIX environment the use of things like UNIX signals is not a problem at all.

Here’s an overview of the ControllableProcess class:

Superclass: PropertyManager
Special methods: __str__()
Public methods: kill(), kill_helper(), terminate(), terminate_helper() and wait_for_process()
Properties: command_line, is_running, logger and pid

You can set the values of the command_line, logger and pid properties by passing keyword arguments to the class initializer.

command_line[source]

A list of strings with the command line used to start the process.

This property may be set or implemented by subclasses to enable __str__() to render a human friendly representation of a ControllableProcess object.

Note

The command_line property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

is_running

True if the process is running, False otherwise.

This property must be implemented by subclasses to enable wait_for_process(), terminate() and kill() to work properly.

logger[source]

The logging.Logger object to use (defaults to the executor.process logger).

If you are using Python’s logging module and you find it confusing that command manipulation is logged under the executor.process name space instead of the name space of the application or library using executor you can set this attribute to inject a custom (and more appropriate) logger.

Note

The logger property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

pid[source]

The process ID (a number) or None.

This property must be set or implemented by subclasses:

Note

The pid property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

wait_for_process(timeout=0, use_spinner=None)[source]

Wait until the process ends or the timeout expires.

Parameters:
  • timeout – The number of seconds to wait for the process to terminate after we’ve asked it nicely (defaults to zero which means we wait indefinitely).
  • use_spinner –

    Whether or not to display an interactive spinner on the terminal (using Spinner) to explain to the user what they are waiting for:

    • True enables the spinner,
    • False disables the spinner,
    • None (the default) means the spinner is enabled when the program is connected to an interactive terminal, otherwise it’s disabled.
Returns:

A Timer object telling you how long it took to wait for the process.
terminate(wait=True, timeout=10, use_spinner=None)[source]

Gracefully terminate the process.

Parameters:
  • wait – Whether to wait for the process to end (a boolean, defaults to True).
  • timeout – The number of seconds to wait for the process to terminate after we’ve signaled it (defaults to DEFAULT_TIMEOUT). Zero means to wait indefinitely.
  • use_spinner – See the wait_for_process() documentation.
Returns:

True if the process was terminated, False otherwise.
Raises:

Any exceptions raised by terminate_helper() implementations of subclasses or kill().

This method works as follows:

  1. Signal the process to gracefully terminate itself. Processes can choose to intercept termination signals to allow for graceful termination (many UNIX daemons work like this) however the default action is to simply exit immediately.
  2. If wait is True and we’ve signaled the process, we wait for it to terminate gracefully or timeout seconds have passed (whichever comes first).
  3. If wait is True and the process is still running after timeout seconds have passed, it will be forcefully terminated using kill() (the value of timeout that was given to terminate() will be passed on to kill()).

This method does nothing when is_running is False.

terminate_helper()[source]

Request the process to gracefully terminate itself (needs to be implemented by subclasses).

kill(wait=True, timeout=10, use_spinner=None)[source]

Forcefully kill the process.

Parameters:
  • wait – Whether to wait for the process to end (a boolean, defaults to True).
  • timeout – The number of seconds to wait for the process to terminate after we’ve signaled it (defaults to DEFAULT_TIMEOUT). Zero means to wait indefinitely.
  • use_spinner – See the wait_for_process() documentation.
Returns:

True if the process was killed, False otherwise.
Raises:

This method does nothing when is_running is False.

kill_helper()[source]

Forcefully kill the process (needs to be implemented by subclasses).

__str__()[source]

Render a human friendly representation of a ControllableProcess object.

Returns:A string describing the process. Includes the process ID and the command line (when available).
exception executor.process.ProcessTerminationFailed(*args, **kw)[source]

Raised when process termination fails.

Here’s an overview of the ProcessTerminationFailed class:

Superclasses: PropertyManager and Exception
Special methods: __init__()
Properties: message and process

When you initialize a ProcessTerminationFailed object you are required to provide values for the message and process properties. You can set the values of the message and process properties by passing keyword arguments to the class initializer.

__init__(*args, **kw)[source]

Initialize a ProcessTerminationFailed object.

This method’s signature is the same as the initializer of the PropertyManager class.

process[source]

The ControllableProcess object that triggered the exception.

message[source]

An error message that explains how the process termination failed.

The executor.schroot module

Secure command execution in chroot environments.

The executor.schroot module defines the SecureChangeRootCommand class which makes it easy to run commands inside chroots that are managed using the schroot program.

executor.schroot.SCHROOT_PROGRAM_NAME = 'schroot'

The name of the schroot program (a string).

executor.schroot.DEFAULT_NAMESPACE = 'chroot'

The default chroot namespace (a string).

Refer to the schroot documentation for more information about chroot namespaces.

class executor.schroot.SecureChangeRootCommand(*args, **options)[source]

SecureChangeRootCommand objects use the schroot program to execute commands inside chroots.

Here’s an overview of the SecureChangeRootCommand class:

Superclass: ExternalCommand
Special methods: __init__()
Properties: chroot_directory, chroot_name, chroot_user, command_line, directory and schroot_command

When you initialize a SecureChangeRootCommand object you are required to provide a value for the chroot_name property. You can set the values of the chroot_directory, chroot_name, chroot_user and schroot_command properties by passing keyword arguments to the class initializer.

__init__(*args, **options)[source]

Initialize a SecureChangeRootCommand object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the ExternalCommand class.
  • options – Any keyword arguments are passed on to the initializer of the ExternalCommand class.

If the keyword argument chroot_name isn’t given but positional arguments are provided, the first positional argument is used to set the chroot_name property.

The command is not started until you call start() or wait().

chroot_directory[source]

The working directory _inside the chroot (a string or None, defaults to /).

When chroot_directory is None the schroot program gets to pick the working directory inside the chroot (refer to the schroot documentation for the complete details).

For non-interactive usage (which I anticipate to be the default usage of SecureChangeRootCommand) the schroot program simply assumes that the working directory outside of the chroot also exists inside the chroot, then fails with an error message when this is not the case.

Because this isn’t a very robust default, chroot_directory defaults to / instead.

Note

The chroot_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_name[source]

The name of a chroot managed by schroot (a string).

This is expected to match one of the names configured in the directory /etc/schroot/chroot.d.

Note

The chroot_name property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot_name (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

chroot_user[source]

The name of the user inside the chroot to run the command as (a string or None).

This defaults to None which means to run as the current user.

Note

The chroot_user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The complete schroot command including the command to run inside the chroot.

This is a list of strings with the schroot command line to enter the requested chroot and execute command.

directory

Set the working directory inside the chroot.

When you set this property you change chroot_directory, however reading back the property you’ll just get DEFAULT_WORKING_DIRECTORY. This is because the superclass ExternalCommand uses directory as the working directory for the schroot command, and directories inside chroots aren’t guaranteed to exist on the host system.

schroot_command[source]

The command used to run the schroot program.

This is a list of strings, by default the list contains just SCHROOT_PROGRAM_NAME. The chroot_directory, chroot_name and chroot_user properties also influence the schroot command line used.

Note

The schroot_command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

The executor.ssh.client module

Remote command execution using SSH.

The executor.ssh.client module defines the RemoteCommand class and the foreach() function which make it easy to run a remote command in parallel on multiple remote hosts using SSH.

The foreach() function also serves as a simple example of how to use CommandPool and RemoteCommand objects (it’s just 16 lines of code if you squint in the right way and that includes logging :-).

The SecureTunnel class makes it easy to use “tunnel only” SSH connections by waiting until the tunnel becomes connected and automatically selecting a free ephemeral port number if a local port number isn’t provided by the caller.

executor.ssh.client.DEFAULT_CONCURRENCY = 10

The default concurrency value to use for CommandPool objects created by foreach().

executor.ssh.client.DEFAULT_CONNECT_TIMEOUT = 10

The default connect_timeout value to use for RemoteCommand objects.

executor.ssh.client.SSH_PROGRAM_NAME = 'ssh'

The name of the SSH client executable (a string).

executor.ssh.client.SSH_ERROR_STATUS = 255

The exit status used by the ssh program if an error occurred (an integer).

Used by RemoteCommand.error_message and RemoteCommand.error_type to distinguish when the ssh program itself fails and when a remote command fails.

executor.ssh.client.foreach(hosts, *command, **options)[source]

Execute a command simultaneously on a group of remote hosts using SSH.

Parameters:
  • hosts – An iterable of strings with SSH host aliases.
  • command – Any positional arguments are converted to a list and used to set the command property of the RemoteCommand objects constructed by foreach().
  • concurrency – Set the command pool concurrency property (defaults to DEFAULT_CONCURRENCY).
  • delay_checks – Set the command pool delay_checks property (defaults to True).
  • logs_directory – Set the command pool logs_directory property (not set by default).
  • spinner – Set the command pool spinner property (not set by default).
  • options – Additional keyword arguments can be used to conveniently override the default values of the writable properties of the RemoteCommand objects constructed by foreach() (see RemoteCommand.__init__() for details).
Returns:

The list of RemoteCommand objects constructed by foreach().
Raises:

Any of the following exceptions can be raised:

  • CommandPoolFailed if delay_checks is enabled (the default) and a command in the pool that has check enabled (the default) fails.
  • RemoteCommandFailed if delay_checks is disabled (not the default) and an SSH connection was successful but the remote command failed (the exit code of the ssh command was neither zero nor 255). Use the keyword argument check=False to disable raising of this exception.
  • RemoteConnectFailed if delay_checks is disabled (not the default) and an SSH connection failed (the exit code of the ssh command is 255). Use the keyword argument check=False to disable raising of this exception.

Note

The foreach() function enables the check and delay_checks options by default in an attempt to make it easy to do “the right thing”. My assumption here is that if you are running the same command on multiple remote hosts:

  • You definitely want to know when a remote command has failed, ideally without manually checking the succeeded property of each command.
  • Regardless of whether some remote commands fail you want to know that the command was at least executed on all hosts, otherwise your cluster of hosts will end up in a very inconsistent state.
  • If remote commands fail and an exception is raised the exception message should explain which remote commands failed.

If these assumptions are incorrect then you can use the keyword arguments check=False and/or delay_checks=False to opt out of “doing the right thing” ;-)

executor.ssh.client.remote(ssh_alias, *command, **options)[source]

Execute a remote command (similar to execute()).

Parameters:
Returns:

Refer to execute_prepared().
Raises:

RemoteCommandFailed when the command exits with a nonzero exit code (and check is True).
class executor.ssh.client.RemoteAccount(*args, **options)[source]

Trivial SSH alias parser.

This class acts as a base class for RemoteCommand and RemoteContext that provides three simple features:

[1]This enables RemoteCommand.have_superuser_privileges to know that superuser privileges are available when the caller sets ssh_alias to a value like root@host. Of course the SSH client configuration can also override the remote username without the executor package knowing about it, but at least executor will be able to use the information that it does have.
[2]This calling convention enables backwards compatibility with executor versions 14 and below which required ssh_alias to be set using the first positional argument to the initializer of the RemoteCommand class.
[3]This new calling convention provides a uniform calling convention for the initializers of the local/remote command/context classes.

Here’s an overview of the RemoteAccount class:

Superclass: PropertyManager
Special methods: __init__()
Properties: ssh_alias and ssh_user

When you initialize a RemoteAccount object you are required to provide a value for the ssh_alias property. You can set the values of the ssh_alias and ssh_user properties by passing keyword arguments to the class initializer.

__init__(*args, **options)[source]

Initialize a RemoteAccount object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the PropertyManager class (for future extensibility).
  • options – Any keyword arguments are passed on to the initializer of the PropertyManager class.

If the keyword argument ssh_alias isn’t given but positional arguments are provided, the first positional argument is used to set the ssh_alias property.

ssh_alias[source]

The SSH alias of the remote host (a string).

If you set this property to a string that contains two nonempty tokens delimited by an @ character, the first token is used to set ssh_user and the second token is used to set ssh_alias. Here’s an example:

>>> from executor.ssh.client import RemoteAccount
>>> RemoteAccount('root@server')
RemoteAccount(ssh_alias='server', ssh_user='root')
>>> RemoteAccount(ssh_alias='server', ssh_user='root')
RemoteAccount(ssh_alias='server', ssh_user='root')
>>> RemoteAccount('server')
RemoteAccount(ssh_alias='server', ssh_user=None)

Note

The ssh_alias property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named ssh_alias (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

ssh_user[source]

The username on the remote system (a string or None).

If the value of ssh_user is None the SSH client program gets to decide about the remote username.

Note

The ssh_user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

class executor.ssh.client.RemoteCommand(*args, **options)[source]

RemoteCommand objects use the SSH client program to execute remote commands.

Here’s an overview of the RemoteCommand class:

Superclasses: RemoteAccount and ExternalCommand
Special methods: __init__()
Properties: batch_mode, command, command_line, compression, connect_timeout, directory, error_message, error_type, have_superuser_privileges, identity_file, ignore_known_hosts, known_hosts_file, log_level, port, ssh_command and strict_host_key_checking

You can set the values of the batch_mode, command, compression, connect_timeout, error_message, error_type, identity_file, known_hosts_file, log_level, port, ssh_command and strict_host_key_checking properties by passing keyword arguments to the class initializer.

__init__(*args, **options)[source]

Initialize a RemoteCommand object.

Parameters:

The remote command is not started until you call start() or wait().

batch_mode[source]

Control the SSH client option BatchMode (a boolean, defaults to True).

The following description is quoted from man ssh_config:

If set to “yes”, passphrase/password querying will be disabled. In addition, the ServerAliveInterval option will be set to 300 seconds by default. This option is useful in scripts and other batch jobs where no user is present to supply the password, and where it is desirable to detect a broken network swiftly. The argument must be “yes” or “no”. The default is “no”.

This property defaults to True because it can get really awkward when a batch of SSH clients query for a passphrase/password on standard input at the same time.

Note

The batch_mode property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command[source]

A list of strings with the command to execute (optional).

The value of command is optional for RemoteCommand objects (as opposed to ExternalCommand objects) because the use of SSH implies a remote (interactive) shell that usually also accepts (interactive) commands as input. This means it is valid to create a remote command object without an actual remote command to execute, but with input that provides commands to execute instead.

This “feature” can be useful to control non-UNIX systems that do accept SSH connections but don’t support a conventional UNIX shell. For example, I added support for this “feature” so that I was able to send commands to Juniper routers and switches over SSH with the purpose of automating the failover of a connection between two datacenters (the resulting Python program works great and it’s much faster than I am, making all of the required changes in a couple of seconds :-).

Note

The command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The complete SSH client command including the remote command.

This is a list of strings with the SSH client command to connect to the remote host and execute command.

compression[source]

Whether compression is enabled (a boolean, defaults to False).

Note

The compression property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

connect_timeout[source]

Control the SSH client option ConnectTimeout (an integer).

The following description is quoted from man ssh_config:

Specifies the timeout (in seconds) used when connecting to the SSH server, instead of using the default system TCP timeout. This value is used only when the target is down or really unreachable, not when it refuses the connection.

Defaults to DEFAULT_CONNECT_TIMEOUT so that non-interactive SSH connections created by RemoteCommand don’t hang indefinitely when the remote system doesn’t respond properly.

Note

The connect_timeout property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

directory

Set the remote working directory.

When you set this property you change the remote working directory, however reading back the property you’ll just get DEFAULT_WORKING_DIRECTORY. This is because the superclass ExternalCommand uses directory as the local working directory for the ssh command, and a remote working directory isn’t guaranteed to also exist on the local system.

error_message[source]

A user friendly explanation of how the remote command failed (a string or None).

Note

The error_message property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

error_type[source]

An exception class applicable to the kind of failure detected or None.

RemoteConnectFailed when returncode is set and matches SSH_ERROR_STATUS, RemoteCommandFailed when returncode is set and not zero, None otherwise.

Note

The error_type property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

have_superuser_privileges

True if ssh_user is set to ‘root’, False otherwise.

There’s no easy way for RemoteCommand to determine whether any given SSH alias logs into a remote system with superuser privileges so unless ssh_user is set to ‘root’ this is always False.

identity_file[source]

The pathname of the identity file used to connect to the remote host (a string or None).

Note

The identity_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ignore_known_hosts

Whether host key checking is disabled.

This is True if host key checking is completely disabled:

If you set this to True host key checking is disabled and log_level is set to ‘error’ to silence warnings about automatically accepting host keys.

If you set this to False then known_hosts_file, log_level and strict_host_key_checking are reset to their default values.

log_level[source]

Control the SSH client option LogLevel (a string, defaults to ‘info’).

The following description is quoted from man ssh_config:

Gives the verbosity level that is used when logging messages from ssh. The possible values are: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG1, DEBUG2, and DEBUG3. The default is INFO. DEBUG and DEBUG1 are equivalent. DEBUG2 and DEBUG3 each specify higher levels of verbose output.

Note

The log_level property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ssh_command[source]

The command used to run the SSH client program.

This is a list of strings, by default the list contains just SSH_PROGRAM_NAME. The batch_mode, connect_timeout, log_level, ssh_alias and strict_host_key_checking properties also influence the SSH client command line used.

Note

The ssh_command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

port[source]

The port number of the SSH server (defaults to None which means the SSH client program decides).

Note

The port property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

strict_host_key_checking[source]

Control the SSH client option StrictHostKeyChecking.

This property accepts the values True and False and the strings ‘yes’, ‘no’ and ‘ask’. The following description is quoted from man ssh_config:

If this flag is set to “yes”, ssh will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed. This provides maximum protection against trojan horse attacks, though it can be annoying when the /etc/ssh/ssh_known_hosts file is poorly maintained or when connections to new hosts are frequently made. This option forces the user to manually add all new hosts. If this flag is set to “no”, ssh will automatically add new host keys to the user known hosts files. If this flag is set to “ask”, new host keys will be added to the user known host files only after the user has confirmed that is what they really want to do, and ssh will refuse to connect to hosts whose host key has changed. The host keys of known hosts will be verified automatically in all cases. The argument must be “yes”, “no”, or “ask”. The default is “ask”.

This property defaults to False so that when you connect to a remote system over SSH for the first time the host key is automatically added to the user known hosts file (instead of requiring interaction). As mentioned in the quote above the host keys of known hosts are always verified (but see ignore_known_hosts).

Note

The strict_host_key_checking property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

known_hosts_file[source]

Control the SSH client option UserKnownHostsFile (a string).

The following description is quoted from man ssh_config:

Specifies one or more files to use for the user host key database, separated by whitespace. The default is ~/.ssh/known_hosts, ~/.ssh/known_hosts2.

Note

The known_hosts_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

class executor.ssh.client.RemoteCommandPool(concurrency=10, **options)[source]

Execute multiple remote commands concurrently.

After constructing a RemoteCommandPool instance you add commands to it using add() and when you’re ready to run the commands you call run().

Note

The only difference between CommandPool and RemoteCommandPool is the default concurrency. This may of course change in the future.

Here’s an overview of the RemoteCommandPool class:

Superclass: CommandPool
Special methods: __init__()
__init__(concurrency=10, **options)[source]

Initialize a RemoteCommandPool object.

Parameters:
  • concurrency – Override the value of concurrency (an integer, defaults to DEFAULT_CONCURRENCY for remote command pools).
  • options – Any additional keyword arguments are passed on to the CommandPool constructor.
class executor.ssh.client.SecureTunnel(*args, **options)[source]

Easy to use SSH tunnels.

The SecureTunnel class combines RemoteCommand with EphemeralPortAllocator and WaitUntilConnected to implement easy to use SSH tunnel support.

The -L option of the SSH client program is used to open a tunnel between the local system and a remote system that persists until the command is terminated (tip: use a with statement).

Additionally the -N option is used to prevent the client from executing a remote command, this enables compatibility with “tunnel only” SSH accounts that have their shell set to something like /usr/sbin/nologin.

The start() method waits for the tunnel to become available before returning control to the caller.

Here’s an overview of the SecureTunnel class:

Superclass: RemoteCommand
Public methods: start()
Properties: asynchronous, command_line, compression, local_port, remote_host, remote_port and tty

When you initialize a SecureTunnel object you are required to provide a value for the remote_port property. You can set the values of the asynchronous, compression, local_port, remote_host and remote_port properties by passing keyword arguments to the class initializer.

asynchronous[source]

Whether to enable asynchronous command execution (a boolean, defaults to True).

Note

The asynchronous property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The complete SSH client command to open the tunnel (a list of strings).

This property overrides RemoteCommand.command_line to inject the command line options -L and -N.

compression[source]

Whether to enable compression (a boolean, defaults to True).

Note

The compression property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

local_port[source]

The port number on the side of the SSH client (an integer).

When the value of local_port isn’t specified a free ephemeral port number is automatically selected using EphemeralPortAllocator.

Note

The local_port property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached. To clear the cached value you can use del or delattr().

remote_host[source]

The remote host name to connect to (a string, defaults to ‘localhost’).

Note

The remote_host property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

remote_port[source]

The remote port number to connect to (an integer).

Note

The remote_port property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named remote_port (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

tty

Override tty to False.

start(**options)[source]

Start the SSH client and wait for the tunnel to become available.

Parameters:

options – Any keyword arguments are passed to the start() method of the superclass.
Raises:

Any exceptions raised by the following methods:
exception executor.ssh.client.RemoteConnectFailed(command, **options)[source]

Raised by RemoteCommand when an SSH connection itself fails (not the remote command).

Here’s an overview of the RemoteConnectFailed class:

Superclass: ExternalCommandFailed
exception executor.ssh.client.RemoteCommandFailed(command, **options)[source]

Raised by RemoteCommand when a remote command executed over SSH fails.

Here’s an overview of the RemoteCommandFailed class:

Superclass: ExternalCommandFailed
exception executor.ssh.client.RemoteCommandNotFound(command, **options)[source]

Raised by RemoteCommand when a remote command returns COMMAND_NOT_FOUND_STATUS.

Here’s an overview of the RemoteCommandNotFound class:

Superclasses: RemoteCommandFailed and CommandNotFound

The executor.ssh.server module

OpenSSH server automation for testing.

The executor.ssh.server module defines the SSHServer class which can be used to start temporary OpenSSH servers that are isolated enough from the host system to make them usable in the executor test suite (to test remote command execution).

executor.ssh.server.SSHD_PROGRAM_NAME = 'sshd'

The name of the SSH server executable (a string).

class executor.ssh.server.SSHServer(**options)[source]

Subclass of ExternalCommand that manages a temporary SSH server.

The OpenSSH server spawned by the SSHServer class doesn’t need superuser privileges and doesn’t require any changes to /etc/passwd or /etc/shadow.

Here’s an overview of the SSHServer class:

Superclass: EphemeralTCPServer
Special methods: __init__()
Public methods: cleanup(), generate_config(), generate_key_file() and start()
Properties: client_options and sshd_path
__init__(**options)[source]

Initialize an SSHServer object.

Parameters:options – All keyword arguments are passed on to executor.ExternalCommand.__init__().
temporary_directory = None

The pathname of the temporary directory used to store the files required to run the SSH server (a string).

client_key_file = None

The pathname of the generated OpenSSH client key file (a string).

config_file = None

The pathname of the generated OpenSSH server configuration file (a string).

host_key_file = None

The random port number on which the SSH server will listen (an integer).

sshd_path

The absolute pathname of SSHD_PROGRAM_NAME (a string).

client_options

The options for the OpenSSH client (required to connect with the server).

This is a dictionary of keyword arguments for RemoteCommand to make it connect with the OpenSSH server (assuming the remote command connects to an IP address in the 127.0.0.0/24 range).

start(**options)[source]

Start the SSH server and wait for it to start accepting connections.

Parameters:options – Any keyword arguments are passed to the start() method of the superclass.
Raises:Any exceptions raised by the start() method of the superclass.

The start() method automatically calls the generate_key_file() and generate_config() methods.

generate_key_file(filename)[source]

Generate a temporary host or client key for the OpenSSH server.

The start() method automatically calls generate_key_file() to generate host_key_file and client_key_file. This method uses the ssh-keygen program to generate the keys.

generate_config()[source]

Generate a configuration file for the OpenSSH server.

The start() method automatically calls generate_config().

cleanup()[source]

Clean up temporary_directory after the test server finishes.

class executor.ssh.server.EphemeralTCPServer(*command, **options)[source]

Make it easy to launch ephemeral TCP servers.

The EphemeralTCPServer class makes it easy to allocate an ephemeral port number that is not (yet) in use.

Here’s an overview of the EphemeralTCPServer class:

Superclasses: ExternalCommand and EphemeralPortAllocator
Public methods: start()
Properties: asynchronous
asynchronous

Ephemeral TCP servers always set ExternalCommand.asynchronous to True.

start(**options)[source]

Start the TCP server and wait for it to start accepting connections.

Parameters:options – Any keyword arguments are passed to the start() method of the superclass.
Raises:Any exceptions raised by start() and wait_until_connected().

If the TCP server doesn’t start accepting connections within the configured timeout (see wait_timeout) the process will be terminated and the timeout exception is propagated.

exception executor.ssh.server.TimeoutError[source]

Raised when a TCP server doesn’t start accepting connections quickly enough.

This exception is raised by wait_until_connected() when the TCP server doesn’t start accepting connections within a reasonable time.

The executor.tcp module

Miscellaneous TCP networking functionality.

The functionality in this module originated in the executor.ssh.server module with the purpose of facilitating a robust automated test suite for the executor.ssh.client module. While working on SSH tunnel support I needed similar logic again and I decided to extract this code from the executor.ssh.server module.

class executor.tcp.WaitUntilConnected(**kw)[source]

Wait for a TCP endpoint to start accepting connections.

Here’s an overview of the WaitUntilConnected class:

Superclass: PropertyManager
Public methods: wait_until_connected()
Properties: connect_timeout, endpoint, hostname, is_connected, port_number, scheme and wait_timeout

When you initialize a WaitUntilConnected object you are required to provide a value for the port_number property. You can set the values of the connect_timeout, hostname, port_number, scheme and wait_timeout properties by passing keyword arguments to the class initializer.

connect_timeout[source]

The timeout in seconds for individual connection attempts (a number, defaults to 2).

Note

The connect_timeout property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

endpoint

A human friendly representation of the TCP endpoint (a string containing a URL).

hostname[source]

The host name or IP address to connect to (a string, defaults to localhost).

Note

The hostname property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

is_connected

True if a connection was accepted, False otherwise.

port_number[source]

The port number to connect to (an integer).

Note

The port_number property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named port_number (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

scheme[source]

A URL scheme that indicates the purpose of the ephemeral port (a string, defaults to ‘tcp’).

Note

The scheme property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

wait_timeout[source]

The timeout in seconds for wait_until_connected() (a number, defaults to 30).

Note

The wait_timeout property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

wait_until_connected()[source]

Wait until connections are being accepted.

Raises:TimeoutError when the SSH server isn’t fast enough to initialize.
class executor.tcp.EphemeralPortAllocator(**kw)[source]

Allocate a free ephemeral port number.

Here’s an overview of the EphemeralPortAllocator class:

Superclass: WaitUntilConnected
Properties: ephemeral_port_number and port_number
port_number[source]

A dynamically selected free ephemeral port number (an integer between 49152 and 65535).

Note

The port_number property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

ephemeral_port_number

A random ephemeral port number (an integer between 49152 and 65535).

class executor.tcp.EphemeralTCPServer(*command, **options)[source]

Make it easy to launch ephemeral TCP servers.

The EphemeralTCPServer class makes it easy to allocate an ephemeral port number that is not (yet) in use.

Here’s an overview of the EphemeralTCPServer class:

Superclasses: ExternalCommand and EphemeralPortAllocator
Public methods: start()
Properties: asynchronous
asynchronous

Ephemeral TCP servers always set ExternalCommand.asynchronous to True.

start(**options)[source]

Start the TCP server and wait for it to start accepting connections.

Parameters:options – Any keyword arguments are passed to the start() method of the superclass.
Raises:Any exceptions raised by start() and wait_until_connected().

If the TCP server doesn’t start accepting connections within the configured timeout (see wait_timeout) the process will be terminated and the timeout exception is propagated.

exception executor.tcp.TimeoutError[source]

Raised when a TCP server doesn’t start accepting connections quickly enough.

This exception is raised by wait_until_connected() when the TCP server doesn’t start accepting connections within a reasonable time.