executor: Programmer friendly subprocess wrapper

https://travis-ci.org/xolox/python-executor.svg?branch=master https://coveralls.io/repos/xolox/python-executor/badge.png?branch=master

The executor package is a simple wrapper for Python’s subprocess module that makes it very easy to handle subprocesses on UNIX systems with proper escaping of arguments and error checking:

  • An object oriented interface is used to execute commands using sane but customizable (and well documented) defaults.
  • Remote commands (executed over SSH) are supported using the same object oriented interface, as are commands inside chroots (executed using schroot).
  • There’s also support for executing a group of commands concurrently in what’s called a “command pool”. The concurrency level can be customized and of course both local and remote commands are supported.

The package is currently tested on Python 2.6, 2.7, 3.4, 3.5, 3.6 and PyPy. For usage instructions please refer to following sections and the documentation.

Installation

The executor package is available on PyPI which means installation should be as simple as:

$ pip install executor

There’s actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).

Usage

There are two ways to use the executor package: As the command line program executor and as a Python API. The command line interface is described below and there are also some examples of simple use cases of the Python API.

Command line

Usage: executor [OPTIONS] COMMAND ...

Easy subprocess management on the command line based on the Python package with the same name. The “executor” program runs external commands with support for timeouts, dynamic startup delay (fudge factor) and exclusive locking.

You can think of “executor” as a combination of the “flock” and “timelimit” programs with some additional niceties (namely the dynamic startup delay and integrated system logging on UNIX platforms).

Supported options:

Option Description
-t, --timeout=LIMIT Set the time after which the given command will be aborted. By default LIMIT is counted in seconds. You can also use one of the suffixes “s” (seconds), “m” (minutes), “h” (hours) or “d” (days).
-f, --fudge-factor=LIMIT This option controls the dynamic startup delay (fudge factor) which is useful when you want a periodic task to run once per given interval but the exact time is not important. Refer to the --timeout option for acceptable values of LIMIT, this number specifies the maximum amount of time to sleep before running the command (the minimum is zero, otherwise you could just include the command “sleep N && ...” in your command line :-).
-e, --exclusive Use an interprocess lock file to guarantee that executor will never run the external command concurrently. Refer to the --lock-timeout option to customize blocking / non-blocking behavior. To customize the name of the lock file you can use the --lock-file option.
-T, --lock-timeout=LIMIT By default executor tries to claim the lock and if it fails it will exit with a nonzero exit code. This option can be used to enable blocking behavior. Refer to the --timeout option for acceptable values of LIMIT.
-l, --lock-file=NAME Customize the name of the lock file. By default this is the base name of the external command, so if you’re running something generic like “bash” or “python” you might want to change this :-).
-v, --verbose Make more noise than usual (increase logging verbosity).
-q, --quiet Make less noise than usual (decrease logging verbosity).
-h, --help Show this message and exit.

Python API

Below are some examples of how versatile the execute() function is. Refer to the API documentation on Read the Docs for (a lot of) other use cases.

Checking status codes

By default the status code of the external command is returned as a boolean:

>>> from executor import execute
>>> execute('true')
True

If an external command exits with a nonzero status code an exception is raised, this makes it easy to do the right thing (never forget to check the status code of an external command without having to write a lot of repetitive code):

>>> execute('false')
Traceback (most recent call last):
  File "executor/__init__.py", line 124, in execute
    cmd.start()
  File "executor/__init__.py", line 516, in start
    self.wait()
  File "executor/__init__.py", line 541, in wait
    self.check_errors()
  File "executor/__init__.py", line 568, in check_errors
    raise ExternalCommandFailed(self)
executor.ExternalCommandFailed: External command failed with exit code 1! (command: bash -c false)

The ExternalCommandFailed exception exposes command and returncode attributes. If you know a command is likely to exit with a nonzero status code and you want execute() to simply return a boolean you can do this instead:

>>> execute('false', check=False)
False

Providing input

Here’s how you can provide input to an external command:

>>> execute('tr a-z A-Z', input='Hello world from Python!\n')
HELLO WORLD FROM PYTHON!
True

Getting output

Getting the output of external commands is really easy as well:

>>> execute('hostname', capture=True)
'peter-macbook'

Running commands as root

It’s also very easy to execute commands with super user privileges:

>>> execute('echo test > /etc/hostname', sudo=True)
[sudo] password for peter: **********
True
>>> execute('hostname', capture=True)
'test'

Enabling logging

If you’re wondering how prefixing the above command with sudo would end up being helpful, here’s how it works:

>>> import logging
>>> logging.basicConfig()
>>> logging.getLogger().setLevel(logging.DEBUG)
>>> execute('echo peter-macbook > /etc/hostname', sudo=True)
DEBUG:executor:Executing external command: sudo bash -c 'echo peter-macbook > /etc/hostname'

Running remote commands

To run a command on a remote system using SSH you can use the RemoteCommand class, it works as follows:

>>> from executor.ssh.client import RemoteCommand
>>> cmd = RemoteCommand('localhost', 'echo $SSH_CONNECTION', capture=True)
>>> cmd.start()
>>> cmd.output
'127.0.0.1 57255 127.0.0.1 22'

Running remote commands concurrently

The foreach() function wraps the RemoteCommand and CommandPool classes to make it very easy to run a remote command concurrently on a group of hosts:

>>> from executor.ssh.client import foreach
>>> from pprint import pprint
>>> hosts = ['127.0.0.1', '127.0.0.2', '127.0.0.3', '127.0.0.4']
>>> commands = foreach(hosts, 'echo $SSH_CONNECTION')
>>> pprint([cmd.output for cmd in commands])
['127.0.0.1 57278 127.0.0.1 22',
 '127.0.0.1 52385 127.0.0.2 22',
 '127.0.0.1 49228 127.0.0.3 22',
 '127.0.0.1 40628 127.0.0.4 22']

Contact

The latest version of executor is available on PyPI and GitHub. The documentation is hosted on Read the Docs. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.

License

This software is licensed under the MIT license.

© 2017 Peter Odding.

API documentation

The following documentation is based on the source code of version 17.1 of the executor package. The following modules are available:

The executor module

Core functionality of the executor package.

If you’re looking for an easy way to run external commands from Python take a look at the execute() function. When you need more flexibility consider using the underlying ExternalCommand class directly instead.

execute() versus ExternalCommand

In executor 1.x the execute() function was the only interface for external command execution. This had several drawbacks:

  • The documentation for the execute() function was getting way too complex given all of the supported options and combinations.
  • There was no way to execute asynchronous external commands (running in the background) without sidestepping the complete executor module and going straight for subprocess.Popen (with all of the verbosity that you get for free with subprocess :-).
  • There was no way to prepare an external command without starting it immediately, making it impossible to prepare a batch of external commands before starting them (whether synchronously or asynchronously).

To solve these problems executor 2.x introduced the ExternalCommand class. This explains why execute() is now a trivial wrapper around ExternalCommand: It’s main purpose is to be an easy to use shortcut that preserves compatibility with the old interface.

Classes and functions

executor.DEFAULT_ENCODING = 'UTF-8'

The default encoding of the standard input, output and error streams (a string).

executor.DEFAULT_WORKING_DIRECTORY = '.'

The default working directory for external commands (a string). Defaults to the working directory of the current process using os.curdir.

executor.DEFAULT_SHELL = 'bash'

The default shell used to evaluate shell expressions (a string).

This variable isn’t based on the $SHELL environment variable because:

  1. Shells like sh, dash, bash and zsh all have their own subtly incompatible semantics.
  2. People regularly use shells like fish as their default login shell :-).

At an interactive prompt this is no problem (advanced users have obviously learned to context switch) but when you’re writing source code the last thing you want to worry about is which shell is going to evaluate your commands! The executor package expects this shell to support the following features:

  • The -c option to evaluate a shell command provided as a command line argument.
  • The - argument to instruct the shell to read shell commands from its standard input stream and evaluate those.

Apart from these two things nothing else is expected from the default shell so you’re free to customize it if you really want to write your shell commands in fish or zsh syntax :-).

executor.COMMAND_NOT_FOUND_CODES = (2,)

Numeric error codes returned when a command isn’t available on the system (a tuple of integers).

executor.COMMAND_NOT_FOUND_STATUS = 127

The exit status used by shells when a command is not found (an integer).

executor.execute(*command, **options)[source]

Execute an external command and make sure it succeeded.

Parameters:
  • command – All positional arguments are passed on to the constructor of ExternalCommand.
  • options – All keyword arguments are passed on to the constructor of ExternalCommand.
Returns:

Refer to execute_prepared().

Raises:

ExternalCommandFailed when the command exits with a nonzero exit code (and check is True).

If async is True then execute() will automatically start the external command for you using start() (but it won’t wait for it to end). If you want to create an ExternalCommand object instance without immediately starting the external command then you can use ExternalCommand directly.

Some examples

By default the status code of the external command is returned as a boolean:

>>> from executor import execute
>>> execute('true')
True

However when an external command exits with a nonzero status code an exception is raised, this is intended to “make it easy to do the right thing” (never forget to check the status code of an external command without having to write a lot of repetitive code):

>>> execute('false')
Traceback (most recent call last):
  File "executor/__init__.py", line 124, in execute
    cmd.start()
  File "executor/__init__.py", line 516, in start
    self.wait()
  File "executor/__init__.py", line 541, in wait
    self.check_errors()
  File "executor/__init__.py", line 568, in check_errors
    raise ExternalCommandFailed(self)
executor.ExternalCommandFailed: External command failed with exit code 1! (command: false)

What’s also useful to know is that exceptions raised by execute() expose command and returncode attributes. If you know a command is likely to exit with a nonzero status code and you want execute() to simply return a boolean you can do this instead:

>>> execute('false', check=False)
False
executor.execute_prepared(command)[source]

The logic behind execute() and remote().

Parameters:command – An ExternalCommand object (or an object created from a subclass with a compatible interface like for example RemoteCommand).
Returns:The return value of this function depends on several options:
Raises:See execute() and remote().
class executor.ExternalCommand(*command, **options)[source]

Programmer friendly subprocess.Popen wrapper.

The ExternalCommand class wraps subprocess.Popen to make it easier to do the right thing (the simplicity of os.system() with the robustness of subprocess.Popen) and to provide additional features (e.g. asynchronous command execution that preserves the ability to provide input and capture output).

ExternalCommand inherits from ControllableProcess which means that all of the process manipulation supported by ControllableProcess is also supported by ExternalCommand objects.

Because the ExternalCommand class has a lot of properties and methods here is an attempt to summarize them (this overview will no doubt become out of date):

Writable properties
The async, callback, capture, capture_stderr, check, directory, encoding, environment, fakeroot, input, ionice, logger, merge_streams, shell, silent, stdout_file, stderr_file, uid, user, sudo and virtual_environment properties allow you to configure how the external command will be run (before it is started).
Computed properties
The command, command_line, decoded_stderr, decoded_stdout, encoded_input, error_message, error_type, failed, have_superuser_privileges, ionice_command, is_finished, is_running, is_terminated, output, pid, result, returncode, stderr, stdout, succeeded, sudo_command and was_started properties allow you to inspect if and how the external command was started, what its current status is and what its output is.
Public methods
The public methods start(), wait(), terminate() and kill() enable you to start external commands, wait for them to finish and terminate them if they take too long.
Internal methods
The internal methods check_errors(), load_output() and cleanup() are used by methods like start(), wait(), terminate() and kill() so unless you’re reimplementing one of those methods you probably don’t need these internal methods.
Context manager

ExternalCommand objects can be used as context managers by using the with statement:

  • When the scope of the with statement starts the start() method is called (if the external command isn’t already running).
  • When the scope of the with statement ends terminate() is called if the command is still running. The load_output() and cleanup() functions are used to cleanup after the external command. If an exception isn’t already being raised check_errors() is called to make sure the external command succeeded.
__init__(*command, **options)[source]

Initialize an ExternalCommand object.

Parameters:

The external command is not started until you call start() or wait().

async[source]

Enable asynchronous command execution.

If this option is True (not the default) preparations are made to execute the external command asynchronously (in the background). This has several consequences:

  • Calling start() will start the external command but will not block until the external command is finished, instead you are responsible for calling wait() at some later point in time.

  • When input is set its value will be written to a temporary file and the standard input stream of the external command is connected to read from the temporary file.

    By using a temporary file the external command can consume its input as fast or slow as it pleases without needing a separate thread or process to “feed” the external command.

  • When capture is True the standard output of the external command is redirected to a temporary file whose contents are read once the external command has finished.

    By using a temporary file the external command can produce output as fast or slow as it pleases without needing a thread or subprocess on our side to consume the output in real time.

Note

The async property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

callback[source]

Optional callback used to generate the value of result.

The callback and result properties were created for use in command pools, where it can be useful to define how to process (parse) a command’s output when the command is constructed.

Note

The callback property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

capture[source]

Enable capturing of the standard output stream.

If this option is True (not the default) the standard output of the external command is captured and made available to the caller via stdout and output.

The standard error stream will not be captured, use capture_stderr for that. You can also silence the standard error stream using the silent option.

If callback is set capture defaults to True (but you can still set capture to False if that is what you want).

Note

The capture property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

capture_stderr[source]

Enable capturing of the standard error stream.

If this option is True (not the default) the standard error stream of the external command is captured and made available to the caller via stderr.

Note

The capture_stderr property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

check[source]

Enable automatic status code checking.

If this option is True (the default) and the external command exits with a nonzero status code ExternalCommandFailed will be raised by start() (when async isn’t set) or wait() (when async is set).

Note

The check property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command[source]

A list of strings with the command to execute.

Note

In executor version 14.0 it became valid to set input and shell without providing command (in older versions it was required to set command regardless of the other options).

Note

The command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The command line of the external command.

The command line used to actually run the external command requested by the user (a list of strings). The command line is constructed based on command according to the following rules:

  • If shell is True the external command is run using bash -c '...' (assuming you haven’t changed DEFAULT_SHELL) which means constructs like semicolons, ampersands and pipes can be used (and all the usual caveats apply :-).
  • If virtual_environment is set the command is converted to a shell command line and prefixed by the applicable source ... command.
  • If uid or user is set the sudo -u command will be prefixed to the command line generated here.
  • If fakeroot or sudo is set the respective command name is prefixed to the command line generated here (sudo is only prefixed when the current process doesn’t already have super user privileges).
  • If ionice is set the appropriate command is prefixed to the command line generated here.
decoded_stdout

The value of stdout decoded using encoding.

This is a unicode() object (in Python 2) or a str object (in Python 3).

decoded_stderr

The value of stderr decoded using encoding.

This is a unicode() object (in Python 2) or a str object (in Python 3).

dependencies[source]

The dependencies of the command (a list of ExternalCommand objects).

The dependencies property enables low level concurrency control in command pools by imposing a specific order of execution:

Note

The dependencies property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

directory[source]

The working directory for the external command.

A string, defaults to DEFAULT_WORKING_DIRECTORY.

Note

The directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

encoded_input

The value of input encoded using encoding.

This is a str object (in Python 2) or a bytes object (in Python 3).

encoding[source]

The character encoding of standard input and standard output.

A string, defaults to DEFAULT_ENCODING. This option is used to encode input and to decode output.

Note

The encoding property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

environment[source]

A dictionary of environment variables for the external command.

You only need to specify environment variables that differ from those of the current process (that is to say the environment variables of the current process are merged with the variables that you specify here).

Note

The environment property is a custom_property. You can change the value of this property using normal attribute assignment syntax. This property’s value is computed once (the first time it is accessed) and the result is cached.

error_message[source]

A string describing how the external command failed or None.

Note

The error_message property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

error_type[source]

An appropriate exception class or None (when no error occurred).

CommandNotFound if the external command exits with return code COMMAND_NOT_FOUND_STATUS or ExternalCommandFailed if the external command exits with any other nonzero return code.

Note

The error_type property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

failed

Whether the external command has failed.

  • True if returncode is a nonzero number or error_type is set (e.g. because the external command doesn’t exist).
  • False if returncode is zero.
  • None when the external command hasn’t been started or is still running.
fakeroot[source]

Run the external command under fakeroot.

If this option is True (not the default) and the current process doesn’t have superuser privileges the external command is run with fakeroot. If the fakeroot program is not installed the external command will fail.

Note

The fakeroot property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

finish_event[source]

Optional callback that’s called just after the command finishes.

The start_event and finish_event properties were created for use in command pools, for example to report to the operator when specific commands are started and when they finish. The invocation of the start_event and finish_event callbacks is performed inside the ExternalCommand class though, so you’re free to repurpose these callbacks outside the context of command pools.

Note

The finish_event property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

group_by[source]

Identifier that’s used to group the external command (any hashable value).

The group_by property enables high level concurrency control in command pools by making it easy to control which commands are allowed to run concurrently and which are required to run serially:

  • Command pools will never start more than one command within a group of commands that share the same value of group_by (for values that aren’t None).
  • If group_by is None it has no effect and concurrency is controlled by dependencies and concurrency.

Note

The group_by property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

have_superuser_privileges

Whether the parent Python process is running under superuser privileges.

True if running with superuser privileges, False otherwise. Used by command_line to decide whether sudo needs to be used.

input[source]

The input to feed to the external command on the standard input stream.

Defaults to None. When you provide a unicode() object (in Python 2) or a str object (in Python 3) as input it will be encoded using encoding. To avoid the automatic conversion you can simply pass a str object (in Python 2) or a bytes object (in Python 3).

The conversion logic is implemented in the encoded_input attribute.

Note

The input property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ionice[source]

The I/O scheduling class for the external command (a string or None).

When this property is set then ionice will be used to set the I/O scheduling class for the external command. This can be useful to reduce the impact of heavy disk operations on the rest of the system.

Raises:Any exceptions raised by validate_ionice_class().

Note

The ionice property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ionice_command

The ionice command based on ionice (a list of strings).

is_finished

Whether the external command has finished execution.

True once the external command has been started and has since finished, False when the external command hasn’t been started yet or is still running.

is_running

True if the process is currently running, False otherwise.

is_terminated

Whether the external command has been terminated (a boolean).

True if the external command was terminated using SIGTERM (e.g. by terminate()), False otherwise.

merge_streams[source]

Whether to merge the standard output and error streams.

A boolean, defaults to False. If this option is enabled stdout will contain the external command’s output on both streams.

Note

The merge_streams property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

output

The value of stdout decoded using encoding.

This is a unicode() object (in Python 2) or a str object (in Python 3).

This is only available when capture is True. If capture is not True then output will be None.

After decoding any leading and trailing whitespace is stripped and if the resulting string doesn’t contain any remaining newlines then the string with leading and trailing whitespace stripped will be returned, otherwise the decoded string is returned unchanged:

>>> from executor import ExternalCommand
>>> cmd = ExternalCommand('echo naïve', capture=True)
>>> cmd.start()
>>> cmd.output
u'na\xefve'
>>> cmd.stdout
'na\xc3\xafve\n'

This is intended to make simple things easy (output makes it easy to deal with external commands that output a single line) while providing an escape hatch when the default assumptions don’t hold (you can always use stdout to get the raw output).

result

The result of calling the value given by callback.

If the command hasn’t been started yet start() is called. When the command hasn’t finished yet func:wait() is called. If callback isn’t set None is returned.

returncode[source]

The return code of the external command (an integer) or None.

This will be None until the external command has finished.

Note

The returncode property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

shell[source]

Whether to evaluate the external command as a shell command.

A boolean, the default depends on the value of command:

When shell is True the external command is evaluated by the shell given by DEFAULT_SHELL, otherwise the external command is run without shell evaluation.

Note

The shell property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

silent[source]

Whether the external command’s output should be silenced.

If this is True (not the default) any output of the external command is silenced by redirecting the output streams to os.devnull.

You can enable capture and silent together to capture the standard output stream while silencing the standard error stream.

Note

The silent property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

start_event[source]

Optional callback that’s called just before the command is started.

The start_event and finish_event properties were created for use in command pools, for example to report to the operator when specific commands are started and when they finish. The invocation of the start_event and finish_event callbacks is performed inside the ExternalCommand class though, so you’re free to repurpose these callbacks outside the context of command pools.

Note

The start_event property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

stderr

The output of the external command on its standard error stream.

This is a str object (in Python 2) or a bytes object (in Python 3).

This is only available when capture_stderr is True. If capture_stderr is not True then stderr will be None.

stderr_file[source]

Capture the standard error stream to the given file handle.

When this property is set to a writable file object the standard error stream of the external command is redirected to the given file. The default value of this property is None.

This can be useful to (semi) permanently store command output or to run commands whose output is hidden but can be followed using tail -f if the need arises. By setting stdout_file and stderr_file to the same file object the output from both streams can be merged and redirected to the same file. This accomplishes roughly the same thing as setting merge_streams but leaves the caller in control of the file.

If this property isn’t set but capture is True the external command’s output is captured to a temporary file that’s automatically cleaned up after the external command is finished and its output has been cached (read into memory).

Note

The stderr_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

stdout

The output of the external command on its standard output stream.

This is a str object (in Python 2) or a bytes object (in Python 3).

This is only available when capture is True. If capture is not True then stdout will be None.

stdout_file[source]

Capture the standard output stream to the given file handle.

When this property is set to a writable file object the standard output stream of the external command is redirected to the given file. The default value of this property is None.

This can be useful to (semi) permanently store command output or to run commands whose output is hidden but can be followed using tail -f if the need arises. By setting stdout_file and stderr_file to the same file object the output from both streams can be merged and redirected to the same file. This accomplishes roughly the same thing as setting merge_streams but leaves the caller in control of the file.

If this property isn’t set but capture is True the external command’s output is captured to a temporary file that’s automatically cleaned up after the external command is finished and its output has been cached (read into memory).

Note

The stdout_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

subprocess[source]

A subprocess.Popen object or None.

The value of this property is set by start() and it’s cleared by wait() (through cleanup()) as soon as the external command has finished. This enables garbage collection of the resources associated with the subprocess.Popen object which helps to avoid IOError: [Errno 24] Too many open files errors.

Note

The subprocess property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

succeeded

Whether the external command succeeded.

  • True if returncode is zero.
  • False if returncode is a nonzero number or error_type is set (e.g. because the external command doesn’t exist).
  • None when the external command hasn’t been started or is still running.
sudo[source]

Whether sudo should be used to gain superuser privileges.

If this option is True (not the default) and the current process doesn’t have superuser privileges the external command is run with sudo to ensure that the external command runs with superuser privileges.

The use of this option assumes that the sudo command is available.

Note

The sudo property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

sudo_command

The sudo command used to change privileges (a list of strings).

This option looks at the sudo, uid and user properties to decide whether command should be run using sudo or not. If it should then a prefix for command is constructed from sudo, uid, user and/or environment and returned. Some examples:

>>> from executor import ExternalCommand
>>> ExternalCommand('true', sudo=True).sudo_command
['sudo']
>>> ExternalCommand('true', uid=1000).sudo_command
['sudo', '-u', '#1000']
>>> ExternalCommand('true', user='peter').sudo_command
['sudo', '-u', 'peter']
>>> ExternalCommand('true', user='peter', environment=dict(gotcha='this is tricky')).sudo_command
['sudo', '-u', 'peter', 'gotcha=this is tricky']
tty[source]

Whether the command will be attached to the controlling terminal (a boolean).

By default tty is True when:

If any of these conditions don’t hold tty defaults to False. When tty is False the standard input stream of the external command will be connected to os.devnull.

Note

The tty property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

uid[source]

The user ID of the system user that’s used to run the command.

If this option is set to an integer number (it defaults to None) the external command is prefixed with sudo -u #UID to run the command as a different user than the current user.

The use of this option assumes that the sudo command is available.

Note

The uid property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

user[source]

The name of the system user that’s used to run the command.

If this option is set to a string (it defaults to None) the external command is prefixed with sudo -u USER to run the command as a different user than the current user.

The use of this option assumes that the sudo command is available.

Note

The user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

virtual_environment[source]

The Python virtual environment to activate before running the command.

If this option is set to the directory of a Python virtual environment (a string) then the external command will be prefixed by a source shell command that evaluates the bin/activate script in the Python virtual environment before executing the user defined external command.

Note

The virtual_environment property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

was_started[source]

Whether the external command has already been started.

True once start() has been called to start executing the external command, False when start() hasn’t been called yet.

Note

The was_started property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

prefix_shell_command(preamble, command)[source]

Prefix a shell command to a command line.

Parameters:
  • preamble – Any shell command (a string).
  • command – The command line (a string, tuple or list).
Returns:

The command line to run the combined commands through a shell (a list of strings).

This function uses reduce_shell_command() to convert command into a string and then prefixes the preamble to the command, delimited by &&.

reduce_shell_command(command)[source]

Reduce a command line to a shell command.

Parameters:command – The command line (a string, tuple or list).
Returns:The shell command (a string).

If the given command is a DEFAULT_SHELL invocation that uses the -c option it is reduced to the argument of the -c option. All other command lines are simply quoted and returned.

This method is used in various places where a command needs to be transformed into a shell command so that a command like cd or source can be prefixed to the command line.

start()[source]

Start execution of the external command.

Raises:

This method instantiates a subprocess.Popen object based on the defaults defined by ExternalCommand and the overrides configured by the caller. What happens then depends on async:

  • If async is set start() starts the external command but doesn’t wait for it to end (use wait() for that).
  • If async isn’t set the communicate() method on the subprocess object is called to synchronously execute the external command.
wait(check=None, **kw)[source]

Wait for the external command to finish.

Parameters:
  • check – Override the value of check for the duration of this call to wait(). Defaults to None which means check is not overridden.
  • kw – Any keyword arguments are passed on to wait_for_process().
Raises:

ExternalCommandFailed when check is True, async is True and the external command exits with a nonzero status code.

The wait() function is only useful when async is True, it performs the following steps:

  1. If was_started is False the start() method is called.
  2. If is_running is True the wait_for_process() method is called to wait for the child process to end.
  3. If subprocess isn’t None the cleanup() method is called to wait for the external command to end, load its output into memory and release the resources associated with the subprocess object.
  4. Finally check_errors() is called (in case the caller didn’t disable check).
terminate_helper()[source]

Gracefully terminate the process.

Raises:Any exceptions raised by the subprocess module.

This method sets check to False, the idea being that if you consciously terminate a command you don’t need to be bothered with an exception telling you that you succeeded :-).

kill_helper()[source]

Forcefully kill the process.

Raises:Any exceptions raised by the subprocess module.

This method sets check to False, the idea being that if you consciously kill a command you don’t need to be bothered with an exception telling you that you succeeded :-).

load_output()[source]

Load output captured from the standard output/error streams.

Reads the contents of the temporary file(s) created by start() (when async and capture are both set) into memory so that the output doesn’t get lost when the temporary file is cleaned up by cleanup().

cleanup()[source]

Clean up after the external command has ended.

This internal method is called by methods like start() and wait() to clean up the following temporary resources:

  • The temporary file(s) used to to buffer the external command’s input, stdout and stderr (only when async is True).
  • File handles to the previously mentioned temporary files and os.devnull (used to implement the silent option).
  • The reference to the subprocess.Popen object stored in subprocess. By destroying this reference as soon as possible we enable the object to be garbage collected and its related resources to be released.
reset()[source]

Reset internal state created by start().

check_errors(check=None)[source]

Raise an exception if the external command failed.

This raises error_type when check is set and the external command failed.

Parameters:check – Override the value of check for the duration of this call. Defaults to None which means check is not overridden.
Raises:error_type when check is set and error_type is not None.

This internal method is used by start() and wait() to make sure that failing external commands don’t go unnoticed.

invoke_event_callback(name)[source]

Invoke one of the event callbacks.

Parameters:name – The name of the callback (a string).
__enter__()[source]

Start the external command if it hasn’t already been started.

Returns:The ExternalCommand object.

When you use an ExternalCommand as a context manager in the with statement, the command is automatically started when entering the context and terminated when leaving the context.

If the proces hasn’t already been started yet async is automatically set to True (if it’s not already True), otherwise the command will have finished execution by the time the body of the with statement is executed (which isn’t really all that useful :-).

__exit__(exc_type=None, exc_value=None, traceback=None)[source]

Automatically terminate and clean up after the external command.

Terminates the external command if it is still running (using terminate()), cleans up (using cleanup()) and checks for errors (using check_errors(), only if an exception is not already being handled).

class executor.CachedStream(command, kind)[source]

Manages a temporary file with input for / output from an external command.

__init__(command, kind)[source]

Initialize a CachedStream object.

Parameters:
  • command – The ExternalCommand object that’s using the CachedStream object.
  • kind – A simple (alphanumeric) string with the name of the stream.
prepare_temporary_file()[source]

Prepare the stream’s temporary file.

prepare_input()[source]

Initialize the input stream.

Returns:A value that can be passed to the constructor of subprocess.Popen as the stdin argument.
prepare_output(file, capture)[source]

Initialize an (asynchronous) output stream.

Parameters:
  • file – A file handle or None.
  • captureTrue if capturing is enabled, False otherwise.
Returns:

A value that can be passed to the constructor of subprocess.Popen as the stdout and/or stderr argument.

redirect(obj)[source]

Capture the stream in a file provided by the caller.

Parameters:obj – A file-like object that has an associated file descriptor.
load()[source]

Load the stream’s contents from the temporary file.

Returns:The output of the stream (a string) or None when the stream was never initialized.
finalize(output=None)[source]

Load or override the stream’s contents and cleanup the temporary file.

Parameters:output – Override the stream’s contents (defaults to None which means the contents are loaded from the temporary file instead).
cleanup()[source]

Cleanup temporary resources.

reset()[source]

Reset internal state.

executor.quote(*args)[source]

Quote a string or a sequence of strings to be used as command line argument(s).

This function is a simple wrapper around pipes.quote() which adds support for quoting sequences of strings (lists and tuples). For example the following calls are all equivalent:

>>> from executor import quote
>>> quote('echo', 'argument with spaces')
"echo 'argument with spaces'"
>>> quote(['echo', 'argument with spaces'])
"echo 'argument with spaces'"
>>> quote(('echo', 'argument with spaces'))
"echo 'argument with spaces'"
Parameters:args – One or more strings, tuples and/or lists of strings to be quoted.
Returns:A string containing quoted command line arguments.
executor.which(program, mode=1, path=None)[source]

Find the pathname(s) of a program on the executable search path ($PATH).

Parameters:program – The name of the program (a string).
Returns:A list of pathnames (strings) with found programs.

Some examples:

>>> from executor import which
>>> which('python')
['/home/peter/.virtualenvs/executor/bin/python', '/usr/bin/python']
>>> which('vim')
['/usr/bin/vim']
>>> which('non-existing-program')
[]
executor.get_search_path(path=None)[source]

Get the executable search path ($PATH).

Parameters:path – Override the value of $PATH (a string or None).
Returns:A list of strings with pathnames of directories.

The executable search path is constructed as follows:

  1. The search path is taken from the environment variable $PATH.
  2. If $PATH isn’t defined the value of os.defpath is used.
  3. The search path is split on os.pathsep to get a list.
  4. On Windows the current directory is prepended to the list.
  5. Duplicate directories are removed from the list.
executor.get_path_extensions(extensions=None)[source]

Get the executable search path extensions ($PATHEXT).

Returns:A list of strings with unique path extensions (on Windows) or a list containing an empty string (on other platforms).
executor.is_executable(filename, mode=1)[source]

Check whether the given file is executable.

Parameters:filename – A relative or absolute pathname (a string).
Returns:True if the file is executable, False otherwise.
executor.validate_ionice_class(value)[source]

Ensure that the given value is a valid I/O scheduling class for ionice.

Parameters:value – The value to validate (a string).
Returns:The validated value (one of the strings ‘idle’, ‘best-effort’ or ‘realtime’).
Raises:ValueError when the given value isn’t one of the strings mentioned above.
exception executor.ExternalCommandFailed(command, **options)[source]

Raised when an external command exits with a nonzero status code.

This exception is raised by execute(), start() and wait() when an external command exits with a nonzero status code.

__init__(command, **options)[source]

Initialize an ExternalCommandFailed object.

Parameters:
command[source]

The ExternalCommand object that triggered the exception.

pool[source]

The CommandPool object that triggered the exception.

This property will be None when the exception wasn’t raised from a command pool.

returncode

Shortcut for the external command’s returncode.

error_message[source]

An error message explaining what went wrong (a string).

Defaults to error_message but can be overridden using the keyword argument of the same name to __init__().

exception executor.CommandNotFound(command, **options)[source]

Raised when an external command is not available on the system.

This exception is raised by execute(), start() and wait() when an external command can’t be started because the command isn’t available.

It inherits from ExternalCommandFailed to enable uniform error handling but it also inherits from OSError for backwards compatibility (see errno and strerror).

errno

The numeric error code ENOENT from errno (an integer).

strerror

The text corresponding to errno (a string).

The executor.chroot module

Simple command execution in chroot environments.

The executor.chroot module defines the ChangeRootCommand class which makes it easy to run commands inside chroots.

Warning

This is low level functionality. This module performs absolutely no chroot initialization, for example /etc/resolv.conf may be incorrect and there won’t be any bind mounts available in the chroot (unless you’ve prepared them yourself).

If you need your chroot to be initialized for you then consider using the executor.schroot module instead. It takes a bit of time to set up schroot but it provides a more high level experience than chroot.

executor.chroot.CHROOT_PROGRAM_NAME = 'chroot'

The name of the chroot_ program (a string).

class executor.chroot.ChangeRootCommand(*args, **options)[source]

ChangeRootCommand objects use the chroot_ program to execute commands inside chroots.

__init__(*args, **options)[source]

Initialize a ChangeRootCommand object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the ExternalCommand class.
  • options – Any keyword arguments are passed on to the initializer of the ExternalCommand class.

If the keyword argument chroot isn’t given but positional arguments are provided, the first positional argument is used to set the chroot property.

The command is not started until you call start() or wait().

chroot[source]

The pathname of the root directory of the chroot (a string).

Note

The chroot property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

chroot_command[source]

The command used to run the chroot program.

This is a list of strings, by default the list contains just CHROOT_PROGRAM_NAME. The chroot, chroot_group and chroot_user properties also influence the chroot command line used.

Note

The chroot_command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_directory[source]

The working directory _inside the chroot_ (a string, defaults to None).

Note

The chroot_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_group[source]

The name or ID of the system group that runs the command (a string or number, defaults to ‘root’).

Note

The chroot_group property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_user[source]

The name or ID of the system user that runs the command (a string or number, defaults to ‘root’).

Note

The chroot_user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The complete chroot command including the command to run inside the chroot.

This is a list of strings with the chroot command line to enter the requested chroot and execute command.

directory

Set the working directory inside the chroot.

When you set this property you change chroot_directory, however reading back the property you’ll just get DEFAULT_WORKING_DIRECTORY. This is because the superclass ExternalCommand uses directory as the working directory for the chroot command, and directories inside chroots aren’t guaranteed to exist on the host system.

have_superuser_privileges

Whether the command inside the chroot will be running under superuser privileges.

This is True when chroot_user is the number 0 or the string ‘root’, False otherwise.

This overrides ExternalCommand.have_superuser_privileges to ensure that sudo isn’t used inside the chroot unless it’s really necessary. This is important because not all chroot environments have sudo installed and failing due to a lack of sudo when we’re already running as root is just stupid :-).

The executor.cli module

Usage: executor [OPTIONS] COMMAND ...

Easy subprocess management on the command line based on the Python package with the same name. The “executor” program runs external commands with support for timeouts, dynamic startup delay (fudge factor) and exclusive locking.

You can think of “executor” as a combination of the “flock” and “timelimit” programs with some additional niceties (namely the dynamic startup delay and integrated system logging on UNIX platforms).

Supported options:

Option Description
-t, --timeout=LIMIT Set the time after which the given command will be aborted. By default LIMIT is counted in seconds. You can also use one of the suffixes “s” (seconds), “m” (minutes), “h” (hours) or “d” (days).
-f, --fudge-factor=LIMIT This option controls the dynamic startup delay (fudge factor) which is useful when you want a periodic task to run once per given interval but the exact time is not important. Refer to the --timeout option for acceptable values of LIMIT, this number specifies the maximum amount of time to sleep before running the command (the minimum is zero, otherwise you could just include the command “sleep N && ...” in your command line :-).
-e, --exclusive Use an interprocess lock file to guarantee that executor will never run the external command concurrently. Refer to the --lock-timeout option to customize blocking / non-blocking behavior. To customize the name of the lock file you can use the --lock-file option.
-T, --lock-timeout=LIMIT By default executor tries to claim the lock and if it fails it will exit with a nonzero exit code. This option can be used to enable blocking behavior. Refer to the --timeout option for acceptable values of LIMIT.
-l, --lock-file=NAME Customize the name of the lock file. By default this is the base name of the external command, so if you’re running something generic like “bash” or “python” you might want to change this :-).
-v, --verbose Make more noise than usual (increase logging verbosity).
-q, --quiet Make less noise than usual (decrease logging verbosity).
-h, --help Show this message and exit.
executor.cli.LOCKS_DIRECTORY = '/var/lock'

The pathname of the preferred directory for lock files (a string).

Refer to get_lock_path() for more details.

executor.cli.INTERRUPT_FILE = 'executor-fudge-factor-interrupt'

The base name of the file used to interrupt the fudge factor (a string).

executor.cli.main()[source]

Command line interface for the executor program.

executor.cli.apply_fudge_factor(fudge_factor)[source]

Apply the requested scheduling fudge factor.

Parameters:fudge_factor – The maximum number of seconds to sleep (a number).

Previous implementations of the fudge factor interrupt used UNIX signals (specifically SIGUSR1) but the use of this signal turned out to be sensitive to awkward race conditions and it wasn’t very cross platform, so now the creation of a regular file is used to interrupt the fudge factor.

executor.cli.get_lock_path(lock_name)[source]

Get a pathname that can be used for an interprocess lock.

Parameters:lock_name – The base name for the lock file (a string).
Returns:An absolute pathname (a string).
executor.cli.run_command(arguments, timeout=None)[source]

Run the specified command (with an optional timeout).

Parameters:
  • arguments – The command line for the external command (a list of strings).
  • timeout – The optional command timeout (a number or None).
Raises:

CommandTimedOut if the command times out.

exception executor.cli.CommandTimedOut(command, timeout)[source]

Raised when a command exceeds the given timeout.

__init__(command, timeout)[source]

Initialize a CommandTimedOut object.

Parameters:
  • command – The command that timed out (an ExternalCommand object).
  • timeout – The timeout that was exceeded (a number).

The executor.concurrent module

Support for concurrent external command execution.

The executor.concurrent module defines the CommandPool class which makes it easy to prepare a large number of external commands, group them together in a pool, start executing a configurable number of external commands simultaneously and wait for all external commands to finish. For fine grained concurrency control please refer to the dependencies and group_by properties of the ExternalCommand class.

class executor.concurrent.CommandPool(concurrency=None, **options)[source]

Execute multiple external commands concurrently.

After constructing a CommandPool instance you add commands to it using add() and when you’re ready to run the commands you call run().

__init__(concurrency=None, **options)[source]

Initialize a CommandPool object.

Parameters:
concurrency[source]

The number of external commands that the pool is allowed to run simultaneously.

This is a positive integer number. It defaults to the return value of multiprocessing.cpu_count() (which may not make much sense if your commands are I/O bound instead of CPU bound).

Setting concurrency to one is a supported use case intended to make it easier for users of the executor.concurrent module to reuse the code they’ve built on top of command pools (if only for debugging, but there are lots of use cases :-).

Note

The concurrency property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

delay_checks[source]

Whether to postpone raising an exception until all commands have run (a boolean).

If this option is True (not the default) and a command with check set to True fails the command pool’s execution is not aborted, instead all commands will be allowed to run. After all commands have finished a CommandPoolFailed exception will be raised that tells you which command(s) failed.

Note

The delay_checks property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

logger[source]

The logging.Logger object to use.

If you are using Python’s logging module and you find it confusing that command pool execution is logged under the executor.concurrent name space instead of the name space of the application or library using executor you can set this attribute to inject a custom (and more appropriate) logger.

Note

The logger property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

logs_directory[source]

The pathname of a directory where captured output is stored (a string).

If this property is set to the pathname of a directory (before any external commands have been started) the merged output of each external command is captured and stored in a log file in this directory. The directory will be created if it doesn’t exist yet.

Output will start appearing in the log files before the external commands are finished, this enables tail -f to inspect the progress of commands that are still running and emitting output.

Note

The logs_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

is_finished

True if all commands in the pool have finished, False otherwise.

num_commands

The number of commands in the pool (an integer).

num_finished

The number of commands in the pool that have already finished (an integer).

num_failed

The number of commands in the pool that failed (an integer).

num_running

The number of currently running commands in the pool (an integer).

running_groups

A set of running command groups.

The value of running_groups is a set with the group_by values of all currently running commands (None is never included in the set).

results

A mapping of identifiers to external command objects.

This is a dictionary with external command identifiers as keys (refer to add()) and ExternalCommand objects as values. The ExternalCommand objects provide access to the return codes and/or output of the finished commands.

spinner[source]

Whether to show an animated spinner or not (a boolean or None).

The value of spinner defaults to None which means a spinner is shown when we’re connected to a terminal but hidden when we’re not connected to a terminal.

Note

The spinner property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

unexpected_failures

A list of ExternalCommand objects that failed unexpectedly.

The resulting list includes only commands where check and failed are both True.

add(command, identifier=None, log_file=None)[source]

Add an external command to the pool of commands.

Parameters:
  • command – The external command to add to the pool (an ExternalCommand object).
  • identifier – A unique identifier for the external command (any value). When this parameter is not provided the identifier is set to the number of commands in the pool plus one (i.e. the first command gets id 1).
  • log_file – Override the default log file name for the command (the identifier with .log appended) in case logs_directory is set.

When a command is added to a command pool the following options are changed automatically:

  • The async property is set to True. If you want the commands to execute with a concurrency of one then you should set concurrency to one.
  • The tty property is set to False when concurrency is higher than one because interaction with multiple concurrent subprocesses in a single terminal is prone to serious miscommunication (when multiple subprocesses present an interactive prompt at the same time and the user tries to answer one of the prompts it will be impossible to tell which of the subprocesses will receive the user’s reply).
run()[source]

Keep spawning commands and collecting results until all commands have run.

Returns:The value of results.
Raises:Any exceptions raised by collect().

This method calls spawn() and collect() in a loop until all commands registered using add() have run and finished. If collect() raises an exception any running commands are terminated before the exception is propagated to the caller.

If you’re writing code where you want to own the main loop then consider calling spawn() and collect() directly instead of using run().

When concurrency is set to one, specific care is taken to make sure that the callbacks configured by start_event and finish_event are called in the expected (intuitive) order.

spawn()[source]

Spawn additional external commands up to the concurrency level.

Returns:The number of external commands that were spawned by this invocation of spawn() (an integer).

The commands to start are picked according to three criteria:

  1. The command’s was_started property is False.
  2. The command’s group_by value is not present in running_groups.
  3. The is_finished properties of all of the command’s dependencies are True.
collect()[source]

Collect the exit codes and output of finished commands.

Returns:

The number of external commands that were collected by this invocation of collect() (an integer).

Raises:
If delay_checks is True:

After all external commands have started and finished, if any commands that have check set to True failed CommandPoolFailed is raised.

If delay_checks is False:

The exceptions ExternalCommandFailed, RemoteCommandFailed and RemoteConnectFailed can be raised if a command in the pool that has check set to True fails. The pool attribute of the exception will be set to the pool.

Warning

If an exception is raised, commands that are still running will not be terminated! If this concerns you then consider calling terminate() from a finally block (this is what run() does).

terminate()[source]

Terminate any commands that are currently running.

Returns:The number of commands that were terminated (an integer).

If terminate() successfully terminates commands, you then call collect() and the check property of a terminated command is True you will get an exception because terminated commands (by definition) report a nonzero returncode.

exception executor.concurrent.CommandPoolFailed(pool)[source]

Raised by collect() when not all commands succeeded.

This exception is only raised when delay_checks is True.

__init__(pool)[source]

Initialize a CommandPoolFailed object.

Parameters:pool – The CommandPool object that triggered the exception.
commands

A shortcut for unexpected_failures.

error_message

An error message that explains which commands failed unexpectedly (a string).

The executor.contexts module

Dependency injection for command execution contexts.

The contexts module defines the LocalContext, RemoteContext and SecureChangeRootContext classes. All of these classes support the same API for executing external commands, they are simple wrappers for ExternalCommand, RemoteCommand and SecureChangeRootCommand.

This allows you to script interaction with external commands in Python and perform that interaction on your local system, on a remote system over SSH or inside a chroot_ using the exact same Python code. Dependency injection on steroids anyone? :-)

Here’s a simple example:

from executor.contexts import LocalContext, RemoteContext
from humanfriendly import format_timespan

def details_about_system(context):
    return "\n".join([
        "Information about %s:" % context,
        " - Host name: %s" % context.capture('hostname', '--fqdn'),
        " - Uptime: %s" % format_timespan(float(context.capture('cat', '/proc/uptime').split()[0])),
    ])

print(details_about_system(LocalContext()))

# Information about local system (peter-macbook):
#  - Host name: peter-macbook
#  - Uptime: 1 week, 3 days and 10 hours

print(details_about_system(RemoteContext('file-server')))

# Information about remote system (file-server):
#  - Host name: file-server
#  - Uptime: 18 weeks, 3 days and 4 hours

Whether this functionality looks exciting or horrible I’ll leave up to your judgment. I created it because I’m always building “tools that help me build tools” and this functionality enables me to very rapidly prototype system integration tools developed using Python:

During development:
I write code on my workstation which I prefer because of the “rich editing environment” but I run the code against a remote system over SSH (a backup server, database server, hypervisor, mail server, etc.).
In production:
I change one line of code to inject a LocalContext object instead of a RemoteContext object, I install the executor package and the code I wrote on the remote system and I’m done!
executor.contexts.create_context(**options)[source]

Create an execution context.

Parameters:options – Any keyword arguments are passed on to the context’s initializer.
Returns:A LocalContext, SecureChangeRootContext or RemoteContext object.

This function provides an easy to use shortcut for constructing context objects:

class executor.contexts.AbstractContext(*args, **options)[source]

Abstract base class for shared logic of all context classes.

The most useful methods of this class are execute(), test(), capture(), cleanup(), start_interactive_shell(), read_file() and write_file().

__init__(*args, **options)[source]

Initialize an AbstractContext object.

Parameters:
  • args – Any positional arguments are passed on to the initializer of the PropertyManager class (for future extensibility).
  • options

    The keyword arguments are handled as follows:

    • Keyword arguments whose name matches a property of the context object are used to set that property (by passing them to the initializer of the PropertyManager class).
    • Any other keyword arguments are collected into the options dictionary.
command_type[source]

The type of command objects created by this context (ExternalCommand or a subclass).

Note

The command_type property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named command_type (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

options[source]

The options that are passed to commands created by the context (a dictionary).

Note

The options property is a writable_property. You can change the value of this property using normal attribute assignment syntax.

parent[source]

The parent context (a context object or None).

The parent property (and the code in prepare_command() that uses the parent property) enables the use of “nested contexts”.

For example find_chroots() creates SecureChangeRootContext objects whose parent is set to the context that found the chroots. Because of this the SecureChangeRootContext objects can be used to create commands without knowing or caring whether the chroots reside on the local system or on a remote system accessed via SSH.

Warning

Support for parent contexts was introduced in executor version 15 and for now this feature is considered experimental and subject to change. While I’m definitely convinced of the usefulness of nested contexts I’m not happy with the current implementation at all. The most important reason for this is that it’s very surprising (and not in a good way) that a context with a parent will create commands with the parent’s command_type instead of the expected type.

Note

The parent property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

get_options()[source]

Get the options that are passed to commands created by the context.

Returns:A dictionary of command options.

By default this method simply returns the options dictionary, however the purpose of get_options() is to enable subclasses to customize the options passed to commands on the fly.

merge_options(overrides)[source]

Merge default options and overrides into a single dictionary.

Parameters:overrides – A dictionary with any keyword arguments given to execute() or start_interactive_shell().
Returns:The dictionary with overrides, but any keyword arguments given to the initializer of AbstractContext that are not set in the overrides are set to the value of the initializer argument.

The ionice option is automatically unset when have_ionice is False, regardless of whether the option was set from defaults or overrides.

prepare_command(command, options)[source]

Create a command_type object based on options.

Parameters:
  • command – A tuple of strings (the positional arguments to the initializer of the command_type class).
  • options – A dictionary (the keyword arguments to the initializer of the command_type class).
Returns:

A command_type object that hasn’t been started yet.

prepare_interactive_shell(options)[source]

Create a command_type object that starts an interactive shell.

Parameters:options – A dictionary (the keyword arguments to the initializer of the command_type class).
Returns:A command_type object that hasn’t been started yet.
prepare(*command, **options)[source]

Prepare to execute an external command in the current context.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The command_type object.

Note

After constructing a command_type object this method doesn’t call start() which means you control if and when the command is started. This can be useful to prepare a large batch of commands and execute them concurrently using a CommandPool.

execute(*command, **options)[source]

Execute an external command in the current context.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The command_type object.

Note

After constructing a command_type object this method calls start() on the command before returning it to the caller, so by the time the caller gets the command object a synchronous command will have already ended. Asynchronous commands don’t have this limitation of course.

test(*command, **options)[source]

Execute an external command in the current context and get its status.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The value of ExternalCommand.succeeded.

This method automatically sets check to False and silent to True.

capture(*command, **options)[source]

Execute an external command in the current context and capture its output.

Parameters:
  • command – All positional arguments are passed on to the initializer of the command_type class.
  • options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:

The value of ExternalCommand.output.

cleanup(*args, **kw)[source]

Register an action to be performed before the context ends.

Parameters:
  • args – The external command to execute or callable to invoke.
  • kw – Options to the command or keyword arguments to the callable.
Raises:

ValueError when cleanup() is called outside a with statement.

This method registers the intent to perform an action just before the context ends. To actually perform the action(s) you need to use (the subclass of) the AbstractContext object as a context manager using the with statement.

The last action that is registered is the first one to be performed. This gives the equivalent functionality of a deeply nested try / finally structure without actually having to write such ugly code :-).

The handling of arguments in cleanup() depends on the type of the first positional argument:

  • If the first positional argument is a string, the positional arguments and keyword arguments are passed on to the initializer of the command_type class to execute an external command just before the context ends.
  • If the first positional argument is a callable, it is called with any remaining positional arguments and keyword arguments before the context ends.

Warning

If a cleanup command fails and raises an exception no further cleanup commands are executed. If you don’t care if a specific cleanup command reports an error, set its check property to False.

start_interactive_shell(**options)[source]

Start an interactive shell in the current context.

Parameters:options – All keyword arguments are passed on to the initializer of the command_type class.
Returns:The command_type object.

Note

After constructing a command_type object this method calls start() on the command before returning it to the caller, so by the time the caller gets the command object a synchronous command will have already ended. Asynchronous commands don’t have this limitation of course.

distributor_id[source]

The distributor ID of the system (a lowercased string like debian or ubuntu).

This is the lowercased output of lsb_release --short --id.

Note

The distributor_id property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

distribution_codename[source]

The code name of the system’s distribution (a lowercased string like precise or trusty).

This is the lowercased output of lsb_release --short --codename.

Note

The distribution_codename property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

have_ionice[source]

True when ionice is installed, False otherwise.

Note

The have_ionice property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

have_superuser_privileges

True if the context has superuser privileges, False otherwise.

cpu_count

The number of CPUs in the system (an integer).

Note

This is an abstract property that must be implemented by subclasses.

exists(pathname)[source]

Check whether the given pathname exists.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname exists, False otherwise.

This is a shortcut for the test -e ... command.

is_file(pathname)[source]

Check whether the given pathname points to an existing file.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname points to an existing file, False otherwise.

This is a shortcut for the test -f ... command.

is_directory(pathname)[source]

Check whether the given pathname points to an existing directory.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname points to an existing directory, False otherwise.

This is a shortcut for the test -d ... command.

is_readable(pathname)[source]

Check whether the given pathname exists and is readable.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname exists and is readable, False otherwise.

This is a shortcut for the test -r ... command.

is_writable(pathname)[source]

Check whether the given pathname exists and is writable.

Parameters:pathname – The pathname to check (a string).
Returns:True if the pathname exists and is writable, False otherwise.

This is a shortcut for the test -w ... command.

read_file(filename)[source]

Read the contents of a file.

Parameters:filename – The pathname of the file to read (a string).
Returns:The contents of the file (a byte string).

This method uses cat to read the contents of files so that options like sudo are respected (regardless of whether we’re dealing with a LocalContext or RemoteContext).

write_file(filename, contents)[source]

Change the contents of a file.

Parameters:
  • filename – The pathname of the file to write (a string).
  • contents – The contents to write to the file (a byte string).

This method uses a combination of cat and output redirection to change the contents of files so that options like sudo are respected (regardless of whether we’re dealing with a LocalContext or RemoteContext). Due to the use of cat this method will create files that don’t exist yet, assuming the directory containing the file already exists and the context provides permission to write to the directory.

atomic_write(*args, **kwds)[source]

Create or update the contents of a file atomically.

Parameters:filename – The pathname of the file to create/update (a string).
Returns:A context manager (see the with keyword) that returns a single string which is the pathname of the temporary file where the contents should be written to initially.

If an exception is raised from the with block and the temporary file exists, an attempt will be made to remove it but failure to do so will be silenced instead of propagated (to avoid obscuring the original exception).

The temporary file is created in the same directory as the real file, but a dot is prefixed to the name (making it a hidden file) and the suffix ‘.tmp-‘ followed by a random integer number is used.

list_entries(directory)[source]

List the entries in a directory.

Parameters:directory – The pathname of the directory (a string).
Returns:A list of strings with the names of the directory entries.

This method uses find -mindepth 1 -maxdepth 1 -print0 to list directory entries instead of going for the more obvious choice ls -A1 because find enables more reliable parsing of command output (with regards to whitespace).

find_program(program_name, *args)[source]

Find the absolute pathname(s) of one or more programs.

Parameters:program_name – Each of the positional arguments is expected to be a string containing the name of a program to search for in the $PATH. At least one is required.
Returns:A list of strings with absolute pathnames.

This method is a simple wrapper around which.

find_chroots(namespace='chroot')[source]

Find the chroots available in the current context.

Parameters:namespace – The chroot namespace to look for (a string, defaults to DEFAULT_NAMESPACE). Refer to the schroot documentation for more information about chroot namespaces.
Returns:A generator of SecureChangeRootContext objects whose parent is set to the context where the chroots were found.
Raises:ExternalCommandFailed (or a subclass) when the schroot program isn’t installed or the schroot --list command fails.
__enter__()[source]

Initialize a new “undo stack” (refer to cleanup()).

__exit__(exc_type=None, exc_value=None, traceback=None)[source]

Execute any commands on the “undo stack” (refer to cleanup()).

class executor.contexts.LocalContext(*args, **options)[source]

Context for executing commands on the local system.

Please refer to the base class AbstractContext for details about initialization of LocalContext objects.

command_type

The type of command objects created by this context (ExternalCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed using multiprocessing.cpu_count().

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

__str__()[source]

Render a human friendly string representation of the context.

class executor.contexts.ChangeRootContext(*args, **options)[source]

Context for executing commands in change roots using chroot_.

__init__(*args, **options)[source]

Initialize a ChangeRootContext object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the AbstractContext class (for future extensibility).
  • options – Any keyword arguments are passed on to the initializer of the AbstractContext class.

If the keyword argument chroot isn’t given but positional arguments are provided, the first positional argument is used to set the chroot property.

chroot[source]

The pathname of the root directory of the chroot (a string).

Note

The chroot property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

command_type

The type of command objects created by this context (ChangeRootCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed using multiprocessing.cpu_count().

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

get_options()[source]

The options including chroot.

__str__()[source]

Render a human friendly string representation of the context.

class executor.contexts.SecureChangeRootContext(*args, **options)[source]

Context for executing commands in change roots using schroot.

__init__(*args, **options)[source]

Initialize a SecureChangeRootContext object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the AbstractContext class (for future extensibility).
  • options – Any keyword arguments are passed on to the initializer of the AbstractContext class.

If the keyword argument chroot_name isn’t given but positional arguments are provided, the first positional argument is used to set the chroot_name property.

chroot_name[source]

The name of a chroot managed by schroot (a string).

Note

The chroot_name property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot_name (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

command_type

The type of command objects created by this context (SecureChangeRootCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed using multiprocessing.cpu_count().

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

get_options()[source]

The options including chroot_name.

__str__()[source]

Render a human friendly string representation of the context.

class executor.contexts.RemoteContext(*args, **options)[source]

Context for executing commands on a remote system over SSH.

Please refer to the base classes RemoteAccount and AbstractContext for details about initialization of RemoteContext objects.

command_type

The type of command objects created by this context (RemoteCommand).

cpu_count[source]

The number of CPUs in the system (an integer).

This property’s value is computed by executing the remote command nproc. If that command fails cpu_count falls back to the command grep -ci '^processor\s*:' /proc/cpuinfo.

Note

The cpu_count property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

get_options()[source]

The options including the SSH alias and remote user.

__str__()[source]

Render a human friendly string representation of the context.

The executor.process module

Portable process control functionality for the executor package.

The executor.process module defines the ControllableProcess abstract base class which enables process control features like waiting for a process to end, gracefully terminating it and forcefully killing it. The process control functionality in ControllableProcess is separated from the command execution functionality in ExternalCommand to make it possible to re-use the process control functionality in other Python packages, see for example the proc.core.Process class.

executor.process.DEFAULT_TIMEOUT = 10

The default timeout used to wait for process termination (number of seconds).

class executor.process.ControllableProcess(**kw)[source]

Abstract, portable process control functionality.

By defining a subclass of ControllableProcess and implementing the pid, command_line and is_running properties and the terminate_helper() and kill_helper() methods you get the wait_for_process(), terminate() and kill() methods for free. This decoupling has enabled me to share a lot of code between two Python projects of mine with similar goals but very different requirements:

  1. The executor package builds on top of the subprocess module in the Python standard library and strives to be as cross platform as possible. This means things like UNIX signals are not an option (although signals exist on Windows they are hardly usable). The package mostly deals with subprocess.Popen objects internally (to hide platform specific details as much as possible).
  2. The proc package exposes process information available in the Linux process information pseudo-file system available at /proc. The package mostly deals with process IDs internally. Because this is completely specialized to a UNIX environment the use of things like UNIX signals is not a problem at all.
command_line[source]

A list of strings with the command line used to start the process.

This property may be set or implemented by subclasses to enable __str__() to render a human friendly representation of a ControllableProcess object.

Note

The command_line property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

is_running

True if the process is running, False otherwise.

This property must be implemented by subclasses to enable wait_for_process(), terminate() and kill() to work properly.

logger[source]

The logging.Logger object to use (defaults to the executor.process logger).

If you are using Python’s logging module and you find it confusing that command manipulation is logged under the executor.process name space instead of the name space of the application or library using executor you can set this attribute to inject a custom (and more appropriate) logger.

Note

The logger property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

pid[source]

The process ID (a number) or None.

This property must be set or implemented by subclasses:

Note

The pid property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

wait_for_process(timeout=0, use_spinner=None)[source]

Wait until the process ends or the timeout expires.

Parameters:
  • timeout – The number of seconds to wait for the process to terminate after we’ve asked it nicely (defaults to zero which means we wait indefinitely).
  • use_spinner

    Whether or not to display an interactive spinner on the terminal (using Spinner) to explain to the user what they are waiting for:

    • True enables the spinner,
    • False disables the spinner,
    • None (the default) means the spinner is enabled when the program is connected to an interactive terminal, otherwise it’s disabled.
Returns:

A Timer object telling you how long it took to wait for the process.

terminate(wait=True, timeout=10, use_spinner=None)[source]

Gracefully terminate the process.

Parameters:
  • wait – Whether to wait for the process to end (a boolean, defaults to True).
  • timeout – The number of seconds to wait for the process to terminate after we’ve signaled it (defaults to DEFAULT_TIMEOUT). Zero means to wait indefinitely.
  • use_spinner – See the wait_for_process() documentation.
Returns:

True if the process was terminated, False otherwise.

Raises:

Any exceptions raised by terminate_helper() implementations of subclasses or kill().

This method works as follows:

  1. Signal the process to gracefully terminate itself. Processes can choose to intercept termination signals to allow for graceful termination (many UNIX daemons work like this) however the default action is to simply exit immediately.
  2. If wait is True and we’ve signaled the process, we wait for it to terminate gracefully or timeout seconds have passed (whichever comes first).
  3. If wait is True and the process is still running after timeout seconds have passed, it will be forcefully terminated using kill() (the value of timeout that was given to terminate() will be passed on to kill()).

This method does nothing when is_running is False.

terminate_helper()[source]

Request the process to gracefully terminate itself (needs to be implemented by subclasses).

kill(wait=True, timeout=10, use_spinner=None)[source]

Forcefully kill the process.

Parameters:
  • wait – Whether to wait for the process to end (a boolean, defaults to True).
  • timeout – The number of seconds to wait for the process to terminate after we’ve signaled it (defaults to DEFAULT_TIMEOUT). Zero means to wait indefinitely.
  • use_spinner – See the wait_for_process() documentation.
Returns:

True if the process was killed, False otherwise.

Raises:

This method does nothing when is_running is False.

kill_helper()[source]

Forcefully kill the process (needs to be implemented by subclasses).

__str__()[source]

Render a human friendly representation of a ControllableProcess object.

Returns:A string describing the process. Includes the process ID and the command line (when available).
exception executor.process.ProcessTerminationFailed(*args, **kw)[source]

Raised when process termination fails.

__init__(*args, **kw)[source]

Initialize a ProcessTerminationFailed object.

This method’s signature is the same as the initializer of the PropertyManager class.

process[source]

The ControllableProcess object that triggered the exception.

message[source]

An error message that explains how the process termination failed.

The executor.schroot module

Secure command execution in chroot environments.

The executor.schroot module defines the SecureChangeRootCommand class which makes it easy to run commands inside chroots that are managed using the schroot program.

executor.schroot.SCHROOT_PROGRAM_NAME = 'schroot'

The name of the schroot program (a string).

executor.schroot.DEFAULT_NAMESPACE = 'chroot'

The default chroot namespace (a string).

Refer to the schroot documentation for more information about chroot namespaces.

class executor.schroot.SecureChangeRootCommand(*args, **options)[source]

SecureChangeRootCommand objects use the schroot program to execute commands inside chroots.

__init__(*args, **options)[source]

Initialize a SecureChangeRootCommand object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the ExternalCommand class.
  • options – Any keyword arguments are passed on to the initializer of the ExternalCommand class.

If the keyword argument chroot_name isn’t given but positional arguments are provided, the first positional argument is used to set the chroot_name property.

The command is not started until you call start() or wait().

chroot_directory[source]

The working directory _inside the chroot_ (a string or None, defaults to /).

When chroot_directory is None the schroot program gets to pick the working directory inside the chroot (refer to the schroot documentation for the complete details).

For non-interactive usage (which I anticipate to be the default usage of SecureChangeRootCommand) the schroot program simply assumes that the working directory outside of the chroot also exists inside the chroot, then fails with an error message when this is not the case.

Because this isn’t a very robust default, chroot_directory defaults to / instead.

Note

The chroot_directory property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

chroot_name[source]

The name of a chroot managed by schroot (a string).

This is expected to match one of the names configured in the directory /etc/schroot/chroot.d.

Note

The chroot_name property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named chroot_name (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

chroot_user[source]

The name of the user inside the chroot to run the command as (a string or None).

This defaults to None which means to run as the current user.

Note

The chroot_user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The complete schroot command including the command to run inside the chroot.

This is a list of strings with the schroot command line to enter the requested chroot and execute command.

directory

Set the working directory inside the chroot.

When you set this property you change chroot_directory, however reading back the property you’ll just get DEFAULT_WORKING_DIRECTORY. This is because the superclass ExternalCommand uses directory as the working directory for the schroot command, and directories inside chroots aren’t guaranteed to exist on the host system.

schroot_command[source]

The command used to run the schroot program.

This is a list of strings, by default the list contains just SCHROOT_PROGRAM_NAME. The chroot_directory, chroot_name and chroot_user properties also influence the schroot command line used.

Note

The schroot_command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

The executor.ssh.client module

Remote command execution using SSH.

The executor.ssh.client module defines the RemoteCommand class and the foreach() function which make it easy to run a remote command in parallel on multiple remote hosts using SSH. The foreach() function also serves as a simple example of how to use CommandPool and RemoteCommand objects (it’s just 16 lines of code if you squint in the right way and that includes logging :-).

executor.ssh.client.DEFAULT_CONCURRENCY = 10

The default concurrency value to use for CommandPool objects created by foreach().

executor.ssh.client.DEFAULT_CONNECT_TIMEOUT = 10

The default connect_timeout value to use for RemoteCommand objects.

executor.ssh.client.SSH_PROGRAM_NAME = 'ssh'

The name of the SSH client executable (a string).

executor.ssh.client.SSH_ERROR_STATUS = 255

The exit status used by the ssh program if an error occurred (an integer).

Used by RemoteCommand.error_message and RemoteCommand.error_type to distinguish when the ssh program itself fails and when a remote command fails.

executor.ssh.client.foreach(hosts, *command, **options)[source]

Execute a command simultaneously on a group of remote hosts using SSH.

Parameters:
Returns:

The list of RemoteCommand objects constructed by foreach().

Raises:

Any of the following exceptions can be raised:

  • CommandPoolFailed if delay_checks is enabled (the default) and a command in the pool that has check enabled (the default) fails.
  • RemoteCommandFailed if delay_checks is disabled (not the default) and an SSH connection was successful but the remote command failed (the exit code of the ssh command was neither zero nor 255). Use the keyword argument check=False to disable raising of this exception.
  • RemoteConnectFailed if delay_checks is disabled (not the default) and an SSH connection failed (the exit code of the ssh command is 255). Use the keyword argument check=False to disable raising of this exception.

Note

The foreach() function enables the check and delay_checks options by default in an attempt to make it easy to do “the right thing”. My assumption here is that if you are running the same command on multiple remote hosts:

  • You definitely want to know when a remote command has failed, ideally without manually checking the succeeded property of each command.
  • Regardless of whether some remote commands fail you want to know that the command was at least executed on all hosts, otherwise your cluster of hosts will end up in a very inconsistent state.
  • If remote commands fail and an exception is raised the exception message should explain which remote commands failed.

If these assumptions are incorrect then you can use the keyword arguments check=False and/or delay_checks=False to opt out of “doing the right thing” ;-)

executor.ssh.client.remote(ssh_alias, *command, **options)[source]

Execute a remote command (similar to execute()).

Parameters:
Returns:

Refer to execute_prepared().

Raises:

RemoteCommandFailed when the command exits with a nonzero exit code (and check is True).

class executor.ssh.client.RemoteAccount(*args, **options)[source]

Trivial SSH alias parser.

This class acts as a base class for RemoteCommand and RemoteContext that provides three simple features:

[1]This enables RemoteCommand.have_superuser_privileges to know that superuser privileges are available when the caller sets ssh_alias to a value like root@host. Of course the SSH client configuration can also override the remote username without the executor package knowing about it, but at least executor will be able to use the information that it does have.
[2]This calling convention enables backwards compatibility with executor versions 14 and below which required ssh_alias to be set using the first positional argument to the initializer of the RemoteCommand class.
[3]This new calling convention provides a uniform calling convention for the initializers of the local/remote command/context classes.
__init__(*args, **options)[source]

Initialize a RemoteAccount object.

Parameters:
  • args – Positional arguments are passed on to the initializer of the PropertyManager class (for future extensibility).
  • options – Any keyword arguments are passed on to the initializer of the PropertyManager class.

If the keyword argument ssh_alias isn’t given but positional arguments are provided, the first positional argument is used to set the ssh_alias property.

ssh_alias[source]

The SSH alias of the remote host (a string).

If you set this property to a string that contains two nonempty tokens delimited by an @ character, the first token is used to set ssh_user and the second token is used to set ssh_alias. Here’s an example:

>>> from executor.ssh.client import RemoteAccount
>>> RemoteAccount('root@server')
RemoteAccount(ssh_alias='server', ssh_user='root')
>>> RemoteAccount(ssh_alias='server', ssh_user='root')
RemoteAccount(ssh_alias='server', ssh_user='root')
>>> RemoteAccount('server')
RemoteAccount(ssh_alias='server', ssh_user=None)

Note

The ssh_alias property is a required_property. You are required to provide a value for this property by calling the constructor of the class that defines the property with a keyword argument named ssh_alias (unless a custom constructor is defined, in this case please refer to the documentation of that constructor). You can change the value of this property using normal attribute assignment syntax.

ssh_user[source]

The username on the remote system (a string or None).

If the value of ssh_user is None the SSH client program gets to decide about the remote username.

Note

The ssh_user property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

class executor.ssh.client.RemoteCommand(*args, **options)[source]

RemoteCommand objects use the SSH client program to execute remote commands.

__init__(*args, **options)[source]

Initialize a RemoteCommand object.

Parameters:

The remote command is not started until you call start() or wait().

batch_mode[source]

Control the SSH client option BatchMode (a boolean, defaults to True).

The following description is quoted from man ssh_config:

If set to “yes”, passphrase/password querying will be disabled. In addition, the ServerAliveInterval option will be set to 300 seconds by default. This option is useful in scripts and other batch jobs where no user is present to supply the password, and where it is desirable to detect a broken network swiftly. The argument must be “yes” or “no”. The default is “no”.

This property defaults to True because it can get really awkward when a batch of SSH clients query for a passphrase/password on standard input at the same time.

Note

The batch_mode property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command[source]

A list of strings with the command to execute (optional).

The value of command is optional for RemoteCommand objects (as opposed to ExternalCommand objects) because the use of SSH implies a remote (interactive) shell that usually also accepts (interactive) commands as input. This means it is valid to create a remote command object without an actual remote command to execute, but with input that provides commands to execute instead.

This “feature” can be useful to control non-UNIX systems that do accept SSH connections but don’t support a conventional UNIX shell. For example, I added support for this “feature” so that I was able to send commands to Juniper routers and switches over SSH with the purpose of automating the failover of a connection between two datacenters (the resulting Python program works great and it’s much faster than I am, making all of the required changes in a couple of seconds :-).

Note

The command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

command_line

The complete SSH client command including the remote command.

This is a list of strings with the SSH client command to connect to the remote host and execute command.

connect_timeout[source]

Control the SSH client option ConnectTimeout (an integer).

The following description is quoted from man ssh_config:

Specifies the timeout (in seconds) used when connecting to the SSH server, instead of using the default system TCP timeout. This value is used only when the target is down or really unreachable, not when it refuses the connection.

Defaults to DEFAULT_CONNECT_TIMEOUT so that non-interactive SSH connections created by RemoteCommand don’t hang indefinitely when the remote system doesn’t respond properly.

Note

The connect_timeout property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

directory

Set the remote working directory.

When you set this property you change the remote working directory, however reading back the property you’ll just get DEFAULT_WORKING_DIRECTORY. This is because the superclass ExternalCommand uses directory as the local working directory for the ssh command, and a remote working directory isn’t guaranteed to also exist on the local system.

error_message[source]

A user friendly explanation of how the remote command failed (a string or None).

Note

The error_message property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

error_type[source]

An exception class applicable to the kind of failure detected or None.

RemoteConnectFailed when returncode is set and matches SSH_ERROR_STATUS, RemoteCommandFailed when returncode is set and not zero, None otherwise.

Note

The error_type property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

have_superuser_privileges

True if ssh_user is set to ‘root’, False otherwise.

There’s no easy way for RemoteCommand to determine whether any given SSH alias logs into a remote system with superuser privileges so unless ssh_user is set to ‘root’ this is always False.

identity_file[source]

The pathname of the identity file used to connect to the remote host (a string or None).

Note

The identity_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ignore_known_hosts

Whether host key checking is disabled.

This is True if host key checking is completely disabled:

If you set this to True host key checking is disabled and log_level is set to ‘error’ to silence warnings about automatically accepting host keys.

If you set this to False then known_hosts_file, log_level and strict_host_key_checking are reset to their default values.

log_level[source]

Control the SSH client option LogLevel (a string, defaults to ‘info’).

The following description is quoted from man ssh_config:

Gives the verbosity level that is used when logging messages from ssh. The possible values are: QUIET, FATAL, ERROR, INFO, VERBOSE, DEBUG, DEBUG1, DEBUG2, and DEBUG3. The default is INFO. DEBUG and DEBUG1 are equivalent. DEBUG2 and DEBUG3 each specify higher levels of verbose output.

Note

The log_level property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

ssh_command[source]

The command used to run the SSH client program.

This is a list of strings, by default the list contains just SSH_PROGRAM_NAME. The batch_mode, connect_timeout, log_level, ssh_alias and strict_host_key_checking properties also influence the SSH client command line used.

Note

The ssh_command property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

port[source]

The port number of the SSH server (defaults to None which means the SSH client program decides).

Note

The port property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

strict_host_key_checking[source]

Control the SSH client option StrictHostKeyChecking.

This property accepts the values True and False and the strings ‘yes’, ‘no’ and ‘ask’. The following description is quoted from man ssh_config:

If this flag is set to “yes”, ssh will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed. This provides maximum protection against trojan horse attacks, though it can be annoying when the /etc/ssh/ssh_known_hosts file is poorly maintained or when connections to new hosts are frequently made. This option forces the user to manually add all new hosts. If this flag is set to “no”, ssh will automatically add new host keys to the user known hosts files. If this flag is set to “ask”, new host keys will be added to the user known host files only after the user has confirmed that is what they really want to do, and ssh will refuse to connect to hosts whose host key has changed. The host keys of known hosts will be verified automatically in all cases. The argument must be “yes”, “no”, or “ask”. The default is “ask”.

This property defaults to False so that when you connect to a remote system over SSH for the first time the host key is automatically added to the user known hosts file (instead of requiring interaction). As mentioned in the quote above the host keys of known hosts are always verified (but see ignore_known_hosts).

Note

The strict_host_key_checking property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

known_hosts_file[source]

Control the SSH client option UserKnownHostsFile (a string).

The following description is quoted from man ssh_config:

Specifies one or more files to use for the user host key database, separated by whitespace. The default is ~/.ssh/known_hosts, ~/.ssh/known_hosts2.

Note

The known_hosts_file property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

class executor.ssh.client.RemoteCommandPool(concurrency=10, **options)[source]

Execute multiple remote commands concurrently.

After constructing a RemoteCommandPool instance you add commands to it using add() and when you’re ready to run the commands you call run().

Note

The only difference between CommandPool and RemoteCommandPool is the default concurrency. This may of course change in the future.

__init__(concurrency=10, **options)[source]

Initialize a RemoteCommandPool object.

Parameters:
  • concurrency – Override the value of concurrency (an integer, defaults to DEFAULT_CONCURRENCY for remote command pools).
  • options – Any additional keyword arguments are passed on to the CommandPool constructor.
exception executor.ssh.client.RemoteConnectFailed(command, **options)[source]

Raised by RemoteCommand when an SSH connection itself fails (not the remote command).

exception executor.ssh.client.RemoteCommandFailed(command, **options)[source]

Raised by RemoteCommand when a remote command executed over SSH fails.

exception executor.ssh.client.RemoteCommandNotFound(command, **options)[source]

Raised by RemoteCommand when a remote command returns COMMAND_NOT_FOUND_STATUS.

The executor.ssh.server module

OpenSSH server automation for testing.

The executor.ssh.server module defines the SSHServer class which can be used to start temporary OpenSSH servers that are isolated enough from the host system to make them usable in the executor test suite (to test remote command execution).

executor.ssh.server.SSHD_PROGRAM_NAME = 'sshd'

The name of the SSH server executable (a string).

class executor.ssh.server.EphemeralTCPServer(*command, **options)[source]

Make it easy to launch ephemeral TCP servers.

The EphemeralTCPServer class makes it easy to allocate an ephemeral port number that is not (yet) in use.

async

Ephemeral TCP servers always set ExternalCommand.async to True.

scheme[source]

A URL scheme that indicates the purpose of the ephemeral port (a string, defaults to ‘tcp’).

Note

The scheme property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

hostname[source]

The host name or IP address to connect to (a string, defaults to localhost).

Note

The hostname property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

port_number[source]

A dynamically selected port number that was not in use at the moment it was selected (an integer).

Note

The port_number property is a lazy_property. This property’s value is computed once (the first time it is accessed) and the result is cached.

connect_timeout[source]

The timeout in seconds for connection attempts (a number, defaults to 2).

Note

The connect_timeout property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

wait_timeout[source]

The timeout in seconds for wait_until_connected() (a number, defaults to 30).

Note

The wait_timeout property is a mutable_property. You can change the value of this property using normal attribute assignment syntax. To reset it to its default (computed) value you can use del or delattr().

start(**options)[source]

Start the TCP server and wait for it to start accepting connections.

Parameters:options – Any keyword arguments are passed to the start() method of the superclass.
Raises:Any exceptions raised by wait_until_connected() and/or the start() method of the superclass.

If the TCP server doesn’t start accepting connections within the configured timeout (see wait_timeout) the process will be terminated and the timeout exception will be propagated.

wait_until_connected(port_number=None)[source]

Wait until the TCP server starts accepting connections.

Parameters:port_number – The port number to check (an integer, defaults to the computed value of port_number).
Raises:TimeoutError when the SSH server isn’t fast enough to initialize.
is_connected(port_number=None)[source]

Check whether the TCP server is accepting connections.

Parameters:port_number – The port number to check (an integer, defaults to the computed value of port_number).
Returns:True if the TCP server is accepting connections, False otherwise.
render_location(scheme=None, hostname=None, port_number=None)[source]

Render a human friendly representation of an EphemeralTCPServer object.

class executor.ssh.server.SSHServer(**options)[source]

Subclass of ExternalCommand that manages a temporary SSH server.

The OpenSSH server spawned by the SSHServer class doesn’t need superuser privileges and doesn’t require any changes to /etc/passwd or /etc/shadow.

__init__(**options)[source]

Initialize an SSHServer object.

Parameters:options – All keyword arguments are passed on to executor.ExternalCommand.__init__().
temporary_directory = None

The pathname of the temporary directory used to store the files required to run the SSH server (a string).

client_key_file = None

The pathname of the generated OpenSSH client key file (a string).

config_file = None

The pathname of the generated OpenSSH server configuration file (a string).

host_key_file = None

The random port number on which the SSH server will listen (an integer).

sshd_path

The absolute pathname of SSHD_PROGRAM_NAME (a string).

client_options

The options for the OpenSSH client (required to connect with the server).

This is a dictionary of keyword arguments for RemoteCommand to make it connect with the OpenSSH server (assuming the remote command connects to an IP address in the 127.0.0.0/24 range).

start(**options)[source]

Start the SSH server and wait for it to start accepting connections.

Parameters:options – Any keyword arguments are passed to the start() method of the superclass.
Raises:Any exceptions raised by the start() method of the superclass.

The start() method automatically calls the generate_key_file() and generate_config() methods.

generate_key_file(filename)[source]

Generate a temporary host or client key for the OpenSSH server.

The start() method automatically calls generate_key_file() to generate host_key_file and client_key_file. This method uses the ssh-keygen program to generate the keys.

generate_config()[source]

Generate a configuration file for the OpenSSH server.

The start() method automatically calls generate_config().

cleanup()[source]

Clean up temporary_directory after the test server finishes.

exception executor.ssh.server.TimeoutError[source]

Raised when a TCP server doesn’t start accepting connections quickly enough.

This exception is raised by wait_until_connected() when the TCP server doesn’t start accepting connections within a reasonable time.