Changelog

The purpose of this document is to list all of the notable changes to this project. The format was inspired by Keep a Changelog. This project adheres to semantic versioning.

Release 23.2 (2020-11-19)

Enhancements:

• Enable control of spinners with foreach() using the spinner keyword (previously this would result in TypeError exceptions).

Changes to Travis CI:

• Stabilized the Python 3.5 job (testing requirements).

• Stabilized the PyPy (2.7) job (cryptography incantations).

• Reordered the build matrix by descending runtime (to optimize total runtime).

• Allow PyPy job failures without failing the complete Travis CI build, because failures in the PyPy job tend to be caused by PyPy incompatibilities.

  In fact I’ve seen hundreds of PyPy job failures on Travis CI over the years while developing my open source projects and less than 1% of these pertained to actual PyPy incompatibilities in the code bases I develop…

Release 23.1 (2020-05-14)

As requested in issue #2: Enable control over the spinner that’s shown when a command pool is active. For more details refer to the CommandPool.spinner documentation.

Release 23.0 (2020-05-14)

This release includes two notable changes:

Improved compatibility of release discovery

As reported in issue #17 official Debian Docker images don’t include the file /etc/lsb-release nor the program lsb_release and this breaks “release discovery” using the distributor_id and distribution_codename properties. This has been fixed by implementing a fall back that tries to parse the /etc/apt/sources.list file to determine the package mirror URL which is matched against the mirror URLs in MIRROR_TO_DISTRIB_MAPPING.

Note

Because this concerns a fall back the risk of regressions is small.

Fixed Unicode inconsistency on Python 2.7

It turns out that while shlex.split() on Python 2.7 accepts Unicode strings it doesn’t actually support them: The result is a list of byte strings and any values that ASCII doesn’t support result in an error.

This caused the values (but not the keys) in the dictionary provided by lsb_release_variables to become byte strings which in turn caused distributor_id and distribution_codename to become byte strings (when those properties are based on lsb_release_variables).

However when distributor_id and distribution_codename are based on the output of the lsb_release program the values become Unicode strings, so this unfortunate behavior of shlex.split() was causing rather inconsistent behavior in the executor package.

This has been fixed by applying a workaround on Python 2.7 (the text is encoded before passing it to shlex.split() and the result is decoded before use).

Note

While this concerns a small detail in the grand scheme of things it is technically backwards incompatible and version numbers are cheap, hence why this is being released as 23.0.

Release 22.0 (2020-03-02)

Maintenance release that changes the compatible Python versions.

Noteworthy changes:

• Documented support for Python 3.8.
• Dropped support for Python 2.6 and 3.4.
• Avoid cyclic dependencies in the executor.tcp module. I’ve been working on readying a new project for publication and started using the EphemeralTCPServer class in its test suite, however I ran into an unnecessary cyclic dependency that caused logging to print dramatic tracebacks (although nothing actually failed):
  • The __init__() method needed access to the ephemeral TCP port (because it is passed to the command that’s started) which implies running a whole lot of code (to pick a port that isn’t in use yet) and this code logged EphemeralTCPServer.__str__().
  • This was intended to use WaitUntilConnected.__str__() however due to incorrect superclass ordering it called ExternalCommand.__str__() instead which needs access to the command property which in turn requires __init__() to have already been run!
  • This catch-22 was broken by removing the __str__() from logging and using a newly added endpoint property instead (explicit is better than implicit).

Miscellaneous changes:

• Bumped humanfriendly to 8.0 and property-manager to 3.0 to fix deprecated imports and resolve a backwards incompatibility in the test suite (introduced by the humanfriendly 8.0 release).
• Changed Makefile to use Python 3 during development.
• Improved the ionice tests.

Release 21.3 (2018-11-17)

Merged pull request #16 that changes the ionice integration to accept the strings ‘1’, ‘2’ and ‘3’ in addition to ‘idle’, ‘best-effort’ and ‘realtime’ because busybox doesn’t support the verbose strings.

It’s still up to the caller to pick the right kind of value and I’m a bit conflicted about that because it’s creating a leaky abstraction. I may at a later point decide to add automatic translation from the verbose labels to the numeric codes (which seem to be the lowest common denominator that’s always supported) …

Release 21.2 (2018-10-11)

Enable context.read_file(..., sudo=True) and context.write_file(..., sudo=True). In fact all optional keyword arguments are supported (not just sudo) but for me the most important one is sudo=True because I strongly prefer “selective sudo” over “just run everything using sudo”.

Release 21.1.1 (2018-10-08)

Bug fix of sorts: Guard against binary data in /etc/lsb-release on Travis CI. This problem became apparent after #10 triggered some new development. Since then I created #15 to track this specific issue.

Release 21.1 (2018-10-07)

Improve compatibility with “vanilla Ubuntu 18.04 docker images” by parsing the file /etc/lsb-release when the program /usr/bin/lsb_release isn’t installed (fixes #10).

This enables the distributor_id and distribution_codename properties to work even when the /usr/bin/lsb_release program isn’t installed, by parsing the /etc/lsb-release file instead. Tested on Ubuntu 14.04, 16.04 and 18.04.

Release 21.0 (2018-10-07)

Implemented Python 3.7 compatibility.

Python 3.7 was released in June 2018 and introduced the reserved keyword async which made the definition of the ExternalCommand.async property a syntax error. Should have seen that coming 😒.

In any case, due to personal circumstances I haven’t had time for any open source programming in the past few months which meant feedback on this issue piled up in the form of issue #9 and pull requests #11 and #13:

• Pull request #11 proposed switching to _async.
• Pull request #13 proposed switching to asynchronous.

Apart from the naming difference both pull requests represented the same change, however I prefer asynchronous over _async because I have a strong dislike for leading and trailing underscores that have no semantic value except to avoid using a reserved keyword (I’m looking at you SQLAlchemy 😛).

There was one thing that bugged me about all of this though: While it was clear that ExternalCommand.async needed to be renamed I didn’t feel like breaking backwards compatibility with lots of existing Python 2 code using executor with the old async naming. That’s why I’ve updated the code to programatically add an async alias that defers to the real asynchronous property. Because this is done using the setattr() function no reserved keywords are harmed in the process 😇.

I’ve also added Python 3.7 to the supported and tested Python releases.

Release 20.0.1 (2018-10-07)

• Bug fix: Merged pull request #14 to make ionice_command compatible with older ionice versions not supporting the --class option.
• Lots of commit noise to debug Python 2.6 support on Travis CI. I’m not sure why I still bother…

Release 20.0 (2018-05-21)

While intended to be fully backwards compatible (because the new behavior is opt-in) I decided to bump the major version number in this release because adding retry support touched on some of the most critical pieces of code in this project.

• Experimental support for retrying of commands that fail. Retrying of asynchronous commands is only supported in the context of command pools.
• Bug fix: Pass keyword arguments of wait() to wait_for_process().
• Fix Sphinx warnings (mostly broken references).

Notes about retry support

I’ve been wanting to add retry support to executor for quite a while now. One thing that I struggled with until recently was how to support retrying of synchronous and asynchronous commands in a way that made sense for both types of commands, without compromising too much on the simplicity of the Python API or the actual implementation code.

In a pragmatic “just implement something and see how it works” moment I decided to add support for retrying of synchronous commands to the ExternalCommand class while requiring the use of a command pool to retry asynchronous commands. Although this implementation doesn’t cover every possible use case I do believe it covers the most important use cases. Some high-level implementation notes:

• Synchronous commands are retried inside of the start() method. The second part of this method was extracted into a new start_once() method and then a loop was added to start() that calls start_once() until the command succeeds.
• Asynchronous commands allow for retry behavior to be configured but won’t actually run a command more than once unless used in the context of command pools. I did experiment with retrying of asynchronous commands inside the wait() method but this ended up creating an API whose behavior was very unintuitive (changing its behavior from non blocking to blocking in order to retry on failure).

Release 19.3 (2018-05-04)

• Added SecureTunnel class for easy to use SSH tunnels (ssh -NL ...).
• Added RemoteCommand.compression property to enable ssh -C.
• Extracted generic TCP functionality from the executor.ssh.server module into a new executor.tcp module (so that the functionality could be reused by the new SSH tunnel support).

Release 19.2 (2018-04-27)

• Added a glob() method to contexts (this was triggered by the feature request in rotate-backups issue #10).
• Improved documentation using property_manager.sphinx.
• Added this changelog, restructured the online documentation.
• Include documentation in source distributions.
• Added license key to setup.py script.

Release 19.1 (2018-03-25)

Added context.is_executable() shortcut.

Release 19.0 (2018-02-25)

Backwards incompatible: Report command output on failure.

Refer to the new really_silent property for details about how this is backwards incompatible. I suspect this to bite less than 1% of use cases and I want executor to have sane defaults, so there :-).

Release 18.1 (2018-01-21)

• Enable runtime processing of stdin/stdout/stderr (#7).
• Enable iteration over lines of text in output (related to #7).
• Changed the Sphinx documentation theme.
• Fixed a broken reStructuredText reference.

Release 18.0 (2017-06-28)

Several backwards incompatible changes were made in an attempt to improve the consistency of error handling:

• Bug fix: Set returncode on OSError exception
• Bug fix: Don’t leave std{out,err} unset on OSError
• Don’t raise exceptions from lsb_release shortcuts.
• Update usage in readme.
• Move test helpers to humanfriendly.testing.

Release 17.1 (2017-06-21)

Added support for Python callbacks in context.cleanup().

Release 17.0 (2017-06-10)

• Rename ChangeRoot* to SecureChangeRoot* to avoid an upcoming name collision (backwards incompatible!).
• Added support for command execution in chroots using the chroot command.
• Reduced code duplication of && logic.

Release 16.1 (2017-06-08)

• Give contexts some lsb_release shortcuts.
• Add Python 3.6 to tested versions.

Release 16.0.1 (2017-04-13)

Bug fix: Allow explicitly setting ionice=None.

Release 16.0 (2017-04-13)

• Make it very easy to use ionice.
• Add simple wrapper for which (context.find_program()).
• Avoid nested shell in context.prepare_interactive_shell().
• Don’t add trailing -- in ChangeRootCommand.command_line.
• Change default working directory in chroots (backwards incompatible, although I wouldn’t be surprised if there are zero uses of the executor.schroot module outside of the code bases I maintain :-).

Release 15.1 (2017-01-10)

• Merged pull request #3: Allow disabling of spinners.
• Bug fix: Stop timer used by wait_for_process() after waiting.
• Bumped humanfriendly requirement for upstream bug fix.

Release 15.0 (2016-12-20)

• Added support for command execution in chroots using schroot.
• Added experimental support for nested contexts.

Release 14.1 (2016-10-12)

Added support for atomic file writes using execution contexts.

Release 14.0 (2016-08-10)

Enable passing shell commands via stdin without specifying a command. Strictly speaking this change is not backwards compatible but my impression is that this won’t break any valid, existing use cases.

Release 13.0 (2016-07-09)

Improve concurrency control for command pools

Previously there was only CommandPool.concurrency to control how many commands were allowed to run concurrently, now the caller can control which commands are allowed to run concurrently (using the two new properties ExternalCommand.dependencies and group_by).

Release 12.0 (2016-07-09)

Connect stdin to /dev/null in command pools (backwards incompatible!)

Recently I ran into some spectacularly weird failures and it took me a while to realize that it was happening because a command pool with SSH client commands was running multiple SSH clients concurrently and each of the SSH clients was allocating a pseudo-tty (ssh -t).

I’m currently under the impression that this new behavior is the only sane choice, even if it is backwards incompatible. Here’s hoping I thought that through well enough before releasing this change :-).

Release 11.0.1 (2016-07-09)

• Bug fix: Allow assignment of individual environment variables.
• Refactored makefile and setup.py script (checkers, docs, wheels, twine, etc).

Release 11.0 (2016-06-03)

Connect stdin to /dev/null when tty=False (backwards incompatible!)

Recently I ran into several external commands whose output was being captured and thus not visible, but which nevertheless rendered an interactive prompt, waiting for a response on standard input (which I wasn’t providing because I never saw the interactive prompt :-). The option to connect stdin and /dev/null was never available in executor, however given the recent addition of the tty option it seemed logical to combine the two.

Two changes in this commit backwards incompatible:

1. The standard input stream of external commands was never connected to /dev/null before and this is changing without an explicit opt-in or opt-out mechanism. I’m making this choice because I believe it to be the only sane approach.
2. The interface of the CachedStream class has changed even though this is a documented, externally available class. However I don’t actually see anyone using CachedStream outside of the executor project, so in the grand scheme of things this is a minor thing (99% of users will never even notice, I’m guessing).

Release 10.1 (2016-06-03)

Added support for start_event and finish_event callbacks.

Release 10.0 (2016-06-01)

Large refactoring concerning executor / proc separation of concerns, backwards incompatible!

In executor 7.7 the process management functionality was decoupled from external command execution in order to re-use the process management functionality in my proc package (this was integrated into proc 0.4). In retrospect I implemented this refactoring (in November ‘15) too hastily because the UNIX signal handling doesn’t belong in the executor package (it’s meant to be portable). Last weekend I decided to finally do something about this! I’m only committing this now because it took me days to clean up, stabilize, document and test the refactoring :-). A high level summary:

• All process manipulation that uses UNIX signals is being moved to the ‘proc’ package, that includes things like SIGSTOP / SIGCONT. This means that the methods ControllableProcess.suspend() and ControllableProcess.resume() are no longer available. This will break fresh installations of my ‘proc’ package until I release a new version, because I haven’t pinned the max version of dependencies I control. The new release of ‘proc’ is waiting to be uploaded though :-).

• The ‘executor’ package no longer keeps references to subprocess.Popen objects after the process has finished, to allow garbage collection. This should resolve an issue I was seeing recently when I was pushing the limits of executor command pools and ran into IOError: [Errno 24] Too many open files.

  Someone on StackOverflow with the same problem: http://stackoverflow.com/questions/6669996/python-subprocess-running-out-of-file-descriptors

  Someone on StackOverflow who knows how to fix it: http://stackoverflow.com/a/23763193/788200

  While implementing this refactoring I had a lot of trouble making sure that ExternalCommand.pid and returncode would be preserved when the subprocess reference was destroyed (it seems so obvious, but nevertheless this tripped me up). The test suite agrees with me that I got things right eventually, so here’s hoping for no external breakage :-).

Release 9.11 (2016-05-27)

Make it possible to disable command pool spinners.

Release 9.10 (2016-05-27)

ExternalCommand and RemoteCommand objects now have a tty option to express whether they need to and/or will be connected to an interactie terminal.

Release 9.9 (2016-04-21)

Bug fix: Preserve environment variables when using sudo.

Release 9.8 (2016-04-13)

Make it easy to test contexts for superuser privileges.

Release 9.7 (2016-04-09)

Added a shortcut for context creation (executor.contexts.create_context()).

Release 9.6.1 (2016-04-07)

Bug fix for previous commit.

Release 9.6 (2016-04-07)

Make remote commands optional (stdin only is a valid use case).

Release 9.5 (2016-04-03)

Provide contexts shortcuts for various test program invocations.

Release 9.4 (2016-04-03)

Automatically get the SSH username from the given SSH alias when available (delimited by an @ sign).

Release 9.3 (2016-03-22)

• Added support for listing directory entries using execution contexts.
• Stop Travis CI from testing tagged releases (I create a lot of them :-).
• Introduce context manager for temporary directories in test suite.

Release 9.2 (2016-03-22)

Improved RemoteContext.cpu_count (by adding a fallback for nproc).

Release 9.1 (2016-03-22)

Support for reading and writing of files using execution contexts.

Release 9.0.1 (2016-03-21)

Bug fix: Proper error messages for RemoteCommandNotFound.

Release 9.0 (2016-02-20)

• Backwards incompatible: Removed fakeroot → sudo fallback behavior.
• Added more documentation of the uid and user options.
• Documented tested interpreters with trove classifiers.

Release 8.4 (2016-02-20)

• Make it possible to run commands as specific users (via sudo).
• Add Python 3.5 to tested versions and document support.
• Refactored setup.py script, add trove classifiers.
• Moved Sphinx customizations to humanfriendly package.

Release 8.3 (2016-01-24)

• Make it possible to explicitly enable/disable shell evaluation.
• Expand documentation of callback/result properties.

Release 8.2 (2016-01-14)

Experimental support for ‘result processing’ callbacks.

Release 8.1.1 (2016-01-13)

Enable custom loggers for remote commands.

Release 8.1 (2016-01-13)

• Added remote() shortcut (execute() for remote commands).
• Simplified RemoteCommand.command_line.
• Improved documentation of execute() function.

Release 8.0.1 (2015-11-14)

Silence ‘make check’ (now failing on Travis CI).

Release 8.0 (2015-11-13)

• Added a command line interface: The executor program.
• Improved documentation after previous refactoring.

Release 7.7 (2015-11-10)

Better process management, decoupled from ExternalCommand.

Release 7.6 (2015-11-10)

• Automatically set async=True when used as context manager.
• Minor improvements to executor.ssh.server module.
• Improve how Sphinx generates the documentation:
  • Configure Sphinx not to skip magic methods by default.
  • Order autodoc entries by source, not alphabetically.

Release 7.5 (2015-11-08)

• Change default logger of commands executed in pools.
• Extract ephemeral TCP server support from executor.ssh.server.SSHServer.

Release 7.4 (2015-11-08)

• Decompose ExternalCommand.start().
• Introduce CommandNotFound subclass of ExternalCommandFailed.

Release 7.2 (2015-11-08)

• Decompose executor.which() and add Windows support.
• Disable capturing in pytest.ini (because it breaks sudo tests).

Release 7.1.1 (2015-10-18)

• Bug fix for integration of ExternalCommandFailed / TimeoutError exceptions.
• Improve documentation of virtual_environment option.

Release 7.1 (2015-10-18)

Make it easy to run commands in Python virtual environments.

Release 7.0.1 (2015-10-06)

Bug fix: Only raise CommandPoolFailed for commands with check=True.

Release 7.0 (2015-10-06)

foreach() now sets delay_checks=True by default.

This change is not backwards compatible but IMHO it fits in the scheme of “making it easy to do the right thing”. For further argumentation refer to the updated documentation.

Release 6.2 (2015-10-06)

Enable delayed error checking for command pools.

Release 6.1 (2015-10-05)

Tag exceptions with the command pool from which they were raised.

Release 6.0 (2015-10-05)

Make CommandPool.run() terminate commands before aborting.

This bumps the major version number because the change isn’t backwards compatible (although I believe it does make for more sane default behavior) and version numbers are cheap :-).

Release 5.1 (2015-10-05)

Make it possible to terminate command pools.

Release 5.0.1 (2015-10-05)

• Bug fix: Make CommandPool.collect() resumable after failing commands.
• Enable intersphinx mapping from executor to property-manager.
• Removed minor (trivial) code duplication from CommandPool.run().
• Renamed ‘construct’ to ‘initialize’ where applicable: A constructor in Python is called __new__() and overriding it is the exception, not the norm. Overriding the __init__() method is the norm, but then __init__() is not a constructor, it’s an “initializer”.

Release 5.0 (2015-10-04)

Promote executor.property_manager to a separate property-manager package (I’d been wanting to reuse this functionality in several other packages for a while now).

Release 4.9 (2015-10-02)

Change executor.ssh.client.foreach() to use SSH aliases as identifiers.

Release 4.8 (2015-10-02)

Change command pool output logging to append instead of overwrite.

Release 4.7 (2015-10-02)

Support capturing foreach() command pool output to logs directory.

Release 4.6 (2015-10-02)

Support capturing command pool output to logs directory.

Release 4.5 (2015-10-02)

• Bug fix: Python 3 doesn’t support ur”strings” (Unicode raw strings)
• Support redirecting standard streams to files provided by caller.
• Implement and enforce PEP-8 and PEP-257 compliance.

Release 4.4.1 (2015-08-30)

• Bug fix for obscure UnicodeDecodeError in setup.py (on Python 3 only).
• Make Travis CI builds fail when coverage isn’t >= 90%.
• Also run the tests under PyPy on Travis CI.

Release 4.4 (2015-05-30)

Expose the CPU count of execution contexts.

Release 4.3 (2015-05-30)

Give contexts a test() method.

Release 4.2 (2015-05-30)

Enable context users to prepare commands without starting them.

Release 4.1 (2015-05-29)

Make it possible to nest ‘unwind contexts’ (executor.contexts).

Release 4.0.1 (2015-05-28)

Bug fix for remote working directory logic.

Release 4.0 (2015-05-27)

Added support for external command contexts (agnostic to local vs. remote execution).

Release 3.6 (2015-05-27)

Support non-default remote working directories.

Release 3.5 (2015-05-26)

Added a RemoteCommandPool class.

Release 3.4.1 (2015-05-26)

Default to StrictHostKeyChecking=no for SSH commands.

Release 3.4 (2015-05-26)

Make the decoded values of stdout/stderr available.

Release 3.3 (2015-05-26)

Made it possible to merge the standard output and error streams.

Release 3.2 (2015-05-26)

Made it possible to capture the standard error stream.

Release 3.1 (2015-05-25)

Added ExternalCommand.succeeded and failed properties.

Release 3.0.2 (2015-05-25)

Don’t set the SSH port number to 22 by default (let the SSH client program figure it out instead).

Release 3.0.1 (2015-05-25)

Bug fix for setup.py (forgot to remove import).

Release 3.0 (2015-05-25)

• Added support for remote command execution using SSH.
• Improved ExternalCommand documentation.

Release 2.4 (2015-05-24)

Make ExternalCommand a context manager.

Release 2.3 (2015-05-24)

Made it possible to terminate external commands.

Release 2.2.2 (2015-05-24)

Improved logging output of CommandPool.run().

Release 2.2.1 (2015-05-24)

Bug fix for import error in executor.compat module.

Release 2.2 (2015-05-24)

Properly distinguish writable properties from ‘reset-able’ properties.

Release 2.1 (2015-05-23)

Added support for concurrent external command execution (command pools).

Release 2.0 (2015-05-23)

• Added support for asynchronous command execution (and lots of small things).
• Improve formatting of ExternalCommandFailed attributes in documentation.

Release 1.7.1 (2015-03-05)

Fixed __version__ variable corruption introduced in 1.7 :-S.

Release 1.7 (2015-03-05)

Make it possible to provide overrides for environment variables (#1).

Release 1.6.2 (2015-03-04)

• Stop mixing SH and Bash usage (consistently use Bash everywhere).
• Documented that the encoding option is used for input and output
• Added tox.ini for easy testing and execute tox using make test.

Release 1.6.1 (2015-03-04)

Bug fix: Properly close open file handle to /dev/null.

This fixes the following warning emitted by Python 3.4:

ResourceWarning: unclosed file <_io.BufferedWriter name='/dev/null'>

Release 1.6 (2014-10-18)

Expose pipes.quote() wrapping logic as executor.quote().

Release 1.5 (2014-10-18)

Added support for execute(..., silent=True) which silences the standard output and error streams.

Release 1.4 (2014-10-18)

• Extend ExternalCommandFailed to expose command and returncode attributes.
• Get test coverage up to 100%.
• Fixed Sphinx documentation warning about missing static directory.
• Added a simple Makefile for common project maintenance tasks.

Release 1.3 (2014-06-07)

• Added support for fakeroot.
• Added a which() function.
• Submit test coverage from Travis CI to Coveralls.

Release 1.2 (2014-05-10)

• Improved Python 3 compatibility: - Remove irregular raise syntax. - First experience with bytes vs strings.
• Documented supported Python versions (2.6, 2.7 and 3.4).
• Started using Travis CI to automatically run the test suite.

Release 1.1 (2014-05-04)

Improved the documentation.

Release 1.0 (2014-05-04)

Initial commit.