### Navigation

- index
- modules |
- next |
- previous |
- Gate One Documentation »
- Developer Documentation »

# `termio.py` - Terminal Input/Output Module

## About termio

This module provides a Multiplex class that can perform the following:

> - Fork a child process that opens a given terminal program.
> - Read and write data to and from the child process (synchronously or asynchronously).
> - Examine the output of the child process in real-time and perform actions (also asynchronously!) based on what is "expected" (aka non-blocking, pexpect-like functionality).
> - Log the output of the child process to a file and/or syslog.

The Multiplex class was built for asynchronous use in conjunction with a running `tornado.ioloop.IOLoop` instance but it can be used in a synchronous (blocking) manner as well. Synchronous use of this module is most likely to be useful in an interactive Python session but if blocking doesn't matter for your program please see the section titled, "Blocking" for tips & tricks.

Here's an example instantiating a Multiplex class:

    multiplexer = termio.Multiplex(
        'nethack',
        log_path='/var/log/myapp',
        user='bsmith@CORP',
        term_id=1,
        syslog=True
    )

Note

Support for event loops other than Tornado is in the works!

Then multiplexer can create and launch a new controlling terminal (tty) running the given command (e.g. 'nethack'):

    env = {
        'PATH': os.environ['PATH'],
        'MYVAR': 'foo'
    }
    fd = multiplexer.spawn(80, 24, env=env)
    # The fd is returned from spawn() in case you want more low-level control.

Asynchronous input and output from the controlled program is handled via IOLoop. It will automatically write all output from the terminal program to an instance of self.terminal_emulator (which defaults to Gate One's `terminal.Terminal`). So if you want to perform an action whenever the running terminal application has output (like, say, sending a message to a client) you'll need to attach a callback:

    def screen_update():
        'Called when new output is ready to send to the client'
        output = multiplexer.dump_html()
        socket_or_something.write(output)
    multiplexer.callbacks[multiplexer.CALLBACK_UPDATE] = screen_update

In this example, `screen_update()` will `write()` the output of `multiplexer.dump_html()` to socket_or_something whenever the terminal program has some sort of output. You can also make calls directly to the terminal emulator (if you're using a custom one):

    def screen_update():
        output = multiplexer.term.my_custom_func()
        whatever.write(output)

Writing characters to the controlled terminal application is pretty straightforward:

    multiplexer.write(u'some text')

Typically you'd pass in keystrokes or commands from your application to the underlying program this way and the screen/terminal emulator would get updated automatically. If using Gate One's `terminal.Terminal()` you can also attach callbacks to perform further actions when more specific situations are encountered (e.g. when the window title is set via its respective escape sequence):

    def set_title():
        'Hypothetical title-setting function'
        print("Window title was just set to: %s" % multiplexer.term.title)
    multiplexer.term.callbacks[multiplexer.CALLBACK_TITLE] = set_title

## Module Functions and Classes

`termio.``debug_expect`(m_instance, match, pattern)

  This method is used by `BaseMultiplex.expect()` if `self.debug` is True. It facilitates easy debugging of regular expressions. It will print out precisely what was matched and where.

  Note

  This function only works with post-process patterns.

`termio.``retrieve_first_frame`(golog_path)

  Retrieves the first frame from the given golog_path.

`termio.``retrieve_last_frame`(golog_path)

  Retrieves the last frame from the given golog_path. It does this by iterating over the log in reverse.

`termio.``get_or_update_metadata`(golog_path, user, force_update=False)

  Retrieves or creates/updates the metadata inside of golog_path.

  If force_update the metadata inside the golog will be updated even if it already exists.

  Note

  All logs will need "fixing" the first time they're enumerated like this since they won't have an end_date. Fortunately we only need to do this once per golog.

exception `termio.``Timeout`

  Used by `BaseMultiplex.expect()` and `BaseMultiplex.await()`; called when a timeout is reached.

exception `termio.``ProgramTerminated`

  Called when we try to write to a process that's no longer running.

class `termio.``Pattern`(pattern, callback, optional=False, sticky=False, errorback=None, preprocess=False, timeout=30)

  Used by `BaseMultiplex.expect()`, an object to store patterns (regular expressions) and their associated properties.

  Note

  The variable m_instance is used below to mean the current instance of BaseMultiplex (or a subclass thereof).

  Pattern : | A regular expression or iterable of regular expressions that will be checked against the output stream.

  Callback :

  A function that will be called when the pattern is matched. Callbacks are called like so:

      >>> callback(m_instance, matched_string)

  Tip

  If you provide a string instead of a function for your callback it will automatically be converted into a function that writes the string to the child process. Example:

      >>> p = Pattern('(?i)password:', 'mypassword\n')

  Optional : | Indicates that this pattern is optional. Meaning that it isn't required to match before the next pattern in `BaseMultiplex._patterns` is checked.
  Sticky : | Indicates that the pattern will not time out and won't be automatically removed from self._patterns when it is matched.

  Errorback :

  A function to call in the event of a timeout or if an exception is encountered. Errorback functions are called like so:

      >>> errorback(m_instance)

  Preprocess : | Indicates that this pattern is to be checked against the incoming stream before it is processed by the terminal emulator. Useful if you need to match non-printable characters like control codes and escape sequences.
  Timeout : | A `datetime.timedelta` object indicating how long we should wait before calling `errorback()`.
  Created : | A `datetime.datetime` object that gets set when the Pattern is instantiated by `BaseMultiplex.expect()`. It is used to determine if and when a timeout has been reached.

class `termio.``BaseMultiplex`(cmd, terminal_emulator=None, log_path=None, user=None, term_id=None, syslog=False, syslog_host=None, syslog_facility=None, debug=False)

  A base class that all Multiplex types will inherit from.

  Cmd : | string - The command to execute when calling `spawn()`.
  Terminal_emulator :
  | terminal.Terminal or similar - The terminal emulator to write to when capturing the incoming output stream from cmd.
  Log_path : | string - The absolute path to the log file where the output from cmd will be saved.
  Term_id : | string - The terminal identifier to associated with this instance (only used in the logs to identify terminals).
  Syslog : | boolean - Whether or not the session should be logged using the local syslog daemon.
  Syslog_host : | string - An optional syslog host to send session log information to (this is independent of the syslog option above--it does not require a syslog daemon be present on the host running Gate One).
  Syslog_facility :
  | integer - The syslog facility to use when logging messages. All possible facilities can be found in `utils.FACILITIES` (if you need a reference other than the syslog module).
  Debug : | boolean - Used by the `expect` methods... If set, extra debugging information will be output whenever a regular expression is matched.

  `add_callback`(event, callback, identifier=None)

    Attaches the given callback to the given event. If given, identifier can be used to reference this callback leter (e.g. when you want to remove it). Otherwise an identifier will be generated automatically. If the given identifier is already attached to a callback at the given event, that callback will be replaced with callback.

    event - The numeric ID of the event you're attaching callback to (e.g. Multiplex.CALLBACK_UPDATE). callback - The function you're attaching to the event. identifier - A string or number to be used as a reference point should you wish to remove or update this callback later.

    Returns the identifier of the callback. to Example:

        >>> m = Multiplex()
        >>> def somefunc(): pass
        >>> id = "myref"
        >>> ref = m.add_callback(m.CALLBACK_UPDATE, somefunc, id)

    Note

    This allows the controlling program to have multiple callbacks for the same event.

  `remove_callback`(event, identifier)

    Removes the callback referenced by identifier that is attached to the given event. Example:

        >>> m.remove_callback(m.CALLBACK_BELL, "myref")

  `remove_all_callbacks`(identifier)

    Removes all callbacks associated with identifier.

  `_call_callback`(callback)

    This method is here in the event that subclasses of `BaseMultiplex` need to call callbacks in an implementation-specific way. It just calls callback.

  `spawn`(rows=24, cols=80, env=None, em_dimensions=None)

    This method must be overridden by suclasses of `BaseMultiplex`. It is expected to execute a child process in a way that allows non-blocking reads to be performed.

  `isalive`()

    This method must be overridden by suclasses of `BaseMultiplex`. It is expected to return True if the child process is still alive and False otherwise.

  `term_write`(stream)

    Writes `stream` to `BaseMultiplex.term` and also takes care of logging to `log_path` (if set) and/or syslog (if `syslog` is `True`). When complete, will call any callbacks registered in `CALLBACK_UPDATE`.

    Stream : | A string or bytes containing the incoming output stream from the underlying terminal program.

    Note

    This kind of logging doesn't capture user keystrokes. This is intentional as we don't want passwords winding up in the logs.

  `preprocess`(stream)

    Handles preprocess patterns registered by `expect()`. That is, those patterns which have been marked with `preprocess = True`. Patterns marked in this way get handled before the terminal emulator processes the `stream`.

    Stream : | A string or bytes containing the incoming output stream from the underlying terminal program.

  `postprocess`()

    Handles post-process patterns registered by `expect()`.

  `_handle_match`(pattern_obj, match)

    Handles a matched regex detected by `postprocess()`. It calls `Pattern.callback` and takes care of removing it from `_patterns` (if it isn't sticky).

  `writeline`(line='')

    Just like `write()` but it writes a newline after writing line.

    If no line is given a newline will be written.

  `writelines`(lines)

    Writes lines (a list of strings) to the underlying program, appending a newline after each line.

  `dump_html`(full=False, client_id='0')

    Returns the difference of terminal lines (a list of lines, to be specific) and its scrollback buffer (which is also a list of lines) as a tuple:

        (scrollback, screen)

    If a line hasn't changed since the last dump said line will be replaced with an empty string in the output.

    If full, will return the entire screen (not just the diff). if client_id is given (string), this will be used as a unique client identifier for keeping track of screen differences (so you can have multiple clients getting their own unique diff output for the same Multiplex instance).

  `dump`()

    Dumps whatever is currently on the screen of the terminal emulator as a list of plain strings (so they'll be escaped and look nice in an interactive Python interpreter).

  `timeout_check`(timeout_now=False)

    Iterates over `BaseMultiplex._patterns` checking each to determine if it has timed out. If a timeout has occurred for a `Pattern` and said Pattern has an errorback function that function will be called.

    Returns True if there are still non-sticky patterns remaining. False otherwise.

    If timeout_now is True, will force the first errorback to be called and will empty out self._patterns.

  `expect`(patterns, callback, optional=False, sticky=False, errorback=None, timeout=15, position=None, preprocess=False)

    Watches the stream of output coming from the underlying terminal program for patterns and if there's a match callback will be called like so:

        callback(multiplex_instance, matched_string)

    Tip

    You can provide a string instead of a callback function as a shortcut if you just want said string written to the child process.

    patterns can be a string, an `re.RegexObject` (as created by `re.compile()`), or a iterator of either/or. Returns a reference object that can be used to remove the registered pattern/callback at any time using the `unexpect()` method (see below).

    Note

    This function is non-blocking!

    Warning

    The timeout value gets compared against the time `expect()` was called to create it. So don't wait too long if you're planning on using `await()`!

    Here's a simple example that changes a user's password:

        >>> def write_password(m_instance, matched):
        ...     print("Sending Password... %s patterns remaining." % len(m_instance._patterns))
        ...     m_instance.writeline('somepassword')
        >>> m = Multiplex('passwd someuser') # Assumes running as root :)
        >>> m.expect('(?i)password:', write_password) # Step 1
        >>> m.expect('(?i)password:', write_password) # Step 2
        >>> print(len(m._patterns)) # To show that there's two in the queue
            2
        >>> m.spawn() # Execute the command
        >>> m.await(10) # This will block for up to 10 seconds waiting for self._patterns to be empty (not counting optional patterns)
        Sending Password... 1 patterns remaining.
        Sending Password... 0 patterns remaining.
        >>> m.isalive()
        False
        >>> # All done!

    Tip

    The `await()` method will automatically call `spawn()` if not `isalive()`.

    This would result in the password of 'someuser' being changed to 'somepassword'. How is the order determined? Every time `expect()` is called it creates a new `Pattern` using the given parameters and appends it to `self._patterns` (which is a list). As each `Pattern` is matched its callback gets called and the `Pattern` is removed from `self._patterns` (unless sticky is `True`). So even though the patterns and callbacks listed above were identical they will get executed and removed in the order they were created as each respective `Pattern` is matched.

    Note

    Only the first pattern, or patterns marked as sticky are checked against the incoming stream. If the first non-sticky pattern is marked optional then the proceeding pattern will be checked (and so on). All other patterns will sit in `self._patterns` until their predecessors are matched/removed.

    Patterns can be removed from `self._patterns` as needed by calling `unexpect(<reference>)`. Here's an example:

        >>> def handle_accepting_ssh_key(m_instance, matched):
        ...     m_instance.writeline(u'yes')
        >>> m = Multiplex('ssh someuser@somehost')
        >>> ref1 = m.expect('(?i)Are you sure.*\(yes/no\)\?', handle_accepting_ssh_key, optional=True)
        >>> def send_password(m_instance, matched):
        ...    m_instance.unexpect(ref1)
        ...    self.writeline('somepassword')
        >>> ref2 = m.expect('(?i)password:', send_password)
        >>> # spawn() and/or await() and do stuff...

    The example above would send 'yes' if asked by the SSH program to accept the host's public key (which would result in it being automatically removed from `self._patterns`). However, if this condition isn't met before send_password() is called, send_password() will use the reference object to remove it directly. This ensures that the pattern won't be accidentally matched later on in the program's execution.

    Note

    Even if we didn't match the "Are you sure..." pattern it would still get auto-removed after its timeout was reached.

    About pattern ordering: The position at which the given pattern will be inserted in `self._patterns` can be specified via the position argument. The default is to simply append which should be appropriate in most cases.

    About Timeouts: The timeout value passed to expect() will be used to determine how long to wait before the pattern is removed from self._patterns. When this occurs, errorback will be called with current Multiplex instance as the only argument. If errorback is None (the default) the pattern will simply be discarded with no action taken.

    Note

    If sticky is True the timeout value will be ignored.

    Notes about the length of what will be matched: The entire terminal 'screen' will be searched every time new output is read from the incoming stream. This means that the number of rows and columns of the terminal determines the size of the search. So if your pattern needs to look for something inside of 50 lines of text you need to make sure that when you call `spawn` you specify at least `rows = 50`. Example:

        >>> def handle_long_search(m_instance, matched)
        ...     do_stuff(matched)
        >>> m = Multiplex('someCommandWithLotsOfOutput.sh')
        >>> # 'begin', at least one non-newline char, 50 newlines, at least one char, then 'end':
        >>> my_regex = re.compile('begin.+[\n]{50}.+end', re.MULTILINE)
        >>> ref = m.expect(my_regex, handle_accepting_ssh_key)
        >>> m.spawn(rows=51, cols=150)
        >>> # Call m.read(), m.spawn() or just let an event loop (e.g. Tornado's IOLoop) take care of things...

    About non-printable characters: If the postprocess argument is True (the default), patterns will be checked against the current screen as output by the terminal emulator. This means that things like control codes and escape sequences will be handled and discarded by the terminal emulator and as such won't be available for patterns to be checked against. To get around this limitation you can set preprocess to True and the pattern will be checked against the incoming stream before it is processed by the terminal emulator. Example:

        >>> def handle_xterm_title(m_instance, matched)
        ...     print("Caught title: %s" % matched)
        >>> m = Multiplex('echo -e "\033]0;Some Title\007"')
        >>> title_seq_regex = re.compile(r'\x1b\][0-2]\;(.*?)(\x07|\x1b\\)')
        >>> m.expect(title_seq_regex, handle_xterm_title, preprocess=True) # <-- 'preprocess=True'
        >>> m.await()
        Caught title: Some Title
        >>>

    Notes about debugging: Instead of using `await` to wait for all of your patterns to be matched at once you can make individual calls to `read` to determine if your patterns are being matched in the way that you want. For example:

        >>> def do_stuff(m_instance, matched):
        ...     print("Debug: do_stuff() got %s" % repr(matched))
        ...     # Do stuff here
        >>> m = Multiplex('someLongComplicatedOutput.sh')
        >>> m.expect('some pattern', do_stuff)
        >>> m.expect('some other pattern', do_stuff)
        >>> m.spawn()
        >>> # Instead of calling await() just call one read() at a time...
        >>> print(repr(m.read()))
        ''
        >>> print(repr(m.read())) # Oops, called read() too soon.  Try again:
        'some other pattern'
        >>> # Doh!  Looks like 'some other pattern' comes first.  Let's start over...
        >>> m.unexpect() # Called with no arguments, it empties m._patterns
        >>> m.terminate() # Tip: This will call unexpect() too so the line above really isn't necessary
        >>> m.expect('some other pattern', do_stuff) # This time this one will be first
        >>> m.expect('some pattern', do_stuff)
        >>> m.spawn()
        >>> print(repr(m.read())) # This time I waited a moment :)
        'Debug: do_stuff() got "some other pattern"'
        'some other pattern'
        >>> # Huzzah!  Now let's see if 'some pattern' matches...
        >>> print(repr(m.read()))
        'Debug: do_stuff() got "some pattern"'
        'some pattern'
        >>> # As you can see, calling read() at-will in an interactive interpreter can be very handy.

    About asynchronous use: This mechanism is non-blocking (with the exception of `await`) and is meant to be used asynchronously. This means that if the running program has no output, `read` won't result in any patterns being matched. So you must be careful about timing or you need to ensure that `read` gets called either automatically when there's data to be read (IOLoop, EPoll, select, etc) or at regular intervals via a loop. Also, if you're not calling `read` at an interval (i.e. you're using a mechanism to detect when there's output to be read before calling it e.g. IOLoop) you need to ensure that `timeout_check` is called regularly anyway or timeouts won't get detected if there's no output from the underlying program. See the `MultiplexPOSIXIOLoop.read` override for an example of what this means and how to do it.

  `unexpect`(ref=None)

    Removes ref from self._patterns so it will no longer be checked against the incoming stream. If ref is None (the default), `self._patterns` will be emptied.

  `await`(timeout=15, rows=24, cols=80, env=None, em_dimensions=None)

    Blocks until all non-optional patterns inside self._patterns have been removed or if the given timeout is reached. timeout may be an integer (in seconds) or a `datetime.timedelta` object.

    Returns True if all non-optional, non-sticky patterns were handled successfully.

    Warning

    The timeouts attached to Patterns are set when they are created. Not when when you call `await()`!

    As a convenience, if `isalive()` resolves to False, `spawn()` will be called automatically with rows, cols, and env given as arguments.

    await
      To wait with expectation.
  `terminate`()

    This method must be overridden by suclasses of `BaseMultiplex`. It is expected to terminate/kill the child process.

  `_read`(bytes=-1)

    This method must be overridden by subclasses of `BaseMultiplex`. It is expected that this method read the output from the running terminal program in a non-blocking way, pass the result into `term_write`, and then return the result.

  `read`(bytes=-1)

    Calls `_read` and checks if any timeouts have been reached in `self._patterns`. Returns the result of `_read`.

class `termio.``MultiplexPOSIXIOLoop`(*args, **kwargs)

  The MultiplexPOSIXIOLoop class takes care of executing a child process on POSIX (aka Unix) systems and keeping track of its state via a terminal emulator (`terminal.Terminal` by default). If there's a started instance of `tornado.ioloop.IOLoop`, handlers will be added to it that automatically keep the terminal emulator synchronized with the output of the child process.

  If there's no IOLoop (or it just isn't started), terminal applications can be interacted with by calling `MultiplexPOSIXIOLoop.read` (to write any pending output to the terminal emulator) and `MultiplexPOSIXIOLoop.write` (which writes directly to stdin of the child).

  Note

  `MultiplexPOSIXIOLoop.read` is non-blocking.

  `_call_callback`(callback)

    If the IOLoop is started, adds the callback via `IOLoop.add_callback()` to ensure it gets called at the next IOLoop iteration (which is thread safe). If the IOLoop isn't started callback will get called immediately and directly.

  `_reenable_output`()

    Restarts capturing output from the underlying terminal program by disengaging the rate limiter.

  `_blocked_io_handler`(signum=None, frame=None)

    Handles the situation where a terminal is blocking IO (usually because of too much output). This method would typically get called inside of `MultiplexPOSIXIOLoop._read` when the output of an fd is too noisy.

  `spawn`(rows=24, cols=80, env=None, em_dimensions=None, exitfunc=None)

    Creates a new virtual terminal (tty) and executes self.cmd within it. Also attaches `self._ioloop_read_handler()` to the IOLoop so that the terminal emulator will automatically stay in sync with the output of the child process.

    Cols : | The number of columns to emulate on the virtual terminal (width)
    Rows : | The number of rows to emulate (height).
    Env : | Optional - A dictionary of environment variables to set when executing self.cmd.
    Em_dimensions : | Optional - The dimensions of a single character within the terminal (only used when calculating the number of rows/cols images take up).
    Exitfunc : | Optional - A function that will be called with the current Multiplex instance and its exit status when the child process terminates (exitfunc(m_instance, statuscode)).

  `isalive`()

    Checks the underlying process to see if it is alive and sets self._alive appropriately.

  `resize`(rows, cols, em_dimensions=None, ctrl_l=True)

    Resizes the child process's terminal window to rows and cols by first sending it a TIOCSWINSZ event and then sending ctrl-l.

    If em_dimensions are provided they will be updated along with the rows and cols.

    The sending of ctrl-l can be disabled by setting ctrl_l to False.

  `terminate`()

    Kill the child process associated with `self.fd`.

    Note

    If dtach is being used this only kills the dtach process.

  `_ioloop_read_handler`(fd, event)

    Read in the output of the process associated with fd and write it to `self.term`.

    Fd : | The file descriptor of the child process.
    Event : | An IOLoop event (e.g. IOLoop.READ).

    Note

    This method is not meant to be called directly... The IOLoop should be the one calling it when it detects any given event on the fd.

  `_read`(bytes=-1)

    Reads at most bytes from the incoming stream, writes the result to the terminal emulator using `term_write`, and returns what was read. If bytes is -1 (default) it will read `self.fd` until there's no more output.

    Returns the result of all that reading.

    Note

    Non-blocking.

  `_timeout_checker`()

    Runs `timeout_check` and if there are no more non-sticky patterns in `self._patterns`, stops `scheduler`.

  `read`(bytes=-1)

    Note

    This is an override of `BaseMultiplex.read` in order to take advantage of the IOLoop for ensuring `BaseMultiplex.expect` patterns timeout properly.

    Calls `_read` and checks if any timeouts have been reached in `self._patterns`. Returns the result of `_read()`. This is an override of `BaseMultiplex.read` that will create a `tornado.ioloop.PeriodicCallback` (as `self.scheduler`) that executes `timeout_check` at a regular interval. The `PeriodicCallback` will automatically cancel itself if there are no more non-sticky patterns in `self._patterns`.

  `_write`(chars)

    Writes chars to `self.fd` (pretty straightforward). If IOError or OSError exceptions are encountered, will run `terminate`. All other exceptions are logged but no action will be taken.

  `write`(chars)

    Calls `_write(*chars*)` via `_call_callback` to ensure thread safety.

`termio.``spawn`(cmd, rows=24, cols=80, env=None, em_dimensions=None, *args, **kwargs)

  A shortcut to:

      >>> m = Multiplex(cmd, *args, **kwargs)
      >>> m.spawn(rows, cols, env)
      >>> return m

`termio.``getstatusoutput`(cmd, **kwargs)

  Emulates Python's commands.getstatusoutput() function using a Multiplex instance.

  Optionally, any additional keyword arguments (**kwargs) provided will be passed to the spawn() command.

`termio.``Multiplex`

  alias of `MultiplexPOSIXIOLoop`

### Navigation

- index
- modules |
- next |
- previous |
- Gate One Documentation »
- Developer Documentation »