mozilla

Circus Library

The Circus package is composed of a high-level get_arbiter() function and many classes. In most cases, using the high-level function should be enough, as it creates everything that is needed for Circus to run.

You can subclass Circus’ classes if you need more granularity than what is offered by the configuration.

The get_arbiter function

get_arbiter() is just a convenience on top of the various circus classes. It creates a arbiter (class Arbiter) instance with the provided options, which in turn runs a single Watcher with a single Process.

circus.get_arbiter()

Example:

from circus import get_arbiter

arbiter = get_arbiter([{"cmd": "myprogram", "numprocesses": 3}])
try:
    arbiter.start()
finally:
    arbiter.stop()

Classes

Circus provides a series of classes you can use to implement your own process manager:

  • Process: wraps a running process and provides a few helpers on top of it.
  • Watcher: run several instances of Process against the same command. Manage the death and life of processes.
  • Arbiter: manages several Watcher.
class circus.process.Process(wid, cmd, args=None, working_dir=None, shell=False, uid=None, gid=None, env=None, rlimits=None, executable=None, use_fds=False, watcher=None, spawn=True, pipe_stdout=True, pipe_stderr=True, close_child_stdout=False, close_child_stderr=False)

Wraps a process.

Options:

  • wid: the process unique identifier. This value will be used to replace the $WID string in the command line if present.
  • cmd: the command to run. May contain any of the variables available that are being passed to this class. They will be replaced using the python format syntax.
  • args: the arguments for the command to run. Can be a list or a string. If args is a string, it’s splitted using shlex.split(). Defaults to None.
  • executable: When executable is given, the first item in the args sequence obtained from cmd is still treated by most programs as the command name, which can then be different from the actual executable name. It becomes the display name for the executing program in utilities such as ps.
  • working_dir: the working directory to run the command in. If not provided, will default to the current working directory.
  • shell: if True, will run the command in the shell environment. False by default. warning: this is a security hazard.
  • uid: if given, is the user id or name the command should run with. The current uid is the default.
  • gid: if given, is the group id or name the command should run with. The current gid is the default.
  • env: a mapping containing the environment variables the command will run with. Optional.
  • rlimits: a mapping containing rlimit names and values that will be set before the command runs.
  • use_fds: if True, will not close the fds in the subprocess. default: False.
  • pipe_stdout: if True, will open a PIPE on stdout. default: True.
  • pipe_stderr: if True, will open a PIPE on stderr. default: True.
  • close_child_stdout: If True, redirects the child process’ stdout to /dev/null after the fork. default: False.
  • close_child_stderr: If True, redirects the child process’ stdout to /dev/null after the fork. default: False.
age()

Return the age of the process in seconds.

children()

Return a list of children pids.

info()

Return process info.

The info returned is a mapping with these keys:

  • mem_info1: Resident Set Size Memory in bytes (RSS)
  • mem_info2: Virtual Memory Size in bytes (VMS).
  • cpu: % of cpu usage.
  • mem: % of memory usage.
  • ctime: process CPU (user + system) time in seconds.
  • pid: process id.
  • username: user name that owns the process.
  • nice: process niceness (between -20 and 20)
  • cmdline: the command line the process was run with.
is_child(pid)

Return True is the given pid is a child of that process.

pid

Return the pid

send_signal(*args, **kw)

Sends a signal sig to the process.

send_signal_child(*args, **kw)

Send signal signum to child pid.

send_signal_children(*args, **kw)

Send signal signum to all children.

status

Return the process status as a constant

  • RUNNING
  • DEAD_OR_ZOMBIE
  • UNEXISTING
  • OTHER
stderr

Return the stdout stream

stdout

Return the stdout stream

stop(*args, **kw)

Stop the process and close stdout/stderr

If the corresponding process is still here (normally it’s already killed by the watcher), a SIGTERM is sent, then a SIGKILL after 1 second.

The shutdown process (SIGTERM then SIGKILL) is normally taken by the watcher. So if the process is still there here, it’s a kind of bad behavior because the graceful timeout won’t be respected here.

Example:

>>> from circus.process import Process
>>> process = Process('Top', 'top', shell=True)
>>> process.age()
3.0107998847961426
>>> process.info()
'Top: 6812  N/A tarek Zombie N/A N/A N/A N/A N/A'
>>> process.status
1
>>> process.stop()
>>> process.status
2
>>> process.info()
'No such process (stopped?)'
class circus.watcher.Watcher(name, cmd, args=None, numprocesses=1, warmup_delay=0.0, working_dir=None, shell=False, shell_args=None, uid=None, max_retry=5, gid=None, send_hup=False, stop_signal=15, stop_children=False, env=None, graceful_timeout=30.0, prereload_fn=None, rlimits=None, executable=None, stdout_stream=None, stderr_stream=None, priority=0, loop=None, singleton=False, use_sockets=False, copy_env=False, copy_path=False, max_age=0, max_age_variance=30, hooks=None, respawn=True, autostart=True, on_demand=False, virtualenv=None, close_child_stdout=False, close_child_stderr=False, **options)

Class managing a list of processes for a given command.

Options:

  • name: name given to the watcher. Used to uniquely identify it.

  • cmd: the command to run. May contain $WID, which will be replaced by wid.

  • args: the arguments for the command to run. Can be a list or a string. If args is a string, it’s splitted using shlex.split(). Defaults to None.

  • numprocesses: Number of processes to run.

  • working_dir: the working directory to run the command in. If not provided, will default to the current working directory.

  • shell: if True, will run the command in the shell environment. False by default. warning: this is a security hazard.

  • uid: if given, is the user id or name the command should run with. The current uid is the default.

  • gid: if given, is the group id or name the command should run with. The current gid is the default.

  • send_hup: if True, a process reload will be done by sending the SIGHUP signal. Defaults to False.

  • stop_signal: the signal to send when stopping the process. Defaults to SIGTERM.

  • stop_children: send the stop_signal to the children too. Defaults to False.

  • env: a mapping containing the environment variables the command will run with. Optional.

  • rlimits: a mapping containing rlimit names and values that will be set before the command runs.

  • stdout_stream: a mapping that defines the stream for the process stdout. Defaults to None.

    Optional. When provided, stdout_stream is a mapping containing up to three keys:

    • class: the stream class. Defaults to circus.stream.FileStream
    • filename: the filename, if using a FileStream
    • max_bytes: maximum file size, after which a new output file is opened. defaults to 0 which means no maximum size (only applicable with FileStream).
    • backup_count: how many backups to retain when rotating files according to the max_bytes parameter. defaults to 0 which means no backups are made (only applicable with FileStream)

    This mapping will be used to create a stream callable of the specified class. Each entry received by the callable is a mapping containing:

    • pid - the process pid
    • name - the stream name (stderr or stdout)
    • data - the data
  • stderr_stream: a mapping that defines the stream for the process stderr. Defaults to None.

    Optional. When provided, stderr_stream is a mapping containing up to three keys: - class: the stream class. Defaults to circus.stream.FileStream - filename: the filename, if using a FileStream - max_bytes: maximum file size, after which a new output file is

    opened. defaults to 0 which means no maximum size (only applicable with FileStream)

    • backup_count: how many backups to retain when rotating files according to the max_bytes parameter. defaults to 0 which means no backups are made (only applicable with FileStream).

    This mapping will be used to create a stream callable of the specified class.

    Each entry received by the callable is a mapping containing:

    • pid - the process pid
    • name - the stream name (stderr or stdout)
    • data - the data
  • priority – integer that defines a priority for the watcher. When the Arbiter do some operations on all watchers, it will sort them with this field, from the bigger number to the smallest. (default: 0)

  • singleton – If True, this watcher has a single process. (default:False)

  • use_sockets – If True, the processes will inherit the file descriptors, thus can reuse the sockets opened by circusd. (default: False)

  • on_demand – If True, the processes will be started only at the first connection to the socket (default: False)

  • copy_env – If True, the environment in which circus is running run will be reproduced for the workers. (default: False)

  • copy_path – If True, circusd sys.path is sent to the process through PYTHONPATH. You must activate copy_env for copy_path to work. (default: False)

  • max_age: If set after around max_age seconds, the process is replaced with a new one. (default: 0, Disabled)

  • max_age_variance: The maximum number of seconds that can be added to max_age. This extra value is to avoid restarting all processes at the same time. A process will live between max_age and max_age + max_age_variance seconds.

  • hooks: callback functions for hooking into the watcher startup and shutdown process. hooks is a dict where each key is the hook name and each value is a 2-tuple with the name of the callable or the callabled itself and a boolean flag indicating if an exception occuring in the hook should not be ignored. Possible values for the hook name: before_start, after_start, before_spawn, after_spawn, before_stop, after_stop., before_signal, after_signal or extended_stats.

  • options – extra options for the worker. All options found in the configuration file for instance, are passed in this mapping – this can be used by plugins for watcher-specific options.

  • respawn – If set to False, the processes handled by a watcher will not be respawned automatically. (default: True)

  • virtualenv – The root directory of a virtualenv. If provided, the watcher will load the environment for its execution. (default: None)

  • close_child_stdout: If True, closes the stdout after the fork. default: False.

  • close_child_stderr: If True, closes the stderr after the fork. default: False.

kill_process(*args, **kwargs)

Kill process (stop_signal, graceful_timeout then SIGKILL)

kill_processes(*args, **kwargs)

Kill all processes (stop_signal, graceful_timeout then SIGKILL)

manage_processes(*args, **kwargs)

Manage processes.

notify_event(topic, msg)

Publish a message on the event publisher channel

reap_and_manage_processes(*args, **kwargs)

Reap & manage processes.

reap_processes(*args, **kw)

Reap all the processes for this watcher.

send_signal_child(*args, **kw)

Send signal to a child.

spawn_process()

Spawn process.

Return True if ok, False if the watcher must be stopped

spawn_processes(*args, **kwargs)

Spawn processes.