Python API Reference¶

Fault tolerance¶

sidecar.call.fault_tolerant(dependency_id=None, circuit_reset_timeout=10, load_balancer=None, http_pool_size=10, fallback=None, timeout=5, retry_attempts=3, max_open_per_netloc=40, retry_on_timeout=False, expected_in=None, statsd=None)[source]¶

Wraps a function with a fault tolerant version that supports timeouts, retries, load balancing, and failovers.

The wrapped function can optionally take a dace.call.Call argument, which will contain services for the specific call.

The following services will wrap executions of the wrapped function:

Logging of the operation, specifically of any errors
Timeout tracking to ensure the operation doesn’t hang
Rate limiting to ensure the operation doesn’t hammer a particular server
Circuit breaker for making sure multiple errors cause the target service to not be contacted again until the reset timeout
Load balancing function execution to have each operation spread load across multiple targets
Retry handling
Fallback function execution in case of failure

Parameters:

dependency_id – The id to use in stats and logs
circuit_reset_timeout (int) – How long in seconds to wait until the circuit is reset and tested again
load_balancer – A function that returns a (protocol, netloc) pair for load balancing
http_pool_size (int) – The number of concurrent HTTP connections to allow
fallback – The function to call when the operation fails to generate the return value
timeout – The timeout for the entire operation execution
retry_attempts (int) – The number of retry attempts to make if the operation fails
max_open_per_netloc (int) – The maximum number of concurrent operations per netloc
retry_on_timeout (bool) – Whether to retry the request on HTTP timeout or not
expected_in (int) – The time the operation is expected to return in. Used to determine a circuit failure.
statsd – The statsd service instance to use for stats tracking

class sidecar.call.Call(dependency, run, fallback, timeout, retry_attempts, max_open_per_netloc, retry_on_timeout, expected_in, statsd, load_balancer, circuit_reset_timeout)[source]¶

Object added to the fault_tolerant decorated method’s kwargs to provide fault tolerance.

http_connector¶: The HTTP connector

http_session¶: The HTTP connector

netloc¶: The load-balanced host to use for any calls

path¶: The API path

scheme¶: The load-balanced scheme to use for any calls

class sidecar.call.Dependency(dependency_id, http_pool_size)[source]¶: Used by the Call instance that is provided to fault_tolerant decorated functions.

sidecar.decorator.circuit(statsd, circuit_reset_timeout, expected_in, func)[source]¶: Apply a circuit breaker, keyed to the netloc from the load balancer

sidecar.decorator.find_property(name, value, args, default=None)[source]¶

Helper for the fault_tolerant decorator to use values from the decorated method’s bound object to fulfill the fault_tolerant logic.

If the decorated function is indeed an instance method of a class, this helper does one of two things: 1. If value is None, look for an attribute on the object with the given name to populate the value. 2. If value is a callable, add the self arg to the callable.

Parameters:

name (str) – The attribute name to look up on the object for the value if value is None.
value (Any) – The value to fill if None or to add self if it is a callable.
args (tuple) – Positional args provided to the decorated method.
default (Any) – Default value to use if the object has no attribute with the given name, or if the decorated function is not a bound instance method.

sidecar.decorator.load_balance(load_balancer, func)[source]¶: Retrieve load balancer results and set on function as “scheme” and “netloc”

sidecar.decorator.logger(func)[source]¶: Log exceptions as they get swallowed in order to return fallback results

sidecar.decorator.optional_call_arg(call_arg, func)[source]¶: Allow run method to contain a ‘call’ argument to access its context

sidecar.decorator.rate_limit(statsd, max_per_netloc, func)[source]¶: Limits the number of open requests

sidecar.decorator.retry(statsd, tries, retry_on_timeout, func)[source]¶: Will retry certain results from the run method execution

sidecar.decorator.timeout(statsd, timeout_secs, func)[source]¶: Ensure the run method executes in a certain amount of time

sidecar.decorator.wraps_run(func)[source]¶: Ensure all run function wrappers get the same dict, as it is used for context information

Service Discovery¶

class sidecar.service_discovery.ServiceDiscovery(services)[source]¶

service_load_balancer(name)[source]¶

Returns a load balancer function for service ‘name’.

The returned function:

Returns:

a (scheme, netloc) tuple. (scheme is always ‘http’ for now.)

Raises:

UnknownServiceError: if the ‘name’ isn’t known

ServiceUnavailableError: if the ‘name’ resolves to an empty list of host:ports

services = None¶: type – sidecar.services.Services

class sidecar.service_discovery.ServiceLocationReplacer[source]¶

This class holds the replacements for service definitions. These replacements are provided via command line. This allow to perform calls against services on addresses different than specified in service configuration file, or even without such file.

get(name)[source]¶

replacements = {}¶

static set_replacements_cmdline(replacement_option_list)[source]¶: Parse replacements list from command line :param replacement_option_list:

class sidecar.services.Services(path='/hipchat/config/sidecar-discovery.json')[source]¶

FILE = '/hipchat/config/sidecar-discovery.json'¶

get(name)[source]¶

Returns a list of netlocs (host:port pairs) for the service ‘name’.

Returns None if the service is not known. Returns an empty list if there are now hosts for the service. Raises RuntimeError if data hasn’t been loaded yet. (See reload() or start().)

keys()[source]¶

reload()[source]¶

Reloads (or first time loads) the json data from self.path.

Keeps the stale data if there’s a problem reloading. Raises ValueError if there’s a problem with initial load.

start(loop=None)[source]¶: Starts the file watcher looking for updates in the file.

Proxy Handler¶

class sidecar.proxy_handler.ProxyHandler(statsd, service_discovery)[source]¶

proxy_request(request, service_ids, path, async=False)[source]¶

proxy_request_handler(request)[source]¶

stop()[source]¶

Circuit Breaker¶

Functionality for managing errors when interacting with a remote service.

The circuit breaker monitors the communication and in the case of a high error rate may break the circuit and not allow further communication for a short period. After a while the breaker will let through a single request to probe to see if the service feels better. If not, it will open the circuit again.

A L{CircuitBreakerSet} can handle the state for multiple interactions at the same time. Use the C{context} method to pick which interaction to track:

try:
    with circuit_breaker.context('x'):
       # something that generates errors
    pass
except DaceCircuitOpenError:
    # the circuit was open so we did not even try to communicate
    # with the remote service.
    pass

class sidecar.breaker.CircuitBreaker(clock, log, error_types, maxfail, reset_timeout, time_unit)[source]¶

A single circuit with breaker logic.

error(err=None)[source]¶: Update the circuit breaker with an error event.

open(err=None)[source]¶

reset()[source]¶: Reset the breaker after a successful transaction.

success()[source]¶

test()[source]¶

Check state of the circuit breaker.

@raise DaceCircuitOpenError: if the circuit is still open

class sidecar.breaker.CircuitBreakerSet(clock, log, maxfail=3, reset_timeout=10, time_unit=60, factory=<class 'sidecar.breaker.CircuitBreaker'>)[source]¶

Controller for a set of circuit breakers.

Variables:

clock – A callable that takes no arguments and return the current time in seconds.
log – A L{logging.Logger} object that is used for the circuit breakers.
maxfail – The maximum number of allowed errors over the last minute. If the breaker detects more errors than this, the circuit will open.
reset_timeout – Number of seconds to have the circuit open before it moves into C{half-open}.

context(circuit_id)[source]¶: Return a circuit breaker for the given ID.

handle_error(err_type)[source]¶: Register error C{err_type} with the circuit breakers so that it will be handled as an error.

Middleware¶

sidecar.middleware.logging_middleware_factory(app, handler)[source]¶

Errors¶

exception sidecar.error.BaseSidecarException(message: str = '', code: int = None)[source]¶

Exception base class that provides HTTP and/or XMPP error codes as well as messages.

If errors contain both a code and a message, the string representation of the message will prepend the code to the message.

exception sidecar.error.ConfigNotFoundError(message: str = '', code: int = None)[source]¶

The requested config couldn’t be found

code = 500¶

message = 'Invalid system configuration'¶

exception sidecar.error.ServiceUnavailableError(message: str = '', code: int = None)[source]¶

The requested service was known but had no viable hosts to route to.

code = 500¶

message = 'No viable hosts found for the requested service'¶

exception sidecar.error.SidecarCircuitOpenError(message: str = '', code: int = None)[source]¶

The circuit breaker is open.

code = 503¶

message = 'Temporarily unable to make requests to the requested service'¶

exception sidecar.error.SidecarRateLimitedError(message: str = '', code: int = None)[source]¶

The operation has hit its rate limit

code = 503¶

message = 'Request ratelimited'¶

exception sidecar.error.SidecarRetryExhaustedError(message: str = '', code: int = None)[source]¶

The operation has exceeded its number of retries

When this exception is raised by sidecar.decorator.retry, its code and message are copied from the exception that triggered it.

code = 502¶

message = 'Request failed (retries exhausted)'¶

exception sidecar.error.SidecarRetryRequestedError(message: str = '', code: int = None)[source]¶

The operation has raised an error, but would like to retry

code = 502¶

message = 'Request failed'¶

exception sidecar.error.SidecarTimeoutError(message: str = '', code: int = None)[source]¶

The operation has timed out

code = 504¶

message = 'Request timed out'¶

exception sidecar.error.UnknownServiceError(message: str = '', code: int = None)[source]¶

The requested service was unknown

code = 500¶

message = 'Unknown service requested'¶

Statsd¶

class sidecar.statsd.MultiStatsD(statsd=None, statsd_enabled=False, dogstatsd=None, dogstatsd_enabled=False)[source]¶

Sends statistics to statsd and / or dogstatsd, only update one stat at a time.

decrement(stat, sample_rate=1, value=1, tags=None)[source]¶

Decrements a single counter

>>> decrement('some.int')

gauge(stat, value, sample_rate=1, tags=None)[source]¶

Sets a single gauge to a value

>>> gauge('some.int', 'some_value')

increment(stat, sample_rate=1, value=1, tags=None)[source]¶

Increments a single counter

>>> increment('some.int')
>>> increment('some.int', 0.5)

start()[source]¶

timing(stat, time, sample_rate=1, tags=None)[source]¶

Log timing information

>>> timing('some.time', '500')

class sidecar.statsd.StatsD(loop, my_hostname, host, port, prefix)[source]¶

Sends statistics to the stats daemon over UDP

decrement(stats, sample_rate=1, value=1)[source]¶

Decrements one or more stats counters

>>> # noinspection PyUnresolvedReferences
>>> decrement('some.int')

gauge(stats, value, sample_rate=1)[source]¶

Sets one or more gauges to a value

>>> # noinspection PyUnresolvedReferences
>>> gauge('some.int', 'some_value')

increment(stats, sample_rate=1, value=1)[source]¶

Increments one or more stats counters

>>> # noinspection PyUnresolvedReferences
>>> increment('some.int')
>>> # noinspection PyUnresolvedReferences
>>> increment('some.int', 0.5)

protocol = None¶: type – StatsdProtocol

start()[source]¶

timing(stats, time, sample_rate=1)[source]¶

Log timing information

>>> # noinspection PyUnresolvedReferences
>>> timing('some.time', '500')

class sidecar.statsd.StatsdProtocol(prefix)[source]¶

connection_lost(exc)[source]¶

connection_made(transport)[source]¶

send(data, sample_rate=1)[source]¶: Squirt the metrics over UDP

transport = None¶: type – asyncio.DatagramTransport

class sidecar.statsd.WrappedStatsD(key_pattern, delegate)[source]¶

Sends statistics to the stats daemon over UDP

decrement(stats, sample_rate=1, value=1)[source]¶

Decrements one or more stats counters

>>> # noinspection PyUnresolvedReferences
>>> decrement('some.int')

gauge(stats, value=1, sample_rate=1)[source]¶

Sets one or more gauges to a value

>>> # noinspection PyUnresolvedReferences
>>> gauge('some.int', 'some_value')

increment(stats, value=1, sample_rate=1)[source]¶

Increments one or more stats counters

>>> # noinspection PyUnresolvedReferences
>>> increment('some.int')
>>> # noinspection PyUnresolvedReferences
>>> increment('some.int', 0.5)

timing(stats, time, sample_rate=1)[source]¶

Log timing information

>>> # noinspection PyUnresolvedReferences
>>> timing('some.time', '500')

Web¶

class sidecar.web.Api(statsd, service_discovery)[source]¶

class sidecar.web.HealthCheckHandler[source]¶

report_health(*args)¶

class sidecar.web.HostsHandler(service_discovery, statsd)[source]¶

get_random_host(*args)¶

class sidecar.web.SidecarApplication(statsd, **kwargs)[source]¶

run(host='localhost', port=12000)[source]¶

Web Decorator¶

sidecar.web_decorator.unhandled_exception_500(wrapped)[source]¶: Decorator of a coroutine that returns a 500 error containing the stack trace of any unhandled exception.

Utilities¶

sidecar.memory.memory(since=0)[source]¶

sidecar.memory.resident(since=0)[source]¶

class sidecar.defaultdict.DefaultDict[source]¶