boogio.aws_informer module¶

Manage connecting to and retrieving information for specified AWS entities.

An instance of AWSInformer corresponds to some type of AWS entity, and includes functionality to help manage the data associated with that entity and its related AWS elements. Some examples include:

An EC2 instance includes references to entities such as VPCs, subnets, security groups and so on. Using the EC2InstanceInformer subclass of AWSInformer, you can easily associate records for these entities and see their relationships with the record for the EC2 instance.

AWS entities have Tags that are represented internally as a mapping {"Key": key, "Value": value}. Using an informer you can treat these as a mapping {"key": "value"} for easier manipulation.

Elastic Load Balancer entities have a DNS entry field, whose corresponding IP address an ELBInformer instance can look up and treat as a full fledged attribute of the informer.

AWSInformer instances retrieve their data through an instance of the` AWSMediator class. The AWSMediator instance caches entity information so that, for example, multiple EC2 instances with the same AWS Security Group assigned don’t trigger multiple AWS queries for the Security Group’s data.

You can define filters in an AWSMediator instance that will be applied to limit the entities retrieved from AWS by that mediator. Mediator filters are defined per entity type using the add_filters() method. The allowed filters are generally those for the relevant client’s describe_X() method, where X depends on the entity type. Thus the legal filters for an “eip” entity type are those documented in the ec2 client’s describe_addresses() method, etc.

class boogio.aws_informer.AWSInformer(resource, region_name=None, profile_name=None, mediator=None, **kwargs)[source]¶

Bases: object

Manage selected information for specified AWS entities.

Parameters:

resource (dict) – An AWS resource being handled by this informer.
profile_name (string) – The name of the AWS profile to use for this informer’s mediator.
region_name (string) – The name of the AWS region to use for this informer’s mediator.
mediator (aws_informer.AWSMediator) – An AWSMediator instance this resource should use.
required_resource_type – The type() of the AWS resource boto class the informer is intended to handle.

identifier¶: str – A string which uniquely identifies this informer, appropriate for the entity type. For example, the InstanceID for an EC2InstanceInformer or the LoadBalancerName for an ELBInformer.

resource¶: dict – The original record for the AWS entity this Informer instance manages.

expansions¶: dict – The child resources found when expanding this informer.

supplementals¶: dict – Additional information assigned to this informer.

is_expanded¶: bool – True if this informer’s expand() method has been called; False otherwise.

promote_to_top_level¶: list – A list of top level resource keys that will be handled specially in AWSInformer.to_dict(). See the documentation for to_dict() for details.

elisions¶: list – A list of top level resource keys that will be handled specially in AWSInformer.to_dict(). See the documentation for to_dict() for details.

entity_type¶: Return the AWSMediator entity type for this informer.

expand()[source]¶

Fetch selected entity details and populate the expansions attribute.

The expansions attribute is a dictionary whose keys match the name of some attribute of the resource attribute, and whose values are either AWSInformer instances or lists of AWSInformer instances.

Expanding an informer pulls the records from AWS (via the informer’s mediator) for other AWS entities only referenced by id in the original resource record.

Each AWSInformer instance in the expansions will in turn be expanded.

identifier: Return the unique identifier string for this informer.

json_dumps(flat=False)[source]¶

Return a JSON representation of the informer.

Parameters:

flat (bool) –

If the flat parameter is True, the data structure returned will be a list of flattened (no nested structures) dicts obtained by calling the informer’s to_dict() method with flat=True.

For more information on flattening, see the documentation for utensils.flatten.

to_dict(entity_identifier=False, flat=False)[source]¶

Return a dict structure using expanded subelements when available.

Parameters:

entity_identifier (bool, default=False) – If True, the dict will include an entity_identifier field whose value is the entity’s identifier attribute. If false, the entity_identifier field won’t be included.
flat (bool, default=False) – If the flat parameter is True, the data structure returned will be a list of flattened (no nested structures) dicts. For more information on flattening, see the documentation for utensils.flatten.

The result of to_dict() is guaranteed to be a serializable structure suitable for JSON conversion. If any subentities are encountered that raise an error when converting to JSON, they will be replaced with their string representation via``str()``.

The result returned by to_dict() will include the informer instance information with the following adjustments.

Any top level keys in the informer’s elisions list will be omitted from the result of to_dict(). As working with expanded entities can generate significant amounts of data, you may sometimes want to use this to omit unneeded substructures to speed up subsequent processing.
The expansions value will replace the resource value for any top level key present in both.
AWS entities store the names and values of their tags as the value of the Key and Value items in a dict. In the structure returned by to_dict(), a tag some_tag
'Tags': {'Key': 'some_tag', 'Value': 'some_value'}
will be represented as
'Tags:some_tag': 'some_value'
at the top level of the returned dict.
Any keys in the top level of the informer’s resource that are present in the informer’s promote_to_top_level list will be treated similarly to tags. If the value of that key is a dict, it will be replaced with a set of top level entries. For example, if TopKey appears in the promote_to_top_level list, then the top level item
'TopKey': {'Key1': 'Val1', 'Key2': 'Val2'}
will be replaced with the top level items
'TopKey:Key1': 'Val1' 'TopKey:Key2': 'Val2'
at the top level of the returned dict.

Note that this will raise an error if the value to be promoted isn’t a dict.
The resulting dict will be updated with the items in the informer’s supplementals dict. If any of these are themselves informers, their to_dict() method will be called.
The resulting dict will be flat if the flat argument is True.

exception boogio.aws_informer.AWSInformerError(*args, **kwargs)[source]¶

Bases: exceptions.Exception

An error associated with an AWS Informer has occurred.

exception boogio.aws_informer.AWSInformerRegionError(msg=None, *args, **kwargs)[source]¶

Bases: boogio.aws_informer.AWSInformerError

An AWSInformer error associated with an AWS region has occurred.

exception boogio.aws_informer.AWSInformerResourceError(msg=None, *args, **kwargs)[source]¶

Bases: boogio.aws_informer.AWSSessionError

An associated with an AWS Informer resource has occurred.

exception boogio.aws_informer.AWSInformerResourceTypeError(msg=None, *args, **kwargs)[source]¶

Bases: boogio.aws_informer.AWSSessionError

An error associated with the type of an AWS Informer has occurred.

class boogio.aws_informer.AWSMediator(**kwargs)[source]¶

Bases: boogio.aws_informer.AWSSession

Manage wholesale AWS entity retrieval in a lazy fashion.

See AWSSession for argument descriptions.

The AWSMediator class extends AWSSession to provide AWS entity data retrieval and caching.

Through its entities() method, an AWSMediator instance makes transparent the distinction between AWS entities that are full fledged AWS services (e.g., ec2, s3) and entities that are retrieved through other services (e.g., security group, network interface).
An AWSMediator instance caches records it has retrieved so that subsequent requests for the same entity don’t required an additional AWS query. It also caches AWSInformers so that the same informers are used when the same mediator is present.
In some cases, an AWSMediator parallelizes AWS queries for faster retrieval.

Entity retrieval filters can be added to the mediator on a per- entity-type basis. These filters will then limit the entities retrieved from AWS for that entity type. See the add_filters() method for details.

PARALLEL_FETCH_PROCESS_COUNT = 48¶

add_filters(entity_type, new_filters)[source]¶

Add AWS entity retrieval filters to the mediator.

Parameters:	entity_type (string) – An informer entity type to which this filter will apply. new_filters (dict) – The Name strings and Values lists for the filters.

Example:

>>> mediator = aws_informer.AWSMediator()
>>> mediator.add_filters(
...     'ec2', {'instance-state-name': ['running']}
...     )
>>> mediator.filters
{'ec2': {'instance-state-name': ['running']}}}

>>> mediator.add_filters(
...     'eip', {
...         'public-ip': ['123.45.67.89', '56.78.91.234'],
...         'domain': ['vpc']
...         }
...     )
>>> mediator.filters
{'ec2': {'instance-state-name': ['running']}},
'eip', {'public-ip': ['123.45.67.89', '56.78.91.234'],
'domain': ['vpc']}}

client(client_type)[source]¶

Return a client from the _clients dict, creating one if needed.

Parameters:	client_type (string) – The type of AWS client to create; e.g., `ec2` or `s3`.

An AWSMediator instance maintains an internal list of AWS clients for different AWS services, created using this mediator’s Session instance. This method returns the client for the indicated entity type, creating a new client if one isn’t already present.

Note that client types are a subset of the legal AWSInformer entity types.

entities(entity_type, use_filters=True)[source]¶

Retrieve entities from AWS (if necessary) and return them as a list.

Parameters:	entity_type (string) – The entity type requested. use_filters (bool, default=True) – If `_fetch()` is called to retrieve entities, this will be passed to control entity filtering. Note that this only happens if this entity type hasn’t previously been fetched.

This is a wrapper that hides the distinction between entity types like ec2 and s3 that correspond to AWS services and other entity types that don’t.

flush(*cached_entity_types)[source]¶

Remove indicated entity types from the internal cache.

Parameters:	cached_entity_types (tuple of string) – The entity types to flush. If not specified, all cached entities will be flushed.

classmethod get_aws_info_in_parallel(requests)[source]¶

Use subprocess pools to retrieve multiple data requests in parallel.

Parameters:

requests (list of dicts) –

A mapping defining the parameters for the requests in question. The expected key/value pairs include:

origin: A passback field for the caller to identify to which entity the data returned for this request belongs.
client_type: The type of client needed to handle this request.
client_method: The client method needed to handle this request.
method_args, method_kwargs: The args and kwargs to pass to the client method.
response_data_keys: The keys for the items in the method response that contain the actual data, rather than response metadata or other information.

Returns: A list of pairs [[request, response]...] where each response has the client method return value for this request. Each response will be a dict whose keys are response_data_keys and whose values are the values of those keys in client_method call return dicts.

informer_cache = {}¶

profile_name¶: Get the profile property.

remove_all_filters(*remove_types)[source]¶: Remove AWS entity retrieval filters from the mediator.

services(service_type)[source]¶: Retrieve a list of services, if necessary, and return the list.

exception boogio.aws_informer.AWSMediatorError(msg=None, *args, **kwargs)[source]¶

Bases: boogio.aws_informer.AWSSessionError

An error associated with an AWS Mediator has occurred.

class boogio.aws_informer.AWSSession(region_name=None, profile_name=None, session=None)[source]¶

Bases: object

Manage AWS sessions.

Parameters:

region_name (string, optional) – The name of the AWS region to use for session creation. This is passed through to initialize a boto3.session.Session instance.
profile_name (string, optional) – The name of the profile record in $HOME/.aws/credentials to use for session creation. This is passed through to initialize a boto3.session.Session instance.
session (boto3.session.Session, optional) – An existing boto3.session.Session instance to use for connections.

Raises:

ValueError – If an existing session and either a
profile_name or region_name are provided.

An AWSSession is a simple wrapper for the underlying boto3 session.

profile_name¶: Property getter.

refresh_session()[source]¶: Recreate self.session with the current parameters.

exception boogio.aws_informer.AWSSessionError(msg=None, *args, **kwargs)[source]¶

Bases: exceptions.Exception

An error associated with an AWS Session has occurred.

__str__()[source]¶: Return a string representation.

class boogio.aws_informer.AutoScalingGroupInformer(resource, *args, **kwargs)[source]¶