boogio.aws_surveyor module¶

Manage sets of AWSInformer instances across accounts and regions.

An AWSSurveyor instance manages AWSMediator and AWSInformer instances across multiple profiles and regions.

Use AWSSurveyor instances with a workflow like the following.

Initialize an AWSSurveyor instance, defining values for profiles, regions and entity types if desired.

Use AWSSurveyor.survey() to retrieve records from AWS and create AWSInformer instances to manage them. You can use the existing presets for profiles, regions and entity types, update the presets, or specify survey-specific values.

Use AWSSurveyor.informers() to retrieve informers from the surveyor for further use.

AWSSurveyor Initialization¶

An AWSSurveyor maintains preset lists of the profiles, regions and entity types that will be used by default when survey() is called. These preset values can be assigned when a surveyor is created and can be updated afterwards. Common preset values can be saved in configuration files as blocks of JSON such as the following.

{
    "profiles": ["some_profile", "another_profile"],
    "regions": ["us-east-1", "us-west-1", "us-west-2"],
    "entity_types": ["ec2", "security_group", "subnet", "vpc"]
}

If no presets are provided when an AWSSurveyor instance is created and $HOME/.aws/aws_surveyor.cfg` exists, it will be parsed as a JSON file to determine presets.

To read from another configuration file, pass the local or absolute path to the file as the value of the config_path initialization argument.

To suppress reading of preset values from $HOME/.aws/aws_surveyor.cfg`, pass an empty string as the value of the config_path argument at initialization.

Examples
Initialize presets from $HOME/.aws/aws_surveyor.cfg:
>>> surveyor = AWSSurveyor()
Initialize presets from aws_surveyor.cfg in the current working directory:
>>> surveyor = AWSSurveyor(config_path='aws_surveyor.cfg')
Don’t read any configuration file at initialization, even if the default configuration file $HOME/.aws/aws_surveyor.cfg exists:
>>> surveyor = AWSSurveyor(config_path='')

You can also configure any of the profiles, regions and entity_types presets individually via initialization arguments. Any values for these presets that would otherwise be read from a configuration file will be overwritten by initialization arguments. To add to, rather than overwriting, configuration file presets set the add_to_config argument to True.

Examples
Define profiles, regions and entity_types presets at initialization, overriding any of their presets defined in $HOME/.aws/aws_surveyor.cfg:
>>> surveyor = AWSSurveyor(
...     profiles=['some_profile', 'another_profile'],
...     regions=['us-east-1', 'us-west-1', 'us-west-2'],
...     entity_types=['ec2', 'security_group', 'subnet', 'vpc']
...     )
Examine the presets property to see the current presets:
>>> surveyor.presets
{'profiles': ['some_profile', 'another_profile'], 'regions': [
'us-east-1', 'us-west-1', 'us-west-2'], 'entity_types': ['ec2',
'security_group', 'subnet', 'vpc']}
Define the entity_types preset at initialization, overriding the entity_types preset if one is defined in the default configuration file $HOME/.aws/aws_surveyor.cfg. This will still use the default configuration file presets for profiles and regions:
>>> surveyor = AWSSurveyor(
...     entity_types=['ec2', 'security_group', 'subnet', 'vpc']
...     )
Read the presets for profiles, regions and entity_types from the specified configuration file, and add an additional “elb” entity type preset:
>>> surveyor = AWSSurveyor(
...     entity_types=['elb'],
...     config_path='my_aws_surveyor.cfg',
...     add_to_config_path=True
...     )
When creating an AWSSurveyor instance, you can also configure the regions preset by setting the set_all_regions flag to True.
Read presets from the default configuration file and then update the regions preset to include all regions:
>>> surveyor = AWSSurveyor(
...     set_all_regions=True
...     )

Configuration Files¶

AWSSurveyor configuration files define the preset values for the AWS credentials profiles, the AWS regions and the AWS entity types (corresponding to AWSInformer subclasses) that the survey() method will use.

An example of a typical AWSSurveyor configuration file:

{
    "profiles": ["some_profile", "another_profile"],
    "regions": ["us-east-1", "us-west-1", "us-west-2"],
    "entity_types": ["ec2", "security_group", "subnet", "vpc"]
}

The default location for an AWSSurveyor configuration file is $HOME/.aws/aws_surveyor.cfg. If this file exists, it will be read automatically unless explicitly suppressed. You can also use configuration files in other locations or use initialization arguments for configuration. See AWSSurveyor Initialization for more detailed information on surveyor initialization and configuration.

Updating AWSSurveyor Configuration¶

If you want to change the presets for an existing AWSSurveyor instance, you can assign directly to the presets property. You can also set the profiles, regions and entity_types explicily, or use the add_* and remove_* methods.

Examples

Set the presets property directly:

>>> surveyor = AWSSurveyor()
>>> surveyor.presets = {
...     'profiles': ['some_profile', 'another_profile'],
...     'regions': ['us-east-1', 'us-west-1', 'us-west-2'],
...     'entity_types': ['ec2', 'security_group', 'subnet', 'vpc']
...     }
>>> surveyor.presets
{'profiles': ['some_profile', 'another_profile'], 'regions': [
'us-east-1', 'us-west-1', 'us-west-2'], 'entity_types': ['ec2',
'security_group', 'subnet', 'vpc']}

Assign values to individual presets:

>>> surveyor = AWSSurveyor()
>>> surveyor.profiles = ['this_profile', 'that_profile']
>>> surveyor.profiles
['this_profile', 'that_profile']

>>> surveyor.remove_profile(['that_profile'])
>>> surveyor.add_profiles(['another_profile'])
>>> surveyor.profiles
['this_profile', 'another_profile']

Adding duplicate preset values or trying to remove values that aren’t already present will have no effect on existing presets:

>>> surveyor = AWSSurveyor()
>>> surveyor.profiles = ['this_profile', 'that_profile']
>>> surveyor.profiles
['this_profile', 'that_profile']

>>> surveyor.remove_profile(['another_profile'])
>>> surveyor.profiles
['this_profile', 'that_profile']

>>> surveyor.regions = ['us-west-1']
>>> surveyor.regions
['us-west-1']

>>> surveyor.add_regions(['us-west-1'])
>>> surveyor.regions
['us-west-1']

Executing Surveys¶

Use the AWSSurveyor.survey() method to query Amazon Web Services and create AWSInformer instances for entity types in regions and using profiles currently configured as presets or to a subset of the preset regions and profiles as defined in arguments passed to survey().

Note

Arguments for profiles and regions passed to survey() must already be present in the AWSSurveyor instance’s presets. If you want to survey new profiles or regions you must first use the add_profiles() and add_regions() methods to update the presets.

Use the AWSSurveyor.instances() method to retrieve the results of the survey as a list of AWSInformer instances.

Examples
Survey using presets defined at instance initialization:
>>> surveyor = AWSSurveyor(
...     profiles=['some_profile'],
...     regions=['us-east-1'],
...     entity_types=['ec2']
...     )
>>> surveyor.survey()
>>> instances = surveyor.instances()
Pass arguments to survey() to override presets. This will survey vpc entity_types in region us-west-2 using credentials defined for another_profile, regardless of the configuration in the aws_surveyor configuration file:
>>> surveyor = AWSSurveyor(config_path='aws_surveyor.cfg')
>>> surveyor.survey(
...     profiles=['another_profile'],
...     regions=['us-west-2'],
...     entity_types=['vpc']
...     )
>>> instances = surveyor.instances()
This surveyor will survey vpc entity_types using the settings for profile and region in the aws_surveyor configuration file:
>>> surveyor = AWSSurveyor(config_path='aws_surveyor.cfg')
>>> surveyor.survey(
...     entity_types=['vpc']
...     )
>>> instances = surveyor.instances()

class boogio.aws_surveyor.AWSSurveyor(profiles=None, regions=None, entity_types=None, config_path=None, add_to_config=False, set_all_regions=False)[source]¶

Bases: object

Manage AWS queries and AWSInformer instances with the results.

Parameters:

profiles (list of str, optional) – A list of the AWS credentials profiles to use for AWS sessions.
regions (list of str, optional) – A list of the AWS regions to connect to with AWS sessions.
entity_types (list of str, optional) – A list of the entity types to retrieve when AWSSurveyor.survey() is called.
config_path (string, optional) – The path to a JSON format configuration file. See Configuration Files.
add_to_config (bool, default=False) – If True, any presets values defined by arguments will be added to the presets values read from any configuration files. If False, values passed as arguments will overwrite values from configuration files.
set_all_regions (bool, default=False) – If True, the regions preset will be set to a list of all currently available regions. This requires a call to aws_surveyor.all_regions(), which connects to AWS.

mediators[source]¶: list of AWSMediator – A list of AWSMediator instances, one for each profile and region assigned to the surveyor or obtained from default configuration selection (e.g. the default profile or an EC2 instance profile).

accounts¶

The surveyors’ AWSMediator‘s AWS account attributes.

Each element of the list is a dict with the account information for one of the surveyor’s mediators, of the form:

{
'account_id': <AWS Account ID>,
'account_name': <Account Name, if defined in boogio.json>,
'account_desc': <account_name if defined, or account_id>,
'region_name': <mediator's region_name attribute>
}

add_entity_types(entity_types)[source]¶

Add new values to the entity_types attribute.

Parameters:	entity_types (list of str) – The entity_type values to add.

add_filters(entity_type, filters)[source]¶

Add AWS entity retrieval filters to the surveyor’s mediators.

Parameters:	entity_type (string) – An informer entity type to which this filter will apply. new_filters (dict) – The Name strings and Values lists for the filters.

See the documentation for AWSMediator for further details.

add_profiles(profiles)[source]¶

Add new values to the profiles attribute.

Parameters:	profiles (list of str) – The profile values to add.

add_regions(regions)[source]¶

Add new values to the regions attribute.

Parameters:	regions (list of str) – The region values to add.

all_paths(*entity_types)[source]¶

Find prunable paths available in informers of given types.

Find all the prune specification paths present in the list of to_dict() output for a list of AWSInformer instances.

Parameters:	entity_types (tuple of str) – A tuple of informer entity types.
Returns:	A list of all the `prune` path specifications found in and informer in this instance’s `informers()` list.
Return type:	list

See the documentation for the utensils prune module for information on pruning and prune paths.

classmethod default_config_dir(value=None)[source]¶

Set or get the class default configuration file directory.

Parameters:	value (string, optional) – If provided, the string to set as the default configuration directory.
Returns:	(string) The current value (updated with the new value, if provided) of the default configuration directory.

classmethod default_config_filename(value=None)[source]¶

Set or get the class default configuration filename.

Parameters:	value (string, optional) – If provided, the string to set as the default configuration filename.
Returns:	(string) The current value (updated with the new value, if provided) of the default configuration filename.

classmethod default_config_path()[source]¶

Return the path to the default configuration file.

Return the path formed by joining the default configuration filename to the default configuration directory.

entity_types¶: Get or set the entity_types attribute.

expand_informers(*entity_types)[source]¶

Expand the current list of surveyed AWSInformer instances.

Parameters:	entity_types (list of str) – A list of informer entity types.

Calling the AWSSurveyor.expand_informers() method will call the expand() method on each instance in the current list of surveyed informers, or on each instance of one of the entity types indicated.

informers(*entity_types)[source]¶

Return the current list of surveyed AWSInformer instances.

Parameters:	entity_types (list of str) – A list of informer entity types.
Returns:	A list of the informers of the specified entity types retrieved in the last call to `survey()`.
Return type:	list

mediators()[source]: Return the list of AWSMediator instances in use.

presets¶

Get or set all of the presets attributes at once.

The presets attributes are profiles, regions and entity_types The presets derived property returns them as a dictionary with the attribute names as keys and the attribute values as values.

Setting presets to a dictionary with the same keys will assign the corresponding values, and assign an empty list for any of the presets attributes not present in the dictionary.

Examples

Set all presets to the values shown:

>>> surveyor = AWSSurveyor()
>>> surveyor.presets = {
...     'profiles': ['some_profile', 'another_profile'],
...     'regions': ['us-east-1', 'us-west-1'],
...     'entity_types': ['vpc', 'subnet']
...     }

Set the profile preset to [another_profile] and empty the other presets:

>>> surveyor = AWSSurveyor()
>>> surveyor.presets = {'profiles': ['another_profile']}

profiles¶: Get or set the profiles attribute.

regions¶: Get or set the regions attribute.

remove_all_filters(*remove_types)[source]¶: Remove AWS entity filters from the surveyor’s mediators.

remove_entity_types(entity_types)[source]¶

Remove values from the entity_types attribute.

Parameters:	entity_types (list of str) – The entity_type values to remove.

remove_profiles(profiles)[source]¶

Remove values from the profiles attribute.

Parameters:	profiles (list of str) – The profile values to remove.

remove_regions(regions)[source]¶

Remove values from the regions attribute.

Parameters:	regions (list of str) – The region values to remove.

set_ec2_elb_supplementals()[source]¶

Record LoadBalancer names in EC2 Instance supplementals.

Calling set_ec2_elb_supplementals() will add a load_balancer_names item to each EC2InstanceInformer in the surveyor’s informers.

Note that this will retrieve ELB records from AWS if they aren’t already loaded.

survey(*entity_types, **kwargs)[source]¶

Conduct a survey of preset or specified AWS targets.

Parameters:

*entity_types (str) – A tuple of the entity types to retrieve for this survey, instead of any instance presets.
profiles (list of str, optional) – A subset of instance profiles to use for this survey.
regions (list of str, optional) – A subset of instance regions to use for this survey.
refresh (optional) – If True (the default), delete all existing informer records and load anew. If False, preserve existing records for any entity type that has already been loaded.

Raises:

ValueError – If any item in profiles or regions isn’t already present in the corresponding presets.
ValueError – If any of the entity types in entity_types isn’t a valid AWSInformer entity type.

The survey() method populates the internal list of AWSInformer instances maintained by this AWSSurveyor instance, which can then be accessed via the instances() method.

By default, the profiles, regions and entity_types used will be the presets values. Any values passed in as arguments will override the corresponding presets values. This can be used to limit the profiles and regions surveyed to a subset of those already configured as presets, but no new profiles or regions can be added. There is no such restriction for entity_types, and any new types will be added to the instance’s entity_types preset.

If no profile was specified (so that default, environment variable specified or instance profile associated credentials will be used), any profiles passed to survey() will be ignored.

survey_timestamp¶: Get the _survey_timestamp attribute.

timestamp_format = '%Y%m%dT%H%M%S'¶

boogio.aws_surveyor.all_regions(profile_name=None, region_name=None, *filters)[source]¶

Return all region names matching any of the strings in filters.

Parameters:	profile_name (str) – An AWS credentials profile name to use call the boto3 `describe_regions()` API to retrieve the names of all regions currently supported. region_name (str) – An AWS region name to use call the boto3 `describe_regions()` API to retrieve the names of all regions currently supported. filters (list of str) – A list of strings which occur as substrings of the desired regions.
Returns:	All available regions that contain at least one filter as a substring.
Return type:	(list of str)

Examples:

>>> all_regions('us')
['us-east-1', 'us-west-1', 'us-west-2']

>>> all_regions('north', 'south')
['ap-northeast-1', 'ap-southeast-1', 'ap-southeast-2']

>>> all_regions('-2')
['ap-southeast-2', 'us-west-2']