boogio.aws_reporter module¶

Report information from collections of AWSInformer instances.

So you’ve used AWSSurveyor to pull records for a few hundred EC2 instances, a handful of VPCs and a few dozen subnets, maybe some ELBs. What now? With the aws_reporter package, you can easily define and re-use report definitions to extract specific fields from a set of AWSInformer instances and return the results in a variety of formats:

Text separated value formats such as csv and tsv.

Lists of dicts or lists of structured lists.

Raw or JSON format nested data structures.

Excel worksheets.

Example

A typical work flow proceeds as follows:

>>> import boogio.aws_surveyor
>>> import boogio.aws_reporter
>>> surveyor = aws_surveyor.AWSSurveyor(
...     profiles=['default'], regions=['us-east-1', 'us-west-1']
...     )
>>> surveyor.survey('ec2', 'subnet', 'vpc')
>>> reporter = aws_reporter.AWSReporter()
>>> from boogio import report_definitions
>>> reporter.add_packaged_report_definitions([report_definitions])
>>> report_names = ['EC2Instances', 'EC2Existential', 'Subnets', 'VPCs']
>>> for name in report_names:
...     reporter.write_tsv(
...     surveyor=surveyor,
...     report_names=[name],
...     output_path=name
...     )

This will create files EC2Instances.tsv, EC2Existential.tsv, Subnets.tsv and VPCs.tsv in the current working directory with the records retrieved by the surveyor.

A few things of which to take note:

The example demonstrates passing AWS entity records by passing an AWSSurveyor instance. You can also pass a list of AWSInformer instances directly with the informers argument.

Instances of aws_reporter.ReportDefinition can be assigned to an AWSReporter instance. The add_packaged_report_definitions() method loads report definitions from a python package, by default from the aws_reporter package. The add_report_definitions() method adds instances of aws_reporter.ReportDefinition directly.

You can use individual reports that are assigned to a reporter by specifying their name attribute when calling reporting methods as shown. You can also pass report definitions to reporting methods directly using the report_definitions argument in place of the report_names argument.

The appropriate file suffix for the output file format is automatically appended.

Report Definitions¶

The ReportDefinition class defines the fields that will be extracted for an atomic report. Each ReportDefinition specifies paths into the to_dict() representation of an AWSInformer type as prune specifications. See the documentation for utensils.prune for details.

An example report definition:

elb_report = aws_reporter.ReportDefinition(
    name='LoadBalancers',
    entity_type='elb',
    prune_specs=[
        {'path': 'meta.profile_name', 'path_to_none': False},
        {'path': 'meta.region_name', 'path_to_none': False},
        {'path': 'LoadBalancerName', 'path_to_none': True},
        {'path': 'VPCId.Tags:Name', 'path_to_none': True},
        {'path': 'Scheme', 'path_to_none': True},
        {'path': 'CreatedTime', 'path_to_none': True},
        {'path': 'DNSName', 'path_to_none': True},
        {
            'path': 'AvailabilityZones',
            'path_to_none': True,
            'value_refiner': lambda x: ' '.join([str(y) for y in x])
            },
        {
            'path': 'DNSIpAddress.INET.[]',
            'flatten_leaves': True,
            'path_to_none': True
            },
        {
            'path': 'Instances',
            'value_refiner': len,
            'path_to_none': True
            },
        ],
    default_column_order=[
        'meta.profile_name', 'meta.region_name',
        'VPCId.Tags:Name', 'LoadBalancerName', 'DNSIpAddress.INET',
        'Scheme', 'CreatedTime',
        'DNSName', 'AvailabilityZones', 'Instances',
        ]
    )

The ReportDefinition instance extract_from() method is used to extract the fields from the appropriate AWSInformer subclass. The extract_from() method can return results either as nested structures directly extracted from the Informer’s dictionary representation or as flattened structures: list of dicts, each dict of which has only scalars among its values. The boolean flat argument to extract_from() selects which structure is returned; flat=True (the default) for flattened structures, flat=False for nested.

Examples

Extract a flattened report from an aws_informer.EC2InstanceInformer instance:

>>> surveyor = aws_surveyor.AWSSurveyor(
...     profiles=['default'], regions=['us-east-1']
...     )
>>> surveyor.survey('ec2')
>>> informer = surveyor.informers()[0]
>>> definition = boogio.report_definitions.ec2_report
>>> definition.extract_from(informer)

Extract a nested report from the same instance:

>>> definition.extract_from(informer, flat=False)

class boogio.aws_reporter.AWSReporter(packaged_report_definitions=False, report_definitions=None)[source]¶

Bases: object

Report information from collections of AWSInformer instances.

Parameters:	packaged_report_definitions (bool, optional) – Assign reports defined in the `boogio.report_definitions` module to this reporter. report_definitions (list of ReportDefinition, optional) – Assign the specified list of report definitions to this reporter.

report_definitions[source]¶: list of ReportDefinition – The report definitions, if any, assigned to this instance.

add_packaged_report_definitions(packages)[source]¶

Assign additional report definitions to this instance.

Parameters:	packages (list of packages) – The list of packages in which to find report definitions.
Raises:	`NameError` – If any item in report_definitions has a `name` attribute that already occurs as the `name` attribute of an assigned report definition.

The add_packaged_report_definitions() method finds any instances of aws_reporter.ReportDefinition in the indicated packages and assigns them to this aws_reporter.AWSReporter instance. The packages must have already been imported.

add_report_definitions(report_definitions)[source]¶

Assign additional report definitions to this instance.

Parameters:	report_definitions (list of aws_reporter.ReportDefinition) – The list of report definitions to assign.
Raises:	`NameError` – If any item in report_definitions has a `name` attribute that already occurs as the `name` attribute of an assigned report definition.

add_worksheets(workbook, informers=None, surveyors=None, report_names=None, report_definitions=None)[source]¶

Generate reports and write to an xlsxwriter workbook.

Parameters:	workbook (xlsxwriter.Worksheet) – The workbook to populate.

See the documentation for report() for details on other arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

This method writes reports to worksheets in an xlsxwriter.Workbook instance passed as its first positional argument. It will set the name of each worksheet to the name of the report written to that worksheet.

Returns: None.

csv_list(informers=None, surveyors=None, report_name=None, report_definition=None)[source]¶

Generate a comma separated report as a list of lines.

See the documentation for report() for details on arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

Returns: A list of comma-separated strings.

json_dumps(informers=None, surveyors=None, report_name=None, report_definition=None, flat=True)[source]¶

Generate a report in JSON format.

See the documentation for report() for details on arguments and exceptions.

Returns: A JSON string.

report(informers=None, surveyors=None, report_name=None, report_definition=None, flat=True)[source]¶

Generate a report as plain python.

Parameters:	informers (list of AWSInformer, optional) – A list of informers on which to report. surveyors (list of AWSSurveyor, optional) – A list of surveyors on whose `informers()` list to report. report_name (str, optional) – The name of aa `aws_reporter.ReportDefinition` instance assigned to this reporter to be generated. report_definition (AWSReporter, optional) – An `aws_reporter.ReportDefinition` instance to be generated. flat (bool, default=True) – A flag to pass through to the report definition `extract_from()` method. See the documentation of that method for details.
Raises:	`IndexError` – If `report_names` contains any values that aren’t the `name` attribute of a member of `self.report_definitions()`. `TypeError` – If no report definitions are assigned to this reporter and `report_definitions` is not defined. `TypeError` – If neither `informers` nor `surveyors` is defined.
Returns:	The result of the selected `ReportDefinition` extracting from the list of informer targets.
Return type:	list

At least one of informers and surveyors is required, although the lists may be empty.

If both informers an surveyors are specified, the resulting report will include data from all informers in the former list and the latter’s informers() method.

Exactly one of report_name and report_definition is required.

Report Structure

The value returned from report() is the result of calling the ReportDefinition.extract_from() method on lists of informers. Thus the result will be a list of flat dicts (if the flat argument had the default True value) or nested dicts (if the flat argument was explicitly set to False).

E.g., if flat is True:

[{<flat record dict>}, {<flat record dict>}, ... ]

And if flat is explicitly set to False:

[{<nested informer dict>}, {<nested informer dict>}, ... ]

report_definitions(*names)[source]

Get assigned report definitions by name.

Parameters:	names (tuple of str) – The names of the reports to return.
Returns:	A list of report definitions whose names are those in the `names` tuple.
Return type:	list

report_names(*report_definitions)[source]¶

Get names of report definitions.

Returns:	All names of report definitions in the report_definitions tuple or in the report definitions assigned to this reporter.
Return type:	list

reports(informers=None, surveyors=None, report_names=None, report_definitions=None, flat=True)[source]¶

Generate multiple reports as plain python.

Parameters:	report_names (list of str, optional) – A list of names of `aws_reporter.ReportDefinition` instances assigned to this reporter that define the reports to be generated. report_definitions (list of AWSReporter, optional) – A list of `aws_reporter.ReportDefinition` instances that define the reports to be generated.
Raises:	`IndexError` – If `report_names` contains any values that aren’t the `name` attribute of a member of `self.report_definitions()`. `TypeError` – If no report definitions are assigned to this reporter and `report_definitions` is not defined.

See the documentation for report() for additional arguments, return values and exceptions raised.

Returns:	Each key in the dict is the name of a report, and the corresponding value is the result of that named `ReportDefinition` extracting from the list of informer targets.
Return type:	dict

This method calls the singular report() method once for each report name and report definition, and returns the results as a dict whose keys are the names of each report.

Reports Structure

The value returned from reports() is a dict where each value represents the result of calling the singular report() method for one of the selected report definitions with other arguments unchanged. Thus each value will be a list of flat dicts (if the flat argument had the default True value) or nested dicts (if the flat argument was explicitly set to False).

E.g., if flat is True:

{
    report_name_1: [{<flat dict>}, {<flat dict>}, ... ],
    report_name_2: [{<flat dict>}, {<flat dict>}, ... ],
    ...
    }

And if flat is explicitly set to False:

{
    report_name_1: [{<nested dict>}, {<nested dict>}, ... ],
    report_name_2: [{<nested dict>}, {<nested dict>}, ... ],
    ...
    }

sv_list(separator=', ', informers=None, surveyors=None, report_name=None, report_definition=None)[source]¶

Generate a string separated report as a list of lines.

Parameters:	separator (str, default=',') – The string with which to separate the values in each line.

See the documentation for report() for details on other arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

Returns: A list of comma-separated strings.

tabulizer(informers=None, surveyors=None, report_name=None, report_definition=None)[source]¶

Generate a report stored in a utensils.tabulizer.

See the documentation for report() for details on arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

Returns: A utensils.tabulizer.Tabulizer instance.

tabulizers(informers=None, surveyors=None, report_names=None, report_definitions=None)[source]¶

Generate multiple reports in utensils.tabulizer instances.

See the documentation for reports() for details on arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

Returns:	Each key in the dict is the name of a report, and the corresponding value is the result of calling `tabulizer` with that report specified.
Return type:	dict

tsv_list(informers=None, surveyors=None, report_name=None, report_definition=None)[source]¶

Generate a tab separated report as a list of lines.

See the documentation for report() for details on arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

Returns: A list of tab-separated strings.

write_csv(output_path, informers=None, surveyors=None, report_name=None, report_definition=None, overwrite=False)[source]¶

Write a comma separated report to a specified file.

This method is a pseudonym for write_sv() with separator=','. See the documentation for write_sv() for details on arguments and exceptions.

write_json(output_path, informers=None, surveyors=None, report_name=None, report_definition=None, flat=True, overwrite=False)[source]¶

Write a JSON report to a specified file.

Parameters:	output_path (string) – The path to the resulting workbook. overwrite (bool, default=False) – If `True`, overwrite any existing file already present.

See the documentation for write_report() for details on other arguments and exceptions.

write_sv(output_path, separator=', ', informers=None, surveyors=None, report_name=None, report_definition=None, overwrite=False)[source]¶

Write a string separated report to a specified file.

Parameters:	separator (str, default=',') – The string with which to separate the values in each line. output_path (string) – The path to the resulting workbook. overwrite (bool, default=False) – If `True`, overwrite any existing file already present.

See the documentation for report() for details on other arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

This method writes reports to a separated value file using the specified separator string.

Raises:	`ValueError` – If `overwrite` is `False` and a file already exists at the location specified by `output_path`.

Returns: None.

write_tsv(output_path, informers=None, surveyors=None, report_name=None, report_definition=None, overwrite=False)[source]¶

Write a tab separated report to a specified file.

This method is a pseudonym for write_sv() with separator='\t'. See the documentation for write_sv() for details on arguments and exceptions.

write_workbook(output_path, informers=None, surveyors=None, report_names=None, report_definitions=None, overwrite=False)[source]¶

Generate a report written to xlsxwriter worksheets.

Parameters:	output_path (string) – The path to the resulting workbook. overwrite (bool, default=False) – If `True`, overwrite any existing file already present.

See the documentation for report() for details on other arguments and exceptions. Note that flat is not an allowed argument for this method, as the reports must be flat for this format.

This method writes reports to worksheets in a xlsxwriter.Workbook instance passed as its first positional argument.

Raises:	`ValueError` – If `overwrite` is `False` and a file already exists at the location specified by `output_path`.

Returns: None.

class boogio.aws_reporter.ReportDefinition(name, entity_type, prune_specs=None, default_column_order=None, default_path_to_none=True)[source]¶

Bases: object

Manage AWSReporter report definitions.

Parameters:

name (string) – The descriptive name for the report.
entity_type (string) – The entity type of the AWSInformer subclass the report extracts.
prune_specs (list of dict, optional) – The specifications for the prune paths the report extracts.
default_column_order (list of str, optional) – An optional list of the prune paths defining the default columns and their order for columnar reports from this report definition.
default_path_to_none (bool, default=True) – The default value for the path_to_none item in each of the report definition’s prune specs. The path_to_none value for any prune spec entry that doesn’t explicitly define it will be set to this value.

copy()[source]¶: Return a copy of a ReportDefinition instance.

extract_from(informers, flat=True)[source]¶

Extract the fields specified by prune_specs.

Parameters:

informers (list of AWSInformer) – AWSInformer instances from which to extract report data. Any informer whose entity_type attribute doesn’t match the report definition’s entity_type attribute will be skipped.
flat (bool) – If True, return a flat result; if False, return a nested result. See the documentation for utensils.prune and utensils.flatten for more information.

prune_specs¶: The ReportDefinition prune_specs property.