Data Quality Package

Inheritance

Submodules

Models

SEED Platform (TM), Copyright (c) Alliance for Sustainable Energy, LLC, and other contributors. See also https://github.com/SEED-platform/seed/blob/main/LICENSE.md

exception seed.models.data_quality.ComparisonError: Bases: Exception

class seed.models.data_quality.DataQualityCheck(*args, **kwargs)

Bases: Model

Object that stores the high level configuration per organization of the DataQualityCheck

exception DoesNotExist: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: MultipleObjectsReturned

REQUIRED_FIELDS = {'PropertyState': ['address_line_1', 'custom_id_1', 'pm_property_id'], 'TaxLotState': ['address_line_1', 'custom_id_1', 'jurisdiction_tax_lot_id']}

add_goal_rule_labels(rule)

add_invalid_geometry_entry_provided(row_id, rule, display_name, value)

add_result_comparison_error(row_id, rule, display_name, value, rule_check)

add_result_dimension_error(row_id, rule, display_name, value)

add_result_is_null(row_id, rule, display_name, value)

add_result_max_error(row_id, rule, display_name, value, rule_max)

add_result_min_error(row_id, rule, display_name, value, rule_min)

add_result_missing_and_none(row_id, rule, display_name, value)

add_result_missing_req(row_id, rule, display_name, value)

add_result_range_error(row_id, rule, display_name, value)

add_result_string_error(row_id, rule, display_name, value)

add_result_type_error(row_id, rule, display_name, value)

add_rule(rule)

Add a new rule to the Data Quality Checks

Parameters:: rule – dict to be added as a new rule
Returns:: None

add_rule_if_new(rule)

Add a new rule to the Data Quality Checks only if rule does not exist

Parameters:: rule – dict to be added as a new rule
Returns:: None

static cache_key(identifier, organization_id)

Static method to return the location of the data_quality results from redis.

Parameters:: identifier – Import file primary key
Returns:

check_data(record_type, rows)

Send in data as a queryset from the Property/Taxlot ids.

Parameters:

record_type – one of PropertyState | TaxLotState
rows – rows of data to be checked for data quality

Returns:

None

check_data_cross_cycle(goal_id, state_pairs): Send in data as properties and their goal state pairs. Update GoalNote.passed_checks with data quality check return :param goal: :param state_pairs: [{property: Property, baseline: PropertyState, current: PropertyState}, {}, …]

get_fieldnames(record_type): Get fieldnames to apply to results.

get_value(property_obj, data_type, goal, cycle_key)

id: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

init_result(record_type, row, fields)

static initialize_cache(identifier, organization_id)

Initialize the cache for storing the results. This is called before the celery tasks are chunked up.

The cache_key is different than the identifier. The cache_key is where all the results are to be stored for the data quality checks, the identifier, is the random number (or specified value that is used to identifier both the progress and the data storage

Parameters:: identifier – Identifier for cache, if None, then creates a random one
Returns:: list, [cache_key and the identifier]

initialize_rules()

Initialize the default rules for a DataQualityCheck object

Returns:: None

name: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <django.db.models.manager.Manager object>

organization

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

organization_id

prune_results()

remove_all_rules()

Removes all the rules associated with this DataQualityCheck instance.

Returns:: None

remove_status_label(label_class, rule, linked_id)

Remove label because it did not match any of the range exceptions

Parameters:

label_class – statuslabel object, either property label or taxlot label
rule – rule object
linked_id – id of propertystate or taxlotstate object

Returns:

boolean, if labeled was applied

reset_all_rules()

Delete all rules and reinitialize the default set of rules

Returns:: None

reset_default_rules()

Reset only the default rules

Returns:

reset_results()

classmethod retrieve(organization_id)

DataQualityCheck was previously a simple object but has been migrated to a django model. This method ensures that the data quality model will be backwards compatible.

This is the preferred method to initialize a new object.

Parameters:: organization – instance of Organization
Returns:: obj, DataQualityCheck

retrieve_result_by_address(address)

Retrieve the results of the data quality checks for a specific address.

Parameters:: address – string, address to find the result for
Returns:: dict, results of data quality check for specific building

retrieve_result_by_tax_lot_id(tax_lot_id)

Retrieve the results of the data quality checks by the jurisdiction ID.

Parameters:: tax_lot_id – string, jurisdiction tax lot id
Returns:: dict, results of data quality check for specific building

rules

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Parent.children is a ReverseManyToOneDescriptor instance.

Most of the implementation is delegated to a dynamically defined manager class built by create_forward_many_to_many_manager() defined below.

save_to_cache(identifier, organization_id)

Save the results to the cache database. The data in the cache are stored as a list of dictionaries. The data in this class are stored as a dict of dict. This is important to remember because the data from the cache cannot be simply loaded into the above structure.

Parameters:: identifier – Import file primary key
Returns:: None

update_status_label(label_class, rule, linked_id, row_id, add_to_results=True)

Parameters:

label_class – statuslabel object, either propertyview label or taxlotview label
rule – rule object
linked_id – id of propertyview or taxlotview object
row_id –
add_to_results – bool

Returns:

boolean, if labeled was applied

exception seed.models.data_quality.DataQualityTypeCastError: Bases: Exception

class seed.models.data_quality.Rule(*args, **kwargs)

Bases: Model

Rules for DataQualityCheck

DATA_TYPES = [(0, 'number'), (1, 'string'), (2, 'date'), (3, 'year'), (4, 'area'), (5, 'eui')]

DEFAULT_RULES = [{'condition': 'not_null', 'data_type': 1, 'field': 'address_line_1', 'not_null': True, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'not_null', 'data_type': 1, 'field': 'pm_property_id', 'not_null': True, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'not_null', 'field': 'custom_id_1', 'not_null': True, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'not_null', 'field': 'jurisdiction_tax_lot_id', 'not_null': True, 'rule_type': 0, 'severity': 0, 'table_name': 'TaxLotState'}, {'condition': 'not_null', 'data_type': 1, 'field': 'address_line_1', 'not_null': True, 'rule_type': 0, 'severity': 0, 'table_name': 'TaxLotState'}, {'condition': 'range', 'data_type': 4, 'field': 'conditioned_floor_area', 'max': 7000000, 'min': 0, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState', 'units': 'ft**2'}, {'condition': 'range', 'data_type': 4, 'field': 'conditioned_floor_area', 'min': 100, 'rule_type': 0, 'severity': 1, 'table_name': 'PropertyState', 'units': 'ft**2'}, {'condition': 'range', 'data_type': 0, 'field': 'energy_score', 'max': 100, 'min': 0, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'range', 'data_type': 0, 'field': 'energy_score', 'min': 10, 'rule_type': 0, 'severity': 1, 'table_name': 'PropertyState'}, {'condition': 'range', 'data_type': 2, 'field': 'generation_date', 'max': '20241231', 'min': 18890101, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'range', 'data_type': 0, 'field': 'gross_floor_area', 'max': 7000000, 'min': 100, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState', 'units': 'ft**2'}, {'condition': 'range', 'data_type': 0, 'field': 'occupied_floor_area', 'max': 7000000, 'min': 100, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState', 'units': 'ft**2'}, {'condition': 'range', 'data_type': 2, 'field': 'recent_sale_date', 'max': '20241231', 'min': 18890101, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'range', 'data_type': 2, 'field': 'release_date', 'max': '20241231', 'min': 18890101, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'range', 'data_type': 5, 'field': 'site_eui', 'max': 1000, 'min': 0, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState', 'units': 'kBtu/ft**2/year'}, {'condition': 'range', 'data_type': 5, 'field': 'site_eui', 'min': 10, 'rule_type': 0, 'severity': 1, 'table_name': 'PropertyState', 'units': 'kBtu/ft**2/year'}, {'condition': 'range', 'data_type': 5, 'field': 'site_eui_weather_normalized', 'max': 1000, 'min': 0, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState', 'units': 'kBtu/ft**2/year'}, {'condition': 'range', 'data_type': 5, 'field': 'source_eui', 'max': 1000, 'min': 0, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState', 'units': 'kBtu/ft**2/year'}, {'condition': 'range', 'data_type': 5, 'field': 'source_eui', 'min': 10, 'rule_type': 0, 'severity': 1, 'table_name': 'PropertyState', 'units': 'kBtu/ft**2/year'}, {'condition': 'range', 'data_type': 5, 'field': 'source_eui_weather_normalized', 'max': 1000, 'min': 10, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState', 'units': 'kBtu/ft**2/year'}, {'condition': 'range', 'data_type': 3, 'field': 'year_built', 'max': '2024', 'min': 1700, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'range', 'data_type': 2, 'field': 'year_ending', 'max': '20241231', 'min': 18890101, 'rule_type': 0, 'severity': 0, 'table_name': 'PropertyState'}, {'condition': 'range', 'cross_cycle': True, 'data_type': 5, 'field': 'eui', 'max': 40, 'name': 'High EUI % Change', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'range', 'cross_cycle': True, 'data_type': 5, 'field': 'eui', 'min': -40, 'name': 'Low EUI % Change', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'range', 'cross_cycle': True, 'data_type': 4, 'field': 'area', 'max': 5, 'name': 'High Area % Change', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'range', 'cross_cycle': True, 'data_type': 4, 'field': 'area', 'min': -5, 'name': 'Low Area % Change', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'range', 'data_type': 5, 'field': 'eui', 'max': 1000, 'name': 'High EUI', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'range', 'data_type': 5, 'field': 'eui', 'min': 40, 'name': 'Low EUI', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'range', 'data_type': 4, 'field': 'area', 'max': 1000000, 'name': 'High Area', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'range', 'data_type': 4, 'field': 'area', 'min': 1000, 'name': 'Low Area', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'not_null', 'data_type': 5, 'field': 'eui', 'name': 'Missing EUI', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}, {'condition': 'not_null', 'data_type': 4, 'field': 'area', 'name': 'Missing Area', 'rule_type': 0, 'severity': 0, 'table_name': 'Goal'}]

exception DoesNotExist: Bases: ObjectDoesNotExist

exception MultipleObjectsReturned: Bases: MultipleObjectsReturned

RULE_EXCLUDE = 'exclude'

RULE_INCLUDE = 'include'

RULE_NOT_NULL = 'not_null'

RULE_RANGE = 'range'

RULE_REQUIRED = 'required'

RULE_TYPE = [(0, 'default'), (1, 'custom')]

RULE_TYPE_CUSTOM = 1

RULE_TYPE_DEFAULT = 0

SEVERITY = [(0, 'error'), (1, 'warning'), (2, 'valid')]

SEVERITY_ERROR = 0

SEVERITY_VALID = 2

SEVERITY_WARNING = 1

TYPE_AREA = 4

TYPE_DATE = 2

TYPE_EUI = 5

TYPE_NUMBER = 0

TYPE_STRING = 1

TYPE_YEAR = 3

condition: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

cross_cycle: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

data_quality_check

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

data_quality_check_id

data_type: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

description: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

enabled: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

field: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

for_derived_column: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

format_strings(value)

get_data_type_display(*, field=<django.db.models.fields.IntegerField: data_type>)

get_rule_type_display(*, field=<django.db.models.fields.IntegerField: rule_type>)

get_severity_display(*, field=<django.db.models.fields.IntegerField: severity>)

id: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

max: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

maximum_valid(value)

Validate that the value is not greater than the maximum specified by the rule.

Parameters:: value – Value to validate rule against
Returns:: bool, True is valid, False if the value is out of range

min: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

minimum_valid(value)

Validate that the value is not less than the minimum specified by the rule.

Parameters:: value – Value to validate rule against
Returns:: bool, True is valid, False if the value is out of range

name: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

not_null: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

objects = <django.db.models.manager.Manager object>

required: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

rule_type: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

severity: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

status_label

Accessor to the related object on the forward side of a many-to-one or one-to-one (via ForwardOneToOneDescriptor subclass) relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

Child.parent is a ForwardManyToOneDescriptor instance.

status_label_id

str_to_data_type(value)

If the check is coming from a field in the database then it will be typed correctly; however, for extra_data, the values are typically strings or unicode. Therefore, the values are typed before they are checked using the rule’s data type definition.

Parameters:: value – variant, value to type
Returns:: typed value

table_name: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

text_match: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

units: A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

valid_text(value)

Validate the rule matches the specified text. Text is matched by regex.

Parameters:: value – Value to validate rule against
Returns:: bool, True is valid, False if the value does not match

exception seed.models.data_quality.UnitMismatchError: Bases: Exception

seed.models.data_quality.format_pint_violation(rule, source_value)

Format a pint min, max violation for human readability.

:param rule :param source_value : Quantity - value to format into range :return (formatted_value, formatted_min, formatted_max) : (String, String, String)

Data Quality Package

Inheritance

Submodules

Models

Tests

Views