Global Function Parameters#

SaQC provides a number of global function parameters that are available in every SaQC function. These parameters modify the execution context of a function rather than its algorithmic behavior, which is controlled by function-specific parameters.

The following global parameters are available.

Aliasing#

label: The label keyword allows assigning a string alias to the execution of a SaQC function. This alias is used in subsequent calls to saqc.SaQC.plot() and can also be accessed within custom flagging schemes.

Flagging#

flag

By default, all SaQC functions mark detected anomalies using the anomaly flag and its flagging scheme specific external representation. This behavior can be modified by passing a flag argument. Detected anomalies will then be marked with the given value.

The provided value must be part of the active flagging scheme.

dfilter

Overrides the default filtering threshold defined by the active flagging scheme. For details on the filtering mechanism, see Filtering.

The data type of both arguments depends on the chosen flagging scheme.

For FloatScheme and AnnotatedFloatScheme, a floating-point value must be provided.
For SimpleScheme, only the literals "UNFLAGGED", "OK", and "BAD" are valid.

Temporal Specification#

start_date: Extends the flag-related masking mechanism by a temporal component. Only observations with timestamps greater than or equal to start_date are passed to the function.
end_date: Extends the flag-related masking mechanism by a temporal component. Only observations with timestamps less than or equal to end_date are passed to the function.

Both arguments may be provided as strings, pandas.Timestamp, or datetime.datetime objects.

While the latter two are interpreted exactly as given, string representations allow partial datetime specifications to restrict the temporal context of function execution.

Examples:

start_date="01:00" and end_date="04:00" Only observations between the first and fourth minute of every hour are processed.
start_date="15:00:00" and end_date="17:00:00" Only observations between 15:00 and 17:00 of every day are processed.
start_date="01T15:00:00" and end_date="13T17:30:00" Only observations between the first day at 15:00 and the 13th day at 17:30 of every month are processed.
start_date="01-01T00:00:00" and end_date="02-28T23:59:59" Only observations from January and February of every year are processed.

Examples#

Flagging Scheme Constraint#

The following examples assume the default flagging scheme, which is FloatScheme.

Example Data#

Let us generate some example data and plot it:

>>> import pandas as pd
>>> import numpy as np
>>> from saqc import SaQC
>>> noise = np.random.normal(0, 1, 200) # some normally distributed noise
>>> data = pd.Series(noise, index=pd.date_range('2020','2021',periods=200), name='data') # index the noise with some dates
>>> data.iloc[20] = 16 # add some artificial anomalies:
>>> data.iloc[100] = -17
>>> data.iloc[160:180] = -3
>>> qc = SaQC(data)
>>> qc.plot('data')

Label Keyword#

The label argument assigns a custom identifier to a SaQC function call. This directly affects subsequent calls to saqc.SaQC.plot().

It is especially useful for enriching figures with contextual information and for distinguishing results from different function calls.

>>> qc = SaQC(data)
>>> qc = qc.flagRange('data', max=15, label='values < 15')
>>> qc = qc.flagRange('data', min=-16, label='values > -16')
>>> qc.plot('data')

`dfilter` and `flag`#

To illustrate the interplay of flag and dfilter, we first assign custom flag levels:

>>> qc = SaQC(data)
>>> qc = qc.flagRange('data', max=15, label='flaglevel=200', flag=200)
>>> qc = qc.flagRange('data', min=-16, label='flaglevel=100', flag=100)
>>> qc.plot('data')

Using the dfilter keyword, we can control which observations are passed to a function. For example:

>>> qc.plot('data', dfilter=50)

Flag Separation#

Usually dfilter equals to the anomaly flag of the active flagging scheme.

If a function assigns this same flag value, subsequent calls will not process already flagged observations.

>>> qc = SaQC(data)
>>> qc = qc.flagRange('data', max=15, label='value > 15')
>>> qc = qc.flagRange('data', max=0, label='value > 0')
>>> qc.plot('data')

To re-test already flagged observations, increase or disable the filtering threshold:

>>> from saqc.constants import FILTER_NONE
>>> qc = qc.flagRange('data', max=0, label='value > 0', dfilter=FILTER_NONE)
>>> qc.plot('data')

Unflagging#

The flag keyword can also be used to remove flags from observations.

For FloatScheme, the internal unchecked flag is -np.inf. Alternatively, the constant UNFLAGGED may be used.

To override existing flags, the input filter must be raised or disabled:

>>> from saqc.constants import UNFLAGGED, FILTER_NONE
>>> qc = qc.flagConstants(
...     'data',
...     window='2D',
...     thresh=0,
...     dfilter=FILTER_NONE,
...     flag=UNFLAGGED
... )
>>> qc.plot('data')