5.10. pypfilt.io

The pypfilt.io module provides functions for reading tabular data from data files.

5.10.1. Reading tabular data

pypfilt.io.read_table(path, columns, comment='#', encoding='utf-8')

Read data from a space-delimited text file with column headers defined in the first non-comment line.

Parameters:
  • path – The path to the data file.
  • columns – The columns to read from the data file, represented as a sequence of (name, type) tuples where type must be a NumPy scalar type, or (name, type, converter) tuples where converter is a function that converts the column string into the desired value.
  • comment – The characters, or list of characters, that indicate the start of a single-line comment.
  • encoding – The name of the encoding used to decode the file content.
Examples:
>>> from pypfilt.io import date_column, read_table
>>> import numpy as np
>>> import datetime
>>> path = "input_data.ssv"
>>> with open(path, 'w') as f:
...    _ = f.write('date value\n')
...    _ = f.write('2020-01-01 1\n')
...    _ = f.write('2020-01-02 3\n')
...    _ = f.write('2020-01-03 5\n')
>>> columns = [date_column('date'), ('value', np.int_)]
>>> data = read_table(path, columns)
>>> isinstance(data['date'][0], datetime.datetime)
True
>>> observations = [{'date': row['date'], 'value': row['value']}
...                 for row in data]
pypfilt.io.date_column(name, fmt='%Y-%m-%d')

Return a (name, type, converter) tuple that can be used with read_table() to convert a column into datetime.datetime values.

Note

Where dates are used for observation times, they should be represented as datetime.datetime values, not as datetime.date values. This is why this function returns a converter that returns datetime.datetime values.

Parameters:
  • name (str) – The column name in the data file.
  • fmt (str) – The date format used to parse the column values.
pypfilt.io.datetime_column(name, fmt='%Y-%m-%dT%H:%M:%S')

Return a (name, type, converter) tuple that can be used with read_table() to convert a column into datetime.datetime values.

Parameters:
  • name (str) – The column name in the data file.
  • fmt (str) – The datetime format used to parse the column values.

5.10.2. Lookup tables

The pypfilt.io module also provides lookup tables, which are used to retrieve time-indexed values (e.g., time-varying model inputs).

pypfilt.io.read_lookup_table(path, time, dtype='f8', comment='#', encoding='utf-8')

Read time-indexed data from a space-delimited text file with column headers defined in the first non-comment line.

Parameters:
  • path – The path to the data file.
  • time (pypfilt.time.Time) – The time scale.
  • dtype – The type of the lookup values.
  • comment – The characters, or list of characters, that indicate the start of a single-line comment.
  • encoding – The name of the encoding used to decode the file content.
Examples:
>>> from pypfilt.io import read_lookup_table, lookup
>>> from pypfilt.time import Datetime
>>> import datetime
>>> path = "input_data.ssv"
>>> with open(path, 'w') as f:
...    _ = f.write('date value1 value2 value3\n')
...    _ = f.write('2020-01-01 1.0 1.5 2.0\n')
>>> time = Datetime()
>>> table = read_lookup_table(path, time)
>>> isinstance(table['date'][0], datetime.datetime)
True
>>> when = datetime.datetime(2020, 1, 1)
>>> values = lookup(table, when)
>>> len(values.shape) == 1
True
>>> all(isinstance(value, float) for value in values)
True
pypfilt.io.lookup_values_count(lookup_table)

Return the number of value columns in a lookup table.

pypfilt.io.lookup(lookup_table, when)

Return the values associated with a specific time.