README for DB-All.e Python bindings

The DB-All.e Python bindings provide 2 levels of access to a DB-All.e database: a complete API similar to the Fortran and C++ API, and a high-level API called volnd that allows to automatically export matrices of data out of the database.

Contents

The DB-All.e API

The 'dballe' module has a few global methods:

describe_level(ltype1=None, l1=None, ltype2=None, l2=None)
Return a string description for a level
describe_trange(pind=None, p1=None, p2=None)
Return a string description for a time range
var(varcode[, default])
Query the DB-All.e variable table returning a Var, optionally initialized with a value
varinfo(varcode)
Query the DB-All.e variable table returning a Varinfo

and several classes, documented in their own sections.

dballe.Var

Var holds a measured value, which can be integer, float or string, and a `wreport.Varinfo`_ with all available information (description, unit, precision, ...) related to it.

Var objects can be created from a `wreport.Varinfo`_ object, and an optional value. Omitting the value creates an unset variable.

Examples:

v = wreport.Var(table["B12101"], 32.5)
# v.info returns detailed informations about the variable in a Varinfo object.
print("%s: %s %s %s" % (v.code, str(v), v.info.unit, v.info.desc))

Members

code
variable code
info
Varinfo for this variable
isset
true if the value is set
enq()

enq() -> str|float|long

get the value of the variable, as int, float or str according the variable definition

enqc()

enqc() -> str

get the value of the variable, as a str

enqd()

enqd() -> float

get the value of the variable, as a float

enqi()

enqi() -> long

get the value of the variable, as an int

format(default='')

format(default="") -> str

return a string with the formatted value of the variable

get(default=None)

get(default=None) -> str|float|long|default

get the value of the variable, with a default if it is unset

dballe.Varinfo

Varinfo object holds all possible information about a variable, such as its measurement unit, description and number of significant digits.

Varinfo objects cannot be instantiated directly, and are created by querying `wreport.Vartable`_ objects.

Members

bit_len
number of bits used to encode the value in BUFR
bit_ref
reference value added after scaling, for BUFR decoding
code
variable code
desc
description
len
number of significant digits
scale
scale of the value as a power of 10
type
return a string describing the type of the variable (string, binary, integer, decimal)
unit
measurement unit

dballe.Vartable

Collection of Varinfo objects indexed by WMO BUFR/CREX table B code.

A Vartable is instantiated by the name (without extension) of the table file installed in wreport's data directory (normally, /usr/share/wreport/):

table = wreport.Vartable("B0000000000000023000")
print(table["B12101"].desc)

for i in table:
    print(i.code, i.desc)

Members

get_bufr
Vartable.get_bufr(basename=None, originating_centre=0, originating_subcentre=0,
master_table_number=0, master_table_version_number=None, master_table_version_number_local=0) -> wreport.Vartable

Look up a table B file using the information given, then load BUFR information from it.

You need to provide either basename or master_table_version_number.

get_crex
Vartable.get_crex(basename=None, edition_number=2,
originating_centre=0, originating_subcentre=0, master_table_number=0, master_table_version_number=None, master_table_version_number_bufr=None, master_table_version_number_local=0) -> wreport.Vartable

Look up a table B file using the information given, then load CREX information from it.

You need to provide either basename or master_table_version_number or master_table_version_number_bufr.

load_bufr

Vartable.load_bufr(pathname) -> wreport.Vartable

Load BUFR information from a Table B file and return it as a wreport.Vartable.

load_crex

Vartable.load_crex(pathname) -> wreport.Vartable

Load CREX information from a Table B file and return it as a wreport.Vartable.

pathname
name of the table

dballe.Record

A record holds a number of key->value mappings, similar to a dict.

Keys are strings, from a fixed set of keywords defined in DB-All.e and documented in the Fortran API documentation, plus all varcodes from the DB-All.e BUFR/CREX B table. Using unknown keys raises an exception.

Values are values internally stored inside dballe.Var objects, with a dballe.Varinfo corresponding to the key. Record's mapping interface directly exposes the variable values, but the dballe.Var objects are still accessible using Record.var(key).

A Record is used to make queries to the database, and read results.

When creating a new record, keyword arguments can be passed and they are set as if Record.update(**kwargs) had been called.

When setting items, either via __setitem__ and via update(), there are special keys that allow to set many items in a single call:

# Set year, month, day, hour, min, sec
rec["datetime"] = None
rec["datetime"] = datetime.datetime(2015, 06, 21, 12, 0, 0)
# Set yearmin..secmin and yearmax..secmax
rec["datetime"] = (None, None)
rec["datetime"] = (datetime.datetime(2014), None)
rec["datetime"] = (None, datetime.datetime(2015))
rec["datetime"] = (datetime.datetime(2014), datetime.datetime(2015))
# Set leveltype1, l1, leveltype2, l2
rec["level"] = None
rec["level"] = (103, 2000)
rec["level"] = (103, 2000, 103, 10000)
# Set pindicator, p1, p2
rec["trange"] = None
rec["trange"] = (205, 0, 10800)

Examples:

rec = Record(lat=44.05, lon=11.03, B12101=22.1)

# Key lookup returns the value of the corresponding variable
print(rec["lat"], rec["B12101"])

# Use .var() to get the variable itself; note that it can return
# None if the value is not set
print(rec.var("lat").info.desc, rec.var("B12101").info.desc)

# Iterating a record iterates all keys
for key in rec:
    print(key, rec.get(key, "undefined"), rec.var(key).info.desc)

# Use .vars() to get a list of only the varcode keys
for varcode in rec.vars():
    print(varcode, rec.get(varcode, "undefined"), rec.var(varcode).info.desc)

Members

clear()
remove all data from the record
clear_vars()
remove all variables from the record, leaving the keywords intact
copy()
return a deep copy of the Record
date_extremes()
(deprecated) get two datetime objects with the lower and upper bounds of the datetime period in this record
get(key, default=None)
lookup a value, returning a fallback value (None by default) if unset
items()
return a list with all the (key, value) tuples set in the Record.
key(key)
(deprecated) return a dballe.Var from the record, given its key.
keys()
return a list with all the keys set in the Record.
set_from_string(str)
(deprecated) set values from a 'key=val' string
set_station_context()
(deprecated) set the date, level and time range values to match the station data context
update(**kwargs)
set many record keys/vars in a single shot, via kwargs
var(key)
return a dballe.Var from the record, given its key.
varitems()
return a list with all the (key, dballe.Var) tuples set in the Record.
vars()
(deprecated) return a sequence with all the variables set on the Record. Note that this does not include keys.

dballe.DB

a DB is used to access the database.

Its members are:

attr_insert_data
Insert new attributes into the database
attr_insert_station
Insert new attributes into the database
attr_query_data
Query attributes
attr_query_station
Query attributes
attr_remove_data
Remove attributes
attr_remove_station
Remove attributes
insert_data
Insert data values in the database
insert_station_data
Insert station values in the database
query_station_data
Query the station variables in the database; returns a Cursor
remove_station_data
Remove station variables from the database
attr_insert(varcode, attrs, reference_id=None, replace=True)
Insert new attributes into the database
attr_remove(varcode, reference_id, attrs=None)
Remove attributes
connect_from_file(filename)
Create a DB connecting to a SQLite file
connect_from_url(url)
Create a DB as defined in an URL-like string
connect_test()
Create a DB for running the test suite, as configured in the test environment
disappear()
Remove all our traces from the database, if applicable.
export_to_file(query, format, filename, generic=False)
Export data matching a query as bulletins to a named file
insert(record, can_replace=False, can_add_stations=False)
(deprecated)Insert a record in the database
is_url(string)
Checks if a string looks like a DB url
load(file)

load(fp, encoding=None)

Load a file object in the database. An encoding can optionally be provided as a string ("BUFR", "CREX", "AOF"). If encoding is None then load will try to autodetect based on the first byte of the file.

query_attrs(varcode, reference_id, attrs=None)
Query attributes
query_data(query)
Query the variables in the database; returns a Cursor
query_stations(query)
Query the station archive in the database; returns a Cursor
query_summary(query)
Query the summary of the results of a query; returns a Cursor
remove(query)
Remove variables from the database
reset([repinfo_filename])
Reset the database, removing all existing Db-All.e tables and re-creating them empty.
vacuum()
Perform database cleanup operations

Examples:

# Connect to a database and run a query
db = dballe.DB.connect_from_file("db.sqlite")
query = dballe.Record(latmin=44.0, latmax=45.0, lonmin=11.0, lonmax=12.0)

# The result is a dballe.Cursor, which can be iterated to get results as
# dballe.Record objects.
# The results always point to the same Record to avoid creating a new one
# for every iteration: if you need to store them, use Record.copy()
for rec in db.query_data(query):
    print rec["lat"], rec["lon"], rec["var"], rec.var().format("undefined")

# Insert 2 new variables in the database
rec = dballe.Record(
    lat=44.5, lon=11.4,
    level=(1,),
    trange=(254,),
    date=datetime.datetime(2013, 4, 25, 12, 0, 0),
    B11101=22.4,
    B12103=17.2,
)
db.insert(rec)

dballe.Cursor

a Cursor is the result of database queries. It is generally not used explicitly and just iterated, but it does have a few members:

attr_query
Query attributes for the current variable
remaining
number of results still to be returned
query_attrs(attrs=None)
Query attributes for the current variable

The volnd API

volnd is an easy way of extracting entire matrixes of data out of a DB-All.e database.

This module allows to extract multidimensional matrixes of data given a list of dimension definitions. Every dimension definition defines what kind of data goes along that dimension.

Dimension definitions can be shared across different extracted matrixes and multiple extractions, allowing to have different matrixes whose indexes have the same meaning.

This example code extracts temperatures in a station by datetime matrix:

query = dballe.Record()
query["var"] = "B12001"
query["rep_memo"] = "synop"
query["level"] = (105, 2)
query["trange"] = (0,)
vars = read(self.db.query(query), (AnaIndex(), DateTimeIndex()))
data = vars["B12001"]
# Data is now a 2-dimensional Masked Array with the data
#
# Information about what values correspond to an index in the various
# directions can be accessed in data.dims, which contains one list per
# dimension with all the information corresponding to every index.
print("Ana dimension is", len(data.dims[0]), "items long")
print("Datetime dimension is", len(data.dims[1]), "items long")
print("First 10 stations along the Ana dimension:", data.dims[0][:10])
print("First 10 datetimes along the DateTime dimension:", data.dims[1][:10])

This is the list of dimensions supported by dballe.volnd:

AnaIndex

Index for stations, as they come out of the database.

The constructor syntax is: AnaIndex(shared=True, frozen=False, start=None).

The index saves all stations as AnaIndexEntry tuples, in the same order as they come out of the database.

NetworkIndex

Index for networks, as they come out of the database.

The constructor syntax is: NetworkIndex(shared=True, frozen=False, start=None).

The index saves all networks as NetworkIndexEntry tuples, in the same order as they come out of the database.

LevelIndex

Index for levels, as they come out of the database

The constructor syntax is: LevelIndex(shared=True, frozen=False), start=None.

The index saves all levels as dballe.Level tuples, in the same order as they come out of the database.

TimeRangeIndex

Index for time ranges, as they come out of the database.

The constructor syntax is: TimeRangeIndex(shared=True, frozen=False, start=None).

The index saves all time ranges as dballe.TimeRange tuples, in the same order as they come out of the database.

DateTimeIndex

Index for datetimes, as they come out of the database.

The constructor syntax is: DateTimeIndex(shared=True, frozen=False, start=None).

The index saves all datetime values as datetime.datetime objects, in the same order as they come out of the database.

IntervalIndex

Index into equally spaced points in time, starting at start, with a point every step time.

Index points are at fixed time intervals, and data is acquired in one point only if it is within a given tolerance from the interval.

The constructor syntax is: IntervalIndex(start, step, tolerance=0, end=None, shared=True, frozen=False).

start is a datetime.datetime object giving the starting time of the time interval of this index.

step is a datetime.timedelta object with the interval between sampling points.

tolerance is a datetime.timedelta object specifying the maximum allowed interval between a datum datetime and the sampling step. If the interval is bigger than the tolerance, the data is discarded.

end is an optional datetime.datetime object giving the ending time of the time interval of the index. If omitted, the index will end at the latest accepted datum coming out of the database.

The data object used by AnaIndex is:

AnaIndexEntry

AnaIndex entry, with various data about a single station.

It is a named tuple of 4 values:
  • id: station id
  • lat: latitude
  • lon: longitude
  • ident: mobile station identifier, or None

The extraction is done using the dballe.volnd.read function:

read(cursor, dims, filter=None, checkConflicts=True, attributes=None)

cursor is a dballe.Cursor resulting from a dballe query

dims is the sequence of indexes to use for shaping the data matrixes

filter is an optional filter function that can be used to discard values from the query: if filter is not None, it will be called for every output record and if it returns False, the record will be discarded

checkConflicts tells if we should raise an exception if two values from the database would fill in the same position in the matrix

attributes tells if we should read attributes as well: if it is None, no attributes will be read; if it is True, all attributes will be read; if it is a sequence, then it is the sequence of attributes that should be read.

The result of dballe.volnd.read is a dict mapping output variable names to a dballe.volnd.Data object with the results. All the Data objects share their indexes unless the xxx-Index definitions have been created with shared=False.

This is the dballe.volnd.Data class documentation:

Data

Container for collecting variable data. It contains the variable data array and the dimension indexes.

If v is a Data object, you can access the tuple with the dimensions as v.dims, and the masked array with the values as v.vals.

The methods of dballe.volnd.Data are:

append(self, rec)

Collect a new value from the given dballe record.

You need to call finalise() before the values can be used.

appendAttrs(self, rec, codes=None)

Collect attributes to append to the record.

You need to call finalise() before the values can be used.

finalise(self)
Stop collecting values and create a masked array with all the values collected so far.

Docutils System Messages

System Message: ERROR/3 (python-dballe.rst, line 36); backlink

Unknown target name: "wreport.varinfo".

System Message: ERROR/3 (python-dballe.rst, line 40); backlink

Unknown target name: "wreport.varinfo".

System Message: ERROR/3 (python-dballe.rst, line 98); backlink

Unknown target name: "wreport.vartable".