Columnar typed data: the currency of deliveries, derived sources and components (spec/01 §1.1).
A table is named columns of declared types, every column the same length:
| Type | Holds | Stored as |
|---|---|---|
:number | integers and floats | one packed binary: signed 64-bit integers when every value is an integer that fits, else 64-bit floats |
:time | DateTimes | one packed binary of signed 64-bit microseconds since the Unix epoch, UTC |
:category, :text, :color, :point, :term | any value | a list |
A missing value is nil, in every type. A packed column keeps a validity bitmap
beside its binary, so a gap stays a gap: absence is never zero (spec/01 §3).
The packed columns live off the process heap. A binary over 64 bytes is reference-counted, so a garbage collection never copies it: a table of a million numbers costs its process a constant number of words, whatever its length.
Row lists are accepted at the edge. from/2 reads every shape data arrives in (a
list of row maps, a map or keyword list of columns, an Nx.Tensor, anything implementing
Table.Reader) into a table, and rows/1 is the row view for code that wants rows. The
row helpers (get/2, key/2, accessor/1, temporal?/1, time_columns/1,
column_types/1) read any source, as visualize's Visualize.Data.Table does; this
module's row readers are its port.
The table package and Nx are optional: without them lists and column maps still work,
and a struct source or a tensor raises UndefinedFunctionError where it is read.
Summary
Types
One column: its type, its storage and, when packed, its validity bitmap.
A column name, as the source spells it.
A row: a map keyed by column name, or a tuple in column order.
Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of
equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing
Table.Reader.
A table.
A column's declared type.
Functions
The accessor fn row -> get(row, name) end; a function is returned unchanged.
The values of column name, in order, nil where a value is missing; nil for a name
the table does not have. A number column of integers reads back integers; of floats, or
of both, floats. A time column reads back DateTimes in UTC at microsecond precision.
Each column's type, by name.
Every named column's type, read whole: all non-nil values temporal is :time, all
numbers :number, all strings :text, all atoms or booleans :category. A mixed or
empty column is left out. A suggestion for a tool, never something a reader relies on;
a table's own declared types are column_type/2's.
A table of any source rows/1 reads.
The value of column name in row, by either spelling of the name: the key itself, else
an atom name as its string and a string name as the atom key spelled the same. nil
when neither is a key. No atom is ever made.
The key of row that spells name: name itself when it is a key, else the key of the
other spelling, nil when neither is. The atom key a string finds is remembered for the
calling process, so reading one field from every row scans one row's keys. No atom is
ever made or looked up by name: the remembered key is one a row held.
A tensor's columns, or tuple rows' elements, under names, in order.
The column names, in order.
A table of columns, each {name, type, values}, in order.
As new/1, raising ArgumentError naming every fault.
Row i (from 0) as a map, or nil past the end.
The rows, one map per row keyed by every column name, nil where missing.
The number of rows.
The rows start to start + count - 1, as a table of the same columns: a packed
column's binary is sliced, not copied.
Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string
in ISO 8601 form that one of those parses.
The names of the columns whose values are all temporal: at least one value not nil,
and every value nil or temporal?/1. The whole column is read.
The column types, in the order spec/01 §1.1 lists them.
Types
One column: its type, its storage and, when packed, its validity bitmap.
A column name, as the source spells it.
A row: a map keyed by column name, or a tuple in column order.
Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of
equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing
Table.Reader.
@type t() :: %Macaw.Data.Table{ columns: %{required(name()) => {type(), column()}}, names: [name()], size: non_neg_integer() }
A table.
@type type() :: :number | :time | :category | :text | :color | :point | :term
A column's declared type.
Functions
The accessor fn row -> get(row, name) end; a function is returned unchanged.
The values of column name, in order, nil where a value is missing; nil for a name
the table does not have. A number column of integers reads back integers; of floats, or
of both, floats. A time column reads back DateTimes in UTC at microsecond precision.
Each column's type, by name.
Every named column's type, read whole: all non-nil values temporal is :time, all
numbers :number, all strings :text, all atoms or booleans :category. A mixed or
empty column is left out. A suggestion for a tool, never something a reader relies on;
a table's own declared types are column_type/2's.
iex> rows = [%{t: ~U[2024-01-01 00:00:00Z], v: 1, k: :a, s: "x", m: 1},
...> %{t: ~U[2024-01-02 00:00:00Z], v: 2.5, k: :b, s: "y", m: "two"}]
iex> Macaw.Data.Table.column_types(rows)
%{t: :time, v: :number, k: :category, s: :text}A table of any source rows/1 reads.
Columns are named by every key of every row map, in term order, a row without the key
missing there. Each column's type is types:'s (a map from name to type) when given,
else the one column_types/1 infers, else :term. A time column's values become
DateTimes here, at the edge: a Date at midnight UTC, a NaiveDateTime as UTC, an
ISO 8601 string parsed. A source whose rows are tuples has
no names; named/2 gives it some first. {:ok, table} or new/1's faults.
The value of column name in row, by either spelling of the name: the key itself, else
an atom name as its string and a string name as the atom key spelled the same. nil
when neither is a key. No atom is ever made.
The key of row that spells name: name itself when it is a key, else the key of the
other spelling, nil when neither is. The atom key a string finds is remembered for the
calling process, so reading one field from every row scans one row's keys. No atom is
ever made or looked up by name: the remembered key is one a row held.
iex> Macaw.Data.Table.key(%{cpu: 1}, "cpu")
:cpu
iex> Macaw.Data.Table.key(%{"cpu" => 1}, :cpu)
"cpu"
iex> Macaw.Data.Table.key(%{cpu: 1}, "mem")
nil@spec named(source(), [name()]) :: {:ok, source()} | {:error, {:width, non_neg_integer(), non_neg_integer()} | {:rank, non_neg_integer()}}
A tensor's columns, or tuple rows' elements, under names, in order.
Column i of a rank-2 tensor is Enum.at(names, i), and a rank-1 tensor is one column.
A tensor whose width is not length(names) is {:error, {:width, length(names), width}},
one of another rank {:error, {:rank, rank}}. Tuple rows are named row by row, element
i under Enum.at(names, i); a tuple of another size is {:error, {:width, n, size}}.
Any other source is {:ok, source}, unchanged.
iex> Macaw.Data.Table.named([{1, 4.0}, {2, 5.0}], ["x", "y"])
{:ok, [%{"x" => 1, "y" => 4.0}, %{"x" => 2, "y" => 5.0}]}
iex> Macaw.Data.Table.named([{1, 4.0, 7}], ["x", "y"])
{:error, {:width, 2, 3}}The column names, in order.
A table of columns, each {name, type, values}, in order.
{:ok, table}, or every fault: {[name], {:enum, types}} for an unknown type,
{[name, i], {:type, type}} for a value its type does not hold, {[name], {:length, n}}
for a column whose length is not the first column's n, {[name], :duplicate} for a
name given twice. A number column holds numbers; a time column DateTimes; a point
column {longitude, latitude} pairs of numbers; the others any value. nil is missing
in every type.
As new/1, raising ArgumentError naming every fault.
@spec row(t(), non_neg_integer()) :: map() | nil
Row i (from 0) as a map, or nil past the end.
The rows, one map per row keyed by every column name, nil where missing.
@spec size(t()) :: non_neg_integer()
The number of rows.
@spec slice(t(), non_neg_integer(), non_neg_integer()) :: t()
The rows start to start + count - 1, as a table of the same columns: a packed
column's binary is sliced, not copied.
Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string
in ISO 8601 form that one of those parses.
The names of the columns whose values are all temporal: at least one value not nil,
and every value nil or temporal?/1. The whole column is read.
@spec types() :: [type(), ...]
The column types, in the order spec/01 §1.1 lists them.