Parquet interop

The Parquet interop package is an Apache Parquet reader, writer, and streamer for l. The Parquet format work is handled by the Arrow Rust parquet crate; l loads the resulting shared library through 2:.

l-labs/parquet_interop

Load from l

.pq.read:  `:target/release/libl_parquet 2: (`pq_read; 1)
.pq.write: `:target/release/libl_parquet 2: (`pq_write; 1)
.pq.stream:`:target/release/libl_parquet 2: (`pq_stream; 1)

First round trip

.pq.write[([]sym:`AAPL`GOOG;price:150.5 175.3); `:/tmp/out.parquet]
t:.pq.read `:/tmp/out.parquet
n:.pq.stream[`:/tmp/huge.parquet; `:/tmp/db/t]

.pq.stream writes native splay files chunk by chunk and patches counts at the end, so get and \l can load the result directly.

Type mapping

l type	Parquet or Arrow type	Read	Write	Notes
KB bool	Boolean	yes	yes	null reads as 0b
KG byte	UInt8, Int8 accepted	yes	yes
KH, KI, KJ	Int16, Int32, Int64	yes	yes	0Ni and 0Nj map to null
KE, KF	Float32, Float64	yes	yes	0n maps to null for f64
KS symbol	Utf8	yes	yes	dictionary-encoded on disk
KD date	Date32 day	yes	yes	epoch shift 2000 to 1970
KP timestamp	Timestamp s, ms, us, ns to ns	yes	yes	epoch shift, null-preserving
KN timespan	Duration ns	yes	yes	via embedded Arrow schema
KZ datetime	Timestamp ns	no	yes	reads back as KP

Nested columns such as List, Struct, and Map raise 'nyi.