Arrow interop

The Arrow interop package is an Apache Arrow IPC reader, writer, and streamer for l. l loads the shared library through 2:.

l-labs/arrow_interop

Load from l

.arrow.read:  `:build/arrow_io 2: (`arrow_read; 1)
.arrow.write: `:build/arrow_io 2: (`arrow_write; 1)
.arrow.stream:`:build/arrow_io 2: (`arrow_stream; 1)

First round trip

.arrow.write[([]sym:`AAPL`GOOG;price:150.5 175.3); `:/tmp/out.arrow]
t:.arrow.read `:/tmp/out.arrow
n:.arrow.stream[`:/tmp/huge.arrow; `:/tmp/db/t]

.arrow.read reads Arrow and Feather v2 files. .arrow.stream splays a large Arrow file to disk while keeping one batch in memory.

Type mapping

l type	Arrow type	Read	Write	Notes
KB bool	Bool	yes	yes	bitpacked on wire, byte-per-value in l
KG byte	UInt8	yes	yes	memcpy
KH, KI, KJ	Int16, Int32, Int64	yes	yes	nulls use validity bitmap
KE, KF	Float32, Float64	yes	yes	NaN maps to null
KS symbol	Utf8	yes	yes	interned on read
KD date	Date32 day	yes	yes	epoch shift 2000 to 1970
KT time	Time32 ms	yes	yes
KP timestamp	Timestamp ns	yes	yes	epoch shift, null-preserving
KN timespan	Duration ns	yes	yes	no epoch shift
KZ datetime	Timestamp ns	no	yes	reads back as KP
dictionary Utf8	Dictionary<Int32, Utf8>	yes	no	index to interned symbol
list	List<T>	yes	no	generic list of typed vectors

Nulls map between l sentinels such as 0Ni, 0Nj, and 0n and Arrow validity bitmaps.