Quickstart

Most databases decompress a column before they can use it: ask for a max and every value expands to full width just to return one number. In l, a compact column stays fully usable — sortable, groupable, summable — through the same operations as any other value. Compression isn’t overhead you add and strip away; it’s the form the data computes in.

Start the REPL

With l on your path (see Installation), start a session:

l

You land at the l> prompt; \\ leaves.

l>1+1
2

Values, lists, tables

q reads right-to-left, no precedence, so 2*3+4 is 2*(3+4). Data is atoms, lists, dictionaries, and tables; a table is named columns side by side, so most of what follows is vector math.

l>2*3+4
14
l>til 5                              / a list
0 1 2 3 4
l>`a`b`c!10 20 30                    / a dictionary: keys!values
a| 10
b| 20
c| 30
l>t:([]sym:`IBM`MSFT`IBM`AAPL;px:120 340 121 175;sz:100 200 300 150)
l>t
sym  px  sz 
------------
IBM  120 100
MSFT 340 200
IBM  121 300
AAPL 175 150

Querying with q-sql

q-sql looks like SQL and reads right-to-left like the rest of q:

l>select sym,px from t where px>150
sym  px 
--------
MSFT 340
AAPL 175
l>select sum sz by sym from t        / group, then aggregate
sym | sz 
----| ---
AAPL| 150
IBM | 400
MSFT| 200
l>update val:px*sz from t            / derive a column
sym  px  sz  val  
------------------
IBM  120 100 12000
MSFT 340 200 68000
IBM  121 300 36300
AAPL 175 150 26250
l>exec sum sz from t                 / pull a single value out
750j

Data in and out

Values serialize to disk as-is and read straight back; tables round-trip through CSV with a header row.

l>`:/tmp/t set t                     / serialize the table (binary)
`:/tmp/t
l>t ~ get `:/tmp/t                   / read it back — identical
1b
l>`:/tmp/t.csv 0: csv 0: t           / write CSV
`:/tmp/t.csv
l>("SII"; enlist ",") 0: `:/tmp/t.csv   / read CSV back into a table
sym  px  sz 
------------
IBM  120 100
MSFT 340 200
IBM  121 300
AAPL 175 150

"SII" types the three columns as symbol, int, int; enlist "," sets the delimiter and asks for a table, taking column names from the header.

Compute on compressed

Assign a large, regular column to a name and l stores it compactly. Inspect it by name with -17!, which returns (compressed?; bytes stored; bytes raw):

l>v:10+til 1000000                   / a million consecutive integers
l>-17!`v
1b
40j
4000000j

Four million bytes to forty. Consecutive integers are almost entirely redundant, so l keeps a base and a step instead of the values. Inspect by name — the backtick matters: passing the value, -17!v, decompresses it first. For a friendlier spelling:

l>.Q.cinfo:{-17!x}
l>.Q.cinfo`v
1b
40j
4000000j

Every verb runs on the compressed column directly, without decompressing it:

l>sum v
500009500000j
l>max v
1000009
l>count where v>1000005
4

Not everything compresses. Store a table and inspect each column:

l>trade:([]sym:1000000?`A`B`C;price:1000000?100.0;size:100*1000000?50)
l>-17!`trade
sym  | 1b 4000032j 8000000j
price| 0b 8000000j 8000000j
size | 1b 2000032j 4000000j

Symbols and sizes compress; the random prices report 0b and keep full width — random doubles have no structure to exploit.

A little math, a little AI

q is also a numeric engine — whole-array math and matrices. A numerically stable softmax is one line:

l>softmax:{e%sum e:exp x-max x}
l>softmax 1 2 3 4.0
0.0320586 0.08714432 0.2368828 0.6439143

A dense layer is a matrix times a vector; the activation is just max against zero:

l>w:(0.1 0.2 0.3;-0.4 0.1 -0.2)      / a 2x3 weight matrix
l>w mmu 1 2 3.0                      / two pre-activations
1.4 -0.8
l>{0.0|x} w mmu 1 2 3.0              / relu zeroes the negative unit
1.4 0

Stored compressed, those weight rows are multiplied on their packed values — the same bandwidth saving, applied to linear algebra.