Record-heterogeneity
Description: Miller documentation
Keywords: John Kerl, Kerl, Miller, miller, mlr, OLAP, data analysis software, regression, correlation, variance, data tools,

Miller Overview: • About Miller • Miller in 10 minutes • File formats • Miller features in the context of the Unix toolkit • Record-heterogeneity • Internationalization Using Miller: • FAQ • Sharing data with other languages • Cookbook part 1 • Cookbook part 2 • Cookbook part 3 • Data-diving examples • Manpage • Reference • Reference: Verbs • Reference: DSL • Documents by release • Installation, portability, dependencies, and testing Background: • Why? • Why C? • Why call it Miller? • How original is Miller? • Performance Repository: • Things to do • Contact information • GitHub repo | Record-heterogeneity Contents: • For I/O • CSV and pretty-print • Key-value-pair, vertical-tabular, and index-numbered formats • For processing We think of CSV tables as rectangular: if there are 17 columns in the header then there are 17 columns for every row, else the data have a formatting error. But heterogeneous data abound (today’s no-SQL databases for example). Miller handles this. # For I/O ## CSV and pretty-print Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:
    $ cat data/het.dkvp
    resource=/path/to/file,loadsec=0.45,ok=true
    record_count=100,resource=/path/to/file
    resource=/path/to/second/file,loadsec=0.32,ok=true
    record_count=150,resource=/path/to/second/file
    resource=/some/other/path,loadsec=0.97,ok=false
    $ mlr --ocsvlite cat data/het.dkvp
    resource,loadsec,ok
    /path/to/file,0.45,true

    record_count,resource
    100,/path/to/file

    resource,loadsec,ok
    /path/to/second/file,0.32,true

    record_count,resource
    150,/path/to/second/file

    resource,loadsec,ok
    /some/other/path,0.97,false
    $ mlr --opprint cat data/het.dkvp
    resource      loadsec ok
    /path/to/file 0.45    true

    record_count resource
    100          /path/to/file

    resource             loadsec ok
    /path/to/second/file 0.32    true

    record_count resource
    150          /path/to/second/file

    resource         loadsec ok
    /some/other/path 0.97    false
You may also find Miller’s `group-like` feature handy (see also Reference):
    $ mlr --ocsvlite group-like data/het.dkvp
    resource,loadsec,ok
    /path/to/file,0.45,true
    /path/to/second/file,0.32,true
    /some/other/path,0.97,false

    record_count,resource
    100,/path/to/file
    150,/path/to/second/file
    $ mlr --opprint group-like data/het.dkvp
    resource             loadsec ok
    /path/to/file        0.45    true
    /path/to/second/file 0.32    true
    /some/other/path     0.97    false

    record_count resource
    100          /path/to/file
    150          /path/to/second/file
## Key-value-pair, vertical-tabular, and index-numbered formats For these formats, record-heterogeneity comes naturally:
    $ cat data/het.dkvp
    resource=/path/to/file,loadsec=0.45,ok=true
    record_count=100,resource=/path/to/file
    resource=/path/to/second/file,loadsec=0.32,ok=true
    record_count=150,resource=/path/to/second/file
    resource=/some/other/path,loadsec=0.97,ok=false
    $ mlr --onidx --ofs ' ' cat data/het.dkvp
    /path/to/file 0.45 true
    100 /path/to/file
    /path/to/second/file 0.32 true
    150 /path/to/second/file
    /some/other/path 0.97 false
    $ mlr --oxtab cat data/het.dkvp
    resource /path/to/file
    loadsec  0.45
    ok       true

    record_count 100
    resource     /path/to/file

    resource /path/to/second/file
    loadsec  0.32
    ok       true

    record_count 150
    resource     /path/to/second/file

    resource /some/other/path
    loadsec  0.97
    ok       false
    $ mlr --oxtab group-like data/het.dkvp
    resource /path/to/file
    loadsec  0.45
    ok       true

    resource /path/to/second/file
    loadsec  0.32
    ok       true

    resource /some/other/path
    loadsec  0.97
    ok       false

    record_count 100
    resource     /path/to/file

    record_count 150
    resource     /path/to/second/file
# For processing Miller operates on specified fields and takes the rest along: for example, if you are sorting on the `count` field then all records in the input stream must have a `count` field but the other fields can vary, and moreover the sorted-on field name(s) don’t need to be in the same position on each line:
    $ cat data/sort-het.dkvp
    count=500,color=green
    count=600
    status=ok,count=250,hours=0.22
    status=ok,count=200,hours=3.4
    count=300,color=blue
    count=100,color=green
    count=450
    $ mlr sort -n count data/sort-het.dkvp
    count=100,color=green
    status=ok,count=200,hours=3.4
    status=ok,count=250,hours=0.22
    count=300,color=blue
    count=450
    count=500,color=green
    count=600