bash - Wrapping a binary data file to self-convert to CSV? -

bash - Wrapping a binary data file to self-convert to CSV? -

i'm writing custom firmware sparkfun logomatic v2 records binary info file on 2gb micro-sd card. info file size range 100 mb 1 gb.

the format of binary info in flux board's firmware evolves (it dynamically reconfigurable @ run-time). rather create , maintain separate decoder/converter programme each version of firmware/configuration, i'd much rather create info files self-converting csv format starting info file bash script written info file before info recording starts.

i know how create here document, suspect bash unable parse , convert gigabyte of binary data, i'd create process run much faster having script first compile c code (assume gcc nowadays , in path), run resulting program, passing binary info stdin.

to create problem more concrete, assume firmware create binary info consisting of 4 16-bit integer values: timestamp (unsigned) followed 3 accelerometer axes (signed). there no separator between records (mainly because i'm saturating spi interface usd card).

so, think need script 2 here documents: 1 c code (parameterized expanded bash variables), , binary data. here's far:

#! env bash # produced firmware version 0.0.0.0.0.1 alpha # configuration info run: header_string = "time, x, y, z" column_count = 4 # create converter executable # utilize "<<-" permit code indented readability. # allow variable expansion/substitution. gcc -xc /tmp/convertit - <<-the_c_code #include <stdio.h> int main (int argc, char **argv) { // write ${header_string} stdout while (1) { // read $(column_count} shorts stdin // break if eof // write $(column_count} comma-delimited values stdout } // close stdout homecoming 0; } the_c_code # pass binary info converter # hard-quote here tag prevent subsequent expansion/substitution /tmp/convertit >./$1.csv <<'the_binary_data' ... ... hundreds of megabytes of semi-random info ... ... the_binary_data rm /tmp/convertit exit 0

does right? don't yet have real info file test with, wanted verify thought before going much further.

will bash complain if closing lines missing? may happen if info capture terminates unexpectedly due shock knocking loose battery or usd card. or if firmware borks.

is there faster or improve method should consider? example, wonder if bash slow re-create binary info fast c programme can consume it: should c programme open info file directly?

tia,

-bobc

you may want have @ makeself. allows alter .tar.gz archive self-extracting file platform independent (something shell script contains here document). allow distribute info , decoder. allows configure script contained within archive run when container script run. way can utilize makeself packaging , within archive can set info files , decoder written in c or bash or whatever language find suitable.

while possible decode binary info using shell tools (e.g. using od), it's cumbersome , ineffective. i'd recommend using either c programme or perl found on machine (check this page).

bash gcc binary-data