Parsing binary files e.g. jpeg files

[Skids]

I am creating an application that should read metadata from jpeg and other standard image files. My first attempt searches the file looking for so called markers. These markers are 2 byte hex pairs e.g. FF E1 . I have converted these to character values and have found the data. However, some of the data in the block is a Tiff file header and reading them as characters does not return the correct result.

A Tiff header is 8 bytes or characters long. Reading my test jpeg file in a hex editor or Exiftool shows that these 8 hex bytes are :

4D 4D 00 2A 00 00 97 22

At the moment my code reads these bytes as :

4D 4D 00 2A 00 00 F3 22

The byte that is in error should be 151 decimal which is above the standard ASCII values so I suspect that HyperXtalk is kindly converting the character to 243decimal on the fly.

I have concluded that relying on character based searches of binary data is not a good idea so I am wondering how else to search for binary/hex values in a file.

One possible method is to read the file into a variable using uint1 :

Case "uInt1"
         read from file pfileName at pStartByte for pByteCount uInt1
         /* Bytes are decoded to decimal and placed in varaible as comma sep data */
         break

As the comment in the code mentions this will output a comma separated list of decimal byte values. While I can work with such a list I am wondering if I am missing a more obvious way of searching for and reading binary data from a file.

Simon

[Brian_Milby]

Backing up a step, how are you getting the data into HXT initially? If reading in as a binfile you should get the exact bits in the file without adjustment by HXT.

[mwieder]

As Brian said, make sure you’re reading the file in binary mode, not text.

Secondly, you should be treating the data as bytes, not as characters.

[Skids]

Thanks for your comments. Yes I am reading the data as binary.

  put URL ("binfile:" & pFilePath) into tData

Working with tData requires some care as when it or any extracts from it are displayed in the IDE then any ASCII null characters are ignored when the variable contents are displayed and all other bytes are displayed as characters which should not be relied on.

this expands to:

showing the high byte values were previously ignored however the nulls are still not indicated so its best checked by copying into BBEdit or hex editor :

Reading a number of bytes from tData is achieved by using this lines similar to this :

if char tPos of tData is not numToByte(255) then exit repeat -- lost sync, bail

“char” and “byte” appear to be synonyms are read 8 bits. NumtoByte(n) is used to ensure the correct byte is searched for.

A second method is to read the file bytes in as a list of decimal numbers with a command like this :

Case "uInt1"
         read from file pfileName at (pStartByte+tOffset) for pByteCount uInt1
         /* Bytes are decoded to decimal and placed in varaible as comma sep data */

from

 Open File pfileName for binary read
   Switch pReadType 
      Case "uInt1"
         read from file pfileName at (pStartByte+tOffset) for pByteCount uInt1
         /* Bytes are decoded to decimal and placed in varaible as comma sep data */
         break
      Case "ASCII"
         read from file pfileName at (pStartByte+tOffset) for pByteCount
         break
      case "Bytes"
         --open file pfileName for binary read
         read from file pfileName at (pStartByte+tOffset) for pByteCount
         --close file pfileName
         break
   end Switch
   
   Close File pfileName
   

Reading the first 16 bytes of the same file yields a list of bytes that is displayed in a logical fashion in the IDE and is simple to understand and work with. Parsing a list of items may be slower than reading bytes directly but I don’t have any figures yet.

tData first 16 bytes displayed in the IDE :

For those who want to know bytes 1-2 indicate that the file is a jpeg file, bytes 3-4 is an App1 marker which indicates metadata, 5-6 is the length of the block, 7 onwards indicate that the block contains Exif data which will be written to comply with the Tiff specification. Bytes 15 and 16 are an ID which is always 00 42 which I take to be a reference to “The Hitch Hikers Guide”, the following bytes are a Tiff IFD.

One advantage of reading bytes as a list is that it is relatively simple to read multi byte ID numbers and pointer values which may be encoded as LSB or MSB depending on camera manufacturer. Jpegs are read MSB except inside EXIF blocks where the data may be LSB (bytes 13 and 14 are the indicator) MM (77,77) indicates the IFD in the sample file is MSB encoded.

I pass a list of bytes to this handler to extract numerical values

Private function HexToDecimal pBytes,pIsLittleEndian
   /* pBytes is a comma seperated list of bytes read from file
   */
   
   ## Each loop starts with the units, followed by squared, followed by pwr 4, pwr 6 ...
   put the number of items in pBytes into tByteCount
   put 0 into tPwr
   Switch pIsLittleEndian
      Case true
         Repeat for each item tByte in pBytes
            --put byteToNum(tByte) into tInt
            put tByte into tInt
            add tInt*(16^tPwr) to tValue
            add 2 to tPwr
         end Repeat
         break
      Case false
         Repeat with N = tByteCount down to 1
            --put byteToNum(item N of pBytes) into tInt
            put (item N of pBytes) into tInt
            add tInt*(16^tPwr) to tValue
            add 2 to tPwr
         end Repeat
         break
   end Switch
   
   return tValue
end HexToDecimal

Having spent some time playing with file bytes I think an option to display the contents of variables as decimal byte values in the IDE would be useful. Certainly being able to see null bytes in data would have saved me some head scratching.

S

[Brian_Milby]

It’s pretty easy to inspect as hex or decimal. Here’s a quick loop that puts the first 16 bytes of a jpg file as hex and decimal:

While true that char and byte seem to return the same value, due to the change to UTF16 in the engine it is better to use the correct term. When you want data, use byte to ensure that is what you are getting. Similarly, you should use byteToNum and numToByte for the conversions. Also, baseconvert is built in and means that you don’t have to do the math manually to convert from base 10 to 16 (only up to 4 bytes though).

You are correct that the variable inspector will not be that useful for binary data.

[Brian_Milby]

Another useful function is binaryDecode:

   put binaryDecode("H8",tData,tDecode) into tResult -- 4 bytes of hex
   put binaryDecode("I1",tData,tLittle) into tResult -- unsigned int from 4 bytes
   put binaryDecode("M1",tData,tBig) into tResult    -- unsigned int from 4 bytes

Big/Little is based on host, so it could be reversed I think. tResult is just the number of variables filled with data, 1 in each of the examples.

[mwieder]

I got there in a similar way:

-- after reading into tBinaryData:
local tConverted, tData
repeat for each byte tByte in tBinaryData
    get binaryDecode("C", tByte, tConverted)
    put tConverted & comma after tData
end repeat

But it also seems like you’re making a push for being able to display variables in the variable watcher in different ways, and that may be something to look into.

[Skids]

Thank you again for further clarification on which handlers should be used when reading bytes from a file. I have struggled to understand the differences between the various byte/char conversions options as explained in the dictionary. The entries appear to have been written from the point of view of someone wanting to read UTF encoded text forgetting that at times reading the base byte values is required.

But it also seems like you’re making a push for being able to display variables in the variable watcher in different ways, and that may be something to look into.

I think I would judge this as a “nice to have”. What would be good is some form of note that explains the limitations when variable contents are displayed in the IDE. In another post I noted that large numbers get displayed in scientific form, and this post identifies that the widget used ignores both null characters and high byte characters.

Perhaps the view single variable contents pane that may be selected in the IDE could be modified to offer the option to display byte values similar to the example above or BBEdit or a hex editor. I wasted time wondering why I was extracting the wrong bytes from the files when I was basing both the selection point and lengths from what I was seeing displayed in the IDE further complicated by Xcode starting files etc at byte 1 where as almost all documentation counts the first byte as byte 0. Who knew that adding one could be so difficult.

I now know better and print conversions of variable contents to a field named debug and or copy data from the IDE and paste it into BBEdit.

best wishes

Simon