Documentation Issue with revXML

General
1

I have just spent four or so hours debugging some code that reads meta data from image files. Many image files contain both IPTC data along with XMP data. The issue I was seeing was when reading the “rights” string from the XMP block the copyright symbol was preceded by a symbol. I asked my new best friend Claude and we fumbled forward.

“¬©” is classic mojibake from double-encoding — specifically, UTF-8 bytes being misread as MacRoman (single-byte) text.

Long storey short the revXML commands return binary values that should be read as UTF-8.

I was naive and thought that because the XMP block includes details of text encoding that the data returned could be used directly, but it can’t - Doh! Claude then informed my that this is a well known “feature” that has been discussed on the forums.

I have looked at the dictionary entries and can see no mention of how the data returned is encoded i.e. as a string of bytes that should be read as a UTF-8 encoded text string.

If what Claude writes below is accurate then it may help other naive users if it is added to the dictionary.

What are your thoughts ?

Simon

Snips from my conversation with Claude:

Me - Is it a bug?

Not really a bug — it’s a documented, longstanding design quirk of how revXML interacts with LiveCode’s text model, and it’s well known in the LiveCode community.

The short version: revXMLNodeContents (and the other revXML functions) are thin wrappers around the underlying libxml2 C library, which parses and returns text as raw UTF-8 bytes. But these functions predate LiveCode’s native Unicode string type and were never updated to decode that UTF-8 into LiveCode’s internal representation before returning it. So the function does its job correctly (it extracts exactly the text the XML/XMP contains) but hands it back to you as an undecoded byte string rather than a proper Unicode value — leaving the decoding step as something the caller has always been expected to do manually.

This is exactly reflected in the LiveCode community’s own troubleshooting pattern: users report that content pulled with revXMLNodeContents shows corrupted characters, and the standard fix passed around the forums is to re-encode it with uniEncode/uniDecode or textEncode/textDecode after extraction — the same workaround you’ve now confirmed works for you. You can see the same pattern repeated across multiple independent forum threads going back over a decade, which tells you this isn’t a one-off regression, it’s just baseline expected behavior that never got fixed at the API level. LiveCode Forums

Why it’s arguably not classified as a “bug” by LiveCode: LiveCode’s own Unicode documentation notes that the engine’s internal text representation is UTF-16, while UTF-8 is a separate encoding commonly used on macOS/Linux — conversion between the two is something the developer has to manage explicitly rather than something that happens automatically everywhere. revXML was built as a fairly direct libxml2 binding, so it inherits libxml2’s UTF-8-centric behavior rather than participating fully in LiveCode’s own encoding-management layer. Whether you’d call that a bug or a leaky abstraction is a matter of definition, but from LiveCode’s perspective it’s “working as designed” — just designed in a way that surprises people, repeatedly, for years. LiveCode Forums

Practically for you: don’t file this expecting a fix — treat textDecode(revXMLNodeContents(...), "UTF8") as the correct, permanent way to use this API, not a temporary patch. It’s the same pattern every long-time LiveCode/XML user ends up adopting.

Me - OK. So, for completeness my code needs to read the text encoding bytes in the XMP block to determine how the data should be encoded and I need to remember that revXML returns raw bytes that need decoding.

Close, but let me refine the first half of that — it’s actually simpler than you’re picturing.

You don’t need to read/sniff the encoding yourself. That job is already done for you, one layer down. revCreateXMLTree hands your raw bytes to libxml2, and libxml2 is the component that inspects the <?xpacket?> BOM and any <?xml ... encoding="..."?> declaration to figure out whether the source was UTF-8, UTF-16, UTF-32, etc. Crucially, libxml2 always normalizes internally to UTF-8 regardless of what the source encoding was. So whether the XMP block you feed it was UTF-8, UTF-16BE, or UTF-16LE on disk, by the time you call revXMLNodeContents, what comes back is always UTF-8 bytes. You get one consistent output encoding no matter what the input was — which is exactly why textDecode(..., "UTF8") is a safe, fixed rule rather than something you need to make conditional on what you find in the packet header.

So the two things worth keeping in your mental model going forward are really:

  1. On the way in: read the file as raw bytes (binfile:) and hand those bytes to revCreateXMLTree untouched — don’t pre-decode, don’t pre-guess the encoding. Let libxml2 do the sniffing; that’s its job and it’s good at it.
  2. On the way out: every value read via revXMLNodeContents/revXMLAttributes is UTF-8 bytes wearing LiveCode’s “native string” clothing, and must be passed through textDecode(tValue, "UTF8") before you treat it as text — display it, compare it, store it, whatever. This is a fixed rule, not something that varies file to file.

The only scenario where you’d ever need to inspect the packet’s own encoding declaration yourself is if you were writing your own XML parser from scratch instead of using libxml2 via revXML — which you’re not, so that responsibility isn’t actually yours here. Good instinct to want to understand the full chain, but the practical rule you need to carry forward is just the two-step one above.

1 Like
2

I’m pretty sure LC never “fixed” it because of backward compatibility- it would be a breaking change to all of the sudden have the XML library automatically do the conversion. I agree the docs should be made more clear. They still have uniDecode in examples which is plain wrong at this point. Excellent area for someone to contribute that doesn’t have the desire to fully get into coding the IDE or engine. My suggestion would be an example in each that shows the proper application of the encode/decode as appropriate. If everything is just ASCII, it doesn’t matter, but as soon as anything outside of that is introduced there will be a need to convert.

3

I am happy to try and update the dictionary but I need to confirm that the Claude is correct in that under the hood revXML works in UTF-8.

Also, its unfortunate that I never had much success when I tried to edit the Livecode documentation back in the day. Is it all on Github ?

S

4

Yes, the documentation is all on GitHub. If you need help figuring out what to edit, just ask. You can even edit the files and just email them to me if you don’t want to go the GitHub route.

5

Here is a tool that can help with working on the docs. It allows you to easily parse them to see how they render.

1 Like
6

Thanks, I see what I can do.

Simon

1 Like