Ideas

I think, there are some structures within the file. Actually a file contains something like a header (magic number, version information) or a footer (description of the file structure), there must be references between the parts. A compiled file should contain lists, arrays, bitmasks etc. Strings can be null-terminated or there is a length information for every string. Repeatable structures must have a given length (in bytes or 32/64bit values). The only thing I need is an entry point to find more and more structures.

If the .fmx file contains descriptions, it should be possible to create simple .fmb files, convert it into .fmx and analyze the differences between a base file and a changed file (ie. change a foreground color, it should switch a few bytes in the .fmx). So I can investigate, which attributes change which bytes and generate a map of file content.