The TDL relies on index files to gain high performance on very slow vintage
architecture.  Index files assist with the following functions:
	- Mapping long titles to 8.3 DOS filenames
	- Identifying long titles with a short, unique hash (useful for
	  tracking "Favorites" across multiple index and file loads)
	- Implementing a fast "search as you type" interface

The remainder of this text file documents the file format of the index
files generated by the TDL indexer.  
All values are little-endian and unsigned unless otherwise noted.

This specification is current as of 20170127.

=====================================================================

Title Index format:

        numEntries:     16-bit word of how many titles we have available
REPEAT  (This structure repeats numEntries times)
        titleOfs:       32-bit word of offset where each variable-length
                        record starts
END
REPEAT  (This structure repeats numEntries times)
        titleID:        16-bit word
        titleHash:      16 bytes of the MD5 hash of the title string
        titleLen:       1 byte of length of title string
        titleStr:       titleLen characters of title string
END

Purpose:
Contains the list of titles available for launching.

Implementation notes:
titleID isn't necessary but is included for error-checking.

titleHash is used to uniquely identify the title in a smaller form that
can be tracked across multiple file/index loads.  Currently it is a hash
of the title string, but later analysis might change it to something
else (like a hash of the .zip file contents, or a MobyGames gameid).

titleLen is just for printing the string from memory; it is not used to
determine how to load the string from disk exactly.  That is an
additional seek operation, and we're trying to minimize seeks on vintage
hardware.  Loading a title record will just be a straight 256-byte load,
and the record fragment that comes after what we want will simply be
ignored.

=====================================================================

Files Index format:

        numEntries:     16-bit word of how many files we have available
REPEAT  (This structure repeats numEntries times)
        fileID:         16-bit word
        fileStr:        12-byte string, including dot ("."), null-padded
END

Purpose:
Contains titleID-to-filename mappings.

Implementation Notes:
Including the dot (".") is redundant, but is included to ensure the
entires are always aligned to word boundaries for speed optimization.
(vintage systems are slow; we're going to need all the help we can get)

numEntries and fileID are redundant, but are included for error-checking
purposes.

=====================================================================

Wordlist index format:

        numEntries:     16-bit word of how many words are in the list
REPEAT  (This structure repeats numEntries times)
        sWord:          16-byte uppercase character sequence
END

Purpose:
Contains all recognized search words/terms.

Implementation Notes:
All words present in all title strings will be converted to uppercase,
truncated to 16 characters, made unique (ie. duplicates will be
removed), and aligned to 8086 paragraph boundaries, sorted in ascii
order, null-padded.  This arrangement will allow for quick searching of
words via binary search.  The offset from the beginning of the wordlist
serves as their ID.

numEntries is redundant, but is included for error-checking purposes.

=====================================================================

Search Word Locations index format:

(This is a mapping table that returns all of the titleIDs that contain a
particular search word.  At each mappingOfs, mappingLen titleIDs follow.)

        numEntries:     16-bit word of how many word mappings we have available
REPEAT  (This structure repeats numEntries times)
        mappingOfs:     32-bit word of offset where each variable-length
                        record starts
        mappingLen:     16-bit word of how many title match records follow
END
REPEAT  (This structure repeats every mappingOfs+mappingLen times)
        titleID:        16-bit word denoting a titleID that contains the word
        (titleID, titleID, titleID, etc. for mappingLen times)
        titleEndMarker: 16-bit value 0xFFFF (used for error-checking)
END

Purpose:
Contains searchwords-to-title mappings.

Implementation Notes:
Lists of titles to display will be based on combinations of the
information found in this index file.

To minimize IO activity, only the titleIDs are included, and not the
word's found position in the title string.  Located words will be found
and highlighted in the launcher running in DOS.  (This decision may be
revisited at a later date if the highlighting process is found to be too
slow in DOS.)

=====================================================================

Favorites export/import format:

        numEntries:     16-bit word of how many favorites are in the file
REPEAT  (This structure repeats numEntries times)
        titleHash:      16 bytes of the MD5 hash of the title string
END

Purpose:
Contains the list of games the user has "favorited".

Implementation Notes:

The actual favorites list is just an array of bitflags that, if
modified, is saved to disk before each handle() call.  However, a user
may want to reload the list of files in the future while still retaining
their favorites.  Since the titleids will change if this happens, and
our existing favorites won't map to the new indexes, we offer the user a
way to export their favorites and then later import them.

One future improvement is to eliminate the need for export/import by
storing the titleids in a block before the titlehashes; that way, the
titleid could be updated whenever a new index load is detected.

=====================================================================

Metadata cache:

        numEntries:     16-bit byte of how many bitflags are in the file
REPEAT  (This structure repeats numEntries times)
        flags:          1 byte containing flags
END

Flags contained in each byte (values in hex):
76543210

0000000x: Title is marked as a Favorite
000000x0: Title was unpacked into data cache at some point
xxxxxx00: Reserved for future expansion

Purpose:
Contains metadata pertinent to each Title, such as whether or not it is
marked as a "favorite", or how many times it has been run.

Implementation Notes:

Metadata is OR'd into a byte or AND'd out of a byte as necessary.  Data is
stored in TITLES.DAT which is flushed to disk before every execution operation,
or at normal program exit.

=====================================================================

Colors spefication:

Array of unsigned 8-bit byte values, each of which have the following
structure:
76543210
bbbb     - background text color
    ffff - foreground text color

UI attributes to be specified at a later date.

=====================================================================

Title audit log (issue #27):

        numEntries:     16-bit word of how many titles are present in the file
REPEAT  (This structure repeats numEntries times)
        launchCount:	16-bit word of how many times title was launched
	TotalPlaytime:  16-bit word of how many minutes title was in play
END

Purpose:
Ability to track what titles have been executed, how many times, and provide
some sort of report. The obvious use case is to generate a report after a
festival/convention ends and see what people were playing, how often, and for
how long.

Implementation Notes:

Audit log is to be kept on disk because it's low-priority data.  This also
allows for atomic updates to the file, which is useful in case people remove
power to the system, reboot, etc.

Counts should be incremented right before a title is launched.
TotalPlaytime should be incremented when *returning* from a launched title.
This will miss playtime minutes if the user reboots or the game cannot be
exited, but at least any minutes recorded will be legitimate.
