PAK Files
PAK is one of the most complex file formats that SS3D has. As mentioned in our filetype overview post, PAK stands for Pack, and the purpose of this filetype is to contain the majority of each client’s actual assets, such as models, textures, animations, etc. An easy way to conceptualize them is as a ZIP file, but without any compression and without being nearly as resilient to issues. Interacting with PAK files in any way requires using SS3D’s FileStorage library, the clients, tools, and SS3D itself all depend on this library to access PAK files.
The actual on-disk format consists of a fairly large header, shown below, containing version information along with the number of files in the PAK. It also contains a number of other fields, however none of these are used in any way by EyaSoft, they appear to have been intended for potential future expansion that never happened.
struct PACK_FILE_HEADER
{
uint32_t dwVersion;
uint32_t dwFileItemNum;
uint32_t dwFlag;
uint32_t dwCRC[4];
uint32_t dwReserved[16];
};
This is followed by a complex variable-length per-file header, containing the total size of the file + header, the size of the file itself, the length of the filename, and the offset, in bytes from the beginning of the PAK file, that the current file should have started at, which is used to ensure the PAK file hasn’t become corrupted or and to detect something going wrong and the file pointer ending up being somewhere it wasn’t expected to be, as well as being how the files are eventually seeked to for access by the client. This is followed by an unused flag variable and then followed by the variable-length filename itself.
struct FSFILE_HEADER
{
uint32_t dwTotalSize;
uint32_t dwRealFileSize;
uint32_t dwFileNameLen;
uint32_t dwFileDataOffset;
uint32_t dwFlag[4];
char szFileName[4];
};
As you can see above, the filename array szFileName[4] is only specified to be up to 4 characters, but in reality that isn’t the case and just serves as a placeholder for the start of the filename array, whose actual size is stored in dwFileNameLen. This filename field actually has a maximum length of _MAX_PATH, or 260 characters.
Upon the client starting, a call is made to SS3D’s InitializeFileStorageWithoutRegistry function (WithoutRegistry meaning to look for FileStorage in the same directory, rather than loading it via COM registration), and it is passed a PACKFILE_NAME_TABLE struct containing the relative path to each PAK file the client needs to be able to access, as well as an unused flag variable.
struct PACKFILE_NAME_TABLE
{
char szFileName[_MAX_PATH];
uint32_t dwFlag;
};
FileStorage then reads each specified PAK file, all of this header information is read and the needed parts are stored in memory, in a container roughly equivalent to a modern std::unordered_map, with the (lowercase) filename as the key. When the client requests a specific file, the FileStorage looks up the filename requested, and if found, seeks inside the PAK file to the file offset specified in the file’s header, and provides a FileStorage specific file handle, which can then be used with PAK specific versions of fread, fscanf, etc, to read data directly from the selected file inside the PAK file. These behave similarly to the traditional functions, but with the FileStorage system performing the necessary offsetting and length limiting to keep reads within the desired file.
Other than using FileStorage’s functions instead of the standard C file functions, access to files inside a PAK is intended to be largely transparent to the client, with FileStorage handling all the complexity involved in seeking to a file inside of a larger file. Normal files, located outside of a PAK, can also be accessed with FileStorage’s file functions, and these behave identically to files inside a PAK. This allows the engine to only use FileStorage’s functions, regardless of whether the file is in a PAK or not. This simplifies things from a programming perspective but does mean that everything using FileStorage, whether in a PAK or not, is limited to only its access functions.
There are a number of limitations with the PAK system that unfortunately make it a common failure point. There is no protection against multiple files in one PAK having the same name, the last file encountered with a particular name is the one that will be accessed when looked up, and in fact the updating mechanism the patcher uses abuses this to “update” files in a PAK, by just adding the newer version to the end of the PAK, leaving the old version as wasted, inaccessible space. In addition, every loaded PAK file’s information is stored inside of one combined hash map, so for example if MAP.PAK and NPC.PAK both have a file named POTATO.MOD, whichever PAK was loaded last will be the one the file is chosen from.
That, combined with the fact that file lookup is case insensitive, means that file collisions are fairly common, and being PAK load order dependent means you can update a file in one PAK, and your changes silently have no effect because a PAK later in the load order had something with the same name. There are also pitfalls when dealing with FSScanf, the PAK equivalent of fscanf, because it does not support all the format specifiers that the normal function does.
Despite all of these issues, EyaSoft continues to use PAK with each game in order to protect assets from modification or even from being stolen and used by others. In addition, most private servers also continue to use FileStorage and PAK files, sometimes with their own tweaks, likely for the same reasons as EyaSoft.
LUNA Private Servers
As we just mentioned, most LUNA private servers make use of FileStorage, although in their case the reason is almost entirely just to keep other private servers from easily stealing assets. While some private servers just use FileStorage as is, providing minimal protection, many end up tweaking the format in various ways to prevent a standard copy of FileStorage from being able to read them. This is sufficient to stop the majority of attempts to steal assets, however is still defeatable with some reverse engineering.
The following header definition structs should be considered to be pseudocode, rather than exact definitions, as some of them have been modified to better indicate some runtime data modifications. As an additional note, EyaSoft makes heavy use of MS specific data types, for example DWORD rather than uint32_t, however the standard types have been used here for clarity as to their size, as well as syntax highlighting that DWORD lacks. This list is roughly in chronological order, except where grouping identical formats was necessary. The list will continue to be expanded as time goes on and new servers are examined, so if you’re interested in new developments, check back occasionally.
*.PNX
RISE LUNA was the first private server I know of that used a customized PAK format. It relies on the default FileStorage library refusing to open anything that doesn’t have a *.PAK file extension, as well as additional padding added to the header, to prevent someone just renaming them back to *.PAK. They use the file extension *.PNX, and have an additional 48 null bytes after the standard PAK header, before the start of the first file. The offsets in each files header do include this extra size, and are readable normally by FileStorage after advancing past the extra main header bytes, making this one of the easier formats to extract.
struct PACK_FILE_HEADER
{
uint32_t dwVersion;
uint32_t dwFileItemNum;
uint32_t dwFlag;
uint32_t dwCRC[4];
uint32_t dwReserved[16];
char ExPadding[48]; //48 bytes extra padding
}
*.GPK, *.HEL, *.DPK, *.LRK
These four variants, each from different private servers (Great LUNA, Helios LUNA, Unicorn LUNA, and Legend LUNA), are all identical. This likely indicates they have a shared codebase and were based off of each other. Similar to *.PNX above, they use a different file extension, as well as additional zeroes added to the end of the main header, in this case, 64 null bytes. However, they don’t stop there and have modified the per-file headers as well, requiring 50 bytes to be subtracted off of each offset before use in order to match the actual offset. They also require skipping forward an extra 10 bytes after the reading of each file has been completed in order to reach the start of the next file’s header.
struct PACK_FILE_HEADER
{
uint32_t dwVersion;
uint32_t dwFileItemNum;
uint32_t dwFlag;
uint32_t dwCRC[4];
uint32_t dwReserved[16];
char ExPadding[64]; //64 bytes extra padding
}
struct FSFILE_HEADER
{
uint32_t dwTotalSize;
uint32_t dwRealFileSize;
uint32_t dwFileNameLen;
uint32_t dwFileDataOffset - 50; //on-disk value minus 50 = actual file offset
uint32_t dwFlag[4];
char FileName[dwFileNameLen+1];
char FileData[dwRealFileSize]; //"actual" file data
char ExPadding[10]; //10 bytes extra padding AFTER file data
}
RALUNA
RALUNA differs from the other variants we have examined so far in that it uses the standard *.PAK file extension, masquerading as normal PAK files, making detecting it at first glance quite difficult unless you know beforehand that the files are of this variant. It’s very similar to the previous four formats, with an extra 128 null bytes at the end of the main header, -50 from each file header offset, and +10 to the actual file offset after reading each file to reach the start of the next file header. This is identical to *.GPK and friends, except for the smaller amount of extra data in the main header. This might suggest common ancestry with these other servers.
struct PACK_FILE_HEADER
{
uint32_t dwVersion;
uint32_t dwFileItemNum;
uint32_t dwFlag;
uint32_t dwCRC[4];
uint32_t dwReserved[16];
char ExPadding[128]; //128 bytes extra padding
}
struct FSFILE_HEADER
{
uint32_t dwTotalSize;
uint32_t dwRealFileSize;
uint32_t dwFileNameLen;
uint32_t dwFileDataOffset - 50; //on-disk value minus 50 = actual file offset
uint32_t dwFlag[4];
char FileName[dwFileNameLen+1];
char FileData[dwRealFileSize]; //"actual" file data
char ExPadding[10]; //10 bytes extra padding AFTER file data
}
ARCANE LUNA
Arcane LUNA has a fairly simple format, consisting of the normal main header, with 12 extra padding bytes at the end, and normal per-file headers. It uses the traditional *.PAK file extension but can be differentiated by its version being 74847996.
struct PACK_FILE_HEADER
{
uint32_t dwVersion;
uint32_t dwFileItemNum;
uint32_t dwFlag;
uint32_t dwCRC[4];
uint32_t dwReserved[16];
char ExPadding[12]; //12 bytes extra padding
}
*.ODY, *.GLD, IPLAY LUNA
These three variants, each from different servers (Odyssey LUNA, Gladius LUNA, and IPLAY LUNA) differ heavily from the standard PAK files, in that they don’t have anything resembling the normal main header. There is seemingly just garbage padding, the file count, followed by more padding. Interestingly, IPLAY uses the normal *.PAK file extension, but is identifiable by the main header version being 74847996, whether this is intended to be a version or not is unclear as we don’t understand the rest of the header for this format, but it is consistent across all available file samples, and thus serves as a suitable type indication. These have normal per-file headers however, making parsing them quite easy once you figure out where the file count is. The header consists of 44 bytes of unknown data, the file count, and then an additional 92 bytes of unknown data.
struct PACK_FILE_HEADER
{
char Padding[44];
uint32_t dwFileItemNum;
char Padding[92];
}
CHRONICLES LUNA
Chronicles LUNA’s PAK files are nearly identical to the *.ODY, *.GLD and IPLAY LUNA ones, with the only difference being an extra 4 bytes of padding at the end of the header, on top of the existing padding. They also use the standard *.PAK extension like IPLAY does and are identifiable by the header version field being 96590396. This suggests that Chronicles LUNA is likely derived from the aforementioned servers.
struct PACK_FILE_HEADER
{
char Padding[44];
uint32_t dwFileItemNum;
char Padding[96];
}