Support mounting compressed archives in the VFS (Virtual File System): save up to 95% disk space and hours of loading time (accumulated per year)

Request

I’d suggest FS2020 to support mounting popular compressed archive formats directly in the VFS (Virtual File System).

At a minimum the the following ones should be supported:

  • .7z because it is very efficient with FS2020 add-ons textures (see below).
  • .zip because it is readily available in Win10.

Here is what your community folder would look like:

instead of this...
FS2020\Community\flybywire-aircraft-a320-neo
FS2020\Community\flybywire-aircraft-a320-neo\SimObjects
FS2020\Community\flybywire-aircraft-a320-neo\effects
FS2020\Community\flybywire-aircraft-a320-neo\html_ui
FS2020\Community\flybywire-aircraft-a320-neo\...
...
FS2020\Community\gaist-msfs-V2\
FS2020\Community\gaist-msfs-V2\... (thousands of folders and files)
you'd have only this...
FS2020\Community\flybywire-aircraft-a320-neo.7z
FS2020\Community\gaist-msfs-V2.7z

Context

FS2020 is using a VFS to manage different mounting points where files can be loaded from. This is essential to support the Community folder or to replace files with newer ones.

However many add-ons comprise hundreds if not thousands of files and this is not without causing a certain number of problems among which hard-drive space size and FS2020 loading times:

Managing add-ons requires moving thousands of files

This can be partly worked around using symbolic links but this is not conveniently done by hand, prone to errors, and this is still putting the burden on the user to store and manually archive add-ons which are no longer used, and to restore and manually unpack archived add-ons whenever the user wants them back.

Loading add-ons consists in indexing thousands of files

Loading add-ons requires not only indexing the subfolders and their files but also checking every single file size on the drive and compare them with the layout.json content. This is causing lots of IOCTL which are adding up to the total load time, and this has a very noticeable impact with no more than 20 “standard” add-on on an SSD, let alone on a HDD.

Solution

Mounting compressed archives directly in the VFS should solve a certain number of problems, both at the user level and at the game level:

  • FS2020 add-on files indexing would be faster because these archive formats are indexing all the files and their sizes in the header already.

  • Add-on management would be easier because you’d move around a single compressed archive file instead of a folder with thousands of subfolders and files in it.

  • User drive space would be considerably reduced in the Community folder (see saving example below)

  • In addition you could archive your add-ons and restore them without having to decompress them first.

Example

Global AI Ship Traffic MSFS V2

This is a “big” add-on with 11802 files and 1763 folders

Total Size (bytes) Compression ratio (%)
Community folder 6,259,674,687
.zip archive (std settings) 927,505,718 14.7 %
.7z archive (std settings) 418,079,126 6.7 %

Appendix

Modernizing layout.json with wildcards

The add-on format requires using a layout.json file which contains the list of all the files the add-on is using. It is meant to tell FS2020 which of the files found into the folder and its subfolders are to load, and therefore which files are to ignore.

However, most of the time nearly all the files in a folder must be loaded and sometimes all of the files in a folder must not (docs, PDF).

I’d suggest you support wildcards so that…

you could replace this...
{
  "content": [
    {
      "path": "ContentInfo/global-ai-ship-traffic-v2/Anzac Destroyer MSFS.JPG",
      "size": 248068,
      "date": 132752292850201898
    },
    {
      "path": "ContentInfo/global-ai-ship-traffic-v2/Arctic Princess LNG MSFS tanker.JPG",
      "size": 249930,
      "date": 132734818938055004
    },
    {
      "path": "ContentInfo/global-ai-ship-traffic-v2/Blue Marlin Heavyload Ship MSFS.JPG",
      "size": 393298,
      "date": 132747380300622415
    },
    {
      "path": "ContentInfo/global-ai-ship-traffic-v2/Blue Star Ferry Naxos MSFS.JPG",
      "size": 338063,
      "date": 132747391718720969
    }
    etc...
  ]
}
with this instead...
{
  "content": [
    {
      "path": "ContentInfo/global-ai-ship-traffic-v2/*.*",
    }
  ]
}
and exclude subfolders like this...
{
  "content": [
    {
      "path-exclude": "SimObjects/Boats/AI_Alice_Austen/extra_info/*.*",
    }
  ]
}

Most add-ons aren’t replacing files, they are just adding theirs, and there is only 1 instance of each files to be mounted in the VFS. The size and date should be only necessary when explicitly wanting to override another existing file (using date).

Otherwise size which is meant to validate the add-on content is adding an unnecessary burden not only at authoring time (regardless of mounting archives or not), but also at loading time if support for archives is added, because the file size is already included in the archive directory and file table headers.

In using wildcards, you’d only keep the more verbose form using size and date for the cases where add-on vendors are specifically overriding a file with the same virtual path. This is the only case where this information is necessary in order to triage which one is the newest.

So many good ideas for the core sim get overlooked in favor of simple “wants”. Glad the forums automatically bump these posts, or else many wouldn’t even know they exist as Wishlist items get buried so fast.

I wish more people would adopt the idea of using 7z over Zip. Zip compression gets the job done, and compression can be improved further by using Deflate64 over the standard Deflate, but 7z using LZMA2 and a dictionary size close to your archive’s uncompressed size produces a much better compressed archive.

This would not only save disk space, but if used server-side for content downloads, it would also substantially decrease download times (less bandwidth) and extraction time would not even exist as long as the files aren’t so big that is has to be broken up into parts.

4 Likes

I saw the Title I I instantly knew it must be a JL (RXP) suggestion , before I even read it …

Gets my vote - (for whats it worth )

2 Likes