ZIPManagement_PhysFS

Personal research about how implement and use PhysFS, in your project

View the Project on GitHub AaronGCProg/ZIPManagement_PhysFS

Assets ZIP management with PhysFS

Welcome to the webpage of how to implement Assets ZIP management with PhysicsFS

I am Aarón Guerrero, student of the Bachelor’s Degree in Video Games by UPC at CITM. This content is generated for the second year’s subject Project 2, under supervision of lecturer Marc Garrigó.

What is PhysicsFS?

Explanation from icculus.org: PhysicsFS is a library to provide abstract access to various archives. It is intended for use in video games, and the design was somewhat inspired by Quake 3's file subsystem. The programmer defines a "write directory" on the physical filesystem. No file writing done through the PhysicsFS API can leave that write directory, for security. For example, an embedded scripting language cannot write outside of this path if it uses PhysFS for all of its I/O, which means that untrusted scripts can run more safely. Symbolic links can be disabled as well, for added safety. For file reading, the programmer lists directories and archives that form a "search path". Once the search path is defined, it becomes a single, transparent hierarchical filesystem. This makes for easy access to ZIP files in the same way as you access a file directly on the disk, and it makes it easy to ship a new archive that will override a previous archive on a per-file basis. Finally, PhysicsFS gives you platform-abstracted means to determine if CD-ROMs are available, the user's home directory, where in the real filesystem your program is running, etc.

In summary, PhysicsFS it's a portable, flexible file i/o abstraction. This API gives you access to a system file system in ways superior to the stdio or system i/o calls. The main benefits highlighted by the API developers are that it is portable, more secure because no file acces is permitted outside the specified dirs, and flexible because archives can be used transparently as directory structures.

We will use it

Using PhysFS involves loading absolutely all files through this API, in a different way than conventional. That means that we must understand how the necessary functions that PhysicsFS brings. The following functions are those that allow us to start working, with which the project will be opened and closed:

About Mounting

Mounting a file is very important in an API like this, which allows you to carry out the rest of the functionalities.

When you mount an archive, it is added to a virtual file system...all files in all of the archives are interpolated into a single hierachical file tree. Two archives mounted at the same place (or an archive with files overlapping another mountpoint) may have overlapping files: in such a case, the file earliest in the search path is selected, and the other files are inaccessible to the application. This allows archives to be used to override previous revisions; you can use the mounting mechanism to place archives at a specific point in the file tree and prevent overlap; this is useful for downloadable mods that might trample over application data or each other, for example.

The mountpoint does not need to exist prior to mounting, which is different than those familiar with the Unix concept of "mounting" may expect. As well, more than one archive can be mounted to the same mountpoint, or mountpoints and archive contents can overlap...the interpolation mechanism still functions as usual.

Necessary Functions

Once we have initialized the API and have mounted a virtual file, we will go to other functions. Among them, the most important will be listed below:

Necessary SDL Functions

To upload files with this new methodology, we will not be able to use the same upload functions with SDL. In this way, from now on, we must know SDL_RWops and all its derivatives.

SDL_RWops is a structure that provides an abstract interface to stream I/O. Applications can generally ignore the specifics of this structure's internals and treat them as opaque pointers; allowing you to use pointers to memory instead of files (though it can handle files too) for things such as images or samples. The primary advantage of this feature is that many libraries load files from the filesystem themselves, leaving you a bit stuck if you want to implement your own special file access, such as an archive format.

The functions we currently use should be replaced by:

How can I implement it?

If you want to use Visual Studio, nmake, or the Platform SDK, you will need CMake. Let's take a little break to explain what is CMake about.

CMake

CMake is a multiplatform code generation or automation tool. The name is an abbreviation for "cross platform brand"; Beyond the use of "make" in the name, CMake is a separate, higher-level suite that the system makes common to Unix, being similar to automated tools.

It is an extensible, open-source system that manages the build process in an operating system and in a compiler-independent manner. Unlike many cross-platform systems, CMake is designed to be used in conjunction with the native build environment. Simple configuration files placed in each source directory (called CMakeLists.txt files) are used to generate standard build files (e.g., makefiles on Unix and projects/workspaces in Windows MSVC) which are used in the usual way. CMake can generate a native build environment that will compile source code, create libraries, generate wrappers and build executables in arbitrary combinations.

We will need CMake to carry out a PhysFS build, obtaining the libraries and .dll that we need to implement it in our project.

Steps to obtain PhysFS .dll and .lib

Remember that it is not necessary to follow all these steps if you are not interested! You have the .dll, .lib and a sample of how everything is integrated into the Solution. You have complete freedom to take what you want from there!

1.Download PhysicsFS ZIP from icculus.org. If you want to download the file directly, click here.

2.Download CMake. I recommend the version with interface since it is more intuitive and is the one that I will explain next. If you want to download it directly, click here.

3.Install CMake and open “CMake (cmake-gui)” executable. Check where CMakeLists.txt is in the downloaded PhysicsFS folder (In this case it is the base folder) and get this path to the source code field. To build the binaries you can create any empty folder.

4.Click configure and set-up the following options with your Visual Studio version and if you will use x64 or x86.

5.Once we have finished the previous window, the following options will be generated. We must discard them all except PHYSFS_BUILD_SHARED and in this case, PHYSFS_ARCHIVE_ZIP. Then, click Generate.

6.This files will be generated. We will need to run the file PhysicsFS.sln and then prepare a build in Release and Win32.

7.In the generated Release folder we will have everything you want. You will find the physfs.h in the following path of the folder you originally download.

Exercises & Solutions

Folder Set-Up

Remember to have all your assets in a ZIP. From now, we will no longer load from conventional folders.

You can organize the ZIP as you wish and intuitive seems to you.

Assets Manager Structure

The functions that we will have to prepare are very few, in a simple and intuitive structure. Then we can see how the structure of the module we will create would look like, simply observing the ".h".

ModuleAssetsManager();
~ModuleAssetsManager();

// Called before render is available
bool Awake(pugi::xml_node&);

// Called before quitting
bool CleanUp();

// Return the bytes of a PhysFS filehandle
uint Load(const char* path, char** buffer) const;

// Allows you to use pointers to memory instead of files or things such as images or samples
SDL_RWops* Load(const char* path) const;

</code>

TODO's

The TODO have various tips and helps, as well as links that will take you to the necessary documentation. Whenever you see that some TODO does not work, check if the following TODO is also related; because some TODO is dependent on each other and until they are both finished they will not work.

Anyway you can see how the progression does not become complicated.

You can download the Handout to do the following exercises clicking here.

TODO 0 & TODO 1:

ModuleAssetsManager::ModuleAssetsManager() : Module()
{

	name = ("assetsManager");

	// TODO 0 (Solved): Open the following link and check it: https://icculus.org/physfs/docs/html/physfs_8h.html
	// TODO 1 (Solved): You must initialize and de-initialize the PhysFS API.

	// Initialize the PhysicsFS library
	// This must be called before any other PhysicsFS function
	// This should be called prior to any attempts to change your process's current working directory
	PHYSFS_init(nullptr);

	// We only need this when compiling in debug. In Release we don't need it.
	PHYSFS_mount(".", nullptr, 1);
}



ModuleAssetsManager::~ModuleAssetsManager()
{

	// Deinitialize the PhysicsFS library.

	// This closes any files opened via PhysicsFS, blanks the search/write paths, frees memory, and invalidates all of your 		file handles.

	// NOTE: This call can FAIL if there's a file open for writing that refuses to close

	PHYSFS_deinit();

}

TODO 2:

bool ModuleAssetsManager::Awake(pugi::xml_node& config)
{
	// Determine if the PhysicsFS library is initialized, we can check it for avoid errors.

	if(PHYSFS_isInit())
		LOG("Asset Manager is succefully loaded");
	else
		LOG("Failed loading Asset Manager");

	// TODO 2 (Solved): Mount the desired ZIP with PhysFS.

	// Add an archive or directory to the search path.

	// If this is a duplicate, the entry is not added again, even though the function succeeds.
	// When you mount an archive, it is added to a virtual file system...
	// all files in all of the archives are interpolated into a single hierachical file tree.
	PHYSFS_mount("Assets.zip", nullptr, 1);

	return true;
}

TODO 3:

uint ModuleAssetsManager::Load(const char* path, char** buffer) const
{

	uint ret;

	// TODO 3 (Solved): You want to return the number of bytes it has read from the file that we passed to this 				function. 	
	// Maybe you want to search readBytes in the documentation, and investigate from there how to build the 				function.
	// The reading offset is set to the first byte of the file.

	// Returns a filehandle on success that we will need for the PHYSFS_fileLength

	PHYSFS_file* file = PHYSFS_openRead(path); 

	// Check for end-of-file state on a PhysicsFS filehandle.

	if (!PHYSFS_eof(file))
	{

		// Get total length of a file in bytes
		uint lenght = PHYSFS_fileLength(file); 
		*buffer = new char[lenght]; 

		// Read data from a PhysicsFS firehandle. Returns a number of bytes read.
		uint bytes = PHYSFS_readBytes(file, *buffer, lenght);

		if (bytes != lenght) 
		{
			LOG("%s" , path, "ERROR: %s" , PHYSFS_getLastError());
			RELEASE_ARRAY(buffer);
		}
		else
			ret = bytes; 
		}
		else
			LOG("%s", path, "ERROR: %s", PHYSFS_getLastError());


	// Close a PhysicsFS firehandle

	PHYSFS_close(file);

	return ret;
}

TODO 4:

	bool ModuleScene::Start() 
  		{ 

		// TODO 4 (Solved): Uncomment all of this and resolve how to load the document from the memory with the link 				below.

		if (!PHYSFS_exists("data.xml"))
		return false;

		char* buffer;

		pugi::xml_document dataFile;
		int bytesFile = app->assetManager->Load("data.xml", &buffer);

		// Loading document from memory with PUGI: https://pugixml.org/docs/manual.html#loading.memory
		pugi::xml_parse_result result = dataFile.load_buffer(buffer, bytesFile);
		RELEASE_ARRAY(buffer);

		// We load all the ZIP texture files
		LoadTexFile(dataFile);

		// We load all the ZIP fx files
		LoadFxFile(dataFile);

		// We load and play the desired music from the ZIP
		LoadMusFile(dataFile);

		return true;
	}

TODO 5:

	SDL_RWops* ModuleAssetsManager::Load(const char* path) const
	{

		char* buffer;
		uint bytes = Load(path, &buffer);

		// TODO 5 (Solved): Check what is: https://wiki.libsdl.org/SDL_RWops
		// We will need a new method to load Music, FX and Textures from the memory.
		// Try to investigate SDL_RWops and Related Functions.

		// Read-only memory buffer for use with RWops, retruns a pointer to a new SDL_RWops structure
		SDL_RWops* ret = SDL_RWFromConstMem(buffer, bytes);

		return ret;
	}

TODO 6:

	// There you don't need to do anything. Only to invest about this methods.

	SDL_Surface* surface = IMG_Load_RW(app->assetManager->Load(path), 1);

	music = Mix_LoadMUS_RW(app->assetManager->Load(path), 1);

	Mix_Chunk* chunk = Mix_LoadWAV_RW(app->assetManager->Load(path), 1);

</code>

Solution

The solution will show us Geralt of Rivia just like this. In addition will play random music, and if you press the key "1", will play random SFX. Progress will not be noticeable until the last TODO is complete. Until then, it is recommended to track the information of the variables through breakpoints to know if the functions are working.

You can download DEMO of Solution clicking here.

You can check the code of the solution, clicking here.

Bibliography