Summary
After almost two decades of development using Subversion as a version control system, we switched to using git. This migration is made complicated by the high amount of binaries we need to handle as a game project.
This page provides a high-level overview of the build, development, and deployment environment of 0 A.D.: depending on your use case and involvement, you can jump to the following:
- If you just want to play the latest development version of the game, jump to the NightlyBuild page. You can keep reading here out of curiosity.
- If you want to bundle/package the game for a Unix distribution or another platform, please read this page. You should also use the NightlyBuild as a base for your work, or you can test our own release bundles, regularly generated with our CI/CD from the nightly build.
- If you want to start contributing to the development, you should take a quick glance at this page. The general overview should be enough to get you started, and you can come back here for details when you are familiar with the development process (see GettingStartedProgrammers).
Overview
A general overview diagram is available below (click to zoom):
Details about each layer follow.
Source code
The source code for the game and a couple tools is versioned under git. The main repository is located at https://gitea.itms.ovh/wfg/0ad.git. Mirrors exist on Github, Gitlab.com, and elsewhere.
The game assets (2D and 3D graphics, sounds and music, game maps, ...) are binary files, which are difficult for git to handle. Since git is a distributed version control system, plainly storing the assets would mean that anyone cloning the repository would download all the versions of all the assets over decades of development. git's algorithms are able to reduce the space used, but that would still put a lot of heavy processing strain on everyone's computer. Thus, we use Git LFS to store binary assets.
Git LFS is part of Git for Windows by default. However, Unix users must install Git LFS through their package manager (usually called git-lfs).
For the user, the use of Git LFS is quite transparent. Binary files are still checked out during a regular clone operation. However, only the current version of each asset is actually downloaded on your disk.
In order to clone the 0 A.D. repository, run
git clone https://gitea.itms.ovh/wfg/0ad.git
This full clone will take around 13GiB of space on your disk.
If you want to save disk space, or if your Internet connection is feeble, you can skip the step where the LFS files are downloaded. Binaries will be checked out in the form of LFS pointer files containing the remote URL and the hash of the binary file. This "text-only" clone of the repo only weighs around 100MiB. To skip the LFS download step, clone with the following environment variable:
GIT_LFS_SKIP_SMUDGE=1 git clone https://gitea.itms.ovh/wfg/0ad.git
After that, you can download any LFS file you wish using git-lfs-pull(1). You can download some specific files/paths using the -I/--include flag.
For instance, if you want to download only the binary files needed for running tests, you can run
git lfs pull -I ...... https://gitea.itms.ovh/wfg/0ad.git
In a clone where binaries were partially fetched, you can fetch the rest of binaries by running
git lfs pull
Libraries
In order to be built, the game needs a dozen libraries. They fall into two categories:
- Most of the libraries are directly built from upstream.
- Some libraries (most importantly SpiderMonkey) are heavily patched to fit our needs, or are maintained by us. We provide them ourselves. We usually call them
source-libs.
Depending on your OS, getting and building the libraries is not done the same way.
On Windows
On Windows, we prebuild all libraries. We distribute built upstream libs through the windows-libs SVN repository. Prebuilt binaries of our source libraries are included in the source-libs SVN repository along with sources.
Getting the libraries is done by running the bat script libraries\get-windows-libs.bat. This scripts checks out both SVN repositories under the libraries directory, and copies DLLs to binaries\system.
Updating libraries is done by running libraries\get-windows-libs.bat again. Running this script also fetches the correct version of libraries when testing a PR or bisecting an old commit.
In case of build errors, do not hesitate to just delete the folders libraries\source and libraries\win32. Retrieving the libraries again is fairly quick and only consumes bandwidth, as everything is already built.
On Linux/BSD
Linux and BSD users will install upstream libraries from their package manager.
Getting the source libraries is done by running the shell script libraries/build-source-libs.sh. This script checks out the source-libs SVN repository, builds them, and copies resulting binaries to the corresponding places (under lib/ directories for static linking and under binaries/system for dynamic linking).
You can pass flags to this script, in order to avoid building libraries and instead use versions from your package manager. For instance, if you installed SpiderMonkey from your system, pass --with-system-mozjs to build-source-libs.sh. It will skip building that library.
Building source libraries, especially SpiderMonkey, is time- and resource-consuming. Hence, the libraries/build-source-libs.sh tries to avoid unnecessary builds. Do not hesitate to run this script every time you update your repository clone, and resort to manual handling of the binaries when you bisect old commits.
On macOS
On macOS, we provide a script which downloads tarballs for tested releases of all dependencies, and builds them. For source libs, the script checks out the source-libs SVN repository and builds the contents.
This script is called libraries/build-osx-libs.sh, and must be run before trying to build the game itself. It is very time- and resource-consuming. In case of issues with a specific library, it is advised to only delete the corresponding folder under libraries/osx, to avoid downloading and rebuilding the rest. For source libs, deleting unversioned files from the libraries/source SVN checkout is an efficient strategy.
Build Workspaces
After all dependencies are downloaded and built, you can build the game itself. First, we setup a build workspace using Premake. We support GNU Make for Unix, VisualStudio for Windows, and XCode for macOS. Using CMake as a Premake alternative was looked into (see issue #1104), but Premake suits our needs for now.
Run the script build/update-workspaces.{sh,bat} depending on your OS. This step is very quick. It builds Premake on Unix (already built on Windows), before generating the workspaces for your OS.
On Linux/BSD, make sure to pass the same flags to update-workspaces.sh as you did to build-source-libs.sh. For instance, if you ran build-source-libs.sh --with-system-mozjs to avoid building SpiderMonkey, now run update-workspaces.sh --with-system-mozjs so that the build environment fetches library binaries from the system-wide install.
When this is done, build the game. More details are found in the BuildInstructions.
In case of build issue, you can try and recreate the build workspace from scratch. On Windows, it is enough to delete build\workspaces\vs20xx. On Unix, we provide a clean-workspaces.sh script which also cleans up the Premake build files and built libraries. It is recommended to try first clean-workspaces.sh --preserve-libs to keep the built libraries intact and avoid wasting time rebuilding them.
To-Do Note: In the future, it would be better if clean-workspaces only cleans workspaces. The use of git clean -fdx can be advised for resetting the repository to a clean state at any point.
Nightly Builds
The CI/CD system (see JenkinsSetup) generates a full build of the game everyday using the latest trunk commit. This build contains all the source libraries, and prebuilt Windows binaries of the game and libraries. It can be used by invested players who want to test the latest version of the game, and also by contributors. This build serves as a basis for release bundles, as part of Continuous Delivery (see next section).
In order to download the nightly build, checkout the repository at https://svn.itms.ovh/nightly-build/trunk.
For more information, consult the page NightlyBuild.
Compared to a built version of the git repository, a nightly build contains extra files:
- Translation files, which are generated and kept in sync with Transifex, the platform our translators use (see Internationalization_and_Localization)
- SPIR-V shaders, needed by the Vulkan backend, which are generated from assets
This nightly build is distributed over SVN. This allows people to watch that build and seamlessly update their copy daily, only downloading the modified files.
Continuous Delivery of release bundles
On a bi-weekly basis, the CD system generates release bundles from the nightly build. These bundles do not serve other functions than testing the release systems regularly. Testers can try them if they wish. These bundles will not be signed, and only the latest will stay up for downloading, directly from Jenkins.
The bundling scripts are run from a macOS worker machine. This allows us to generate macOS release bundles using proprietary Apple tooling. The Windows release bundle is a NSIS installer. The Unix release bundles are compressed tarballs, separating the game engine and the game assets. We encourage package maintainers to test their packaging scripts on those tarballs.
TODO: Add the link to the Jenkins jobs providing the downloads.
Building a 0 A.D. AppImage should be doable using the guide LinuxAppImage inside a nightly build checkout.
Release Branches
(For more details, see also ReleaseProcess)
We follow a trunk-based development model. All of the development happens on the main trunk branch. Contributors prepare their work inside small and short-lived development branches, which are rebased, usually squahed, and merged into trunk. The nightly builds are prepared using the contents of the trunk branch.
When a release is planned, at some point we decide a Feature Freeze for the upcoming release and a release branch is created from trunk. This branch will only receive two types of commits:
- Cosmetic changes or config changes that are only relevant for this release (for instance, the release name display, the public key for signing mods, etc.)
- Bug fixes that are cherry-picked from
trunk. No custom bug fixes should be applied to the release branch without also fixingtrunk, in order to avoid regressions. Of course, if a bug in the release branch cannot be reproduced in the trunk, you have no choice, but there is a risk of regression should the bug reappear intrunkin the future. Cautiously document such exceptional fixes.
A SVN release-build repository is created for the duration of the Feature Freeze. The CI generates builds from the release branch to this repository upon each commit, instead of nightly. Release candidates will be generated by the CD scripts from this SVN repository when the release approaches completion.
Configuration of the Transifex project is changed, so that updates of the .pot files are performed from the release-build SVN repo, instead of the nightly. That way, for the duration of the feature freeze, translators can focus on polishing the upcoming release.
Players are encouraged to stop testing the nightly builds and start testing the release builds instead. This may be a good time to reset the nightly-build to scratch, starting anew for a new development cycle, and cleaning up the accumulated history of binaries from the server.
Upon publication of the release, the release commit is tagged. Two weeks after the release, the branch is deleted and the associated SVN repository is also deleted.