I recently wrote a post which explained how Brackeys' Save & Load system could be fixed so that it isn't victim to a major security flaw regarding BinaryFormatter
, by instead using BinaryWriter
and BinaryReader
. This guide will explain how to write a better, portable, more scalable save & load system from the ground up.
Buckle your seatbelts, we have a lot to cover.
Please be advised that this guide aims to teach the fundamentals of using protobuf-net, and does not focus on recommended code style or best practices.
Introduction
As I hinted at in the end of my post where I fix Brackeys' Save & Load system, we'll be using Protocol Buffers to serialize and deserialize game save data. Protocol Buffers are a language-agnostic specification on binary data serialization, which means there is a way to use them in almost every programming language (C# included!)
Aside from the security flaw, one of the problems with BinaryFormatter
is that it was specifically built to serialize and deserialize CLR objects. If you saved your game data with BinaryFormatter
in C#, and then attempted to load that data in Java, you would have an extremely difficult task ahead of you trying to figure out how exactly the data is encoded.
Protocol Buffers solve this because the specification for how data is encoded is very well documented, and extremely easy to implement; not that we'll have to worry about it.
Why not…
... PlayerPrefs
?
I see a lot of people suggest PlayerPrefs
to save and load game state such as high scores and player positions. I wholeheartedly disapprove of this advice for a few reasons:
On Windows it stores the data in the registry, which can lead to very inefficient and non-optimal programming if not done properly.
It's only capable of handling 3 types - float, int, and string. This is even worse than using
BinaryReader
andBinaryWriter
! At least they support double, long, bool, and all the other built-in types. Unity did not even bother to add overloads forColor
, orVector3
, a type you'd think would be very important to saving the player position or that of objects.
... JSON1 (or similar)?
Of course, another alternative to serializing game state is by using a human-readable format such as JSON1. This would entirely solve the issue of serializing complex types such as vectors, or other custom classes.
The problem, however, is that formats such as these were designed to be easily read by humans - not computers. Computers don't understand JSON1, and so precious CPU time must be spent either constructing a JSON1 string when writing, and attempting to parse it back to an object when reading.
This makes JSON ideal for storing configuration (e.g. the player's chosen settings), things which are okay to be modified by the player - but less than ideal for things which you want to keep relatively untouched.
It also drastically increases the size it occupies on disk. Remember that characters use - at the very least - 8 bits each. Even a simple JSON structure like so:
{
"health": 100.0,
"score": 90
}
If you viewed the hex dump of this string, you would see the following:
EF BB BF 7B 0D 0A 20 20 22 68 65 61 6C 74 68 22 3A 20 31 30 30 2E 30 2C 0D 0A 20 20 22 73 63 6F 72 65 22 3A 20 39 30 0D 0A 7D
A whopping 42 bytes, for a float value and an int value. That's insane. Using binary serialization would mean writing only 8 bytes total; 4 bytes for a float, 4 bytes for an int. That's a massive ~81% reduction. If you're curious, this is the equivalent data represented in pure binary:
00 00 C8 42 5A 00 00 00
Bringing Protocol Buffers to C#
Method 1: Using the NuGet outside of Unity
If you aren't using Unity, all you need to do to follow along with the rest of this guide is install the protobuf-net
NuGet into your project and skip to the next section.
Method 2: Using the NuGet with Unity
If you are using Unity, you can use NuGetForUnity which I personally find to be a fantastic way of adding NuGet dependency support to Unity. I won't go into detail about how to use that, the README explains it. Add the protobuf-net
NuGet to your project, and skip to the next section.
Method 3: I did Method 4 for you
If neither method 1 nor 2 work for you, I have compiled the assembly references you need so that protobuf-net
can be imported into Unity with ease.
Simply download this .unitypackage (version 3.0.101; last updated June 21st, 2021), and import the files into your project. However, be warned: I will not be keeping this file up to date on a strict and regular basis. If you are reading this post far in the future, and the references are dangerously out of date, follow the steps in Method 4.
Method 4: Manually downloading references
Open the NuGet page for protobuf-net
and choose to download the package directly:
The .nupkg file it gives you is actually just a ZIP archive with a misleading file extension. You can either rename it from protobuf-net.x.x.xx.**nupkg**
to protobuf-net.x.x.xx.**zip**
, and open it as normal, or you can right click it and choose to Open With an app such as WinRAR or 7-zip.
Once inside the file, navigate to the lib
directory, and then netstandard2.0
. This directory contains two files: protobuf-net.dll
and protobuf-net.xml
. Extract them both to somewhere in your Assets
.
Repeat these steps for the following NuGets:
Testing the library
For the remainder of this guide, I will be giving my examples in the context of Unity. Do keep in mind, however, that protocol buffers are completely platform independent and so this guide should be useful to you whether or not you are using Unity (by making changes like replacing Debug.Log
with Console.WriteLine
, etc.)
Create a new behaviour and copy the following code into the Start
method.
using (var stream = new MemoryStream())
{
Serializer.SerializeWithLengthPrefix(stream, new { X = int.MaxValue }, PrefixStyle.Base128);
var buffer = stream.ToArray();
Debug.Log(BitConverter.ToString(buffer));
}
Ensure the behaviour is attached to a live game object and play the scene. If you see 06-08-FF-FF-FF-FF-07
in your console, this means everything is working as expected:
As an aside, the four bytes FF FF FF FF
is the X
value on the third line! This is the representation of int.MaxValue
in hexadecimal, a value whose 32 binary digits are all 1
. Try changing the value of X
and watch how the bytes change.
Creating a save data model
Let's start off by assuming you have a behaviour which is attached to the player, creatively named PlayerBehaviour
, within which are a couple of properties that represent some data about the player we wish to save, such as their Score
and how much Health
they have.
using UnityEngine;
public class PlayerBehaviour : MonoBehaviour
{
<mark>[field: SerializeField]</mark>
<mark>public int Score { get; set; } = 0;</mark>
<mark>[field: SerializeField]</mark>
<mark>public float Health { get; set; } = 100;</mark>
private void Awake()
{
// ...
}
private void Update()
{
// ...
}
private void OnApplicationQuit()
{
// ...
}
}
Our first step would be to create a POCO model containing the data - and only the data - we wish to save. Let's create a simple SaveData
class to encapsulate it.
public class SaveData
{
public int Score { get; set; } = 0;
public float Health { get; set; } = 100;
}
There are two attributes we need to use so that this class is easily understood by protobuf-net.
The first is ProtoContractAttribute
. This tells protobuf that the type is one we wish to serialize. It is similar to having to mark a class Serializable
when using BinaryFormatter
.
<mark>using ProtoBuf;</mark>
<mark>[ProtoContract]</mark>
public class SaveData
{
public int Score { get; set; } = 0;
public float Health { get; set; } = 100;
}
The second attribute is ProtoMemberAttribute
. This is similar to using JsonProperty
or JsonPropertyName
when using JSON serialisation libraries - in that it indicates the “name” of the property we want to serialize.
protobuf-net unfortunately does not support named properties, but it does support integer “tagged” properties. Decorate each property with ProtoMemberAttribute
like so:
using ProtoBuf;
[ProtoContract]
public class SaveData
{
<mark>[ProtoMember(1)]</mark>
public int Score { get; set; } = 0;
<mark>[ProtoMember(2)]</mark>
public float Health { get; set; } = 100;
}
You can use any tags you like - they do not have to be 1 and 2, nor do they have to be in any particular order. The only rule with protocol buffers is that each member's tag must be unique. This is because if you decide to change your model later on by removing or adding new properties, it won't interfere with the existing data.
The save model is now complete!
Loading the data
To get started, let's create a field in PlayerBehaviour
which will store the save path of the save file, as well as two new empty methods named LoadGame
and SaveGame
.
<mark>using System.IO;</mark>
<mark>using ProtoBuf;</mark>
using UnityEngine;
public class PlayerBehaviour : MonoBehaviour
{
<mark>private string _saveFile;</mark>
[field: SerializeField]
public int Score { get; set; } = 0;
[field: SerializeField]
public float Health { get; set; } = 100;
private void Awake()
{
<mark>_saveFile = Path.Combine(Application.persistentDataPath, "save001.dat");</mark>
// ...
}
private void Update()
{
// ...
}
private void OnApplicationQuit()
{
// ...
}
<mark>private void LoadGame()</mark>
<mark>{</mark>
<mark>}</mark>
<mark>private void SaveGame()</mark>
<mark>{</mark>
<mark>}</mark>
}
Path.Combine
?Different operating systems use different directory separators. Unix systems such as Linux and macOS separate directories using the /
character, while Windows uses \
.
Calling Path.Combine
will automatically use the correct separator for the current operating system, which ensures your game will work on all platforms.
We'll focus on LoadGame
first. There are only four things we need to do here.
Check if a save file exists
If it does, open a read-only stream to the file
Deserialize the stream to a
SaveData
instanceSet
Score
andHealth
in ourPlayerBehaviour
protobuf-net offers a Serializer
class which, unlike classes such as BinaryFormatter
, is accessed statically. Call the generic Deserialize
method, passing in the input stream.
private void LoadGame()
{
if (!File.Exists(_saveFile))
return;
using var file = File.OpenRead(_saveFile);
<mark>var data = Serializer.Deserialize<SaveData>(file);</mark>
Score = data.Score;
Health = data.Health;
}
All you need to do is call LoadGame
when the game starts (Awake
) or when the player loads their save.
Saving the data
The Serializer
class also contains a Serialize
method, which works in the opposite direction. Construct a SaveData
instance, and then serialize it to a writable stream:
private void SaveGame()
{
using var file = File.<mark>OpenWrite</mark>(_saveFile);
<mark>var data = new SaveData</mark>
<mark>{</mark>
<mark> Score = this.Score,</mark>
<mark> Health = this.Health,</mark>
<mark>};</mark>
<mark>Serializer.Serialize(file, data);</mark>
}
All you need to do now is call SaveGame
when the game quits (OnApplicationQuit
) or the player manually saves their game. Whatever works for your needs.
Serializing Unity types
We can now serialize and deserialize custom types! We can even have nested types (i.e. custom classes inside SaveData
) too as long as those are also decorated with ProtoContract
. However, we can't serialize Unity types such as Vector3
or Color
.
To show what I mean, let's store the player position in SaveData
as a Vector3
, and adjust our SaveGame
and LoadGame
methods to reflect it.
using ProtoBuf;
<mark>using UnityEngine;</mark>
[ProtoContract]
public class SaveData
{
[ProtoMember(1)]
public int Score { get; set; }
[ProtoMember(2)]
public float Health { get; set; }
<mark>[ProtoMember(3)]</mark>
<mark>public Vector3 Position { get; set; }</mark>
}
private void LoadGame()
{
if (!File.Exists(_saveFile))
return;
using var file = File.OpenRead(_saveFile);
var data = Serializer.Deserialize<SaveData>(file);
Score = data.Score;
Health = data.Health;
<mark>transform.position = data.Position;</mark>
}
private void SaveGame()
{
using var file = File.Create(_saveFile);
var data = new SaveData
{
Score = this.Score,
Health = this.Health,
<mark>Position = transform.position</mark>
};
Serializer.Serialize(file, data);
}
This will compile - after all there's nothing syntactically wrong here - but protobuf-net won't handle it. Running this code alone will throw us InvalidOperationException
.
UnityEngine.Vector3
Vector3
(and all other Unity types) are not decorated with the ProtoContract
attribute, therefore protobuf-net has no idea about how to serialize it. We also have no way to access the source code to add it. What can we do in this situation?
Surrogates to the rescue
We can define a surrogate type, which essentially acts as a “middle man” between protobuf-net and the types we don't have access to modify. In a nutshell, surrogates tell protobuf-net “Hey, whenever you see a Vector3
, convert it to a different type and serialize that instead.”
To do this, let's create a very basic struct which will serve as the surrogate for a Vector3
, called Vector3Surrogate
. All we need to care about are the X, Y and Z components - and so that is all we need to declare. For example, we don't need the magnitude
because that is calculated from these 3 components.
using ProtoBuf;
[ProtoContract]
public struct Vector3Surrogate
{
[ProtoMember(1)]
public float X { get; set; }
[ProtoMember(2)]
public float Y { get; set; }
[ProtoMember(3)]
public float Z { get; set; }
}
Next, we need to define a couple of conversion operators so that we can convert between Unity's Vector3
and our surrogate.
using ProtoBuf;
<mark>using UnityEngine;</mark>
[ProtoContract]
public struct Vector3Surrogate
{
[ProtoMember(1)]
public float X { get; set; }
[ProtoMember(2)]
public float Y { get; set; }
[ProtoMember(3)]
public float Z { get; set; }
<mark>public static implicit operator Vector3(Vector3Surrogate vector) =></mark>
<mark>new Vector3(vector.X, vector.Y, vector.Z);</mark>
<mark>public static implicit operator Vector3Surrogate(Vector3 vector) =></mark>
<mark>new Vector3Surrogate { X = vector.x, Y = vector.y, Z = vector.z };</mark>
}
If you don't understand operator overloading yet, you can read the documentation about them. But do feel free to copy/paste this simple surrogate.
Now we need to tell protobuf-net about our struct. For beginners, this is arguably the most complicated part of the entire guide (which is why this guide is classified as “Intermediate”), so bear with me.
The way protobuf-net knows what types to serialize - and how to serialize them - is through something called a “type model”. We can get the model that's in use by accessing Protobuf.Meta.RuntimeTypeModel.Default
, and storing it in a variable.
using ProtoBuf.Meta;
// in a method
var model = RuntimeTypeModel.Default;
Now we can add our struct as a known type to serialize. To do this, call the Add
method and pass the surrogate type we made as a generic argument:
var model = RuntimeTypeModel.Default;
<mark>model.Add<Vector3Surrogate>();</mark>
The last step is to tell protobuf-net to actually use this as a surrogate in place of UnityEngine.Vector3
. Call Add
again, but this time pass Vector3
as a generic argument, and false
as the first argument (this tells protobuf-net not to serialize Vector3
).
Then, call SetSurrogate
on the result, and pass the type of our struct as the argument.
var model = RuntimeTypeModel.Default;
model.Add<Vector3Surrogate>();
<mark>model.Add<Vector3>(false).SetSurrogate(typeof(Vector3Surrogate));</mark>
At this point you're probably asking "but where do I write this?", since earlier my only comment was merely “in a method”. The most important thing to note here is that these 3 lines should only be executed once. And when I say once, I don't mean “once per behaviour” or “once per scene load”, I mean once throughout the entire lifetime of the game. Once protobuf-net knows about the surrogate, it doesn't need to be told again (you will actually get errors if you do this more than once.)
This is down to you. Depending on how your game is structured, you might have a GameManager
which has some one-time initialization logic as soon as the game launches. If you do, put it there. Or, instead favouring the use of the RuntimeInitializeOnLoadMethod
attribute.
But purely for the purposes of demonstration, I'm going to shove this logic into the static constructor for PlayerBehaviour
. This is a dirty trick that will at least let us test our code is working.
The code presented below is for demonstration purposes only. I do not recommend you do this in actual production code. You should be using a proper initialization pipeline.
using System.IO;
using ProtoBuf;
<mark>using ProtoBuf.Meta;</mark>
using UnityEngine;
public class PlayerBehaviour : MonoBehaviour
{
// ...
<mark>static</mark> PlayerBehaviour()
{
<mark>var model = RuntimeTypeModel.Default;</mark>
<mark>model.Add<Vector3Surrogate>();</mark>
<mark>model.Add<Vector3>(false).SetSurrogate(typeof(Vector3Surrogate));</mark>
}
// ...
}
If at any point you lost track of the code, you can see how my full PlayerBehaviour
looks here.
You should be able to launch the game, play around with Score
and Health
, and even move the player around. Stop the game, relaunch it, and it should have loaded your the previous state! Player position included.
Making a structural improvement
The more properties you add, the more cumbersome this can become to exchange values between SaveData
and PlayerBehaviour
; just like with BinaryReader
and BinaryWriter
.
Here you could instead add the Serializable
attribute to SaveData
(so that it shows in the inspector properly) and store an instance of it within the PlayerBehaviour
itself like so:
public class PlayerBehaviour : MonoBehaviour
{
private string _saveFile;
[field: SerializeField]
<mark>public SaveData Data { get; private set; }</mark>
private void Awake()
{
_saveFile = Path.Combine(Application.persistentDataPath, "save001.dat");
LoadGame();
}
private void LoadGame()
{
if (!File.Exists(_saveFile))
return;
using var file = File.OpenRead(_saveFile);
<mark>Data = Serializer.Deserialize<SaveData>(file);</mark>
transform.position = Data.Position;
}
private void SaveGame()
{
Data.Position = transform.position
using var file = File.OpenWrite(_saveFile);
<mark>Serializer.Serialize(file, Data);</mark>
}
// ...
}
We do still have to set the transform position, since that Vector3
and the one defined in SaveData
are unrelated values. You can work around that by passing a reference to the player for SaveData
to access. The Position
property could automatically set/get the player's position.
However, this alone greatly reduces the amount of code in both the Save and Load methods - only having to serialize a single type similar to BinaryFormatter
. 👌
Adding compression
For now this is good. We only have 3 properties (Score
, Health
, and Position
) and they serialize to just a few bytes. But as a game grows, it will need to start serializing more and more data. Skills, inventories, enemy data, game progression. A savefile can get quite large pretty quickly. This is where compression comes in, and thanks to .NET it is remarkably easy to implement.
Keep in mind this will introduce a breaking change, so be sure to delete your current save file so that deserialization errors don't happen! On Windows, Application.persistentDataPath
points to %LOCALAPPDATA%low\<COMPANY>\<PRODUCT>
, the save001.dat
file will be in there.
protobuf-net works with streams. However, it doesn't care what stream. We opened a FileStream
by calling File.OpenRead
and File.OpenWrite
, but we can also encapsulate that stream in another stream - a GzipStream
- and tell protobuf-net to use that instead. GZip compression functionality is built right into .NET, so we don't need any other third party libraries!
Let's modify SaveGame
to create a new GZipStream
under “Compress” mode - wrapping the FileStream
. Then we simply pass that GZipStream
to Serialize
.
<mark>using System.IO.Compression;</mark>
// ...
private void SaveGame()
{
Data.Position = transform.position;
using var file = File.Create(_saveFile);
<mark>using var gzip = new GZipStream(file, CompressionMode.Compress);</mark>
Serializer.Serialize(<mark>gzip</mark>, Data);
}
LoadGame
will also open a GZipStream
but instead under “Decompress” mode.
private void LoadGame()
{
if (!File.Exists(_saveFile))
return;
using var file = File.OpenRead(_saveFile);
<mark>using var gzip = new GZipStream(file, CompressionMode.Decompress);</mark>
Data = Serializer.Deserialize<SaveData>(<mark>gzip</mark>);
transform.position = Data.Position;
}
The last thing we need to do to ensure maximum compatibility is to add a length prefix to our data, so that protobuf-net knows how much data to deserialize.
Thankfully, this functionality is included so we don't need to do it ourselves. Replace Serialize
and Deserialize
with SerializeWithLengthPrefix
and DeserializeWithLengthPrefix
respectively, passing in a prefix encoding. It doesn't necessarily matter which you choose, but they do need to match. I typically choose PrefixStyle.Fixed32BigEndian
.
Endianness, also known as “byte order”, is the order in which bytes are stored in memory. Big and little endian bytes are equal to each other's reversed bytes.
For example, the number 1,234,567,890 can be stored in memory as either 49 96 02 D2
(big endian) or D2 02 96 49
(little endian). Most modern systems use little endian, but some older systems such as PowerPC and SPARC use big endian.
It's important to know which endianness your system uses, because if you try to read a big endian file on a little endian system, you will get the wrong data. This is why we need to specify the endianness when using PrefixStyle.Fixed32BigEndian
.
As an additional fun fact: when transmitting data over the network, it is common to convert the data from whatever endianness your system uses (known as “host order”) into big endian (also known as “network order”).
The final versions of SaveGame
and LoadGame
should now look like this:
private void LoadGame()
{
if (!File.Exists(_saveFile))
return;
using var file = File.OpenRead(_saveFile);
using var gzip = new GZipStream(file, CompressionMode.Decompress);
Data = Serializer.DeserializeWithLengthPrefix<SaveData>(gzip<mark>, PrefixStyle.Fixed32BigEndian</mark>);
transform.position = Data.Position;
}
private void SaveGame()
{
Data.Position = transform.position;
using var file = File.Create(_saveFile);
using var gzip = new GZipStream(file, CompressionMode.Compress);
Serializer.SerializeWithLengthPrefix(gzip, Data<mark>, PrefixStyle.Fixed32BigEndian</mark>);
}
And there we have it. We now have the foundations of save/load system! One which can be easily scaled by simply adding more things to SaveData
- including child classes.
I do want to stress that the code I've shown in this guide is only for demonstration purposes. I'm aware that I've committed some code sins here, but I am confident that as a developer, you will find ways to improve it. You may also feel like this is a rather convoluted way to save data, but the fact is that data serialization is not exactly trivial - especially if you want it done right. Game development takes a lot of effort as it stands. Good game development will take some time.
Besides, we didn't need to use BinaryFormatter
.
Yes, I have finished my mission… Now I can die peacefully.