To illustrate the way in which I program, I figured it would be interesting to show how I wrote the Entity-Component-System (ECS) for ETEngine. In this first part I will go over the basics of what it is and how we can work with it. In the next part I will go into the nitty gritty details of how it was implemented.
What is an Entity Component System?
Entity Component Systems are a design pattern that formalises the relationship between an Entity in a games Scene, it’s Properties (Data / Components) and the functionality that can be applied to them (Systems).
ECS follows the principles of Data Oriented Design, in which Data Structures (the Components) are defined separately from the logic (Systems). Entities are simple identifiers, with which we can find all the components which hold the data that makes up this entity, which are stored separately. Systems then modify the data in components in a way that’s agnostic to specific entities.
This design pattern is an alternative to other common aproaches game engines use for structuring scene data, like for example Scene Actors (in which all properties and functionality are members of an Actor, often implemented using lots of inheritance), or the nowadays somewhat ubiquitous Entity-Component model that both Unity and Unreal implement in their own way (Components contain both data and functionality, and entities may or may not do so too). Both of these approaches follow a more OOP based approach, and while this may seem a bit more intuitive to work with, there are several benefits to using an ECSs based architecture instead:
- The separation of concerns makes ECS very easy to modify. Components can be mixed and matched freely for each entity without needing to rewrite anything
- Components can easily be added and removed from entities at runtime, without needing to worry about breaking complex relationships. Systems will automatically start and stop affecting entities with the relevant components
- ECS based architecture is highly scalable because it uses composition instead of complex inheritance structures, which makes it easy to add new features as one doesn’t need to worry about breaking existing code or respecting strange edge cases that are distributed across a large code base
- Last but not least, using ECS can have big performance benefits, especially when handling scenes with large numbers of entities. This is because data can be structured in a way that makes iteration very cache friendly. They also enable mostly automating parallel execution, because systems can strictly define mutability and execution sequencing
Example of a feature implemented using ECS in ETEngine
One of the the features in my engines demo is a spawn system, which is used to shoot balls with rigid body physics from the camera into the scene. Here is how this system is defined:
//---------------------------
// SpawnSystem
//
// Spawns objects upon user input
//
class SpawnSystem final : public fw::System<SpawnSystem, SpawnSystemView>
{
public:
SpawnSystem();
void Process(fw::ComponentRange<SpawnSystemView>& range) override;
};
There are three aspects we’re going to have a closer look at:
- The Spawn system declares it’s dependencies in it’s constructor
- The Spawn system works with something called
SpawnSystemView(a type ofComponentView) - The Spawn system has a process function in which the main functionality is implemented
Dependencies and Execution Flow
In order to ensure a correct execution order, systems declare their dependencies and dependents in their constructor:
//--------------------
// SpawnSystem::c-tor
//
// system dependencies
//
SpawnSystem::SpawnSystem()
{
DeclareDependencies<fw::TransformSystem::Compute>(); // update after transforms have been updated so we spawn from the most recent spawner position
DeclareDependents<fw::RigidBodySystem>(); // update before rigid bodies so bullet simulates our spheres as soon as they are added
}
Dependents are any systems that need to run after our system, where as dependencies are systems that need to run before our system because it generates information we depend on. The engines ECS automatically calculates the order in which systems execute based on this information. This is handled by the so called EcsController, which handles all top level concerns for dealing with an ECS. In order for it to start running the spawn system, all we have to do is register it:
void MainFramework::OnSystemInit()
{
fw::EcsController& ecs = fw::UnifiedScene::Instance().GetEcs();
ecs.RegisterSystem<FreeCameraSystem>();
ecs.RegisterSystem<SpawnSystem>(); // the order here doesn't particularly matter as all systems will be sorted according to their dependencies after the fact
ecs.RegisterSystem<LightControlSystem>();
ecs.RegisterSystem<SwirlyLightSystem>();
ecs.RegisterSystem<CelestialBodySystem>();
ecs.RegisterSystem<PlaylistSystem>();
ecs.RegisterSystem<DemoUISystem>();
}
If we at any point wanted to disable the spawn behaviour, all we’d need to do is unregister the Spawn System:
ecs.UnregisterSystem<SpawnSystem>();
Component Views
Component views provide a way to access component data in a cache friendly way while iterating entities relevant to a system, while also declaring if we need read or write access to run the system (which is important for multi-threading). Here’s what this looks like for our spawn system:
//---------------------------
// SpawnSystemView
//
// ECS access pattern for spawn behavior
//
struct SpawnSystemView final : public fw::ComponentView
{
SpawnSystemView() : fw::ComponentView()
{
Declare(spawner);
Declare(transform);
}
WriteAccess<SpawnComponent> spawner;
ReadAccess<fw::TransformComponent> transform;
};
We see this view allows us to read from transform components (but not write, their access is const), and write to spawn components (implicitly also read from them of course). Note we don’t need to know that this spawner is attached to the camera so our spawn system view doesn’t include a camera component. This way if we wanted to for example create a fountain in our world it could reuse the same components and systems we use here too.
Lets have a look at what these components look like:
//---------------------------------
// SpawnComponent
//
// Component containing data to spawn entities with model and rigid body components
//
struct SpawnComponent final
{
// definitions
//-------------
ECS_DECLARE_COMPONENT
// construct destruct
//--------------------
public:
SpawnComponent(AssetPtr<render::MeshData> const meshPtr,
AssetPtr<render::I_Material> const materialPtr,
float const s,
btCollisionShape* const shape,
float const shapeMass,
float const interv,
float const imp);
~SpawnComponent() = default;
float interval = 0.f;
float cooldown = 0.f;
float impulse = 0.f;
// hold the assets so that they are loaded in already when spawning
AssetPtr<render::MeshData> mesh;
AssetPtr<render::I_Material> material;
float scale = 1.f;
btCollisionShape* collisionShape = nullptr;
float mass = 1.f;
};
As we can see, it entirely contains data, some of it relating to its spawn behavior, and some of it containing information about what to actually spawn.
I will spare you the details of transform components, but essentially it contains a bunch of transform data, accessed through basic getter and setter functions (but no complex behavior). If you want to see the details you can find it HERE.
Our spawn system will automatically operate on all entities that have both a spawn component and a transform component.
Beyond read and write, here are a few more advanced access patterns we can define in our Component View (which are not required for this example but good to know about)…
EntityRead<ComponentType>While for performance reasons it’s best to avoid it. sometimes it’s necessary to read component data from another entity for a system to function, and this accessor makes this possible. To use it one supplies an entity ID, and the accessor will provide a pointer to the relevant component data:ParentRead<ComponentType>This allows the system to read component data from a parent entity. E.T.Engines ECS supports scene hierarchy (parent child relationships between entities), so this allows us to read data from a parents component. A good example is the transform system accessing a parents transform to update the childs coordinates accordingly.
EntityRead<MyComp> extComp; T_EntityId extEntity; MyComp const* const comp = extComp[extEntity];
- We can also create a System which requires an entity to have a component, but doesn’t actually need to read or write any data from this component using the
Includefunction. This can be useful to limit the amount of entities that a system runs on. For example here is a view for a system that operates only on cameras, but never actually needs to read or write data from a camera component:
//---------------------------
// FreeCameraSystemView
//
// ECS access pattern for editor camera behavior
//
struct FreeCameraSystemView final : public fw::ComponentView
{
FreeCameraSystemView() : fw::ComponentView()
{
Declare(camera);
Declare(transform);
Include<fw::CameraComponent>();
}
WriteAccess<FreeCameraComponent> camera;
WriteAccess<fw::TransformComponent> transform;
};
Processing Systems
Every system implements a Process function. This is where the magic happens, systems use this opportunity to iterate over all entities with the correct combination of components as defined by their ComponentView and execute the behavior the system is intended for.
void Process(fw::ComponentRange<SpawnSystemView>& range) override;
As you can see, the function provides a so called ComponentRange, which provides the means to iterate across relevant entities. Let’s have a look at our spawn systems Process function and then break down what’s happening.
//------------------------------
// SpawnSystem::Process
//
// Spawn upon user input
//
void SpawnSystem::Process(fw::ComponentRange<SpawnSystemView>& range)
{
// Skip execution unless we have the required user input
if (!(core::InputManager::GetInstance()->GetMouseButton(E_MouseButton::Right) >= E_KeyState::Down))
{
return; // nothing needs to be spawned
}
// common variables
fw::EcsCommandBuffer& cb = GetCommandBuffer();
float const dt = core::ContextManager::GetInstance()->GetActiveContext()->time->DeltaTime();
// spawn stuff
for (SpawnSystemView& view : range)
{
view.spawner->cooldown -= dt;
if (view.spawner->cooldown <= 0.f)
{
view.spawner->cooldown = view.spawner->interval;
fw::T_EntityId const spawned = cb.AddEntity();
vec3 const& dir = view.transform->GetForward();
fw::TransformComponent tf;
tf.SetPosition(view.transform->GetWorldPosition() + dir * view.spawner->scale);
tf.SetScale(vec3(view.spawner->scale));
fw::RigidBodyComponent rb(true, view.spawner->mass, view.spawner->collisionShape);
fw::ModelComponent model(view.spawner->mesh, view.spawner->material);
cb.AddComponents(spawned, tf, model, rb);
vec3 const impulse = dir * view.spawner->impulse;
cb.OnMerge(spawned, fw::EcsCommandBuffer::T_OnMergeFn([impulse](fw::EcsController& ecs, fw::T_EntityId const entity)
{
ecs.GetComponent<fw::RigidBodyComponent>(entity).ApplyImpulse(impulse);
}));
}
}
}
Before we start actually iterating, we get an opportunity to do some housekeeping, like computing variables that are consistent for all entities. Let’s skip to the for loop, as what comes before will be self evident.
for (SpawnSystemView& view : range)
{
view.spawner->cooldown -= dt;
if (view.spawner->cooldown <= 0.f)
{
view.spawner->cooldown = view.spawner->interval;
// spawn stuff
}
}
When iterating over a component range, for each relevant entity we get a reference to one of the component views we declared for our system. This allows us to access component data through the operator-> of our accessors (ReadAccess<TComponent> provides a const ref and WriteAccess<TComponent> provides a non-const ref).
We declared WriteAccess<SpawnComponent> spawner, which allows us to modify the cooldown member and read the interval member of SpawnComponent. By doing so we ensure a more reasonable spawn rate for the entities.
The next block of code deals with actually spawning a new entity:
fw::T_EntityId const spawned = cb.AddEntity();
This is orchestrated through a command buffer, which we accessed before entering the for loop, using EcsCommandBuffer::AddEntity() and EcsCommandBuffer::AddComponents(entity, ...). While the EcsController has all the required functions for adding entities and components, we can’t actually use those while iterating over entities in our component range, because adding them immediately may cause undefined behavior due to how the memory gets restructured when entities get added.
So the command buffer allows us to tell it what changes we want to make (Like adding or removing entities or components), and then stores them until it can execute them later when we’re done iterating all the entities. Note that command bufferss are only necessary for systems that make structural changes to our ECS, most systems only modify the data in components that already exist, and therefore don’t need a command buffer.
In our case we start by adding a new entity. Note that all we get back is T_EntityId, remember with ECS an entity is just an identifier, and all the data lives in components. Our new entity doesn’t actually contain any data yet, so let’s give it some:
vec3 const& dir = view.transform->GetForward(); fw::TransformComponent tf; tf.SetPosition(view.transform->GetWorldPosition() + dir * view.spawner->scale); tf.SetScale(vec3(view.spawner->scale)); fw::RigidBodyComponent rb(true, view.spawner->mass, view.spawner->collisionShape); fw::ModelComponent model(view.spawner->mesh, view.spawner->material); cb.AddComponents(spawned, tf, model, rb);
First we create a TransformComponent, RigidBodyComponent and ModelComponent, which we initialize with data from the spawner entity that we can access through our ComponentView. Then, we tell the command buffer to add those components to the entity we had it make in the previous step.
That’s all that’s needed to create a new entity, once we’re done updating all our spawners, the command buffer will tell the ECS to add this entity, and going forward this new entity will be affected by all systems that need transform, rigid body and/or model components.
There’s one last thing we do here though. Once the entity has been created, we want to give it a little push so it goes flying off into the sunset. We have to wait until it actually exists though, and so we use the command buffers OnMerge() callback function to do this as soon as all it’s commands have been executed:
vec3 const impulse = dir * view.spawner->impulse;
cb.OnMerge(spawned, fw::EcsCommandBuffer::T_OnMergeFn([impulse](fw::EcsController& ecs, fw::T_EntityId const entity)
{
ecs.GetComponent<fw::RigidBodyComponent>(entity).ApplyImpulse(impulse);
}));
More examples
And with this we have touched on most of the fundamental concepts required to write features with E.T.Engine’s ECS. If you’re interested in why structuring your code like this can lead to some big performance gains, hold on tight until I write the next part about how some of this works under the hood…
I realize some of the workflow might seem a little aesthetically contrived if you’re used to how things are done in Unreal or Unity, however the quirks mostly start and end with what you’ve already seen in this post. After using this approach to scene managment for a bit I’ve really come to enjoy how well this allows the user to keep all of the code related to implementating a feature logically contained in one place. If you have ever worked on projects with a large codebase you will have learnt the joys of discovering 30 little modifications you have to make to tangentially related systems all across the codebase just to add one small feature – this seems to be much rarer with this architecture.
Here are some examples of how other systems are implemented
- SwirlyLightSystem: Moves lights about in a scene like fireflies
- PlaylistSystem: Allows users to control a playlist on an audio source
- TransformSystem: Updates transform components according to scene hierarchy (this is a unique [to my knowledge] feature of my ECS implementation that bakes entity hierarchy into how it works, watch out for my next post to see more about how it works).
ECS and Scene Serialization
E.T.Engine’s ECS can be automatically serialized and deserialized to binary or human readable formats using the engines reflection system (built on top of RTTR). This is also the process during which a components variables can be intialized, should any such thing be required. Here is an excerpt from a serialized SceneDescriptor that describes one of the spawner entities we looked at before:
{
"id": 0,
"components": [
{
"transform component": {
"position": [ 0.0, 2.0, -10.0 ],
"rotation": [ 0.0, 0.0, 0.0, 1.0 ],
"scale": [ 1.0, 1.0, 1.0 ]
}
},
{
"camera component": {
"is perspective": true,
"field of view": 45.0,
"ortho size": 25.0,
"near plane": 0.1,
"far plane": 1000.0
}
},
{
"audio listener component": {
"gain": 1.0
}
},
{
"spawn comp desc": {
"mesh": "Models/sphere.etmc",
"material": "Materials/MI_TexPBR_Ball.json",
"scale": 0.2,
"shape": {
"sphere collider shape": {
"radius": 0.2
}
},
"mass": 3.0,
"interval": 0.2,
"impulse": 30.0
}
}
],
"children": []
}
This entity is composed of a transform, a camera, an audio listener and a spawn component. Except what you actually see here are ComponentDescriptors, or even more accurately implementations of them. No, the irony that I ended up using inheritance to serialize components in an ECS doesn’t escape me. The abstraction this makes saving and loading very simple, and importantly we do not use this inheritance at runtime, only for loading and saving.
In E.T.Engine, ComponentDescriptors create Components for the ECS to use at runtime. For example, here is the spawn components descriptor:
// SpawnComponent.h
//---------------------------------
// SpawnComponentDesc
//
// Descriptor for serialization and deserialization of spawn components
//
class SpawnComponentDesc final : public fw::ComponentDescriptor<SpawnComponent>
{
// definitions
//-------------
RTTR_ENABLE(ComponentDescriptor<SpawnComponent>)
DECLARE_FORCED_LINKING()
// construct destruct
//--------------------
public:
SpawnComponentDesc() : ComponentDescriptor<SpawnComponent>() {}
SpawnComponentDesc& operator=(SpawnComponentDesc const& other);
SpawnComponentDesc(SpawnComponentDesc const& other);
~SpawnComponentDesc();
// ComponentDescriptor interface
//-------------------------------
SpawnComponent* MakeData() override;
// Data
///////
AssetPtr<render::MeshData> mesh;
AssetPtr<render::I_Material> material;
float scale = 1.f;
fw::CollisionShape* shape = nullptr; // This should be updated to use the engines smart pointer system
float mass = 1.f;
float interval = 1.f;
float impulse = 0.f;
};
// SpawnComponent.cpp
// reflection
//------------
RTTR_REGISTRATION
{
rttr::registration::class_<SpawnComponent>("spawn component");
BEGIN_REGISTER_CLASS(SpawnComponentDesc, "spawn comp desc")
.property("mesh", &SpawnComponentDesc::mesh)
.property("material", &SpawnComponentDesc::material)
.property("scale", &SpawnComponentDesc::scale)
.property("shape", &SpawnComponentDesc::shape)
.property("mass", &SpawnComponentDesc::mass)
.property("interval", &SpawnComponentDesc::interval)
.property("impulse", &SpawnComponentDesc::impulse)
END_REGISTER_CLASS_POLYMORPHIC(SpawnComponentDesc, fw::I_ComponentDescriptor);
}
DEFINE_FORCED_LINKING(SpawnComponentDesc) // force the linker to include this unit
// constructor and assignment implementations are omitted for simplicity
//------------------------------
// SpawnComponentDesc::MakeData
//
// Create a spawn component from a descriptor
//
SpawnComponent* SpawnComponentDesc::MakeData()
{
return new SpawnComponent(mesh, material, scale, shape->MakeBulletCollisionShape(), mass, interval, impulse);
}
The macros RTTR_ENABLE(ComponentDescriptor<SpawnComponent>) and the code following RTTR_REGISTRATION is what is needed to tell the engine how to automatically serialize and deserialize this component descriptor.
Also note that SpawnComponentDesc is a ComponentDescriptor<SpawnComponent>, and implements MakeData(), which creates a new SpawnComponent. During the call to MakeData(), we take the opportunity to get the Physics Engine to create a collision shape. Effectively this allows us to use the MakeData() function to do something similar to what happens in Unreal engine when BeginPlay() is called on an actor.
Not every component needs this kind of initialization. Using a SimpleComponentDescriptor, ECS components can be their own descriptors:
// .h
//---------------------------------
// AudioListenerComponent
//
// Descriptor for serialization and deserialization of sprite components
//
class AudioListenerComponent final : public SimpleComponentDescriptor
{
// definitions
//-------------
ECS_DECLARE_COMPONENT
RTTR_ENABLE(SimpleComponentDescriptor) // for serialization
DECLARE_FORCED_LINKING()
friend class AudioListenerSystem;
// construct destruct
//--------------------
public:
AudioListenerComponent(float const gain = 1.f) : m_Gain(gain) {}
~AudioListenerComponent() {}
// accessors
//-----------
float GetGain() const { return m_Gain; }
// modifiers
//-----------
void SetGain(float const val) { m_Gain = val; }
// Data
///////
private:
vec3 m_PrevPos;
float m_Gain = 1.f;
};
// .cpp
// reflection
//------------
RTTR_REGISTRATION
{
BEGIN_REGISTER_CLASS(AudioListenerComponent, "audio listener component")
.property("gain", &AudioListenerComponent::GetGain, &AudioListenerComponent::SetGain)
END_REGISTER_CLASS_POLYMORPHIC(AudioListenerComponent, I_ComponentDescriptor);
}
DEFINE_FORCED_LINKING(AudioListenerComponent) // force the linker to include this unit
ECS_REGISTER_COMPONENT(AudioListenerComponent);
ECS_REGISTER_COMPONENT(ActiveAudioListenerComponent);
On the flip side for some additional initialization functionality, component descriptors can implement OnScenePostLoad() in order to execute functionality after the component has been loaded and added to the scene, like in the following example, where a GUI Canvas Component creates it’s draw context after it can grab data from a linked entity containing a camera component:
// .h
class GuiCanvasComponent final : public SimpleComponentDescriptor
{
// definitions
//-------------
ECS_DECLARE_COMPONENT
RTTR_ENABLE(SimpleComponentDescriptor) // for serialization
// init deinit
//-------------
static void OnComponentAdded(EcsController& controller, GuiCanvasComponent& component, T_EntityId const entity);
static void OnComponentRemoved(EcsController& controller, GuiCanvasComponent& component, T_EntityId const entity);
// interface
//-----------
bool CallScenePostLoad() const override { return true; }
void OnScenePostLoad(EcsController& ecs, T_EntityId const id) override;
private:
// Some other data members and functions
EntityLink m_Camera; // for screenspace canvases defines the viewport to render to, for worldspace canvases defines where the events are coming from
};
// .cpp
//-------------------------------------
// GuiCanvasComponent::OnScenePostLoad
//
void GuiCanvasComponent::OnScenePostLoad(EcsController& ecs, T_EntityId const id)
{
ET_UNUSED(id);
if ((m_RenderMode == E_RenderMode::ScreenSpaceOverlay) && (m_Id == core::INVALID_SLOT_ID))
{
ET_ASSERT(m_Camera.GetId() != INVALID_ENTITY_ID);
ET_ASSERT(ecs.HasComponent<CameraComponent>(m_Camera.GetId()));
CameraComponent const& camera = ecs.GetComponent<CameraComponent>(m_Camera.GetId());
if (camera.GetViewport() != nullptr)
{
GuiExtension& guiExt = *UnifiedScene::Instance().GetGuiExtension();
m_Id = guiExt.CreateContext(camera.GetViewport());
guiExt.SetLoadedDocument(m_Id, m_GuiDocumentId);
m_DataModel = guiExt.GetDataModel(m_Id);
guiExt.SetContextActive(m_Id, m_IsActive);
}
}
}
Of interest here is m_Camera, which is an EntityLink. EntityLinks are what’s used to reflect entity IDs, because they get patched up after the scene is loaded to contain the correct entity ID.
And then lastly also note the static callback functions OnComponentAdded and OnComponentRemoved. These allow the GuiCanvasComponent to execute some functionality right after they get added or removed from the ECS, by registering those functions with the EcsController:
//---------------
// EcsController
//
// Full context for an entity component system
//
class EcsController final
{
// [...]
// component events
template<typename TComponentType>
T_CompEventId RegisterOnComponentAdded(T_CompEventFn<TComponentType>& fn);
template<typename TComponentType>
T_CompEventId RegisterOnComponentRemoved(T_CompEventFn<TComponentType>& fn);
template<typename TComponentType>
void UnregisterComponentEvent(T_CompEventId& callbackId);
// entity events
T_EntityEventId RegisterOnEntityAdded(T_EntityEventFn& fn);
T_EntityEventId RegisterOnEntityRemoved(T_EntityEventFn& fn);
void UnregisterEntityEvent(T_EntityEventId& callbackId);
// [...]
};
The end
That’s it for today. If you have any questions feel free to comment or write me an email and I’ll try my best to provide clarity. If you’re interested in how this all actually works, keep your eyes peeled for part 2!
