My name is Daniel DiCicco and I am the developer of StarDrive, a 4X space game.
This article is an overview of the general approach we have taken with StarDrive’s AI and is meant to provide some insight into the process of designing an AI for games of this genre. Primarily we will discuss how “hierarchical” artificial intelligence can be organized into tiers, with boss AIs delegating tasks to sub-AIs to keep things clear and flexible for the developer. Furthermore, we will discuss the concept of “Goals” which act in a dual role as scripts and memories for AI behavior.
A few game design basics
Games of the 4X genre present quite a challenge to the AI designer because these games tend to be very complex with many nuanced systems. Generally speaking, every 4x game involves growing a civilization from its infancy through its various stages of maturity and ultimately to dominance over the game world. Along the way you will need to explore this world, decide how to allocate your resources to best exploit what you find in that world, and you will need to compete with the AI players for control of these resources using diplomacy and combat.
To understand some of the higher level concepts of AI design, you should consider a few game design basics. First, in every game there is a bunch of data stored in memory and this data is known as the Game State. In a 4x game this might be information such as where the planets are, the location of all the ships, their health, their velocity, and so on and so forth. In a 4x game the Game State is typically quite large with many thousands of items stored in memory.
The second game design basic to understand is that this Game State is updated at a fixed interval – typically sixty times per second. That is, sixty times every second, the game has some logic that it follows to bring the game state into the next moment. It starts at the top of a list of instructions and works its way to the bottom. Even in turn-based games like Endless Space this is happening – the nebulas swirl, a little light blinks on and off — whatever. It’s updating fast and frequently and typically redrawing everything you see on screen every frame. Thus, frames per second really means how many times the game is able to loop itself per second.
The Artificial Intelligence of most games lives in that little loop, notwithstanding games that utilize multi-threaded AIs which just have their own loops. Every loop, the AI has the duty of analyzing the game state, drawing some conclusions from that game state, and implementing some sort of reaction to the game state (even if the reaction is no reaction). And herein lies the rub and the hard work. What data should the AI analyze? What should it do with that data?
AI in StarDrive and the hierarchical AI concept
In StarDrive I have a big boss AI that I call the GSAI – the Grand Strategic Artificial Intelligence. The Computer Player. It has full access to all of the data about its own empire, just like a player would. It knows where all of its ships are and all of its colonies. It knows what ships are inside its sensor range, what technology it is researching, how much money it has, and so on. And just like a Human player, the AI has a few imperatives programmed into it.
First, the AI wants to be constantly expanding. But rather than bother itself with figuring out which planet exactly to colonize and what to build there, the GSAI passes that task on to a subordinate – the Expansion Manager. The Expansion Manager is a limited little fellow. He can’t see everything the GSAI can see and he can only handle a few orders at a time. But he sure does know how to colonize a planet.
And so every update, the GSAI will tell the Expansion Manager – “hey, I want to have 2 (or however many) plans underway to colonize new planets. Take care of it.” The Expansion Manager then creates “Goals” and executes the subroutines needed to carry out the goal. It figures out which planet is best based on quality and distance. It figures out if it has any colony ships available and if it doesn’t, it orders one to be built. It remembers where it is building that ship so that each update it can check if that ship is ready yet. Once the ship is ready, it orders that ship to go on and colonize the planet. The GSAI meanwhile hasn’t gotten its hands dirty with the details.
Second, an AI player has the imperative of survival. It needs to be able to defend itself from external threats and to do that it is going to need ships and troops. But again, the GSAI is really worried more about what it needs rather than how it needs to get done. So it creates a few new manager positions – a Force Pool Manager and a Defensive Coordinator – and it tells them what it wants. For example, it may say that we need 50 ships and of those ships 25 should be on defense and the other 25 we should leave available for offensive operations. And then off the Managers go. They analyze what they have available, compare it against what the boss wants, and if we’re missing some ships then they’ll create a few private goals to get those ships made.
The Defensive Coordinator highlights a third level of the AI hierarchy: the Unit AI. The Unit AI is essentially the Captain of the ship! While the Defensive Coordinator is responsible for deciding what the ship should guard, the Unit AI is responsible for actually driving the ship and deciding how to execute its orders. It never has to think about bigger picture concepts, instead needing only to know where to fly and what to shoot. Another example of a third-level AI is a planetary governor, who knows only how to manage his planet according to the role assigned it by his boss AI.
But there are even more managers to be had. StarDrive also utilizes a Diplomatic Manager whose job it is to create and respond to diplomatic events. There is an Economic Manager that sets tax rates based on need. A Research Manager that picks technologies to research based on the GSAI’s personality. A War Planner to make decisions about which planets to attack in a time of war and to create fleets for those tasks. A Fleet Admiral AI whose job is to execute the orders given to it by the War Planner. That sure is a lot of managers!
At the core of the AI are Goals
Yet for all of the complexity of an AI like this, the manager system keep things flexible and easy to both enhance and debug. If a developer is unhappy with the logic used for exploration, then he or she knows exactly where to go to retool that Manager’s thinking. If we want some new behaviors out of a ship’s captain in combat, then we go edit his Unit AI. By enforcing the hierarchy and compartmentalizing these AIs we can get reliable and predictable AI performance.
At the core of all of these manager systems is the concept of “Goals”. The GSAI tells the Managers what to do and then the Manager creates “Goals”, which are basically like scripts. The Manager knows what it needs to do to execute the goal and tracks the goal’s progress in steps. For instance, the Expansion Manager might follow logic like this when evaluating its assigned goal of colonizing a planet:
Steps:
- Find a planet to colonize
- Look for a colony ship that is awaiting orders. If none exists, go to step 3. If we have a ship, go to step 4.
- Check if we have requested a new colony ship. If not, create a colony ship goal. Next update, Go to step 2.
- Now that we have a ship, order the unit AI to go colonize that planet.
What would the human player do?
Ultimately, when designing a behavior for an AI, I like to think of what a human player would do. I break that down into discrete steps that the AI can execute using the same rules a human player must follow. I must be careful to consider all of the possible steps. What if the colony ship is destroyed while en route? The Expansion Manager needs to know that its goal failed and start over. But with careful planning and plenty of testing, it is possible to create complex finite state machines for these managers that are extremely effective at doing their jobs.
The downside of a system like this is that it is highly deterministic. Given a set of inputs, the AI is likely to respond in the same way every time because it is performing the same analysis. To combat this, the developer can input elements of randomness into AI decision making. So for instance, if the GSAI says “Give me 5 ships!” then the Force Pool Manager, rather than picking the same five every time, may do a few dice rolls to figure out which out of an array of acceptable ships it might build. Or which planet it will colonize, or what technology it will demand of a player, and so on and so forth.
Memories also play a crucial role in human decision making. If an AI has been a huge jerk to us then we’re going to remember it. We’re going to remember that they took our planet and we’re going to want it back. And, likewise, the AI has memories like this to. Deciding what to remember and how those memories will affect a Manager’s analysis, this is the creative work that the developer has to undertake.
Final thoughts
At the end of the day we have to keep our goal in mind as developers. Our job is to create a fun and challenging experience for the player. Even if we had all of the money and time in the world to create an AI for our game, it could always be better. There are always more human things it can do. But the goal here is not to pass the Turing test. It’s to make a fun game.
What’s fun, well that’s a whole different can of philosophical worms. I believe that would be an excellent topic for conversation here — what do you find fun in AIs from 4x games? What don’t you like? What do you want to see that you haven’t seen before? Who knows, maybe some developers out there are listening…
Until next time,
Dan.
Daniel DiCicco is the developer of the space 4X game StarDrive, due to release in Q1 2013. You can know more about Dan and find more tips on game development – especially concerning space 4X games – by having a look at our interview, where Dan talks about StarDrive, Kickstarter and indie games development.
Subscribe RSS
30 Comments
Related Articles:
- StarDrive Has Reached Its Funding Target!
- Starship Corporation Funding Campaign Complete
- StarDrive Alpha Funding Starts at Desura
- StarDrive: Beta Pre-Orders Pushed to Beginning of 2013
- StarDrive 2 Officially Announced
Loading ...
Good article! I was thinking of doing something similar (Tiered AI), and I’m glad to know that it is actually a good idea and is used in other 4X games!
Wow, what a cool article! Honestly, I had a feeling that AI was written that way and it’s cool to see that the rough concept I had of it was right!
You have a big fan in me, Zero. Thanks for writing this.
Daniel, you’re gonna be the “Sid Meier” of the new Generation.
A lot of the time AI’s in 4X games and others do not seem to have any personality! In fact in a lot of these game AI’s seem to be pretty much like every other AI. I’d love to see an AI on par with say Firaxis Sid Meyer’s Alpha Centuri, that for me at least was my favorite AI and the best AI I have ever encountered in any game.
Nice article. I’m curious, since StarDrive is the first video game you’ve developed, did you survey the different AI schemes employed in similar games before settling on the one you used? If so, what other schemes did you investigate and what were their strengths/weaknesses?
My approach is, I think, fairly standard at least on a conceptual level. Frankly there are no guides out there to how to do it, you know? But there are plenty of text books that discuss basic AI theory, and it’s up to the designer to pick and choose various theories and apply them to the game.
However, the specifics of the Manager implementations, these can vary wildly. I’m using Finite State Machines, a fairly standard approach. But as I noted, these are deterministic with some random elements thrown in. There are other ways for these Managers to make decisions. You could employ genetic algorithms for things like, say, Ship Design. This would have the manager constantly designing and testing new ship designs against each other to figure out which is the best ship. But this is computationally intensive and not suitable for run-time execution I would think.
One might also employ neural networks – learning algorithms – to study player behavior and respond to it. This is complicated stuff and could ultimately lead to some great AI behavior. But, it’s complicated stuff! I’m sure I could implement it, but there is a balancing of effectiveness versus time invested.
For me, the goal is as I said to have a fun and effective AI. The approach I chose has some very emergent elements to it. While it is at a fundamental level a scripted AI, the interaction between the various managers and their boss AIs doesn’t give you a feeling that you’re playing against some little robot.
Or so I hope! You tell me when the time comes.
Wow, thanks for providing such a thorough answer to my question! I’m looking forward to the release of StarDrive.
The new Sid Meier? That’s as maybe, we’ll have to wait and see. Dan’s analysis sounds good, but as always the proof will be in the pudding. I guess we’ll find out in 2-3 months.
To my way of thinking a micromanaging AI should behave according to the high-level directives/policies given to it by the imperator, with occasional direct micromanagement when it is felt necessary. This is how I play Distant Worlds, and although the AI is not always perfect (indeed, sometimes far from it), it is good enough most of the time to make for a decent and enjoyable simulation.
On a related note, I think Global Geopolitical Simulator 2 falls down a bit in this regard. That is, it requires too much micromanagement with the economy, when in reality the President, Premier, Prime Minister, etc, would have a staff of finance ministers and economists making recommendations and taking care of the day-to-day stuff.
Speaking of Sid, I agree with Dan totally about the quirky and inconsistent diplomacy AI in Civ5. As Dan so succinctly noted, it is very frustrating in Civ when you have every reason to believe that a relationship with an AI nation is excellent, only to see them stab you in the back at the drop of a dime.
Good read but has the stardrive AI a grand strategic goal it can pursue from the getgo? The feature is very nice in Civ 5 but only seems to apply to certain victory conditions. Imagine if AI could “choose” certain sets of strategy (influenced by it’s race/faction and such) and follow that strategy making adjustments when necessary and constantly evaluate everything it has in order to optimize that strategy. That’s what humans are doing, strategic as opposed to tactical/situational optimization. This is necessary if AI is ever going to be a true “AI”.
What can be done now is more like having massive code to cover every situation without putting them together and makes some coherent sense from each. I don’t know how far we are from this feature(?) But it’s going to revolutionize more than games :P
Things are going forward though, Civ 5 AI despite it’s flaws is one of the best, if not best AI to date (I’m talking about G&K expansion here :P) given the games complexity. It’s certainly better than civ 4 or any of it’s derived mods (even excellent ones like Tholals AI mod for FFH). GC2 TotA also has better than average AI but has little strategic competence like the one in Civ 5. More 4x games like Civ 5 AI would be much welcomed though I’m not sure this niche 4x genre alone can push the boundaries too much.
I think the AI in Civ V now is pretty darn good. I played a game on the 2nd to highest difficulty last week and really enjoyed the AI. They even try to screw you with peace offers. So I’m beating them in a war and I have like -4 happiness, and then they offer me another city as a peace offering, which would put me way below -10. It’s clever. I like it.
I settled in on this approach to StarDrive’s AI by reading a bunch of AI textbooks.
Actually, I think that indicates a incompetent AI rather than a clever one; it basically offers a peace deal that no one in their right mind would accept. =P
A dev diary from Blue Manchu mentioned the aim of a gaming AI is not to be the best there is but to provide a fun and ultimately winnable experience that mimics playing with a human player.
Personally, I think that’s the key difference between building a great AI and a great gaming AI; it’s all about making something so clever that it let’s you win without you realising it.
Nice to see an article like this from you, Dan. I agree completely with you. Every time I start thinking about AI, I am constantly reminded how easy it is for the human mind to judge an entire situation at a quick glance. The AI, on the other hand, has to take in tons of raw data and then try and make sense of what, if any, effect this data has on it. I kind of view AI in the same way I view magic. If it wow’s the audience and keeps them guessing, who cares if behind all the smoke and mirrors the act is more artificial than it appears. It isn’t always easy to see things this way as the designer, as the magic is removed when you are the one creating the tricks.
Hi Dan,
I found your article really interesting. I had no idea that AI was generally programmed in an “object oriented” manner with AI “objects” interacting with each other, but the way that you described it makes perfect sense. I guess object oriented programming makes everything easier, including AI.
The whole deterministic nature of hierarchical AI is interesting too, the idea that given the same data, it would do exactly the same thing every time, without some sort of randomizing factor. What about a game like chess. A chess AI doesn’t seem to so the same thing every time, so does that mean that there is a random factor somewhere in there as well. For example if it has 3 potential moves that it considers roughly equal in effectiveness, does it roll a 1d3 in order to decide which one to use?
I noticed in many strategy games that the AI will often do stupid things (for example in Civ V, moving its archery units to the front line to be slaughtered by melee troops). Do you think that this is just a failure of the programmers to cover that possibility or perhaps the AI is just using a script to solve a problem that it was never meant to solve?
Thanks again for taking the time to write your article, I really enjoyed reading it.
Cheers,
Mark.
Daniel, I enjoyed reading your article and found it informative. Thank you for taking the time to write it.
You are certainly doing all the right things to promote Stardrive and stir up interest amongst the strategy gaming community.
Please open access to your AI code in an easy to access format as soon as possible upon release. The community has a significant amount of free resource available for you to tap into to improve your game and, ultimately, result in more base units sold. AI is one of the most important factors in game longevity.
I trust that game balance variables will be available in an easy to edit fashion also?
I think it’s great for a developer to discuss details of their AI design. Better than the usual buzzword ridden nonsense you hear – “we’re using fuzzy neural networks and genetic algorithms so that our AI learns to play just like a human player would!”
The main problem with 4x games for AI is that they are just so complicated. AI is very good at dealing with problems that are very well defined and self contained. With games, you reach a level of complexity where it no longer makes sense to model the ‘solution’ to playing the game as a single self contained problem, but rather as a set of different problems. Orchestrating between a set of different solutions to different, but related, problems is an area where the human brain still hugely outperforms computers.
One could write a perfectly good module to handle colonising planets and a module to handle your military fleets. A human player, however, can decide to change his colonising strategy based upon how well hcan defend certain planets. He/she may then alter his fleet management to accommodate the new colonising plans.
In case anyone is interested and hasn’t heard of it, would recommend checking out the google AI challenge – which had two open contests based around writing AIs for simplified expand and conquer games.
Thanks for the heads up on AI Challenge, seems like fun :)
Thanks again for finding the time to write about this Dan.
I think it’s a great introduction to the subject. You provide great insights. The hierarchical structuring of the AI is brilliant. As you say, it gives you flexibility to tweak the code more effectively. It provides a very modular and clean way to structure your code, so that you can debug or tweak it easily in the future. Your approach is deterministic as you say, but with sufficiently randomness injected it could work.
I think this was really helpful on helping demystifying this subject.
let’s be honest – it is not an overview of AI designs. Daniel only shares one possible idea of the hierarchical AI – a simple one, but lacks many aspects for building viable-competitive AI (see amazon for the better game AI books – “AI for games” & etc ). So article’s title is deceptive :)
It is based on an assumption, that goals and decisions are hierarchically independent branches of the goal-decision tree. But that’s not true in most cases – there are several strategies to choose from for the goal with own risks. In one situation economic strategy is more viable, in other – military and diplomatic combo. So supervising AI (root) and it’s children AI have to respond to many stimulus, leading to (state’s machine) state explosion; moreover stimulus events for state transitions usually relate different AI inheritance levels. Of course finite state space can be pruned by merging/disposing some states/transitions in advance (= more predictable behaviour).
Mentioned decision randomness staves off predictability, but superficial and unfounded responses can ruin impression a lot – stucked flying enemy behind a box on the floor looks unreal (reason for that is straight (A* variety) pathfinding algorithm).
So the AI implementation is very hard task and requires substantial field knowledge and experience. Implementation complexity also plays very important role, as large decision space (surely debugging hell ;)) can take to much time to process (player noticables as lag, Daniel mentiones that as a (low) refresh rate), so some games have options for setting advanced AI by a player. Even multithreaded AI architecture isn’t a cure, as it brings aspects of concurrent decision making in a timely fashion (decision is still beeing evaluated, but time frame for that is over). Although multicore hardware exploitation is a right way to go
Maybe the subtitle is not the best, I agree. After all, Dan explains that this is an overview but of his own approach on his game. So, perhaps a better title for the article would be “AI Design in 4X Games – An Approach” or “AI Design in 4X Games – Some Insights”. I made up the title myself, Dan only supplied the “AI Design in 4X Games” part.
What do you think Dan? Should we change the title? And if so, can you help me find the most suitable one?
I think the title is fine. I don’t like editing stuff after its gone to print anyway! The article speaks for itself. It is “an” approach for sure, not “the” approach.
Yes, the text is fine, it was only my choice for the subtitle that was poor. I’m not comfortable with it. It doesn’t seem proper. I screwed it, I’ll fix it. As you said, it is “an approach”. I’m sorry about this.
it is widely accepted practice to have draft versions of an article and to improve it by changing some parts while mentioning it (revised, edited, version and so on) – internet media even encourages that :)
“AI design in games – StarDrive’s approach” or “AI design challenges in StarDrive game” could be more precise (IMO).
Although the topic is very relevant, significant, wide, can integrate several math branches and in most cases is a watchfully protected non-public IP
I disagree. Although the example included in the article is about Dan’s game – because that’s his experience – the principle he used is generic enough to be applied to any other 4X game, in my opinion.
and it is a good example of an ancient and general divide-and-rule principal :)
Why do I think the title (AI design in 4X games) is deceptive? The title is truly generic – Dan talks about state machines and partitioned goal-oriented behaviour for decision making, but what about decision/behaviour trees, fuzzy logic, rules and scripting, etc.? what is wrong about other options – can’t be used in 4X games? why and what implications that would bring? that still covers the generic AI design title and possible approach to go for.
If an example is provided, then it would be nice to understand how an architecture was picked from many possible options – how constraints and requirements influenced the picks, what consequences (tradeoffs) that would have.
Very interesting I will try this approach! Thank you very much for taking your time to talk about it.
I know very little about programming – although I’ve made some popular Fallout 3 mods in the past that had a good amount of scripting work (FWE, Fallout Wanderers Edition).
Anyway – I’d be curious to understand how the concept of “risk” plays into AI decision making.
It would seem to me that balancing acceptable levels of risk across different types of development activities, and how the AI prioritizies responses to risk based on a feed of information would central to making both an effective AI and as an effective means of creating distinct AI “personalities.”
One of the biggest issues I’ve seen with 4X AI’s is that the AI isn’t really proactive and they are often barely reactive. What do I mean? Building up a huge pile of fleets next to the AI as I prepare for an attack doesn’t cause the AI to shift strategies and start fast building tons of ships to prepare defense – it just plugs along doing whatever it was doing before.
Hi,
i developed some mods for Haegemonia (Prophecy, etc.) and others years ago. I read your blog from time to time and this article made a lot of fun to read.
I hope, and after watching different Stardrive Beta Videos i am sure, that this game has very much potential. Its on the right track, definatly. Since MoO2 which generated so much addiction there has never been a good one again. Neihter Soase, Sword, Haegemonia nor others.
So i guess, the right mixture comes through depth with simple control, a convenient graphic with various treats and the attention to detail. And it is always good to be guided by principles of functioning. Copy whats right and fill in your own ideas.
Star Drive seems to have everything, and offers diverse opportunities to expand.
So great job until now. It seems to have the fun-factor.
I would bet that Stardrive would perform significantly better on Kickstarter today. I missed it at that time, unfortunately. But hey, whatever. On Steam i will buy it and i bet, it will sell well for sure and we can hope for a variety of DLCs.
I’m looking forward to it. When will it be on steam?
Thanks for your support. Maybe StarDrive 2 will see Kickstarter someday. We are on Steam now, but not up for sale just yet. We are opening the pre-purchase / instant beta access on Jan 29th I believe. No firm date is set just yet but that’s the target date; we have to get it all worked out with Valve and that’s up to Iceberg to finalize.
I wonder, is THE AI in civ5 that much better than in Galciv2 or DW? How would you rank the AI of THE recent 4X games?
I don’t think it is necessary for an AI to mimic a human player. In many classic 4x games it explicitly did NOT do this (MoO, MoO2, Alpha Centauri) but instead played in a thematic fashion – simulated opposing empires with thematic agendas rather than game opponents.