For my next article I’m going to explain how I implemented modding. Please be aware that this is a more technically focused article.
I wanted to make sure the game was moddable because it can be lots of fun for the players: either by total conversions or just changing a few options and increase the game’s longevity. The game generally sticks around longer. That can only be a good thing!
Mod Format
I decided I would take the common approach of using XML files to hold the majority of the game data. I like XML because I feel it gives the players readable files that can easily be edited. It’s also generally easy to work with. However, I wanted to make sure these were “simple” XML files. So I don’t include any of the more complex features you can do with XML (ie: DTD’s).
What can you mod?
I made it possible so the majority of the game data can be modded. This includes, but is not limited to:
- orbitals (planets)
- stellar objects (stars)
- resources
- technologies
- buildings
- trade guilds
- technology upgrades
- races (which includes all necessary starting values)
- ship types (ie: destroyer, battleship values)
- ship components
- particle effects (there is actually a tool present to make these)
You can also mod parts of the UI, namely the look and feel. Since TWL is defined by XML, this fits in nicely. I do not expect anyone but the hardcore modders to really try to mod the UI. TWL can be fairly complex if you are not familiar with UIs and programming in general. But at least the option is there.
How does the XML look? Here is an example for one of the races:
This is an image of the modifier section and the starting ship components the empire knows. As you can see, the modifiers are listing what are the valid modifiers for the various weapons (WEAPON_1 can be marked as LONG_RANGE or RAPID_FIRE during ship design).
Here is another example of defining a ship type:
This is one of the rare ship types you can find. You can see how it can hold loads of components (4,000 spacial units) but costs A LOT! (27,000 industry). You can add or change these to your liking.
I also wanted to have modding for some uncommon elements of the game. I wanted to be able to allow the player to specify their own unique maps that can be outside of the game’s usual settings. So these are essentially pre-made maps. You’ll be able to specify the exact locations, and every single piece of useful information in these pre-mades. You could make up the Star Wars galaxy (this is pretty popular) if you wanted too, or something else. Yep, I’m a big Star Wars fan.
I think that having the ability to make pre-made maps is important. But I wanted to take it one step further. Since having to define every single piece of information could be a lot of work, I’m going to support random generation for various elements. Maybe you only want to have one or two unique planets. Maybe you just want to place certain stars in certain locations. Or maybe you just care about having their names right and everything else about them random. You’ll be able to do this! Random tags will exist to generate information you don’t care about it.
What can’t you mod?
Right now the main parts that you cannot mod are: any of the AI – fleet AI, game AI or anything else. For the time being, this is going to be hard coded. I may change this depending on how much funding I receive in the future.
You also cannot change any of the locations of the UI or the components on them. I don’t think I’ll ever support this.
Some Pit Falls
I ran into some pitfalls while coding up the game to support modding. The first one is maybe not so obvious to the average person: the time and effort to require to support modding. It took me quite awhile to make up the infrastructure to handle this feature. The reason is because all of the data is not only defined in the code but is now outside, in a file somewhere. This means that the game has to be able to load and read the various data files, then build the in memory objects to represent it. This can be a big project since more code and testing needs to be done.
Another issue was with the XML parsers. I started to write nested XML files in some cases, namely the races XML file. The Race file has everything that defines a race: name, bonuses/negatives, starting locations(which include the stellar objects, orbitals, resources, built structures, etc), starting technologies and so on. As I’m sure you are thinking, it gets nested fast! Also I started using the same tags in different files, such as the modifier tags (gives a modifier to something, like +10 Industry) and resource tags (defining what resources are required to build something). The races.xml and technologies.xml both use the modifier tags. I wanted an easy way to be able to reuse the parsing code but make it obvious what’s going on code-wise. This would help me maintain the game and extend it for the future (which I intend to support for a long time!) easier. We should always strive to have some level of code reuse in our programs.
There are many XML parsers out there, DOM, SAX, and various pulling parsers. I usually use SAX and sometimes I use DOM. I’m personally not a fan of pulling parsers. All of these parsers were not capable of easily parsing many nested documents without writing lots and lots of code. I haven’t been able to find a good/easy way of using SAX to do many nested statements and DOM is just a pain to use sometimes, plus it isn’t as efficient as SAX. So I decided to write my own parser!
The Pew Pew XML Parser
(cheesy banner)
I named this after nukers(casters) in MMORPGS – they blast (pew pew) mobs with spells. You can think of it as being able to blast away the nestings of the XML file with little effort. This is built on top of SAX.
Now, this is a bit technical, so bear with me. Feel free to skip to the Mod Support section if you aren’t interested.
Where in SAX, you simply read a file and then implement the code to read the data in and decide what to do when a tag comes with no state data, the Pew Pew Parser works differently. With Pew, you attach a top level filter to a parser and provide a file name. This filter is responsible for looking at the tags and deciding what to do. Each filter has “tags” and then “group tags”. The “tags” are straight data, like name, description, or ID. The filter will fill this in on the correct in-memory object. So for example, we can have a Race object with Name as a field. The filter would read in the name tag and then insert that data into the Race’s Name field.
Here is an example:
The other part are the group tags. These group tags are for nested tags. You attach a filter to this tag. So when the parser detects a group tag has been read, it’ll pass control to the next filter in a stack format. That filter will be able to handle everything it’s supposed to. For example, I could have a ModifierFilter that handles modifiers for something (in this case a Race). So when the XML gets to a point where the tag appears, the parser will pass control from the RaceFilter to the ModifierFilter.
Then the ModifierFilter will be able to handle everything that is supposed to happen during a Modifier. The beauty of this is…you guessed it…I can reuse this code for any XML file.
Here are the tags for the RaceFilter:
You can see the normal tags (name, description, etc) and then the group tags (such as mod). When a group tag is seen, the parser will automatically change to that filter (so the ModifierFilter will be called). When the ModifierFilter is completed, it’ll return control back to the RaceFilter. The same goes with the improvement tag.
Using this parser I have been able to define sets of nested XML tags – and have successfully hidden it from the modder. I’ll also be able to use these filters to load saved games.
I do have some code to do proper error reporting so if the parser finds something it does not like, it’ll display an error message. This helps figure out what went wrong before you even play the game. As such, a modded game will be more stable.
Even though it took me a few weeks to get this written, I can reuse it constantly in this game, as well in future games I make. Overall, I’m pretty happy with my implementation.
On a side note: This parser cannot be obtained anywhere on the web. I’m thinking about releasing it for free use. I do not know if anyone is interested in using this thing but if there is enough interest, I may release it. Feel free to contact me outside of this article at: dayrinni@dayrinni.com
Mod Support
It is my goal to try my best at supporting the mod community. I have a couple of goals here:
- Ensure mods are stable. I really do not like it when I’m playing a modded game and it is prone to crashing. I will try my best to resolve any issues with mods crashing the game. I’ve tried to write a lot of code that handles error conditions gracefully, as well as having the Pew Pew Parser ensure the data from the XML files is in a more valid format and it has all of the right data.
- I will take mod feature requests and attempt to put them into the game. So maybe a player needs a special function for a weapon type or something. I’ll try to honor this request. Of course, this is no guarantee, and I may get more requests than I can fulfill, but you can be sure I’ll try my best in regards of adding new features and functionality.
I feel this is an important investment of my time as I want the game to be around for many years to come – either in the base game or modded in some form. I really love playing Hearts of Iron 3 mods, but some of them just do not jell well with the engine and they crash a lot. I would love to be able to have my game be stable with mods.
Finally, I have been documenting the XML files with explanations on how things work. I’ve also written documentation on the game’s website. So anyone should be able to easily find out how to mod this game and get to work making their favorite mods faster!
Intended Side Benefit
With everything I’ve done so far for the game in regards to modding, there is one large benefit I am getting: I will be able to design the content using my modding tools. Generally, for me, using a tool (like these XML files) to define the game values is faster than writing the content in the code (ie: defining ship components in tables or classes). So in a way, I save time in the end. This is the same for the documentation – it’ll help me remember what does what if I haven’t looked at it for a while.
I can also tweak game play aspects without having to recompile the code – I can simply tweak values and see how they work. This’ll help with balancing and ensuring good game play quickly.
What is left?
There are a few items left, namely being able to have a separate mod directory and the functionality to turn the mod on/off without editing the main data files. I’ll be working on this later, after the game is mostly completed. Then the other items deal with changing many of the “hard coded” values in the game moddable, such as size of fleet combat nodes, and so on. These are defined in the game’s code at the moment so I’ll need to transfer them to files (I have a class that handles all of this, so it isn’t too much of an effort).
Conclusion
So there you have it – this is how I’ve modded the game. I hope to be able to support many mod types such as Star Wars, Star Trek, Mass Effect, or any of the popular Sci-fi IPs out there.
What am I currently working on?
I decided to add this section to give you all an idea of what I’ve been up too. These articles usually are about topics I completed a long time ago, sometimes up to 6 months. My policy is to mostly write only about what I’ve done, not what I will do. I do not want to create any false impressions about the game.
I’ve been spending time programming the fleet combat system – which includes ship components, ship design, colonization, of course, the combat. I will be doing a large article on this and it’s one I’m personally really excited for! But we’ll just have to wait until that article appears in the coming months.
Thanks for reading and I look forward to your comments!
dayrinni has been a Space Sector contributor since October 2011. This is his first foray into writing articles for any review site. He is an avid gamer in the genres of 4X, Strategy, MMO’s and RPGs. Finally, he has been the implementor of several MUDs and is currently working on a space 4X game that offers large scope and complexity. See all dayrinni’s posts here. In particular, check his “what makes a good game” and “making a space 4X game” series.
Subscribe RSS
24 Comments
Related Articles:
- Making a Space 4X Game: Setting the Stage
- Making a Space 4X Game: An Overview on Graphics Libraries
- Distant Worlds: Universe to Launch in May 2014
- Master of Orion Inspired Beyond Beyaan Goes Open Source
- What Makes A Good Game – User Interface
Loading ...
IMHO, using an XSD for verifying the game XML files is benefical to both the developers and modders, and gives the option to check the resource files and fail gracefully instead of failing in the initialization or loading process, hanging the game.
I do see the benefits of it, but I wanted to make the learning/usage curve as low as possible. From my perspective, I’ve always liked games that were very easy to modify their values – it lets you spend more time modding than trying to figure out what’s going on.
If a bunch of modders want me to implement some sort of XML schema, I would strongly consider it.
Was the title “Modding Support” or “XML Tutorial”? :) But yeah, supporting mods is an excellent way of extending a game’s life.
In Beyond Beyaan, I use XML files to define things as well, but I also use .CS files (C#) for scripts that are used in space combat, galaxy generation, etc. How is this done? C# supports run-time compilation, so I can compile and load any C# files during the game, making it possible for modders to code certain parts of the game, and since it’s compiled, there’s no interpretation of script to slow down the game, it becomes part of the game itself!
The downside to this is that you feel like you’re doing twice the work, once by adding functions in code that calls certain functions in the script, and once by creating the scripts themselves, which usually have same functions defined. But it’s so very worth it! Creating new galaxy shapes is a snap, I just add a new .cs file and the game automatically detects it and compiles it, with error messages popping up if there’s issues with compilation. Creating unique technologies in space combat is feasible as well, and is not limited by the game’s code (i.e. only specifying fire rate, damage, particle, etc. values in XML, but you can actually code the behavior of the item)
Will you support scripting in your game?
Thanks for reading!
The article isn’t intended to show how XML works or what you can and can’t do with it, just some basic background information to get the general jist across. Some games, such as HoI3, like to use their own markup language.
I don’t have any scripting in, I’m not sure I’ll add anything from the get-go. I haven’t done much with it, like integrating lua. I would like to explore this more if I acquire more funding that I expect or when the game is successful.
I’ve seen you mention scripts a lot on your blog. It looks like it does serve you very well.
Ihave no experience with C#, however, compiling files that modders give your program really feels like a big security risk to me. I hope you check those files (if necessary) before compiling them!
Perhaps you could follow up your comment with another one talking about some potential security risks? I’m sure some of the other developers would be interested in hearing about this!
I’ve considered that. While I can’t prevent everything, there is one thing that I can do. People could submit mods for “official” review (me looking at their mod, including all of the .cs files), and I can add it to list of “Approved” mods that people can safely download from my website if it passes review.
Since my game is open source, people could re-distribute a modified Beyond Beyaan that is malicious. So the only way to really trust that they’re using the correct version is if they download from official sources.
This sounds like an interesting idea. Have you given much thought on how much of your time/energy that would take up? I suppose it wouldn’t be a big issue for the majority of mods since they are generally smaller. But for total conversions it may be longer? Though, there could be a chance that you form a relationship with the modders who do TCs just from them asking you for help/questions /etc about the code base.
Can’t say for sure. I work on security components for my day job so I know there is never a magic bullet that solves all issues and the landscape is always changing.
I forgot to mention something. I do allow for equations to be used in some cases. Each one has their own variables and then the modder can come up with their own equation for the situation. I think this is pretty cool and gives a lot of freedom with little work from me.
Another path I’m going is essentially making certain tags have different functionality in the game. So modders (or myself) can toggle them on/off to attain different results. I’m handling that with the espionage system. I think it’ll work well and offer enough variability/flexibility to people without supporting a full-fledged scripting system. As mentioned in the articles, I’ll try my best to extend this system based on requests from modders.
For some odd reason in the email the author was ‘Adam Solo’, reading through right now… :)
One little mistake when I finished my review and hit “publish” :) I noticed it and updated the author field accordingly.
Wooot, code! I love wrinting parsers :)
XMLs remind me of Star Wars: Empire At War modding, the first thing I modded. But I didn’t enjoy the XML that much… :) Also, it used LUA for the AI part iirc.
I don’t know if you encountered similar, but in my JSON parser (that’s Java like yours), before I switched the implementation to stream based it always crashed at the 2^16th character of the file. I hope you won’t have this issue. Btw: you’re using Eclipse, aren’t you? :))
As for my game, I’ll store the information in binary form in the end, however, they’ll be “compiled” from text based files (mostly), so the loading times should be faster, since I won’t have to parse walls of text each time the game starts. I’m not a big fan of XML, so I’m not sure what I’ll use. But, if I have enough free time I could write a “compiler” for any kind of file. :) For the language files, however, I have my own little format that I like, so probably I’ll use it.
I’m happy to hear that you support modders, mods can really extend a game’s life! I hope you’ll have a really flexible AI, or implement a moddable one, however. It’s quite sad to see the AI struggling with an otherwise great mod!
Good luck with the development! :)
It’s funny you mention EaW. My pleasant experience modding that game years ago was a good chunk of the reasoning on going with XML. It was pretty easy to jump right in and get started modding. Plus, some of the more popular mods made HUGE changes. Too bad the game engine didn’t scale well =(
I do use Eclipse – I love it. I grew up on C using pico. So the day I found Eclipse I was astounded. This was years ago, like in 2004 or some such. I pretty much never looked back. I use Java on my day job.
Your path is interesting, so are you the only able to modify the data files, or can you do so outside of your development environment? Or are you going to package your “compilation tool” with the game so others can use it?
The only issue (this was years and years ago) I had with parsing large XML files in Java and SAX was with using a String rather than a StringBuilder when reading data in a file. It seemed after awhile (it could have very well been 2^16 characters) the variable got FUBARed and stopped reading in data. I changed this to a StringBuilder and the problem was solved.
I also find the performance very good on large files, at least for the basic SAX parser. I don’t load very large data files for the game – just many smaller ones. When I say large, I mean upwards up to 40 MB. In past projects (previous Space game) I read in those large files very fast and with little issue. However, I do understand what you mean – there are a lot of redundant tags in XML. So I aim to make sure for very large files (ie: saved games) that the tags are much smaller.
I really do want to include scripted/moddable AI eventually. Just right now with my funding I don’t have the resources. It’s going to be a stretch goal most likely for when I hit Kickstarter in the future.
Thanks for the nice words and comment! It most definitely had lots of great things to say!
Well, since my plans for the game making it quite free (i.e. you can form your ship however you want) I’m not sure what should be moddable. :)
Some candidates are the GUI, the technologies, the races, the AI, and the language files (pirate language YARR!). I don’t know yet how the module-system will look like exactly (the most complicated things, like weapon modules may be premade), so it’s also possible that’ll be a subject to modding too.
What I imagine is the game will check the dates, and if the texts are fresher it’ll convert them and then use them. If not it’ll use the “compiled”, binary files. The format of the binary files will be publicly available, so it’ll be possible to make converters for any kind of file. (In that case the mod would contain the binary files.)
Of course first I’ll check which solution is faster, if there’s little (or no) difference, I won’t introduce crazy things like this (but I do like playing around with binary stuff :)).
On the other hand, the ships will be made with an editor. What I imagine instead of voxels (like ScrumbleShip) is something polygon-based. That means I’ll need a way to store the data effectively, that’s easy (instant) to load, so they’re likely to be binary. Also, this introduces a bunch of craziness, like optimizing the hitbox and the model of the ship on the fly, so it won’t require a supercomputer to play. At least I hope. :)
Thanks for explaining how you’ll handle the files, it makes some more sense.
I have to say for the rest – that sounds all very ambitious but very cool!! How much of it have you completed? It sounds like you’re in the majority of the planning/idea/brainstorming phase?
One cool thing you mention is the ship editor, that definitely sounds really cool. Maybe to solve your problem on the hit box, you can kick off a background thread to perform the operations during normal game play. Odds are that players (at least in TBS 4X games) won’t be using the ship design right away so you have almost an “infinite” amount of time to perform the optimizations on the hit box.
Well, sometimes explaining stuff isn’t my strong point :/
I’m planning to do it exactly as you mentioned btw :)
After you give in the plans to the “planning agency” it’ll start to “research” the plans, and the larger your ship is the more it’ll take to complete. (Small ships, i.e. fighters shouldn’t take time at all.) While this is all the user will see the program will run the calculations needed to convert the plans to something usable. Since I have no idea how long this phase will take the exact details are still a subject to change. :)
While this part is quite hard to implement, it’ll give many freedom to the player, however, I still have no idea where will I get the creative content – effects, artwork, sounds, and music. I’m not a too talented person in this field unfortunately.
And I’m still in the (end) of the planning phase, I have few lines of the code ready, but not too much yet. Still mostly experimenting with things, but I have most of the libraries and similar stuff collected. :)
Also, I have tonns of free time, so it the long time the system will take shouldn’t a problem. :)
Do you have a website or something yet for your game? I’d like to follow it!
I like your idea about freedom for the player and not being too restrictive. I like those qualities in games and try to implement them to some degree (they can be time consuming to do). What I do really try to focus on is presenting the player with choices. I’m going to do an article about this in the near future actually.
If you want to toy around with art, try looking into GIMP and finding some videos on You Tube. GIMP is pretty cool and free, but can be hard to learn to use. But it can do a lot of things with its filters. You may find it useful. I’m no artist either and that is where 90% of my funding is going to go.
Hmm, maybe I’ll follow what you do, and launch a kickstarter for the artistic part. But first I want to have something that can be presented to the “kickstarter masses”. By the way, I played around with Inkscape, but the result was… well, ugly :)
I don’t have a website yet, I’m planning to launch one, however. I can’t afford a paid one, while I couldn’t find a good and free one yet, and then my own connection isn’t the best to launch one through it… So there’ll be one, I’m just not sure how I’ll make it yet, until then I’ll wander around here :)
(And as soon as I’ll have something more presentable than a bunch of flying bricks I’ll probably laucnh a twitter and youtube account dedicated to the game as well.)
I’m in the same boat – I’m waiting until I get the game’s features almost complete and then some art for the races/empires. Then I’ll head to kickstarter and make my case. I want to make sure the backers know that the game is almost complete so they can feel good about making a contribution.
I think it’ll work out well – I don’t plan on asking for a lot of money.
You may want to look into WordPress. This blog is ran on that and mine too, for that matter. It’s easy to use and free!
Good ideas on a You Tube/Twitter angle, it’ll definitely be useful.
One question that I really need to ask – why on earth choose XML?
Compared to the simple “name=value” format (typically seen in .ini files), XML is less readable and far more verbose (in the XML example above, more space was used for the delimiters than the actual data). You have a hierarchal/nested layout in XML, but that can be done with .ini files too (name.subfield1=…, name.subfield2=…).
For a situation like this, XML really does seem like two steps backwards.
C&C Generals used .ini files too, I liked them better than the .xml ones. I believe it shouldn’t be a problem to write a converter from .ini to .xml :)
Btw, does .ini files support arrays? (just curious) As for me I’d prefer something more readable too, maybe .json files.
Well, a few reasons.
1. Parsers are readily out there and I have experience using them. (Yes, I ended up writing my own/extending one but I would of probably had to do that with any file format). I’m not saying there aren’t ones out there for your described file format. In addition, XML and parsers can have different formats such as attributes/etc in them. Maybe I’ll want to add that functionality later on to my files.
2. The general opinion that I’ve encountered is that XML is readily known about and that makes it easier for people to involved with it.
3. White space in the example above can be removed. My personal coding preference is to use a lot of white space and new lines. My eyes and brain work better that way. Because of this, I rarely boost about my LOCs :).
I’ve seen XML and source files that have very little white space/new lines.
Modders can write their own XML however they see fit.
4. While I did not use the functionality, XML parsers have various mechanisms to verify the structure of their contents.
XML has it’s flaws and the duplicate tagging is one of them (which I mentioned in an above comment).
Very rarely you’ll find a magic bullet that is perfect for every situation. I’m not claiming XML is the be all end all to file formats. Sorry if it came off that way.
Thanks for your comment!