Engine

by Ron Gilbert
Jan 05, 2015

After I posted this question on Grumpy Gamer, I started to take a close look at engines that were suitable for a 2D point & click game. Lots of great choices, but deep down (if I'm honest with myself), I'm a game engine snob. Whenever I use an engine, I keep wanting to modify it and hack into it to do what I need. I guess I've found over the years that it's a lot easier (and less stressful) for me if I just create my own.

I have a fairly robust 2D graphics engine written in C/C++ that I used for The Big Big Castle, Scurvy Scallywags and 30+ game-a-week prototypes over the years. It's stable and I'm used to it. It runs on iOS and (mostly) on Android, and I ported it to Windows and the Mac about six months ago.

I use SDL for window creation and handling input, but not for rendering. I just get the surface back and then use my own rendering code. It has an abstraction layer for OpenGL, DirectX, etc, but the DirectX layer is pretty simple and kludgy. I'm not sure I have the desire to fully learn DirectX, so I might get someone to help with that.

I think my engine has everything I need...

Except a scripting language.

I looked at Lua which it's pretty easy to integrate and is highly optimized, but I really hate the syntax. There is just too much that is goofy about Lua. I like my { braces }.

I looked at writing my own scripting language. It's not as complex as it seems, and I figured it would take me about a month, but I'm not sure I want to eat up that much time. 18 months seems like a long time, but it's going to go away fast, it always does.

The other option was Squirrel. I've followed it for several years, but never really had the need to integrate it. I like the syntax and it's pretty simple implement. I'll probably go in and modify the syntax a little to mimic how SCUMM laid out code for rooms and objects.

So, mixed with design and budgeting, I think this week will be spent getting Squirrel implemented and building a quick test game to see how well it works.

I'll let you know how it goes.

- Ron



Jesper B. Hansen - Jan 05, 2015 at 14:51
I am really looking forward to these updates - thanks Ron.

I don't know if you will explain it, perhaps someone else will, but what is 'the engine' doing and what is a 'scripting language'?

Sorry for knowing nothing but would love to start learning now ;)

- Jesper

gaston blanco - Jan 05, 2015 at 14:52
probably not a question for this blog. but why would you need a scripting language? why not code everything un c#?

im not a game dev.

Bruno - Jan 05, 2015 at 15:50
It can be for a variety of reasons. Scripting languages are simpler and thus it could allow a non programmer game designer to tweak and test gameplay elements. Also scripting languages are interpreted at run time rather than compiled which can speed up development as you test and tweak gameplay aspects allowing to test without having to compile the whole game (which can take several minutes, depending on the game of course).

Thomas - Jan 05, 2015 at 19:21
You use a scripting language to make your code modular, reusable and managable. That´s what functions and methods were originally intended to do with less abstraction. The great turning point is when you refactor your logic to be represented by plain data. I.e. your code is so abstract that it can handle anything you like. You don´t do it for one game. You do it once and reuse and improve it every time.
M-Theory is physicists code abstraction layer for data driven universe construction. I like it so I hope it will prove right :)
Or simply: scripting languages kick ass once you have them. :D

Joe O'Connell - Jan 05, 2015 at 14:57
Hi Ron,

I'm loving this project and your level of transparency with the blog. Have you entertained the idea of bringing any developers on board (pro bono / like an open source project)?

I'm pretty sure I owe my career in Computer Science in part to being inspired by the awesome games you built that I grew up on; it would be an honor to contribute to this project in some way. I'm sure I'm not alone.

Joe

Norman - Jan 05, 2015 at 15:22
Or you could mostly use C / C++ as a scripting language, e.g.:

http://stackoverflow.com/questions/1513920/scripting-language-for-c-c

RT - Jan 05, 2015 at 15:37

pankratz - Jan 05, 2015 at 16:35
That was my thought too. I am sure there is a simple reason. Licenses?

Zak Phoenix McKracken - Jan 06, 2015 at 05:49
Unfortunately, they (WD) took everything. Every character, every logo, every every. Probably (Ron or Gary can confirm it) they also took the SC. U. for M. M.

salty-horse - Jan 05, 2015 at 15:47
Jesper,

An "engine" is usually the core of the game's program.
It handles drawing things on the screen, playing sounds, handling input from the mouse and keyboard. Since this is an adventure game engine, it also models things you are familiar with in adventure games: It allows characters to walk inside and between rooms/"screens", pick up items, and use them on other items, converse with each other using dialogue trees.

An engine can usually be used for a variety of games. If it's an adventure game engine, it will usually be used for several adventure games, such as the SCUMM engine that was used in (almost) all of LucasArts's adventure games.

The actual parts that are specific to a game, such as WHICH characters are in the game, WHICH items interact with WHICH other items, how rooms are connected - all that is usually handled outside the "engine", usually written in a custom programming language (usually called a "scripting language") that understands the adventure game concepts of "characters", "items", and "rooms".

This separation is useful for several reasons, mostly because it allows you to have different people from different disciplines working on different parts of the game. The designers who decide how to connect all the different parts of the game, and write the game's text script, don't have to know how to program the engine, which usually requires a lot of technical programming knowledge - they just have to learn the scripting language for the game. Similarly, it also allows the artists to insert art assets easily.

As mentioned in another comment, the two parts don't necessarily have to be separate - especially since the Thimbleweed Park team is so small - but that's how it will probably be in this case :)

Jesper B. Hansen - Jan 06, 2015 at 03:46
Thank you, Salty.

Fantastic explanation :D

Hop - Jan 14, 2015 at 07:47
I'd love to hear some details on how Ron splits the implementation of the point-and-click gameplay features between the C++ engine and the Squirrel scripts. If he already has a 2D engine, will he specialise by add  gameplay features, such as rooms, objects,  inventory, dialogue trees, and provide an API to the scripts to access these scripts.  Or will all of the gameplay elements be implemented purely in the scripts, which just calls the 2D engine to draw and play sounds etc, and leaves it suitable for use with any type of game in the future.

Marco Lizza - Jan 05, 2015 at 15:56
Hello Ron, being a longtime fan of yours I'm glad to hear we are both preferring Squirrel to Lua for our engines. :)

( and we both like to reinvent the wheel the way we like ;) )

Recently I'm considering using SFML instead of SDL for the graphics (on mobile platforms I'm still using a gfx system of mine). Just experimenting in these weeks and I'm liking very much.

Looking forward for your updates and seeing how Squirrel will be adapted!

Gabriel - Jan 05, 2015 at 16:08
If you need help with SDL on Android (or in general), let me know. I happen to be the maintainer for that platform (as well as a backer...and a fan of yours!)

Tomimt - Jan 05, 2015 at 16:12
These are the kind of updates and journals I was hoping to see from several projects I've backed, as many of them did promises of letting us see how the sausage is made so to speak. But more often than not, those updates haven't been very stellar, but with this one you are on a good start.

Rum Rogers - Jan 05, 2015 at 16:53
Being developing a web engine for graphic adventures myself, I can't wait to see how Ron Gilbert makes one.
Eagerly waiting for details, Ron!
P.S. thanks for the Puzzle Dependency Graphs/Charts post @ grumpygamer!

Christopher Stevenson - Jan 05, 2015 at 17:35
Hmmm… interesting.

For anyone interested in SCUMM and in understanding what a scripting language looks like (and is used for), I found this post of Ron explaining a bit about how SCUMM worked/works :

http://www.pagetable.com/?p=614

Sly - Jan 05, 2015 at 18:11
I recently worked on projects that had already implemented Squirrel as the scripting language.  Hated it.  Can't debug.  Syntax errors only reported at runtime.  Semi-colons are optional.  Typeless variables.  We knew when a member of our team was debugging and execution dropped into a Squirrel script because there was a very audible groan come from that part of the office.

The only scripting solution that I have used that I felt comfortable with was the one that I wrote myself.

Ron Gilbert - Jan 05, 2015 at 19:00
That's all true, but Lua and just about any other scripting language will have the same issues. My philosophy is to keep the engine world and the game world very separate. There are few cases where a engine programmer needs to step into the script code. If they are, then something else is wrong.  And script programmers should never be stepping into the engine code. They shouldn't even have access to the engine code.  If they need to step into the engine, then something else is really wrong. But, that said, I have no illusions to how frustrating it's going to be while all the tools come up.

Derrick Reisdorf - Jan 06, 2015 at 00:33
Are we going to get to see some of the code...Some of the classes and members and methods in action?  I'd probably be a little more interested in the game scripting, rather than the engine.

Jonathan Reisdorf - Jan 06, 2015 at 02:43
+1 for seeing some code once it's there
and squirrel does not look too bad imho (no practical experience with it though)

Derrick Reisdorf - Jan 07, 2015 at 21:10
Like Ron said, it looks very much like C++ or  Java...

Iron Curtain - Jan 06, 2015 at 07:19
Of course, if Ron were to do that, the game would be open-source (at least partially).

dazza - Jan 06, 2015 at 18:31
Can confirm, currently using Lua on Smith and Winston and the lua debugging is painful because I haven’t written a debugger and I can’t get any of the available debuggers to work. prints work mostly though. It’s still better than tracking down painful AI C++ leaks and invalid pointers.

existen - Sep 11, 2016 at 12:05
If you develop game for windows, you can use Decoda (http://unknownworlds.com/decoda/) or BabeLua for VisualStudio (https://babelua.codeplex.com/). These IDEs have convenient step-by-step debugger. They work pretty good in 97% of time.
I develop multiplatform c++ game (Windows, OS X, iOS, Android). And, I usually write and debug lua-scripts on Windows. Then, if I need to debug scripts on another OS (but this is rare situation), print-debug - is all I have)

Sly - Jan 06, 2015 at 18:54
Engine and game code should indeed be kept separate.  This particular project was all kinds of messed up.  The engine made heavy use of almost every C++ feature possible.  The game code was mostly C++ (similar to the engine, but written by less skilled coders who made a mess of it), but a lot of the essential pieces were written in Squirrel, which would be called from the C++ game code and would then make calls back into the C++ game code.  To make it worse, most of the UI was written in Flash which used a lot of ActionScript.  So at times we had a callstack that included C++, Squirrel (a black box to the debugger) and ActionScript (yet another black box).

diego - Jan 05, 2015 at 21:33
you could make a dreamcast version like pier solar hd xD

Indrid Cold - Jan 06, 2015 at 02:39
I'm sure you've thought of this, but what concerns me a little is that making your own engine will make porting it to 3 OSes a lot more difficult than using a ready-made multi-platform one, won't it?

Joost - Jan 06, 2015 at 03:02
You missed the "it's stable and I'm used to it" bit :)

Tomimt - Jan 06, 2015 at 03:50
Though Ron did already meantion the engine he uses already is a multiplatform engine from Windows to iOS and Android. Besides, judging from other projects I've backed, using a propreriety engine with multiplatform support isn't a key to happines either. Many games have had unforseen issues, as they might have used some other components, which aren't stable or aren't supported by the other platforms.

Linus - Jan 07, 2015 at 02:30
An engine that never ran on Linux/X11 ... That will probably end well. Not.

Joost - Jan 06, 2015 at 03:02
"squirrel's syntax is similar to C/C++/Java etc... but the language has a very dynamic nature like Python/Lua etc... "
ah. That's a good choice :)

PMIK - Jan 06, 2015 at 03:17
Please tell us you are going to let us try the test game.

Even if it is a stick figure and the puzzle is to walk across a drain without falling through by placing a plank over it, I would be pleased

(Go ahead,  I'll let you use my idea for free :) )

Seb - Jan 06, 2015 at 04:39
Ahoy there,

it would appear that the RSS-Feed is not updating correctly - yesterday's article (Jan 05) is not available on http://blog.thimbleweedpark.com/rss (the two previous entries are).

Could you please have that fixed?

Cheerio!

Seb

Zak Phoenix McKracken - Jan 06, 2015 at 05:59
I'm a programmer, and I agree with the fact to separate the "logic" from the "engine". Is a good practice.
It can really help to maintain/debug one part despite the other part.
Once I programmed a point-n-click adventure, but I used two different engines: AGS and Visionaire. they both integrated script & engine. The result was a good game, but quite difficult to debug and to change something after the game was already done.

So, even if I don't know Squirrel, it seems it appears the most suitable tool, or at least a starting-point. Maybe you could adapt to your needs, changing something in a easier way than using other tools, and in less time than writing your own.

I think you could really spend one week to stress-test Squirrel, and then if it doesn't satisfy you, perhaps writing your own scripting language would be the best way.
You did it once, you can do it twice!

Markus - Jan 06, 2015 at 09:00
That sounds interesting, i always wondered how the code to such games would look like. Do you think that you could share some code with us in the future?

k0SH - Jan 06, 2015 at 11:59
Maybe we can play this little test game later?
(if it´s playable..)

Oskar - Jan 06, 2015 at 14:03
You mentioned you want to keep the engine world and the game world very separate and you are already using C/C++. Have you considered moving out the game part into a loadable DLL and just reload on changes? Would this be significantly more work compared to integrating a scripting language in your engine?

Marco Lizza - Jan 06, 2015 at 18:20
In my opinion, this kind of modularization won't provide the single major benefit in binding a scripting engine with a game engine: expressive power.

I mean, we should strive and reduce the complexity of the scripted code, not realize a one-to-one binding with low level APIs resulting in overcomplex scripts. The balance between "how much" the game engine and the scripted part do is critical. The game engine should do enough to have elegant and compact scripts... not being only a game-loop stub that demands everything to the scripts.

(not to mention DLLs are compiled binaries, which means that designers need to have a suitable compiler/IDE and the skill to use it)

(also, DLLs are available only on Windows platform, altough other OSes have the same tech with different names)

Oskar - Jan 07, 2015 at 13:31
I don't disagree about the benefit of expressive power, but there is also something to be said about using already established tool chains for debugging, profiling, static analysis, etc. Look at Unreal Engine 4 for example. They ditched UnrealScript in favor of run-time compiled C++.

> (not to mention DLLs are compiled binaries, which means that designers need to have a suitable compiler/IDE and the skill to use it)

Compiler, maybe. IDE, no. Using command-line "cl" in Windows and gcc/clang in Linux/OSX it's possible to automate this so it seems interpreted. Setup a remote build system  and you don't even need the compiler installed.

> (also, DLLs are available only on Windows platform, altough other OSes have the same tech with different names)

True. It's just easier to type than shared library and the name is more well known so didn't think that was worth clarifying.

Oskar - Jan 07, 2015 at 13:46
> The game engine should do enough to have elegant and compact scripts... not being only a game-loop stub that demands everything to the scripts.

If this is what you meant by "expressive power" I don't see why a scripting language is necessary for this. Just use the C/C++ interface you would export for your scripting language and you have the same thing?

Marco Lizza - Jan 08, 2015 at 05:15
> True. It's just easier to type than shared library and the name is more well known so didn't think that was worth clarifying.

I didn't want to sound pedant, sorry. :) I just wanted to point out that a DLL-based approach would require to use different binary resources (built differently, most notably to handle entry-points, initialization and general resource usage if needed) on different platform. That is, the porting is (a little bit) more complex. On the contrary, in a script based approach obviously the very same script is used every platform without changes.

In my opinion, if the project is aimed to be multi-platform, adding a layer of complexity (porting-wise speaking) is masochistic.

> [...] there is also something to be said about using already established tool chains for debugging, profiling, static analysis, etc. Look at Unreal Engine 4 for example. They ditched UnrealScript in favor of run-time compiled C++.

I guess real-time requirements are quite different in the Unreal Engine nowadays. ;)

> Compiler, maybe. IDE, no. Using command-line "cl" in Windows and gcc/clang in Linux/OSX it's possible to automate this so it seems interpreted. Setup a remote build system  and you don't even need the compiler installed.

Automation is certainly possible outside an IDE, but I guess the designer will need a IDE to write down the C++ code and possibly debug it (an editor could not be enough). Moreover, knowing C++ could be a major problem to some.

The game engine should do enough to have elegant and compact scripts... not being only a game-loop stub that demands everything to the scripts.

> If this is what you meant by "expressive power" I don't see why a scripting language is necessary for this. Just use the C/C++ interface you would export for your scripting language and you have the same thing?

Sorry, I didn't explain myself well. "More expressive power" of course means "less line of codes". That is, the game-engine binding layer should enable the scripts to do *more* with *few* lines of code (and C++ do not shine on this).

I was referring to some script-base general purpose game engines where you end up having tons of scripts that makes the application from the ground up. The engine is just a sandbox VM with primitives.

This do make sense in some occasions (I built engines like this on purpose), but not everytime.

Ploe - Jan 06, 2015 at 18:28
Are you using vanilla SDL (surfaces on surfaces) or are you a little lower level in the OpenGL context?

Lua is nice and light, if a little peculiar (the meta-table stuff is a little cumbersome and left me scratching my head from time to time) and it has some sexy syntactic sugar so you can write stuff like:

'Actor.create {
--attributes
}'

So you're missing out the parentheses. I do miss certain operators like '+=' and in Lua though.

Why don't you use your former colleague Douglas Crockford's pet serialisation format, JSON?

If you're writing your own tool will it ever see public release or will you be keeping it under lock and key?

Ron Gilbert - Jan 06, 2015 at 18:31
I get an OpenGL context from SDL. I don't use any of their rendering. I use JSON for most of my data file formats.

Pierre-Yves Gérardy - Jan 07, 2015 at 08:20
If your main gripe with Lua is the lack of braces, you could also modify its parser to bend the syntax to your liking.

The Lua source is very clean and easy to modify.

http://www.lua.org/source/5.1/llex.h.html
http://www.lua.org/source/5.1/llex.c.html
http://www.lua.org/source/5.1/lparser.h.html
http://www.lua.org/source/5.1/lparser.c.html

Advantage over Squirrel: more tooling and libraries (e.g. http://lua-users.org/wiki/DebuggingLuaCode). ZeroBrane supports live coding (hot code swapping) for several game engines, for example.

You could still run third party vanilla Lua code by compiling it to bytecode first. C-based libraries will also work out of the box (http://www.kyne.com.au/~mark/software/lua-cjson.php).

IDE/debuggers like Decoda or ZeroBraneStudio would require some tweaks to handle the modified syntax, but they're open source, and all in all it should take half a day to implement.

Marco - Jan 07, 2015 at 08:31
Ron,
I don't want to tell you anything but this "DON'T VASTE A MONTH!".

LUA is great but if you don't like it choose whatever else but don't write your own scripting language.

Ciao.

Marco Lizza - Jan 07, 2015 at 10:06
Spending time and not finding a suitable scripting language is far worse than spending a month in devising your personal one. Mostly because, at last, the resulting language will surely fit your needs.

That being said, I also hope that Ron will find Squirrel to be language of choice.

Max - Jan 07, 2015 at 08:46
You might want to consider TinyScheme. It has a really small core which can easily be extended (only a few thousand lines), and LISP seems like a good fit for that sort of thing. No curly braces of course...

Cerator - Jan 07, 2015 at 11:23
You could just add what you need to Squirrel and call your variant Two-Headed-Squirrel! ;-)

Woof - Jan 07, 2015 at 12:53
Instead of DirectX I recommend looking at the ANGLE project:

https://code.google.com/p/angleproject/

I've been using it for a while, writing only OpenGL ES 2.0 for Windows, with the advantage that the same code can then be used for tablets directly, or with usually a few minor tweaks OS X and Linux (and more recently with both Native Client and Emscripten to run the same in the browser).

Ron Gilbert - Jan 07, 2015 at 12:56
Funny you should mention ANGLE. That is what I'm using right now. It works, and will probably be fine for a while, but it feels like for a shipped game (especially if we start looking at an XBox port), we should have a real DirectX layer.

suntorytimes - Jan 07, 2015 at 14:56
I'm gonna go on a rant and state that this single post is more interesting than all that meaningless rubbish I've been fed by the DFA "we're gonna shove the sausage in your face" fiasco.
Thanks Ron.

Woof - Jan 07, 2015 at 14:57
I guess for Xbox then DirectX is probably the way to go (although I'd seen ANGLE had support for more than Windows, although this may only be WinRT and Windows Phone right now). Besides Google themselves, IdeaWorks use it with their Marmalade Windows versions (used in quite a few commercial titles). I use it at work on quite a large project but the Windows version is only used internally.

Manoel - Jan 07, 2015 at 16:36
Any chance to make the adventure engine public? Not necessarily Open source, but usable by others, or allowing to be ported to other platform like retro one?

bigjocker - Jan 08, 2015 at 12:20
Why not use Javascript? I know there's now a lot of people trying to sell Javascript as the ultimate language for everything, and that may be contributing to undermine the language in areas where it _is_ useful. There are very optimized engines for almost any platform and language, and in some you can even attach a debugger.

And if you are already using JSON ... I would guess you've already considered Javascript and discarded it?

Thomas Arnold - Jan 14, 2015 at 04:52
Hey,
for scripting you might want to look at:
http://munificent.github.io/wren/index.html

Hope that helps.

Cheers

ac - Jan 23, 2015 at 23:32
Looks good though if it was my project I'd have run an automatic translator to port the c++ code to C# and then use C# for the scripting language (using the dynamic keyword where needed and partial classes with static methods and lambdas it's possible to stay functional and dynamic with global scope* across files , while sprinkling OO where desirable). On the plus side, Squirrel does look very much like C#. The real reason to use C# everywhere is that you get integration of best debugging tools, profiling tools and so forth at no cost in the community edition. "non-enterprise organizations, up to 5 users can use Visual Studio Community" .. It's practically the same as VS pro.

* though this global scope isn't really that attractive now with VS 2015 CTP 5 as you get a new feature that allows this:

global.cs: put global stuff here in classes (eg. class Global { })
others.cd: import classes - now instead of writing Global.StaticMethod() , you can write StaticMethod()  - except since this is C# you get intellisense and auto completion and such - I wouldn't touch a scripting language without these and great debugger)

ac - Jan 23, 2015 at 23:39
TYPO FIX

others.CS: import classes - now instead of writing Global.StaticMethod() , you can write StaticMethod()  - except since this is C# you get intellisense and auto completion and such AND NOW YOU DON'T NEED TO WRITE ANYTHING (just type st[space]) - I wouldn't touch a scripting language without these and great debugger). In c++ either you type more or you got very polluted intellisense. Yeah there's F# and such you could also use but while the language may be succint, the rest of the experience is nowhere near C#. And besides, succint languages are not needed when C# auto-completion takes care of the writing for about 90% of the code.

Zuckerberg - Jul 02, 2015 at 15:08
What blog engine are you using here? Super clean and simple, at least the UI. Good choice, and feels very EGA and 8-bit ;).

Zuckerberg - Jul 03, 2015 at 07:23
Arg! Answered it myself: Ron mentions on GrumpyGamer.com that he hacked up his own blogging engine.