Scripting Test

by Ron Gilbert
Jan 20, 2015

This might not seem very impressive, but it represents a major milestone in Thimbleweed Park. If you're not impressed, at least pretend you are. My feelings get hurt pretty easy.

I'm looking at using Squirrel as the scripting language for Thimbleweed Park and one of the things I always liked about SCUMM was how it handled multi-tasking. It was effortless to run scripts in different threads and let objects handle themselves without cumbersome state engines or weird co-routines.

I just want to start a function and have it run until told otherwise. I've seen a few scripting languages that can do this, but Squirrel didn't have the ability "out of the box".

I was chatting with my friend Vincent Hamm (who worked on SCUMMVM in the early days) and he didn't think it would be too hard, so he whipped up a quick proof of concept.

The true test of a game scripting language is being able to do powerful things with very little code.

I spent the few days last week getting his code integrated into my engine and started to implement all the bindings between Squirrel and my code.

I wrote the following Squire code that drives the video above.

function bounceImage(imageName)
{
    local image = Image("GameSheet", imageName);

    local x = random(01280);
    local y = random(0720);

    do
    {
        local steps = random(100.0,150.0);

        local end_x = random(01280);
        local end_y = random(0720);

        local dx = (end_x - x) / steps;
        local dy = (end_y - y) / steps;

        for (local i = 0; i < steps; i++)
        {
            x += dx;
            y += dy;

            image.at(x, y);
            breakhere(1);
        }
    }
    while(1);
}

for (local count = 1; count <= 10; count++)
{
    startthread(bounceImage, "Detective1");
}

for (local count = 1; count <= 10; count++)
{
    startthread(bounceImage, "Detective2");
}

Over the weekend, I started to create the structure of a "room" and figuring out how the scripting will interact with all the objects, characters and verbs. I should have that working by the end of the week and will do another post with some more glorious code. I know everyone lives to see source code.

There is still a long ways to go before it looks like an adventure game, but believe it or not, this is a big step forward. It's pretty much just polish from here on out.

- Ron



Brian - Jan 20, 2015 at 12:27
Nice!   I'm not a game designer but I do a lot of engineering work with scripting languages (Python, Perl, Tcl, etc..).    I'm impressed how concise, yet readable this code example is.  Looks like you are well on your way.  I'm glad you are able to pick the tools you want to use and are taking time to explore the space.

Pietz - Jan 20, 2015 at 12:27
What does breakhere(1)? Seems like a sleep() or wait()...

Ron Gilbert - Jan 20, 2015 at 12:31
Yes, it sleeps/waits for 1 frame. It's what it was called in SCUMM, so that's what I called it here.

Pietz - Jan 20, 2015 at 12:50
Thanks :) And plus, I am impressed about the shortness of the code. Seems like a good and efficient way for the game (although I am not familiar with Squirrel besides the animals and the webmail client ;-)

Brian - Jan 20, 2015 at 12:46
This sort of thing just warms my heart:

I was chatting with my friend Vincent Hamm (who worked on SCUMMVM in the early days) and he didn't think it would be too hard, so he whipped up a quick proof of concept.

Coding help from old friends.  What could be better.

Bobe - Jan 20, 2015 at 12:58
Nothing thrills me more than adventure games and their development.  I just watched 47 seconds of images bouncing around a screen, and it was glorious.

LichiMan - Jan 20, 2015 at 13:16
Thanks for posting this. Hope there's a free adventure game engine with this kind of script language.

But please, tell me that no detectives were harmed in the making of this test. Some of them go through the others like Patrick Swayze in Ghost. That hurts!

Yehuda - Jan 20, 2015 at 13:19
How do you stop a thread? The thread itself doesn't seem to have any clean exit points, is there a check breakhere() whether a thread has been marked for stop?

Ron Gilbert - Jan 20, 2015 at 13:22
If the function just returns, the thread will stop, also you can do this...

myThread = startthread(bounceImage, "Detective1");

...then at some later point, you do a...

stopthread myThread

Rusty - Jan 20, 2015 at 13:22
Knowing that this isn't using coroutines, I'm very curious about how this actually works. Does breakhere() actually reach in and save/swap the Squirrel VM execution state? I'm also looking forward to seeing how load/save will work, later on...

Martin - Mar 13, 2015 at 16:09
I'm also very curious about the manner loading/saving a game will be handled. Can you give us more insight regarding this?

Leandro Pelorosso - Jan 20, 2015 at 13:30
Beautiful, just beautiful!! I started working on my own graphic adventure engine a using LUA a while ago, and at some point got forgotten inside a drawer when I started the University. Ron Gilbert was always an inspiration. Seeing this makes me want to quit my job and continue with the project. Thank you so much for everything!

Thom - Jan 20, 2015 at 13:58
Really cool, I don't think I've seen a scripting language handle threads that neatly. Will you be pushing the changes back upstream to Squirrel?

Mathias - Jan 20, 2015 at 14:06
I want that jerky movement (back) ...

(don't move low-res characters in high-res steps :-)

Ron Gilbert - Jan 20, 2015 at 14:20
This test window was created in 1280x720. The final game will be 320x200 (TBD adjusted slightly to be widescreen) and then scaled up (keeping it's pixelyness) to match window/screen size, so they will move on low-res pixel boundaries.

Brian S - Jan 20, 2015 at 17:22
Music to the ears.

Roland - Jan 20, 2015 at 21:34
Awesome!!

longuist - Jan 20, 2015 at 14:28
I know the question sounds a little bit stupid (and comes too late apparently), but why aren't you using the trusty SCUMM™ ? (You used it outside of LucasArts at Humongous)

Ron Gilbert - Jan 20, 2015 at 14:37
Because SCUMM is a 25 year old engine. There is a lot to be gained from a modern workflow and being able to run more natively on modern platforms. I also don't own the SCUMM engine or have access to any of the tools (SCUMM was more it's tools than it's runtime engine) . It will take me less time to get my own engine up and running that it would to get SCUMM working, and I'll have something new.

longuist - Jan 20, 2015 at 14:53
Thx for elaborating. That makes perfectly sense! breakhere(rand);

Micha - Jan 23, 2015 at 17:05
Sounds logical, even if real Scumm (and furthermore real Amiga(/C64)) would be much more retro and interesting in my opinion. But youre the man ;)
I'm really looking forward to the end result!

Brian Ruff - Jan 20, 2015 at 14:54
Thanks for sharing the code. It's super interesting (I'm a programmer).

Ploe - Jan 20, 2015 at 14:54
That looks really slick! Squirrel looks nice... Like a cross between Lua and C/C++ - in a good way.

I remember reading somewhere about early SCUMM and how there were processes that executed independently of one another and me thinking "BLIMEY THAT SOUNDS A LOT LIKE A KERNEL! SCUMM IS ESSENTIALLY AN OS!" Inspired the design of my own forays!

Brian S - Jan 20, 2015 at 15:32
Is the Image Class something you wrote, or part of some library from elsewhere?  Same question for startthread.  Just curious how the initial test development works.  Seems like there's probably a decent amount of work in both.

Ron Gilbert - Jan 20, 2015 at 15:35
The Image class comes from my engine. startthread and breakhere are commands I added to Squirrel using their API, not by modifying any of the core Squirrel code..

Tobias M. - Jan 20, 2015 at 15:59
If I recall correctly you described how you implemented your own preemptive multitasking system for the SCUMM engine in some online article.
For Thimbleweed did you implement your threads as real process threads (provided by the underlying operating system) or did you do your own thread-like implementation again?

Ron Gilbert - Jan 20, 2015 at 17:42
The script threads are not real threads. Each script is run once per-frame until it hits a breakhere(), then it runs the next script, etc. This is to keep issues of concurrency simple. No two threads will try and access the same data at the same time. Everything is keep orderly. This does mean a thread could take a long to time run, or even lockup the system (not a hard lock, you get a warning), and that's just something to scripter needs to watch out for. The simplification is worth it. Again, we're talking about a scripting language for an adventure game, not OS level code. I always favor simplicity and efficiency.

Matt Lacey - Jan 20, 2015 at 17:46
As soon as I read this I wanted to ask about breakhere(1) but you've already answered what I was wondering. I assumed it mean 'give up control for now'. Does the 1 signify anything in particular at this point like frames to sleep for, or is it part of future proofing?

Ron Gilbert - Jan 20, 2015 at 17:54
It's the number of frames to break.  There is also a breaktime(x), that will break for x seconds (or fraction of). The rendering engine runs as fast as it can, but I limit the scripts frames to 1/10 of a second.  It is possible that a "frame" could last longer then 1/10 of a second, but they won't know it. Again, this is to keep things simple and it works fine for what we're making. It wouldn't for a game that was more realtime like an FPS or RTS.

Davide - Jan 20, 2015 at 20:27
That's exactly how co-routines (for example in Unity3D, that's quite all single-threaded) work.
startthread -> StartCoroutine
stopthread -> StopCoroutine
breakhere(1) -> yield return null;
breakhere(n) -> for (int i=0;i<n; ++i) yield return null;
breaktime(t) -> yield return WaitForSeconds(t);
function return -> coroutine's end
coroutine deadlock -> game and editor deadlock (lol)

Davide - Jan 20, 2015 at 20:28
Of course, "breakhere" is prettier than yield return :P

Davide - Jan 20, 2015 at 20:29
FIX: breaktime(t) -> yield return new WaitForSeconds(t);

Tobias M. - Jan 21, 2015 at 05:00
Really neat!
Thanks for the explanation!

Flowo1974 - Jan 20, 2015 at 17:42
Impressive, most impressive (really)!

Meuti - Jan 20, 2015 at 18:05
What Flowo said!

Ciantic - Jan 21, 2015 at 03:33
I don't understand why you use relative obscure language like Squirrel, which probably does not exist after this game comes out. I understand it is easy to compile with the game engine sources as it is relatively small language with C++ source codes.

For the sake of it, I propose you to try Go language. It is also compiled, and it has great concurrency with channels etc.

Ciantic - Jan 21, 2015 at 03:48
Embedding maybe the key here and Go is not meant for that.

Lua however has a bigger community and could be a better choice for the future, especially with new requirements like concurrency etc.

Ron Gilbert - Jan 21, 2015 at 11:20
The main reason I don't use Lua is that I really dislike the syntax. Squirrel has been around for a long time and has been used in games like Portal 2. It's actively developed, with the latest build being pushed up just a few days ago.  Since it's c-based, it allows for a very clean and tight syntax. Lua is a very bulky syntax with a lot of required crlf.  But in the end, programming languages are just like religious arguments. Lua is right for some people, just not me.

Zak Phoenix McKracken - Jan 21, 2015 at 05:22
Proud to see that the author of Squirrel is an italian who live in Singapore, and that he started to develop Squirrel starting from his hobbies, in the late '80s, playing videogames.
He never mention them, but we can imagine he also played the same games created by Ron, Gary and company.
So, Squirrel derived from the games that Ron created, and now Ron is creating a game using Squirrel, that has been created by a game created by Ron...
Maybe the cat on the bread slice paradox is simpler!!!

Kim Jørgensen - Jan 21, 2015 at 06:33
Nice to see something running!
The scripting language looks simple, but personally I would have preferred to get rid of some of the noise. E.g. the local keyword for variable declarations, semicolons after statements, and parentheses for method calls. On the other hand the language seems like a very reasonable choice for keeping the development effort down.

Hop - Jan 21, 2015 at 09:02
Great stuff. Have you used a similar technique to the "latent functions" implemented in this article?

http://www.irrelevantconclusion.com/2012/01/latent-functions-for-squirrel/

Ploe - Jan 21, 2015 at 16:44
Just found out the Squirrel file extension is ".nut" - surely that's worth it on its own!

Heinz - Jan 21, 2015 at 18:33
".nut"...Hmm... Be care of the two-headed squirrel ;)

Francesco - Jan 21, 2015 at 17:44
Well...
Just for whom do not understand anything about programming...or almost anything..
Is this the game engine then?.

Is this the game engine?

Zak Phoenix McKracken - Jan 21, 2015 at 18:11
Well, it's a test to see if it's possibile to do some tasks at the same time.
For example, if Ron writes a script where the Detective has to move from the corpse location to the bench, and while this action is completing another script is launched (for example, to move the clown to jump on a tree), the Detective should not stop, but continue to move.
All that stuff is not ever so easy to do, and this test demonstrates that not only it's possible, but also quite easy to manage.

Francesco - Jan 21, 2015 at 18:46
mmm...
is this the answer to my question or just a consideration?

longuist - Jan 21, 2015 at 20:08
Yes its the game engine! (More or less) I can understand that from a non-programmer perspective this looks lame, because there are no backgrounds, animations, interface etc. But it shows something and you can program it in an easy way. This means major components "under the hood" are linked together in a working manner. Its a starting point from which it can progress steadily.
Now whats finally missing here (what has he ever done to you?) is that you pretend that you are very impressed indeed! :)

Zak Phoenix McKracken - Jan 22, 2015 at 04:57
It's a fundamental part of the game engine.

Mattias Cedervall - Jan 21, 2015 at 21:52
Hallelujah! I've never been more impressed with something in my life!

By the way, why is the answer to the seckrit question always so hard?

Hop - Jan 22, 2015 at 04:01
started to implement all the bindings between Squirrel and my code

Are you using raw squirrel for binding, or a bindings library such as Sqrat?

Juan Pablo Bettini (aka Zak McKracken) - Jan 22, 2015 at 07:39
Great work, Ron! :D

One question: I know that Squirrel is a high-level scripting language that should speed up things, but why didn't you code your own specific one like SCUMM? I know it sounds like reinventing the wheel, but I remember you saying how fast and easy was adding features and behaviors to the game with SCUMM. How will you achieve the same goal with Squirrel?

Ron Gilbert - Jan 22, 2015 at 11:04
I looked at doing that, but squirrel did just about everything I would have done and so far it's been great and saved the project a good month of my time.

Mnemonic - Jan 23, 2015 at 02:32
What about serialization of running scripts? Is that doable with vanilla Squirrel? I found that it's often an area neglected by existing scripting solutions, simply because usually it's not needed. But for some kind of games it's necessary to be able to "save anywhere".

mr. T - Jan 22, 2015 at 16:57
I'd just like to express my gratitude for being able to follow these posts and the development of the game. It's such a treat as a developer and a fan. Kinda getting an itch to start dabbling with my own adventure game :D

enthusi - Jan 23, 2015 at 07:05
Hehe, exactly NOW I am working on an scripting engine natively for the C64, running quite similar tests.
I.e. I divide the fractional dx, dy by two (so its just shifting) until their sum is <= 1 and as I shifted a counter in parallel I always know how many steps to take. As far as I know Maniac Mansion on C64 didnt use fractional dx,dy but went for the direction with the largest delta.
Running scripts on a per-frame basis in dedicated slots back then was amazing and I happily copied that approach : )
My scripts run themselves repeatedly with on data or are capable to replace themselves, i.e. a walking-to script after an find-proper-path-to-target script. Yes, I am just bragging but it soo much fun :)
Will you implement something similar to the walking-boxes and pre-calced Dijkstra or rather some real-time pathfinding?

Averell - Jan 23, 2015 at 15:27
I like that style how your tom is walking around - it's a special charme games look like in the 80's.
You shouldn't change anything because it's good as it was and quite funny to look at.

Badde - Jan 23, 2015 at 16:05
where is it Garry Winnick? Where is it?

Pedram - Jan 24, 2015 at 09:44
Hello Ron,

I have a question: I know it is just a tech demo, but if this were real, you'd have to allocate images in the main thread and pass a reference to the animation thread, otherwise an image, which is thread local would be deleted once the thread is dead. Also you will have to care of locking (mutex), if you use resources across threads. Do you guys need to extend squirrel to be able to understand these things?


Regards,

Pedram

Ron Gilbert - Jan 24, 2015 at 11:43
These aren't real CPU threads, so it doesn't have any of those mutex issues, which is the reason I do it like this. All objects are reference counted, so if an Image is shared by two "threads" and one deletes the image, if the other is still using it, it don't be deleted,

ac - Jan 25, 2015 at 05:14
In C# I would use Tasks. The cancellation support is really nice and you can choose from various schedulers and ways to syncronize thread context etc. Too bad I did not bother learning about that a year ago when I wrote some Tasks code that really should have used the cancellation support instead - ended up with a runtime bug.

The code in C# looks about identical but is bit shorter and of course less typing with auto-completion.

I would probably implement the game to run completely in browser from server side. Then draft up some legal doc that if someone buys up the company or it goes bust, the buyers get the server code ... if archive.org is running old games in dosbox that was compiled to javascript, surely there's a way to run C# cross platform in browsers too. I just haven't heard of it yet.

Why is this important? Well if I were to buy a game, I don't like it if someone can have it for free (I'd rather have free high quality demos than the paid alpha tests of these days, like back in the 90s). If someone can have it for free, then that may effect the resale value of the physical box, which is bad from investment perspective. To furher increase the value of the boxed version, the boxed edition buyers should get something special in the browser version through use of uncrackable license code (thanks to the game running in unhackable server, thanks to not using C++). Perhaps a unique room full of boxed games? LOL

longuist - Jan 25, 2015 at 05:59
What a disturbing read

ac - Jan 25, 2015 at 09:47
This reminds of sites where one can vote on what other people say and some vote negatively without explaining their rationale. It's completely understandable that it would be pointless if everyone had to restate the same rationale. But there is no reason to implement the system that way. If even one of them explained the rationale and then people voted on the rationale they agreed on instead of simply piling on original comment without any requirement to express what they intend with their act, a lot of irrational arguments and hurt feelings could be avoided.

ac - Jan 25, 2015 at 10:19
After some introspection I found atleast one reason to not implement the proposed system :

If I voted a comment or idea down by agreeing on some critique posted by someone else, then the original poster could post a rebuttal to the critique and invalidate my vote. That would be mighty annoying as then as a perfectionist I'd have desire to answer to that rebuttal. In urban dictionary this is atleast under "tireless rebutter", one who wants to have the last word.

Possible solution to this is that all negative votes (eg. agreed on a critique) are final on version basis - after revision only few votes against the critiques would put those into invalidated state. This should reduce the endless arguments and rebuttals issue to minimum, especially with multiple votes on castable on different critiques, allowing the revision to address all the issues quickly.

O_o - Jan 25, 2015 at 16:54
Probably I would spend more time on the development of the game, to enjoy myself and the others,
than producing a bullet-proof perfect-utopistic comment system :/

Roo - Jan 26, 2015 at 01:32
Will there be any chance of a ps4 release?

BartC - Jan 30, 2015 at 06:20
So the startthread function fires up another instance of the squirrel VM and suspends this VM when breakhere is called ?
Or is there a more elegant way of doing this .

buy grush - Jan 31, 2015 at 04:35
Will you open source the TP engine when you are done? Or/and let everyone write plugins to the game itself? Or license the engine. Or make a TP store, where devs sell their TP engine games and you get a cut?

Rui Figueira - Feb 19, 2015 at 12:43
I added that functionality to Squirrel a couple of years ago for something I was working on. It makes things so much easier. Game logic because much linear and easy to follow.

I guess one of the reasons Squirrel is not used more often is because of the lack of tools.
I'm at the moment refactoring the IDE I've made for another project, to be a dedicated Squirrel IDE, with nice syntax colouring, debugger and all.
I should have something usable in the next few days, as I'm just bringing code from the other project and adding what I need for Squirrel.

Feel free to provide to suggest features. :)
Repository at: https://bitbucket.org/ruifig/nutcracker

Rui Figueira - Feb 19, 2015 at 12:45
Damn grammar mistakes. Can't edit posts?
* Game logic becomes much more linear and easy to follow.

Thiezn - Jul 03, 2015 at 04:42
Not sure if commenting on an older post has any use but was curious to know if you considered using python as the scripting language as well? I came across this older article from Humongous entertainment (http://www.gamasutra.com/view/feature/131372/gdc_2002_game_scripting_in_python.php) where the advantages and catches were described. Perhaps squirrel is a bit more lightweight and more easily to integrate than python? Would be interested to hear any pitfalls you might encountered using python back at Humongous

Ron Gilbert - Jul 03, 2015 at 10:25
They switched to python after I left. Python is too heavy. I also wanted C-like syntax, I really don't like Python's syntax very much, Also Squirrel is a very simple compiler and run-time and it's easy for me to go in and make any changes I need. I don't think many games use Python for a scripting language, it was all rage a while back.

thiezn - Jul 08, 2015 at 15:36
Thanks Ron, this blog is turning out to be the best resource on (adventure) game design ever! Hope you sell enough copies of Thimbleweed to be able to keep your webhosting payments up for years to come :)

Timothy Wright - Jan 19, 2016 at 11:02
I'm currently streaming code development where I take your code examples from the blog and try to get them running with C++ code.  I was curious if you're using the squirrel coroutines behind the scenes with your breakhere(1) thread stuff or just saving the stack for each thread yourself, or something else?