Vassal 4:Design Blog 5

3-Sep-17 Vassal 4 Scripting Language - Lua prototype test

As a project to improve my C++, I undertook a detailed test implementation of Lua as a Vassal 4 scripting language. I apologise that that this discussion may lapse into technical detail and assumes a basic understanding of C++ and Lua.

This work was based in part on Joel’s preliminary document outlining a Game Representation Language for Vassal 4.

Note that my prototype is a test of the interface mechanism only, There is no actual implementation of Vassal 4 functionality.

These notes record how I found Lua stacked up against the requirements that Vassal 4 needs from a scripting language.

Overall Result

Overall, I was extremely happy with the ease with which I was able to integrate Lua with my simulated Vassal objects.

Lua is an extremely flexible language that made it easy to implement some of the facilities I needed within Lua itself, as well as being plenty powerful enough for even complex scripting needs.

Lua also provides a full C API that allows all of the language elements to be driven from a C or C++ program with the improved efficiency of not having to be interpreted. This allows sections of Lua code to be re-coded in C++ if they turn out to be an efficiency bottleneck (For example, the setup code that is required at the start of each script invocation).

Being so simple, Lua is extremely easy to pickup if you have any programming experience at all. There is a very good reference manual, a more detailed programming guide (older version available free online) and a very active and helpful community on Stack Overflow.

The two main complaints against Lua seem to be about variables defaulting to global and arrays commencing from 1 instead of 0.

The Global variable issue is moot, since for Vassal, we will be disabling the use of Lua global variables and generating a run-time error if they are attempted to be used. I am also hoping to be able to detect the use of global variables at ‘compile’ time.

The issue of arrays starting at 1 instead of 0, I found to be inconsequential. Lua has only one data structure, the table, which is essentially an associative array. You can use any indexes you like for table entries, including 0.

If you use sequential numeric indices starting at 1, then Lua will store your data as an array, certain Lua library routines will expect the indices to start at 1.

However, you can use any indices you like. You can have an array that is indexed using the integers -5 to 5. You can start arrays at 0 if you like and your code will work, except that the Lua Length operator and some Lua library routines will not see the correct length.

Because Lua tables are associative arrays, and Lua provides in-built mechanisms to iterate over a table’s contents, there is no great need to use specifically indexed arrays.

While Lua is not a hugely well-know language, it is actually widely used as a scripting language in games and is probably the most likely language that a non-programmer Vassal module developer will know!

The Verdict

With the danger of sounding like a Lua fan-boy, I find it hard to see how we can go past Lua as the scripting language.

General

Prototype written and tested using C Development Tools (CDT) of the Eclipse IDE on windows 7 using the MingW tool chain with no optimization and full debug information generated. The C++ is compiled using C++14 compatibility.

Later testing showed recompiling with full optimization and no debug information provided a 40% performance improvement on a small CPU intensive test script.

Lua was built using the latest 5.3.4 source from lua.org.

Overview

I first created some dummy C++ objects to represent Vassal 4 Game Objects – A Map object, a Piece object and a Collection object.

I created a set of interface classes that sit between Lua and C++ and allow Lua scripts to act on Vassal Game Objects. For example, all access to the Vassal Piece class is mediated by the interface ProxyPiece class. Lua scripts only have access to ProxyPiece and the services it provides.

This design cleanly separates the scripting language from the Vassal Game Objects and allows the Vassal core to be developed without worrying (too much) about the details of the interaction with the scripting language. The Proxy classes handle any necessary translation between the scripting language and Vassal.

Embeddable in and Extensible by C++

Lua provides a straight-forward stack-based interface between C++ and Lua that allows C++ to call Lua scripts and for Lua to call out to C or C++. The actual interface between C++ and Lua is encapsulated into a single class.

Here is an example script that might be attached to a Vassal Piece object.

In general, scripts will be attached to and run against a Vassal Game Object. I create a Lua object named ‘this’ that can be used to reference the Vassal object that the script is being run against.

Here is an example script that will flip all pieces that are sitting above it and rotate them 90 degrees if they have not already been rotated.

   for piece in this:getPiecesAbove() do
     piece:flip()
     if piece.rotate == 0 then
       piece.rotate = piece.rotate + 90
     end
   end

Lightweight, Efficient, Fast

Lua was designed specifically as a lightweight embeddable scripting engine. It is a simple, logical and consistent language providing the basic facilities a scripting language needs. The bytecode format and Virtual Machine have also been carefully crafted with speed as a major objective. Lua runs several times faster than other equivalent interpreted language implementations.

Portability

Lua is provided as a set of C source files written in ANSI standard C. It compiled out of the box with no problems. It should compile on any platform that has a standard C compiler.

Maintainability

The Lua source code is well written and easy to follow. It can be debugged with a standard debugger if required. I needed to make 2 changes to the source to patch two DOS vulnerabilities in the String library (See Sandboxing) and was able to easily locate and fix the required code. I also enabled the Macro to change Lua to use 32 bit integers and floats instead of the default 64 bit integers and doubles. For our needs, smaller sized numbers are more than enough and scripts exhibited an overall 10% performance boost after recompiling. I do not envisage any problems in maintaining our own fork of the Lua source.

Ability to syntax check/compile scripts without running them

Lua scripts are executed in two steps. In step 1, the source code is ‘compiled’ into byte code. In step 2, the byte code is executed by the Lua virtual machine. Step 1, the ‘compilation’ can be done completely separately from step 2, without necessarily running the compiled code. This allows scripts to be syntax checked by the module developer when created, rather than having to wait until the script is executed during module use.

Sandboxing

Lua was a dream to sandbox. Support for sandboxing is built-in to Lua, it does not need to be hacked on later. The majority of the sandboxing is written in Lua itself.

Control access to ‘dangerous’ language features

Lua provides the ability to pass a restricted environment to a piece of code when it is ‘compiled’. The restricted environment can exclude all language elements that may be used to ‘break out’ of the sandbox.

Monitor scripts for excessive CPU or Memory usage

Lua provides a debug ‘hook’ facility where a routine is called regularly during Lua script execution. This routine can monitor the amount of CPU and memory that is being used by the script. If pre-set limits are exceeded, then the hook an raise a Lua ‘error’ to abort the execution of the script.

Patch known vulnerabilities in String library

There are a couple of know vulnerabilities in the Lua String library that can be used to execute a Denial of Service attack using a maliciously coded script. I was able to easily locate and patch these vulnerabilities in the Lua source. Disable access to Global Variables within the language The sandboxing facilities provided by Lua also allow us to detect attempts to create or use Lua Global variables and generate an error.

Consistent error handling

Lua provides 3 mechanisms that allowed me to develop a consistent, useful error handling mechanism. Firstly, ‘chunks’ of Lua code can be executed in ‘Protected mode’ which catches any errors that occur within the execution of the ‘chunk’ and returns the generated error object. Secondly, Lua provides an in-built error handling mechanism that acts in a similar way to the Java/C++ exception handling mechanism. An error can be raised and an error ‘object’ passed back up to the last procedure ‘call’. Finally, Lua provides an introspection facility that allows the source of the error to be determined and a ‘Stack trace’ generated showing the course of script calls that led to the error.

Here is an example error and stack trace.

   Script Error: May not use setmetatable on a locked table
   stack traceback:
   [MapTestLower]:4: in main
   [MapTestUpper]:1: in main

Re-entrant

For efficiency reasons, I designed the interface to use a single Lua ‘state’ that is shared between all script invocations.

When a script calls another script, Lua will call out to C++ which will then call back into Lua at a new level to evaluate the new script, before the original script has finished running.

To prevent different levels of scripts from interfering with each other, each level of script (called within another script), has it’s own ‘context’.

A ‘context’ consists of the ‘this’ object that the script is running against, and a ‘Context Frame’ that contains separate copies of all of the Proxy objects that this level of script uses to access Vassal Game Objects. Earlier ‘Context Frames’ are saved on a stack when a new level of script is invoked and popped off when a script completes, or is aborted by an error.

The consistent and controlled error handling provided by Lua allows the ‘Context Frame’s to be kept in sync with the Lua script invocation levels.