This is an old document describing how users of a ScriptEase Javascript/Ecmascript product, or a product that is created with those libraries, can include Distributed Scripting into their application. Javascript users could use this protocol following the Distributed Scripting Javascript Library manual.

SE:DSP was a technology created by no-longer-existing Nombas.

For more documentation, see Old Nombas User Docs.

DScriptEase

Distributed Scripting Protocol

Thex Distributed Scripting Protocol, or DSP, is Nombas' method for remote scripting. Using DSP, your application can communicate with and run scripts on any other application that uses ScriptEase and is DSP-enabled. For instance, your application can be controlled by a ScriptEase: Desktop script running on another machine, a ScriptEase: Web Server Edition script, or even another instance of your application running on another machine.

There is quite a bit of information about distributed scripting we need to cover, and much of it is interrelated. For that reason, at times, related information to a topic isn't explored until a later section of the manual. It may take you a couple of reads to get the most from this manual.

I: Adding DSP to Your Application

The first part of this document is concerned with how you get DSP running on your application. The second part of this document is using DSP to script remote applications.

For your application to perform remote scripting tasks, you will first need to add the ScriptEase: ISDK to your application. This process is extensively detailed in the ScriptEase: ISDK manual. You can also use DSP with ScriptEase products such as ScriptEase: Desktop or ScriptEase: Web Server Edition. In any case, the external library capability of ScriptEase allows you to use (via the '#link' directive) the DSP external library. If you have decided to turn off the external library capability in your application, you will need to add the DSP library internally to your application. This is a simple process.

First, add the source file 'srclib/dsp/sedsp.c' to your builds. Second, edit your 'jseopt.h' file and add the following line to it:

#define JSE_DSP_ALL

That's it. Recompile and relink your application and you can now use the Distributed Scripting Protocol with it.

A common usage of DSP involves using the internet (TCP-IP) for tranporting the data, as detailed below. This is usually done in ScriptEase via the sesock external library. Again, if you are not using external libraries, but still want to use sockets for your DSP transport, you need to add the socket library internally. This is done just like the sedsp functionality was added, by including the file 'srclib/socket/sesocket.c' and adding a define to your 'jseopt.h' file:

#define JSE_SOCKET_ALL

The Distributed Scripting Transport Mechanism

Distributed scripting consists of two parts, the protocol itself and the transport layer. The protocol is concerned with what information to send while the transport layer sees to it that the information gets passed between the two machines. The protocol is DSP itself; we've done all the work on implementing it, so you need only use it, which we'll discuss extensively later. The protocol is attached to no particular transport mechanism. You choose the one you want. This section is concerned with the transport layer. DSP needs to get particular information to the partner application. You need to get the applications connected and able to pass this information back and forth.

Commonly, you will use TCP-IP to allow applications to script each other over the internet. However, we have an example of using shared files to implement transport as well, and you can use any number of other ways to move the data between two machines. We include an implementation of TCP-IP tranport as well as shared files for you to use. Before we get into how you would write your own mechanism, we will explain how to use these two implementations

We will make all examples in this document scripted examples; you would use these when all DSP interaction is done inside scripts. For instance, in a client/server model, both the client and servers would be scripts running on a ScriptEase ISDK application. You can access the DSP routines directly from the ScriptEase API. We reserve discussion of this until Appendix i so as to not make the explanations we are giving confusing.

Shared Files

The shared file mechanism is implemented in the file 'dspfile.jsh', simply include this file at the top of your DSP script. Now you can create a new DSP connection, using shared files, using the 'new fileDSP()' constructor. This constructor takes two arguments, the name of two files used to pass the data back and forth. The result of this call is a full DSP connection object. Once you have this object, you can use all of the DSP mechanisms described later. You have your ticket, now you are ready see the show.

Here is an example of creating a DSP object using our shared file transport mechanism:

var conn = new fileDSP("dspfile.in","dspfile.out");

You will need to specify the same two files for both DSP partner applications. The only little gotcha is that the file names for the two partners must be swapped; the input file for one application is the output file for the other application, and vice versa. In the example above, one application has set up file DSP. The other application will need to initialize its connection like this:

var conn = new fileDSP("dspfile.out","dspfile.in");

Obviously, shared files can only work on machines that can see the same files. This is easily done if both scripts are run on the same machine, or machines that share files such as via a Netware file server.

TCP-IP

With the popularity of the internet, it is only a natural that people want to use it to do distributed scripting. Many popular internet staples, such as web servers and email, are implemented at their core by a daemon that accepts connections on a TCP-IP socket and then allows a remote application to control it. With DSP, you can do exactly the same thing, but you have the full power of Javascript to allow far more complex commands than the simplistic ones most daemons allow. Once your application has integrated the ScriptEase ISDK, not only can you run scripts locally to control it, you can allow other machines to run them to control your application as well.

TCP-IP is the 'language' of the internet. ScriptEase provides the library calls necessary to create and use TCP-IP sockets in the 'sesock' external library. We provide an include file, "isdp.h", that uses these calls to implement a transport mechanism for DSP.

Like the file transport mechanism above, both partners must create their end of the connection. In the TCP-IP world of sockets, one partner must create a connection and the other partner must connect to that existing connection. This easily lends itself to a client-server model, but as we will see later, that is not the only possible model; once connected, either side may run scripts on the other side, or both may.

TCP-IP Server

We will first examine the listener side. This is the application that creates a socket to accept connections on and waits for those connections to occur. This application creates a socket it can listen on by using the 'new iDSPServer()' constructor. This is traditionally called a "server", but keep in mind that the client-server relationship we are discussing now is only for getting the two applications connected. Once the connection is established, either side may script the other as we will describe in Chapter II.

Here is the code snippet for creating a server:

#include "idsp"

var server = new iDSPServer( port );

In this case, 'port' should be a numeric value, the port to listen on. When using sockets, you must specify a port number so that the connecting client can then find you. Other applications on your machine may be using their own ports to do TCP-IP connections, so it is slightly possible you may get a conflict. If this happens, select a different port number. You should always use port numbers that are 1000 or higher; the lower port numbers are reserved for standard system programs like mail and web servers.

Once you have done this, you will have a socket that you can accept connections on. This value returned is not itself a connection, it is just a socket handle with some extra DSP information attached. It is a permanent place for clients to find you and for to find the clients. You use this socket to get connections to clients; You use that handle's method 'accept()'. Even after you have a connection and are performing DSP scripting using it, the original socket remains. You can continue to accept more connections now and in the future using it. Usually, a server handling a TCP-IP socket runs indefinitely, until it is explicitly shutdown, and this socket exists for the life of the program.

Once we have created a socket to listen on using the above code, we need to accept connections to actually do DSP scripting. Whenever a TCP-IP client connects to us, we use the 'accept()' method of that socket to get this connection and service the client. We can use the 'ready()' method of the socket to know if such a connection has been made. If we just call 'accept()', it will always get a connection for us. If no one is trying to connect when we make a call to 'accept()', we will wait until someone does. This can be fine for some programs, but in others you may want your program to do other tasks rather than be put to sleep waiting for a connection to appear. This is especially important for Windows scripts. While the 'accept()' method is waiting, Windows messages are not processed, and the application will appear to freeze.

So, here is a simple fragment to get new connections using iDSP (this fragment builds on the last simple fragment):

/* This server just runs forever */

while( 1 )

{

/* wait for the next connection to appear */

while( !server.ready(250) );

/* get the next connection */

var conn = server.accept();

/* do some DSP scripting with the remote host */

/* close the connection and loop to service the next connection */

conn.dspClose();

}

Notice that this simple server code handles only a single connection at a time. This is not necessary, you can call the 'accept()' method multiple times and have more than one connection simultaneously, if multiple clients are trying to connect to your server at the same time. Of course, the code to handle their requests simultaneously will be more complex. Handling multiple socket connections at once is demonstrated in the sample 'inndsp.jse', which is discussed fully in Appendix ii.

Of course, any of the methods you would use to handle multiple socket connections applies equally to multiple DSP connections, since at its heart, an iDSP connection is really just a socket. While a few of the socket library calls, such as 'accept()' and 'listen()' are directly mimicked by the iDSPServer() object, not all are. Fortunately, you can get the socket object itself by using the '.dspConnection' field of a DSP connection, such as 'server.connection' or 'conn.connection', in the sample code fragments above. You can then use this socket in any socket calls.

This sample code fragment uses the 'dspClose()' call. This will be discussed as part of the next chapter on how you actually do DSP scripting. Suffice it to say for now, when you finish scripting on a particular connection, you use 'dspClose()' to shut down the connection.

TCP-IP Client

Now that we have the server half of the applications, we need a client. The client will connect to an already existing server's socket. Fortunately, it is much simpler than the work we needed to do to get the server running. We can connect to an existing server like this:

var client = new iDSP(host,port);

The 'host' parameter is the internet machine name that the server is running on. For instance, my machine is named "outworld.nombas.com", so if you wanted to connect to a server running on my machine, that would be the name you use. You can use "localhost" to have the client connect to an application running on the same machine the client is on. 'port' is the port number, the same one that the server used when it initialized its socket. That was described above in the section on the server.

That's it. Again, both partner applications will have a connection they can use to script the other application. We will now dive into actually using those connections. If you would like to build a custom transport mechanism instead of using one of our sample ones, Appendix ii has full details on how you go about doing this.

II: Using Distributed Scripting

Distributed Scripting Models

After you have a connection established, remote scripting can take place. The connection does not have any preferred 'direction'. Either side can call DSP commands that the other will process.

For instance, the TCP-IP server may accept connections from clients, then run scripts on them. Maybe you are writing a server that contains updates for software installed on your company's machines. Each day, as part of a CRON job, every machine in the office connects to that server. The server then queries the client machine to see what updates it has that are not yet on the client machine, and installs them.

On the other hand, you can instead make the client machine run a script which queries the server to see what updates it has, then installs them on itself. Which partner is running the script and which is being scripted is determined by the scripts being run themselves. DSP allows both sides of the connection to run scripts on the other.

Let's look at two common models for doing DSP scripting.

Client-Server

Often, you'll want a Client-Server model. In this case, after the connection is made, one side will run a script that may execute script code on the other side. The other partner will just wait while that script runs. In this case, the side running the script is the client, and the side waiting is the server. The service being provided is the processing of the DSP scripts that the client wishes to run on the server. This is the simplest model and is very easy to understand. Again, don't confuse this with the iDSP client-server connections. Once the connection is established, either side may be the client as we are talking about now. It is most common that the side doing the connecting is the client, though.

There is nothing to say about the client in this section; it just runs some script doing DSP. What you can do with DSP will be fully described below. In this model, the client script is the only one 'doing anything'.

The server is also running a script. However, in this model, its only job is to wait for the client to finish while processing any DSP commands the client wants done. It is possible for DSP errors to occur, such as losing the connection, so the server's script needs to be able to handle that.

Here is a simple fragment that implements a DSP server:

try

{

/* while the other guy has stuff to do, do it */

while( conn.dspService() );

}

catch( e )

{

/* if some error occurs, don't crash, just terminate the connection */

}

conn.dspClose();

You would use this fragment after the connection is made, for instance after the line in the TCP-IP sample fragment that reads:

/* do some DSP scripting with the remote host */

The key routine we learn about here is 'dspService().' DSP commands sent from the remote host are processed with this call. Each call to it processes a single command. It returns 'false' if the remote host closed the connection using 'dspClose()'. We don't close our own connection until after the client has done all of the scripting it desires.

Peer-To-Peer Scripting

Not all DSP applications will want only one side running code. You may want both sides running code and using distributed scripting to communicate. In this case, we actually have a relationship among peers. This is the basic DSP model; the Client-Server really just is a special case of this in which one side, the server, does nothing. As peers, each side is free to run whatever script it wants. It can call functions and modify variables on the other side using DSP.

The main point to remember while doing peer-to-peer communication is that DSP requests from the other side are only handled when your own script calls some DSP function. If you do other scripting tasks, the other side will be waiting on any DSP processing it needs done until you actually call a DSP function. For this reason, peer-to-peer scripting is necessarily more complex than a Client-Server relationship. Several hints are in order. First, make sure your script periodically calls 'dspService()' to process any commands that your partner needs done. Second, you must have some mechanism to signal both sides that you are ready to quit. If one side calls 'dspClose()', then the connection will be lost, even if the other side still wants to make DSP calls.

The 'inndsp.jse' sample, described fully in Appendix ii, uses a peer model, so it will be instructive to look at it to see what can be done.

Using DSP

We've covered a lot of ground, from setting up distributed scripting to the models you will use. Now we get to the good stuff, actually using DSP in your scripts. Fortunately, this section is very short, since DSP scripting is very easy. Once you have a connection established with the remote host, that connection is treated very much like the entire scripting environment on the other side. We have, in our previous code fragments, named the connection variable 'conn', we will use that in the following examples.

So what does 'scripting environment' mean. Simply, all the variables and functions on the remote side can be accessed using this variable. For instance, 'conn.a' will give us the value of the global variable 'a' on the remote machine. 'conn.Clib.puts('Hi there!');' will print out a message on the remote machine, and so forth. Of course, there are a few caveats we will mention shortly, but this is the basic idea. Let's look at the potential gotchas.

DSP References

The first thing to explain is that information is not transferred between machines unless it is needed. If you write,

var a = conn.a;

'a' is not actually the value of 'a' on the other side until you try to use it, such as with:

var b = a + 10;

Now, 'a's value will be fetched from the other side to add to 10. This is especially important with objects, if you write:

var rClib = conn.Clib;

perhaps with the intention of then doing:

rClib.printf("Hello, world.\n");

rClib.puts("Enter your name:);

var name = rClib.gets();

/* etc */

It would be silly to try to copy the entire Clib object across the network due to this statement. Instead, whenever you reference the other side, a special object is constructed that knows how to 'fetch itself' from the remote machine when the value of the object is needed. If you were to printout the 'typeof a', it would register as a "function" (a function is a special type of object), not whatever type the value on the other machine is.

There are two ways you can see what 'a's type really is. The first is to force the value to be copied to the local machine, using 'dspGetValue()', for instance:

Clib.printf(typeof (a.dspGetValue()));

The second, preferred way, requires no copying of data across the machines. You simply check the type on the remote machine, and just get the type string. Here is a code fragment:

Clib.printf(conn.eval("typeof a"));

By using 'conn.eval', we run a short script fragment on the remote machine. It has no trouble checking the type of 'a' there, and then the result is send back to us for printing. This may seem a more work, but if 'a' is a complex object, trying to send the whole thing across the DSP connection can be much more work.

Working With Objects

The same gotcha can manifest itself when you want to enumerate an object's members if the object is on the remote machine. Just as 'typeof' sees the placeholder object, not the actual value, so would the FOR..IN statement. If you use FOR..IN, you will see the members of the placeholder object, not the members of the remote object. So, how do you go about getting them?

Just like seeing its type, you have two solutions. Using 'dspGetValue()' is obviously the easiest sounding one, but it will actually lead to many problems. Javascript objects can often be very complex things that don't make much sense on another machine. Cyclic loops especially can cause a lot of problems when trying to copy objects. For this reason, DSP does not allow an entire object to be copied across. We are considering ways to get this to work in future versions, but for now a workaround is needed. You need to do the enumeration on the remote machine and get the results to the local machine. So, here is the code that you would like to do:

var a = conn.Clib;

for( var x in a ) { ... }

and this is how you should write it so it will work correctly:

var a = conn.Clib;

var a = conn.Clib;

var fields = conn.eval(

"var tmp = \"\"; for( var tmp2 in Clib ) tmp = tmp + \",\"+ tmp2; return tmp;");

fields = fields.substring(1,fields.length).split(",");

for( var x2 in fields )

{

var x = fields[x];

...

}

(Yes, this is ugly, but it works.)

DSP and The ScriptEase API

It is useful to do DSP processing via the ScriptEase ISDK API rather than via scripts in some applications. A application may want to run a server in the background and not devote an entire thread to running a server script. The application can do the same actions via the API as a script would. The source file "dspapi.c" and corresponding header "dspapi.h" provide the routines we examine.

The basic tasks for a ScriptEase API DSP program are the same as for a script; create a DSP connection, run code that uses DSP and/or service incoming DSP requests, and close everything down when done. Each task can be done by simply calling the corresponding DSP wrapper functions, finding them with jseGetMember() and calling them with jseCallFunction(). However, you can call them directly in C using the functions we will talk about next.

First, creating a DSP connection is necessary. A DSP connection is just a jseVariable that is set up with the connection information. In your case, you will need to set up the physical connection with the remote machine, then call the 'dspCreateConnection()' function, passing it two callback parameters. These parameters are C functions you provide to transmit and receive data. The resulting jseVariable can be saved, or assigned to a member of the global object if you want scripts to be able to DSP using it. In addition, this function takes parameters to allow you to set the security of this script (security is described in appendix ii.)

Now that you have a DSP object, you can perform DSP actions on it just like a script can. For instance, you can use 'jseGetMember()' to look up a member of the object, which is the remote value. Underneath, this will eventually call the required DSP routines to service the request. The function 'dspService()' is provided as well. This is a C-callable routine that acts just like the script version, allowing you to call it every so often to handle incoming DSP requests on a particular DSP connection.

Finally, you can close a DSP connection, just like in Javascript, by using the 'dspClose()' function provided.

Writing Your Own Transport

Making your own transport mechanism is pretty simply. You will need to define a new object class that handles DSP creation and transport. Doing this requires a constructor method and a couple of extra methods to handle reading data, writing data, and closing the connection.

First, you will need a constructor function. This function will create and return a new DSP connection.

Such a function looks something like this:

function myDSP(args)

{

var ret = new DSP(myOpen,args);

if( ret!=null ) ret._prototype = myDSP.prototype;

return ret;

}

'myOpen' is another function you will write which we talk about shortly. Arguments are whatever arguments you need to pass to the constructor. For instance, in iDSP, a client needs the host name and port number to connect to. You can pass more than one argument. We call the DSP() constructor with our open function, passing it these arguments. The result is an object of class 'DSP', but we really want it to be of our class (a subclass of DSP), and the second line accomplishes this. If you want to know more about prototypes and _prototypes, see the ISDK manual. For our purposes, it is enough to know that this is the way you do it. Finally, we return the resulting object.

The 'myOpen' function (or whatever you name it) simply takes the arguments and creates a connection to the remote host. A variable is returned representing the connection, of whatever format you choose. Often it is an object whose members store data you need to access the connection. This variable will be passed to all of the routines we will describe next. First, though, because we have changed the objects returned to be of our class, we need to make sure they still will be DSP objects. We do this by making our class an explicit subclass of DSP. You do this with the following code snippet (building on the last example):

myDSP.prototype._prototype = DSP.prototype;

Now that we have our connection established, DSP will be using it to pass its data back and forth. You need to write the necessary transport functions to allow this to happen. These functions must be put in the prototype of the constructor function so that DSP objects created with that constructor will have them. These functions only apply to DSP objects constructed with your function; you may have other DSP objects constructed with a different transport protocol, and they will use their own functions.

Building on the above example:

function myDSP.prototype.dspSend(conn,buffer,timeout)

{

/* tranport the data */

}

function myDSP.prototype.dspReceive(conn,&buffer,length,timeout)

{

/* receive the data into buffer */

}

Both functions take a connection as their first parameter, this being whatever value your DSP open function returned. They both also receive a timeout, the number of milliseconds to try to send or receive the data before giving up. You can ignore this parameter and just wait forever if you like. Both functions also receive a buffer parameter (a variable of type "buffer", a ScriptEase data type.)

For the send function, the buffer contains the data to be send, while for the receive function, it is a parameter you fill in with the data actually received. Remember, you can access the '.length' member of the buffer to determine its actual size.

The receive function receives one additional parameter, the length. This is the maximum size the buffer should be on return, it can be less if you receive some data but not as much as specified. Both functions return the actual length of the data transmitted.

We are almost done, one more function completes the minimal DSP transport requirements. You need a function to close down the connection when it is done. This will be called by DSP, usually as a direct result of the script using dspClose() on the connection, but it can also happen if there is an error that forces the connection to be aborted. The function looks like this:

function myDSP.prototype.dspCloseConnection(conn)

{

/* close the connection */

}

Security and DSP

In an ideal world, we would be able to open up our machine to anyone in the world to connect to us and run scripts on our machines. We would hope that they would only run appropriate ones, never malicious ones. Unfortunately, that is not our world. It is necessary to implement security measures to make sure that distributed scripting is not misused.

The first part of DSP security is in the connection mechanism itself. When you are opening a connection, you can implement a myriad of ways to verify the connector, such as by a password. However, this is usually not enough. If you are writing a chat program for instance, you want anyone to be able to connect. You just want them to only be able to do 'chat' things, but arbitrary functions such as Clib.system(). DSP allows you to do just that by adding security to the code run by remote scripts. You should take the time to read the ScriptEase security manual, since DSP security uses the normal ScriptEase security mechanism.

DSP security is implemented by adding 'dspSecurityInit()', 'dspSecurityTerm()', and 'dspSecurityGuard()' functions to the transport class, just like the 'dspSend()' and 'dspReceive()' we talked about above. These functions correspond exactly to the normal security functions, except they apply only to remote scripts run on a connection of this class. The security variable is a newly created object, but it has the "dsp" member set to the DSP connection variable.

You should make sure to make all of your transport, security, and exported functions read-only. You do this using the 'setAttributes()' function, such as by:

setAttributes(myDSP.prototype.dspSecurityInit,0x06);

If you fail to do this, a clever hacker can on the first pass, change your function to one of his own, then connect again and bypass security. NEVER allow access to the 'setAttributes()' function, or your security will be able to be bypassed.

THE CHAT EXAMPLE

The programs 'inn.jse, inncli.jse', and 'innbard.jse' are long time ScriptEase samples that demonstrate a simple application using the socket library. To demonstrate most of the DSP concepts, 'inndsp.jse' is a version of all three programs that communicates by using DSP. It still uses sockets for its basic transport. This program uses a peer-to-peer model. You should look at the sample to see most of the DSP concepts described in this chapter in action.