Writing a network scheme for Rebol 3 ==================================== :author: Graham Chiu :date: 26-March-2013 :revision: 1.5 :email: This document is a distillation of what I have read and understood regarding networking and URI schemes as defined for Rebol 3. The sources are the Rebol websites along with info obtained in chatting with other Rebol users. Any incorrect information contained herein is entirely due to my own misunderstandings. As an illustration I will talk about creating a basic *Scripting Layer For Android* (SL4A) scheme from the ground up. Opening a Port Concepts ----------------------- When you open a network scheme you are returned a port! object. The port! object contains the functions, objects, and other variables that allow you to track state, and process network events. One of those objects is the underlying sockets-level TCP port which interacts with the TCP device. In Rebol 2, this was generally referred to as the sub-port while Rebol 3 schemes such as HTTP call it conn or a similar name. Make a note as this can be a source of confusion. And if the scheme is anything other than a TCP scheme using a spec block, the underlying TCP port is not opened and will require a second call to *open*. Simple Example -------------- To illustrate how the built-in TCP scheme works, we can look at a basic interaction with a SL4A server at network address 192.168.1.103 on port 4321. I will discuss line by line what happens. .basic tcp example with event handler ---- android-request: {{"params": ["Hello, Android!"], "id":1, "method": "makeToast"}^/} android-port: make port! tcp://192.168.103:4321 ; android-port: make port! [ scheme: 'tcp host: 192.168.1.103 port-id: 4321 ] android-port/awake: func [event /local tcp-port] [ tcp-port: event/port print ["port equality?" equal? tcp-port android-port] print ["==TCP-event:" event/type] switch/default event/type [ lookup [ ; this happens when we use a url!, and the IP address has to be resolved ; using DNS, or when using a specification block and the host is provided ; as a name print "opening tcp port" open tcp-port ] connect [ ; these next two lines are equivalent since we are using a basic tcp scheme. ; write tcp-port android-request write android-port android-request ] wrote [ print "have a wrote event, so read response from server" read tcp-port ] read [ ; data has arrived print ["^\read:" length? tcp-port/data] return true ] ] [ ; returns if receives other events like done, close, error, custom true ] ] open android-port either port! = type? wait [android-port 3] [ print to string! android-port/data ] [ print "Port timed out!" ] close android-port ---- First, we define the string *android-request* to hold the JSON representation of the SL4A function call *makeToast*, with a parameter of "Hello, Android". The terminating newline is needed as the API appears to be line oriented. ---- android-port: make port! tcp://192.168.103:4321 ---- Next, we create a new port! based on the URL of tcp://192.168.103:4321. If we now examine android-port, we have this .output from probe android-port ---- make port! [ spec: make object! [ title: "TCP Networking" scheme: 'tcp ref: tcp://192.168.103:4321 path: none host: "192.168.103" port-id: 4321 ] scheme: make object! [ name: 'tcp title: "TCP Networking" spec: make object! [ title: none scheme: none ref: none path: none host: none port-id: 80 ] info: make object! [ local-ip: none local-port: none remote-ip: none remote-port: none ] actor: make native! [[port!]] awake: make function! [[event][print ['TCP-event event/type] true]] ] actor: make native! [[port!]] awake: make function! [[event][print ['TCP-event event/type] true]] state: none data: none locals: none ] ---- We can see that the port! object contains a *spec* object derived from the URL, tcp://192.168.1.103:4321. This was created by the *parse-url* function which in turn was called by the *make-port* function. The spec object is filled in with all URI scheme components parsed as per RFC 3986 (e.g. tcp://user:password@hostname:port). Next we see the *scheme* object based on the Rebol TCP scheme, followed by actor which is currently a port! (it can also be a block holding actors such as *read*, *write*, etc.), the awake function, state, data and locals. The *data* field will hold any data returned by the tcp port during a read. The *awake* function by default just prints the event to the console, and then by *return true*, exits any wait function that might have passed the event to it. The next lines in our example do something more useful defining an awake function that handles all the different type of events that a port can receive. The events we have not named such as _close_, _error_, _done_ will cause our switch statement to return true by default. We have ordered the events in the switch statement according to the order that we expect them to happen but you might find it more efficient to re-order them in descending order of frequency of use. Now, let's step through each event to see what happens Lookup ~~~~~~ The *lookup* event is generated when either the TCP device has to do a DNS lookup to resolve a name, or, if a URL has to be decoded by *decode-url*. Here we used the url! construction form to create the port!, but with the block form [ scheme: 'tcp host: 192.168.1.103 port-id: 4321 ] the *lookup* event would not be needed and would not occur as a tuple! is provided for the IP address. Connect ~~~~~~~ Here, our TCP device has connected with the server, and a *connect* event is returned. When handling protocols like POP3, and SMTP, we would want to read the welcome string from the server here, but with other protocols like HTTP and SL4A there wouldn't be a welcome string so the request is sent immediately. Wrote ~~~~~ Having written our request to the tcp-port aka android-port, we now get a *wrote* event. Under normal conditions the server would respond to our request, so we issue an immediate _read tcp-port_. Read ~~~~ The TCP device has received some data which is made available in the android-port/data field, and a *read* event generated. We print the data and return *true* since we don't need to deal with any other events. This allows us to exit the *wait* function. Rest of Code ------------ Now, let's look at the remaining code. ---- open android-port ---- passes the port! to the open function. The *open* function in this instance does not actually open the TCP port, but issues a *lookup* event which is stored in a system block which holds all events. If we had done the following instead: ---- android-port: open tcp://192.168.1.103:4321 ---- then *open* would have called *make-port* to first create the port! structure, and then issued the *lookup* event as above. But if we had done this instead ---- android-port: open [ scheme: 'tcp host: 192.168.1.103 port-id: 4321 ] ---- then the same thing happens as above except no *lookup* event is generated. In this last instance the underlying TCP port is actually opened. The difference is due to the fact that when using the url! form the host address is converted to a string! i.e. "192.168.1.103" instead of a tuple! by the *decode-url* function and so still needs to be resolved using a DNS lookup. ----- either port! = type? wait [android-port 3] ----- The *wait* function adds the android-port to the list of system ports, and then does a _wake-up android-port lookup_, i.e. it sends the *lookup* event to the android-port's awake handler. In this case our awake handler on receipt of the *lookup* event now opens the tcp-port with _open tcp-port_. Each subsequent event is passed to the port! where they are handled as follows: * *connect* - We *write* to the port * *wrote* - Next, we *read* the port * *read* - Finally, we *print* the data and return *true* On receipt of the true value, the *wait* function removes the port! from the system list of ports, and then returns from the *wait* returning a port! A timeout is specified so that if the port! fails to return within 3 seconds the *wait* will exit. If we get a port! returned from the *wait*, we know we completed without timing out and have data, so we can print it out. Custom Schemes -------------- It would be tedious to write out the above code each time we wanted to write something to a port when we could do something like this: ---- result: write sl4a://192.168.1.103 {{"params": ["Hello, Android!"], "id":1, "method": "makeToast"}^/} ---- The sl4a scheme would handle the opening of the TCP port, set the default port-id and async handler, take our data, write it to the TCP port on *connect*, and return the data. Since we do not have any reference to the port!, it would then be closed for us by the garbage collector. In the ideal code above, we have a *write* function that takes two arguments, a url!, and a string! and returns a result. How do we do this when *write* is a built-in, native function which knows nothing about our custom sl4a scheme? This is where the magic of the port! actors comes into play. When *write*, or any of the built-in functions that take a port! as a parameter are called, the port! actor block is examined to see if there is a function of the same name present. If so, it is executed in preference to the built-in function, passing the port! as the first argument, followed by any additional arguments if present. You will remember from before that we had the following when we inspected the port! object created for the built-in TCP scheme: ---- actor: make native! [[port!]] ---- In a typical custom scheme, the signatures of the actor functions would instead look something like this: ---- actor: [ write: func [ port [port!] data ][ ... write actions ...] read: func [ port [port!]][ ... read actions ...] open: [ port [port!]][ ... open actions ... ] open?: [ port [port!]][ .. returns a truthy value if open ... ] close: [ port [port!]][ .. closes the port ...] ] ---- and the custom scheme *write* actor is invoked instead of *lib/write*. So, the first thing we need to do is write our port! actors. We can have here *create*, *delete*, *open*, *close*, *read*, *write*, *open?*, *query*, *update*, and *rename* and all these functions take a port! as an argument. We can also add series functions such as *length?*, *append*, *insert* to the actor block, but these function can only take a port! object as the argument, and can not take a url! or spec object. So, instead of ---- insert sl4a://192.168.1.103 "some data" ---- we need to write it something like this: ---- android: open sl4a://192.168.1.103 insert android "some data" ---- The minimum actors we need for our scheme example are *write*, *open*, *open?* and *close*, so let's start! Open? ~~~~~ This just needs to return the port/state. When the port! is first created, it should be *none*, as seen in the probe output above. Once the port! is opened, the port/state is set to an empty string by *open*. So, when we close the port!, we need to set this back to *none*. ---- open?: func [ sl4a-port [port!]][ sl4a-port/state ] ---- Close ----- We need to close the tcp-port, and also set the sl4a-port/state to *none*. The tcp-port is going to be held in the sl4a-port/state/tcp-port. ---- close: func [ sl4a-port [port!]][ close sl4a-port/state/tcp-port sl4a-port/state: none ] ---- Write ----- Our *write* actor is going to first check if the sl4a-port is open. If it is not open, it will open the sl4a-port using the sl4a-port *open* actor (since it is opening a sl4a-port). Once the sl4a-port is opened, we will write the data to the sl4a-port. You will recall from the TCP scheme example that we actually write to the tcp-port in the *connect* event. We will use function named *sync-write* that assigns a custom async handler to do this for us. We also need to return some data. Again, recalling the basic TCP scheme, data is returned to the tcp-port/data, or sl4a-port/state/tcp-port/data in the current code. The problem is that this might not be safe if we have more than one read since interleaving reads and writes clears this field. We could store it in sl4a-port/data, but we may not necessarily have a reference to that from the tcp-port, so we will create an additional tcp-port spec field, JSON, to hold this data. We are naming it JSON since we know the SL4A api returns JSON strings. ---- write: func [sl4a-port [port!] data] [ if not open? sl4a-port [ open sl4a-port ] sync-write sl4a-port data ; and this is the data placed here by the sync-write function which we now return sl4a-port/state/tcp-port/spec/JSON ] ---- Open ---- Our *open* actor has the task of setting up the sl4a-port/state object which is going to hold our tcp-port, our JSON data field which is part of our port spec(ification), and perhaps other fields we might need later on as we expand our scheme functionality. If the sl4a-port is already open, it is simply returned. If a host address is not specified, we create an error. We then create our tcp-port here using *make port!* as with the TCP scheme, using a spec block, and *make port!* creates the scheme and spec objects. Before opening the tcp-port, we set the awake handler to *none*. Normally we would set it to the scheme's awake handler, but we haven't defined one yet. It won't matter yet because even though we then *open* the tcp-port, there wouldn't be any events to process until we call *wait* on the tcp-port. Note that since we are now using *open* on a tcp-port rather than a sl4a port, Rebol will use the *open* function appropriate for a TCP port. Otherwise we would have a stack overflow. ---- open: func [ sl4a-port [port!] /local tcp-port ] [ if sl4a-port/state [return sl4a-port] if none? sl4a-port/spec/host [make-sl4a-error "Missing host address"] sl4a-port/state: context [ tcp-port: none ] sl4a-port/state/tcp-port: tcp-port: make port! [ scheme: 'tcp host: sl4a-port/spec/host port-id: sl4a-port/spec/port-id timeout: sl4a-port/spec/timeout ref: rejoin [tcp:// host ":" port-id] port-state: 'init json: none ] comment { port/state/tcp-port now looks like this [ spec [object!] scheme [object!] actor awake state data locals ] } tcp-port/awake: none open tcp-port sl4a-port ] ---- Sync-write ---------- You will recall we called function *sync-write* inside the *write* function but have not defined it. *sync-write* takes the sl4a-port and a JSON string as arguments. It needs to set up the awake handler for the tcp-port so that a *connect* event sends the JSON string, then gets data back and stores it in the sl4a-port/state/tcp-port/spec/JSON field. It then kicks off event handling by doing a *wait* on the tcp-port. If a *lookup* event occurs, we know we need to *open* the tcp-port. If we want to do another *write* on the port, we can no longer do it on the *connect* event as that only occurs when we first open the tcp-port. Therefore, we need to check if the tcp-port has been connected. If so, we *write* to it and then *wait* to restart the event handler. To track the port state, a flag, port-state, is created in the spec object. ---- sync-write: func [sl4a-port [port!] JSON-string /local tcp-port ] [ unless open? sl4a-port [ open sl4a-port ] tcp-port: sl4a-port/state/tcp-port tcp-port/awake: :awake-handler either tcp-port/spec/port-state = 'ready [ write tcp-port to binary! JSON-string ][ tcp-port/locals: copy JSON-string ] unless port? wait [tcp-port sl4a-port/spec/timeout] [ make-sl4a-error "SL4A timeout on tcp-port" ] ] ---- Awake Handler ------------- We used an awake-handler in the *sync-write* function, and define it now. In the *connect* state, we set the port-state flag to 'ready; in other states we can set it as as appropriate. It is very similar to our prototype awake handler that we discussed at the beginning, but we are also going to use a structured exit at the bottom instead of using returning *true*, to make things clearer. The value left at the end is the value of the *switch* statement. ---- awake-handler: func [event /local tcp-port] [ print ["=== Client event:" event/type] tcp-port: event/port switch/default event/type [ error [ print "error event received" tcp-port/spec/port-state: 'error true ] lookup [ open tcp-port false ] connect [ print "connected " write tcp-port tcp-port/locals tcp-port/spec/port-state: 'ready false ] read [ print ["^\read:" length? tcp-port/data] tcp-port/spec/JSON: copy to string! tcp-port/data clear tcp-port/data true ] wrote [ print "written, so read port" read tcp-port false ] close [ print "closed on us!" tcp-port/spec/port-state: 'ready true ] ] [true] ] comment { the awake handler should return false unless we want to exit the wait which we do either as the default condition ( ie. unspecified event ), or with Error, Read and Close. } ---- Wrapping Up ----------- So we are nearly ready to wrap this up into a scheme. We have to set up the scheme header and define any missing functions that we have used so far without having defined them. .SL4A scheme ---- Rebol [ System: "REBOL [R3] Language Interpreter and Run-time Environment" title: "R3 SL4A" file: %prot-demo.r author: ["Graham"] name: 'sl4a type: 'module version: 0.0.1 Date: [26-Mar-2013] Purpose: "R3 send and receive from Scripting Layer 4 Android" ] make-sl4a-error: func [ message ] [ ; the 'do arms the error! do make error! [ type: 'Access id: 'Protocol arg1: message ] ] awake-handler: func [event /local tcp-port] [ print ["=== Client event:" event/type] tcp-port: event/port switch/default event/type [ error [ print "error event received" tcp-port/spec/port-state: 'error true ] lookup [ open tcp-port false ] connect [ print "connected " write tcp-port tcp-port/locals tcp-port/spec/port-state: 'ready false ] read [ print ["^\read:" length? tcp-port/data] tcp-port/spec/JSON: copy to string! tcp-port/data clear tcp-port/data true ] wrote [ print "written, so read port" read tcp-port false ] close [ print "closed on us!" tcp-port/spec/port-state: none true ] ] [true] ] sync-write: func [sl4a-port [port!] JSON-string /local tcp-port ] [ unless open? sl4a-port [ open sl4a-port ] tcp-port: sl4a-port/state/tcp-port tcp-port/awake: :awake-handler either tcp-port/spec/port-state = 'ready [ write tcp-port to binary! JSON-string ] [ tcp-port/locals: copy JSON-string ] unless port? wait [tcp-port sl4a-port/spec/timeout] [ make-sl4a-error "SL4A timeout on tcp-port" ] ] sys/make-scheme [ name: 'sl4a title: "SL4A Protocol" spec: make system/standard/port-spec-net [port-id: 4321 timeout: 5] actor: [ open: func [ sl4a-port [port!] /local tcp-port ] [ if sl4a-port/state [return sl4a-port] if none? sl4a-port/spec/host [make-sl4a-error "Missing host address"] sl4a-port/state: context [ tcp-port: none ] sl4a-port/state/tcp-port: tcp-port: make port! [ scheme: 'tcp host: sl4a-port/spec/host port-id: sl4a-port/spec/port-id timeout: sl4a-port/spec/timeout ref: rejoin [tcp:// host ":" port-id] port-state: 'init json: none ] tcp-port/awake: none open tcp-port sl4a-port ] open?: func [sl4a-port [port!]] [ sl4a-port/state ] write: func [sl4a-port [port!] data] [ if not open? sl4a-port [ open sl4a-port ] sync-write sl4a-port data sl4a-port/state/tcp-port/spec/JSON ] close: func [sl4a-port [port!]] [ close sl4a-port/state/tcp-port sl4a-port/state: none ] ] ] ---- Testing ------- We can test that it works with the following code, using no port! reference: ---- android-request: {{"params": ["Hello, Android!"], "id":1, "method": "makeToast"}^/} result: write sl4a://192.168.1.103 android-request ---- The output: .console trace ---- >> result: write sl4a://192.168.1.103 android-request === Client event: lookup === Client event: connect connected === Client event: wrote written, so read port === Client event: read ?read: 36 == {{"error":null,"id":1,"result":null} ---- If we want to hold on to the port! we can run this code: ---- >> android: open sl4a://192.168.1.103 >> result: write android android-request === Client event: lookup === Client event: connect connected === Client event: wrote written, so read port === Client event: read ?read: 36 == {{"error":null,"id":1,"result":null} } >> result: write android android-request === Client event: wrote written, so read port === Client event: read ?read: 36 == {{"error":null,"id":1,"result":null} } ---- This should provide you with enough information to write your own Rebol 3 network schemes. If you wish to discuss aspects of this article I can be reached on the Stackoverflow chat room http://chat.stackoverflow.com/rooms/291/rebol-and-red References ---------- . http://www.rebol.net/wiki/Ports:_Synchronous_and_Asynchronous_Operations . http://www.rebol.net/wiki/TCP_Port_Open_Issue . http://www.rebol.net/wiki/Schemes:Notes