========== AAL/Python ========== This is AAL/Python reference documentation. AAL is a pre/postcondition language for defining 1. test steps (how something is tested) 2. conditions for test steps (under which they can be tested). In AAL/Python test steps and their conditions are written in Python. AAL syntax ---------- Syntax of AAL is the following: aal "" { language "" { [prelude] } variables { } initial_state { } adapter_init { } adapter_exit { } action "" [, ""...] { guard() { } body() { } adapter() { } }... tag "" { guard() { } adapter() { } }... } guard(), body() and adapter() blocks in actions and tags are optional. The default guard() is always true, default body and adapter blocks are empty. If tag and action blocks are nested, the guard() of an inner block is evaluated only if the guard() of the outer block returns true. Names of actions must start with either - "i" for input actions. Execution of these actions is triggered by the test generator. - "o" for output actions. Execution of these actions is triggered by the system under test. Test engine cannot suggest executing these actions. Shorthands "input" and "output" can be used instead of "action". In this case names are automatically prefixed by "i:" and "o:", respectively. Shorthand blocks "serial" and "parallel" can contain actions or nested serial and parallel blocks. Actions and subblocks inside a serial block are executed in a loop, starting from the first block. On the other hand, actions or subblocks inside a parallel block can be executed in any order, but they can be executed again only after executing every action and subblock has been executed exactly once. Comments in AAL are written after "#" on their own lines. Commenting syntax inside code blocks is the same as in the programming language in use. Control flow ------------ The main loop in test generation and execution is: 0. Initialise: first the model, then the adapter. 1. Validate the current state (check every tag on this state). 2. Listen to output actions. If an output action is observed, execute it and go to step 1. 3. Choose and execute an enabled input action, then go to step 1. Initialisation When a compiled AAL module is loaded by fmbt or its remote model/adapter like remote_pyaal, the prelude code (see the AAL syntax above) is executed first. Once loaded, the module is initialised: if the AAL module is used as a model in the test configuration, the initial_state block is executed. If it is used as an adapter, the adapter_init block is executed. If it is used as both, both blocks will be executed in this order. State validation First, guard() blocks of all tags are evaluated to find which of them are True in the current state. After that, adapter() blocks of those tags are executed where guard() returned True. If the adapter() block of a tag returns False or raises an AssertionError, state validation has failed, and the test will fail after all tags have been validated. If the adapter() block raises some other Exception, the test will stop with an error. Otherwise, validating the state in the perspective of the tag has succeeded. Listening to outputs Before listening to outputs, the test generator has checked which actions can be executed in the current state by evaluating all guard() blocks. When listening to outputs, adapter() blocks of output actions are executed until one of them reports observation of an action, or none of them reports observations. In more detail, if the adapter() of an action returns False (or None), the action was not observed and nothing else will be done for it. If the adapter() block returns True, this action has been observed. If the adapter() returns some other action, that action has been observed. Next, observed action is validated, that is, it is checked whether the observation is allowed by the model in the current state. If the guard() of the observed action returned False before listening to outputs, test engine tries to validate observed action by re-executing its guard() again. (This allows adapter() block to change model variable in some complex cases like non-deterministic system under test, so that execution of the action is after all allowed under new conditions.) If the guard() of the observed action returned True, execution of observed action is allowed in the current state. The model is updated by executing the body() of the action. On the other hand, if the guard() returned False in both cases, the test will fail due to observing an action that is not allowed in the current state. If the adapter() block raises an error, the test stops with an error. Choosing and executing an enabled input action In the simple case (no simulation), guard() blocks of all input actions are executed. One of the actions whose guard() returned True will be chosen. Next, the adapter() of the chosen action is executed. If the adapter() returns None or True, execution of this action is considered successful. If the adapter() returns False or raises AssertionError, test will fail due to unsuccessful execution of the action. If the adapter() raises some other exception, the test will fail with an error. If execution was successful, the model is updated by executing the body() block of the action. Control flow in case of simulation Test generator simulates test execution before choosing the next step if test configuration defines heuristic lookhead(n) with n > 0. In simulation the generator calls guard() blocks of all input actions, then for each enabled action it: 1. Pushes the values of variables into a stack. 2. Simulates the execution of the action by updating the model as if the action has been successfully tested: by executing its body() block. This may change the values of variables. 3. Simulates executions of next n-1 steps similarly. 4. Pops the original values of variables from the stack. Finally the algorithm chooses the next step, executes the adapter() and, in case of success, the body() of the chosen action as in the simple case. Control flow when AAL is used only as a model Choosing the next step takes place as in simple and simulation cases. However, adapter() blocks are never executed. Instead, the name of the chosen action is sent to configured adapter. In case of successful execution, body() of the chosen test step is executed. Control flow when AAL is used only as an adapter guard() and body() blocks are never executed. Adapter() blocks are executed in the order in which they have been chosen by the test generator. Control flow when the adapter() of an action returns a different action In addition to True, None and False, the adapter() block of an action is allowed to return an integer. The integer is the action that the adapter reports to be executed - instead of the action whose adapter() block was executed. This enables reacting to non-deterministic events. If the adapter() returns an integer (an action), the body() of the action returned (reported) action is executed, instead of body() of the suggested action. Control flow when the adapter() of an action raises an exception If adapter_exception_hander() is not defined, this is an execution of an unidentified action and test fails. If the handler is defined, it is called. If it returns an action, that body() of that action is executed. If it returns None, body() of suggested action is executed. If it raises an exception, test fails with an execution of an unidentified action. End of test adapter_exit is executed at the end of test. If the test ends because of reaching an end condition, "verdict" and "reason" variables contain more detailed information, for instance "pass" and "coverage reached". Otherwise, "verdict" and "reason" are None. AAL/Python syntax ----------------- The code in language, initial_state, adapter_init, adapter_exit, guard(), body() and adapter() blocks is Python. Variables block is a comma-separated list of names of variables. Continued line should end with "\". Example: variables { system_state, \ calculator_mode, \ calendar_view, calendar_active_day } guard() must return True or False. Two examples: action "iAlwaysDisabled" { guard() { return False } } action "iCalendar-NextDay" { guard() { if calendar_view == "year": return False else: return True } } As seen above, single-line Python statements can be written to the same or separate lines as "{" and "}". In multiline blocks each statement must be written to its own line. Indentation depth of the first line in a block is not meaningful, but the rest of the lines must be indented according to Python rules. Actions can be nested in tags. This avoids repeating the same checks in guard()s of actions that have something in common. For instance, launching applications when in the homescreen can be modelled without checking homescreen view in guards of every action: tag "homescreen" { guard() { return view == "homescreen" } input "app1 launch" { body() { view = "app1" } } input "app2 launch" { body() { view = "app2" } } input "app3 launch" { body() { view = "app3" } } } Serial and parallel blocks can save a lot of effort in writing guards. For example, all possible interleavings of two users filling up a questionnaire, can be modelled as follows: parallel { serial { input "Alice login" {} input "Alice answer 1" {} input "Alice answer 2" {} input "Alice logout" {} } serial { input "Bob login" {} input "Bob answer 1" {} input "Bob answer 2" {} input "Bob logout" {} } } Serial and parallel blocks can be parameterised. Parameters: single block can be executed at most once. Example: serial(single) { input "x" {} } AAL/Python tools ---------------- remote_pyaal is an fMBT utility that enables using AAL/Python models and adapters in test configuration. For instance, test configuration model = "aal_remote(remote_pyaal -l aal.log example.aal)" adapter = "aal" uses the same AAL/Python model/adapter instance as both model and adapter. Before loading the AAL/Python file, remote_pyaal uses fmbt-aalc to compile AAL/Python into Python. By default, fmbt-aalc uses fmbt-aalp for preprocessing the AAL/Python file. AAL preprocessor ---------------- fmbt-aalp is the AAL preprocessor. It understands the following directives: ^ifdef "" ^ifndef "" ^endif ^include "" The directives must be used in the beginning of each line. Included filenames are looked for from directories listed in the AAL_INCLUDE_PATH environment variable, and in include directories given with -I to fmbt-aalp, fmbt-aalc or remote_pyaal. AAL compiler ------------ fmbt-aalc compiles the AAL/ code to . Before compilation it runs a preprocessor, that is fmbt-aalp by default. For instance, AAL/Python model is compiled into Python library as follows: $ cat > example.aal << EOF aal "example" { language "python" { import os } variables { pid_printed } initial_state { pid_printed = False } action "iPrintMyPID" { guard() { return pid_printed == False } body() { log("my pid is: %s" % (os.getpid())) pid_printed = True } } } EOF $ fmbt-aalc example.aal > example.py remote_pyaal ------------ remote_pyaal implements remote AAL model/adapter interface. It loads a model/adapter from AAL/Python source or pure Python produced by fmbt-aalc. remote_pyaal can convert AAL/Python models into finite state machines, too. It outputs the state machine in LSTS (labelled state transition system) format. fmbt-editor ----------- fMBT editor visualises AAL models as finite state machines. By default, each visualised state corresponds to a unique combination of values of variables, that are listed in AAL's variables block. The scope and the output of the visualisation can be adjusted with the following special comments understood by the editor. # preview-hide-vars: Hide given variables. Each visualised state corresponds to a unique combination of values of *visible* variables. If two states differ by values of hidden variables only, the states will be equated in the visualisation. # preview-show-vars: Hide all other variables but the ones that are listed here. # preview-hide-trans: Hide all transitions where action matches to the regular expression. # preview-hide-untested-trans: Hide all transitions where action matches to the regular expression and the transitions have not been executed in the generated test. # preview-hide-states: [unvisited] [orphaned] unvisited: hide states that have not been visited in the test. orphaned: hide states where all incoming and outcoming transitions have been hidden because of preview-show-[tested-]trans or preview-hide-[untested-]trans. # preview-show-trans: Show only transitions where action matches the regular expression. # preview-show-tested-trans: Show only transitions where action matches the regular expression and the transitions have been executed in the generated test. Other previewer options are: # preview-depth: Show model state space only to this depth from the initial state. Rest of the states and transitions are not generated. # preview-image-path: In order to show images on tooltips, look for them from directories listed in preview-image-path. Debugging --------- remote_pyaal option --verbose-fmbt-log prints AAL/Python variables and their values to fMBT log. If you need pdb (Python debugger) prompt inside AAL/Python or other Python code during fMBT test run, call fmbt.debug() in the code, run the test, and run command fmbt-debug See python -c 'import fmbt; help(fmbt.debug)' for more information. Special variables and functions in AAL/Python --------------------------------------------- __file__ - string that contains the name of the AAL file. action_name - string that contains the name of an action. Available: in guard(), body() and adapter() blocks of actions. When using action_name in "input" or "output" blocks, action_name contains full name of the action, and input_name/output_name variables contain the user-defined name. See input_name. Example: Actions "iLogin:Alice" and "iLogin:Bob" can be executed when "Alice" or "Bob" is not yet logged in. action "iLogin:Alice", "iLogin:Bob" { guard() { username = action_name.split(":")[1] return not username in logged_in } body() { username = action_name.split(":")[1] logged_in.add(username) } } input_name - constant that contains the name of an input Available: in guard(), body() and adapter() blocks of inputs. Example: Executing action "play music" in input "play music", "play video" { adapter() { print "input name:", input_name print "action name:", action_name } } will print input name: play music action name: i:play music Example: Embedding Python in input names input "callee='Alice'", "callee='Bob'" { body() { exec input_name in globals() } } defines actions i:callee='Alice' and i:callee='Bob'. The actions set a value to the callee variable when the body() block is executed. output_name - constant that contains the name of an output Available: in guard(), body() and adapter() blocks of outputs. tag_name - string that contains the name of a tag. Available: in guard() and adapter() blocks of tags. Example: Embedding Python in tag names tag "songcount==0", "songcount==1", "songcount in [2,3,4]" { guard() { return eval(tag_name) } } action(name) - function that returns corresponding action (integer) Available: guard(), body() and adapter() blocks of inputs, outputs and actions. Adapter blocks that need to report execution of some other action (input or output), do it with: return action("i:play music") return action("o:playback stopped") or with corresponding shorthands return input("play music") return output("o:playback stopped") Example: TODO adapter_exception_handler(name, exc) - hook adapter exceptions Available: N/A (defined by user) If defined, this function will be called in case the adapter() block of an action or a tag raises an exception. The name of the action or tag and the exception object are passed as arguments. If this function returns an integer when handling an exception of an action, execution of the action corresponding to the integer is reported to the test generator (see the action function). Returning integer 0 means reporting special "unidentified" action. Returning an integer when handling an exception of a tag is an error. If this function returns None or True, adapter reports test generator successful execution of the action or successful validation of the tag whose adapter() raised the exception. If this function returns False when handling an exception of a tag, adapter reports failed validation of the tag. Returning False when handling an exception of an output action means that the action was not observed. Returning False for an input action is an error. If this function raises an exception, adapter reports execution of an unidentified action (test will fail). Example: TODO guard_list - list of block names why the guard is executed Available: in guard() blocks of tags and actions. Contains names of blocks because of which the guard is evaluated. The last name in guard_list is the name of the current block. The first name in the list is the innermost block whose guard is evaluated. Example: the guard of "outer" tag disables all actions in "inner-1" tag, leaving "world" as the only enabled action. tag "outer" { guard() { return "inner-1" not in guard_list } tag "inner-1" { input "hello" {} } tag "inner-2" { input "world" {} } } input(name) - see action(name) log(message) - function that writes messages to remote_pyaal's log Available: everywhere. output(name) - see action(name)