FST Format #

Introduction #

The operation scripts of MMDAgent-EX are described using a state transition model, with conditions and actions. The file extension is .fst.

At runtime, MMDAgent-EX is always in a certain state. The module monitors messages flowing within MMDAgent (input messages) and if an input message matches any of the conditions waiting in the current state, it outputs the corresponding message to MMDAgent and transitions to the next state. This process is repeated to execute the dialogue scenario.

In MMDAgent-EX, the .fst specification has been expanded from the original MMDAgent, but compatibility is maintained, and old .fst files for the original can still be used.

In MMDAgent, you can display the .fst debug window internally with the Shift+f key. Also, you can open the running fst file in an editor with the e key. Please utilize this for operation checks.

How to use Sub-FST.

VSCode Extension #

We have released a VS Code extension for .fst files, please give it a try.

https://marketplace.visualstudio.com/items?itemName=MMDAgent-EX.dialogue-fst-editing-support

The extension includes the following features:

Input assistance that tells you about message specifications and arguments
Detection of states to be aware of, such as states with no transitions or states that are not transitioned from anywhere
The ability to move to the definition by specifying the state name
Display a list of references that jump to the state name

Overview #

.fst files are text files. Lines starting with # are ignored as comments.

The following is an example. In this example, it first sets the background image, loads the model file and plays the motion, and sets the camera parameters. Then, it transitions to a state named MAINLOOP. In the MAINLOOP state, it sends corresponding messages for each key input and returns to the MAINLOOP state.

<eps> represents an empty word in FST. That is, if there is an <eps> in the condition field, it is always TRUE and the line proceeds to the next without waiting for input. Also, if there is an <eps> in the output field, it proceeds without outputting anything.

# initial values
${agentPMD}="Agents/mai/mai.pmd"
${camera_default}="1.7,12.7,0.0|0.0,0.0,0.0|44|16|1"

# Begins with state "0"
0 MAINLOOP:
  <eps>                    STAGE|floor.png,back.jpg
  <eps>                    CAMERA|${camera_default}
  <eps>                    MODEL_ADD|0|${agentPMD}
  MODEL_EVENT_ADD|0        MOTION_ADD|0|base|waiting.vmd|FULL|ONCE|OFF|OFF
  MOTION_EVENT_ADD|0|base  <eps>

MAINLOOP MAINLOOP:
  KEY|1 MOTION_ADD|0|ojigi|mei_greeting.vmd|FULL|ONCE|ON|OFF

MAINLOOP MAINLOOP:
  KEY|2 SYNTH_START|0|mei_voice_normal|Hello.

MAINLOOP MAINLOOP:
  KEY|9 AVATAR_LOGSAVE_START|log.txt

MAINLOOP MAINLOOP:
  KEY|0 AVATAR_LOGSAVE_STOP

Basic Format #

Indentation is critical. The state name line should have no indent, and the transition description line should be indented. This is mandatory.

The state name can be any string. In older versions of MMDAgent, it was just a number, but in the latest version, you can use any string. The state ID representing the initial state is fixed at “0” (zero).

Each field is separated by a space or tab. If you want to specify a value that includes a space, such as a path name, use "" or ''.

name1 name2:
    input_message1 output_message2
    input_message2 output_message2
    ...

name2 name3:
    ...

Details of Transition Description #

name1, name2, … are state names. The first one represents the current state, the second one represents the transition destination state. The lines indented from the next line represent the behavior definition sequence between the two states. Each input_message is the field for the transition condition, and output_message is the field for the message to be output at runtime.

The behavior definition can be written over several lines. If it is described in multiple lines, it will be processed sequentially as a sub-state from top to bottom. In other words, in the above example, when the current state becomes name1, first wait for input_message1. When it arrives, output output_message2 and wait for the next line’s input_message2. When a message corresponding to input_message2 arrives, issue output_message2… and so on, connecting in order. When the last line of that block ends, transition to the state name specified by name2.

If you define multiple blocks that start with the same state name, they are evaluated from the one defined earlier in the .fst file. That is, in the following example, if the input matches both input_message1 and input_message1, the top one is prioritized.

name1 name2:
    input_message1 output_message1

name1 name3:
    input_message2 output_message2
    ...

Local Variables #

You can define, assign, and reference local variables for each .fst. They are enclosed for each .fst. (Do not confuse with MMDAgent-EX’s global variables)

The initial value can be specified in the first part of the .fst file (before the first state definition).

${agentPMD}="Agents/mai/mai.pmd"
${camera_default}="1.7,12.7,0.0|0.0,0.0,0.0|44|16|1"

Local variables can be referenced in the condition field and output field. By writing ${variable name}, it will replace that part at runtime (the moment it is evaluated) with the value of that local variable at that time, and evaluation and execution will be performed.

XXX YYY:
    <eps> MODEL_ADD|mei|${agentPMD}

In the condition field, you can use the value of a local variable as a transition condition. Only matches as a string are possible. The comparison operators are == and != only.

XXX YYY:
    ${flag}==xxx  MODEL_ADD|mei|...

WWW ZZZ:
    ${flag}!=yyy  MODEL_ADD|mei|...

Changes in runtime values and assignments are described as additional fields at the end of each transition. It is also possible to reference between variables.

XXX YYY:
  <eps>            <eps>  ${place}=Nagoya

ZZZ QQQ:
  MODEL_ADD|mei|.. <eps>  ${value}=${src}/${dst}

It is also possible to assign multiple values at once.

XXX YYY:
  <eps>  <eps>  ${src}=Nara,${dst}=Tokyo,${pref}=nozomi

By writing ${%global variable name}, you can handle and retrieve global variables as local variables.

XXX YYY:
  <eps>            <eps>  ${place}=${%KeyName}

How to Write Condition Fields #

Plain Text #

If you write a string that does not match any of the following in the condition field, it will match an input message that exactly matches it.

Variable Value #

The value of a local variable can be a transition condition. This condition does not depend on the input message, and the transition occurs when the evaluation of the given expression is TRUE. The comparison operators are == and != only.

XXX YYY:
    ${flag}==xxx  MODEL_ADD|mei|...

WWW ZZZ:
    ${flag}!=yyy  MODEL_ADD|mei|...

Please note that the evaluation of the expression is only once immediately after transitioning to that state. Whether the expression holds true or not is only evaluated immediately after the transition to the state, and it does not respond even if the expression holds true during the state stay.

Regular Expressions #

You can use regular expressions for text matching. To write it, enclose the entire condition field with @. The following is an example of a transition description that conditionally matches when a message containing Station or station arrives in the recognition result (RECOG_EVENT_STOP).

XXX YYY:
    @RECOG_EVENT_STOP\|.*[Ss]tation.*@  <eps>

The range enclosed by @ is thrown directly to the regular grammar engine, so be careful, for example, | needs to be written as \| as in the example. The regular expression library uses Google RE2. Refer to Google’s document etc. for the format.

Note that it is a full match, not a partial match. Regular expressions that only catch part of it will not match. Write the regular expression so that the entire message matches.

# Bad example
XXX YYY:
    @[Ss]tation@  <eps>

After evaluating the regular expression, the sub-match range enclosed in parentheses is automatically assigned to local variables ${1}, ${2}, etc. This allows you to extract the matched part into a local variable. For example, the following is an example of extracting the model alias name and motion alias name from the MOTION_EVENT_ADD message into ${model} and ${motion}.

XXX YYY:
    @MOTION_EVENT_ADD\|(.*)\|(.*)@ <eps> ${model}=${1},${motion}=${2}

%INCLUDE #

Inside .fst,

%INCLUDE("filename.fst")

By doing so, you can include the specified file at that location.

Includes are expanded when reading .fst, and the contents are interpreted as if they were expanded right there. No particular scope processing is done for state names or variables. Be very careful about state name conflicts and processing consistency.

Parallel FST Launch (Sub FST) #

You can run multiple FSTs in parallel. When the main FST file is

foobar.fst

if there are .fst files with names like the following, MMDAgent-EX will also open them at the same time. (xxx is any character)

foobar.fst.xxx.fst

All launched sub FSTs will start in parallel with the main FST. Input is cascaded to the sub FST and the output of the sub FST is streamed into the message queue. Sub FSTs operate independently of the main FST.

Typical uses include, for example,

Playing motions at regular intervals
Launching corresponding messages when certain messages arrive

These allow you to write operations that run independently from the state in dialogue management.