All Messages

  • ”()” is a literal description of the value.
  • “x,y,z” is a position coordinate.
  • “rx,ry,rz” is rotation degrees.
  • ”(A or B..)” shows that valid values are A and B, and the first value (‘A’ in this case) is the default value when not specified.

Model

Show, change, delete and configure 3-D model in the scene. MODEL_ADD newly adds a model into the scene, and MODEL_CHANGE replaces an existing model with another .pmd, keeping its status. You can specify base coordinates or rotation, usage of cartoon rendering in MODEL_ADD. You can also place a model onto another model by specifying parent model and bone name of the parent model where the model should be mounted. The loaded model can be deleted by MODEL_DELETE.

MODEL_ADD|(model alias)|(model file name)
MODEL_ADD|(model alias)|(model file name)|x,y,z
MODEL_ADD|(model alias)|(model file name)|x,y,z|rx,ry,rz
MODEL_ADD|(model alias)|(model file name)|x,y,z|rx,ry,rz|(ON or OFF for cartoon)
MODEL_ADD|(model alias)|(model file name)|x,y,z|rx,ry,rz|(ON or OFF for cartoon)|(parent model alias)
MODEL_ADD|(model alias)|(model file name)|x,y,z|rx,ry,rz|(ON or OFF for cartoon)|(parent model alias)|(parent bone name)
MODEL_CHANGE|(model alias)|(model file name)
MODEL_DELETE|(model alias)

MODEL_EVENT_ADD|(model alias)
MODEL_EVENT_CHANGE|(model alias)
MODEL_EVENT_DELETE|(model alias)
MODEL_EVENT_SELECT|(model alias)

KeyValue-to-bone mapping

Let a KeyValue dominates and controls a bone of a model. You can force a specific bone of a model to ignore motion and be dynamically contolled by a value of KeyValue. The controll will be enabled when MODEL_BINDBONE and MODEL_BINDFACE messages are issued after the model is added.

MODEL_BINDBONE|(key name)|(min)|(max)|(model alias)|(bone name)|x1,y1,z1|rx1,ry1,rz1|x2,y2,z2|rx2,ry2,rz2
MODEL_BINDFACE|(key name)|(min)|(max)|(model alias)|(face name)|rate1|rate2

After the above command was issued, changing the KeyValue key name will directly moves the bone and face. Any motions to the bone and face by a motion file will be ignored. This binding will be valid until the model is deleted.

How it works: at MODEL_BINDBONE, coordinate and rotation of the bone will be controlled by linearly interpolate the value of key name as follows.

r = ($(key name) - min) / (max - min)
(x,y,z) = (x1,y1,z1) * (1 - r)  + (x2,y2,z2) * r
(rx,ry,rz) = (rx1,ry1,rz1) * (1 - r)  + (rx2,ry2,rz2) * r

MODEL_BINDFACE will control the face rate by the value of key name.

r = ($(key name) - min) / (max - min)
rate = rate1 * (1 - r) + rate2 * r

Motion

A motion .vmd file can be applied to a model by MOTION_ADD message. When multiple motions are added, they will be played together. Press b key or select menu to show bone debug information to check how manu motions are running on a model.

FULL or PART in MOTION_ADD sets whether the motion should be treated as full-body motion or part-of-body motion. When FULL is specified, all the motion including the first frame will be applied. When PART is specified, the motion will be modified before applied so that bones or faces that has only the first frame will be ignored. Ones with only initial position / rate are ignored and only moves are applied. The PART is useful when you are overlaying a part-of-body motion in addition to the base model.

ONCE or LOOP sets whether the motion should play once or keep playing by looping from the start. When ONCE is specified, the motion will be automatically deleted after a motion loop has been played. A looping motion will run forever, until deleted by MOTION_DELETE.

MMDAgent-EX has motion smoothing function that compensate gliches at the beginning and ending of a motion at run time. It is enabled by default but you can disable it for the motion by setting OFF for smooth.

Also MMDAgent-EX has another unique function of center reposition on motion start to reset model root position by its center bone at motion start. It is also enabled by default but you can disable it by setting OFF for reposition in MOTION_ADD.

MOTION_CHANGE will swap the running motion with other .vmd, keeping its status. The motion timer will be reset to 0.

MOTION_RESET rewinds the running motion to the first frame.

When several motions are added to a bone at the same time, motions are applied in the order of motion priority, from low to high. The default priority is set to 0, and you can explicitly set priority of a motion at MOTION_ADD. When priority is the same, the lately added one will win.

You can set different motion application rule for each bone. The default rule is replace, in which the highest priority motion override others on the bone. You can modify the rule of a bone to add or none by MOTION_CONFIGURE. add specifies that all motions on the bone should be added, and none specifies that all the motions on the bone should be skipped.

MOTION_ADD|(model alias)|(motion alias)|(motion file name)
MOTION_ADD|(model alias)|(motion alias)|(motion file name)|(FULL or PART)
MOTION_ADD|(model alias)|(motion alias)|(motion file name)|(FULL or PART)|(ONCE or LOOP)
MOTION_ADD|(model alias)|(motion alias)|(motion file name)|(FULL or PART)|(ONCE or LOOP)|(ON or OFF for smooth)
MOTION_ADD|(model alias)|(motion alias)|(motion file name)|(FULL or PART)|(ONCE or LOOP)|(ON or OFF for smooth)|(ON or OFF for reposition)
MOTION_ADD|(model alias)|(motion alias)|(motion file name)|(FULL or PART)|(ONCE or LOOP)|(ON or OFF for smooth)|(ON or OFF for reposition)|priority
MOTION_CHANGE|(model alias)|(motion alias)|(motion file name)
MOTION_DELETE|(model alias)|(model alias)
MOTION_CONFIGURE|(model alias)|(motion alias)|MODE_BONE_REPLACE|bonename[,bonename,..]
MOTION_CONFIGURE|(model alias)|(motion alias)|MODE_BONE_ADD|bonename[,bonename,..]
MOTION_CONFIGURE|(model alias)|(motion alias)|MODE_BONE_NONE|bonename[,bonename,..]
MOTION_CONFIGURE|(model alias)|(motion alias)|MODE_FACE_REPLACE|facename[,facename,..]
MOTION_CONFIGURE|(model alias)|(motion alias)|MODE_FACE_ADD|facename[,facename,..]
MOTION_CONFIGURE|(model alias)|(motion alias)|MODE_FACE_NONE|bonename[,bonename,..]
MOTION_ACCELERATE|(model alias)|(motion alias)|(speed)|(duration)|(specified time for end)
MOTION_EVENT_ADD|(model alias)|(motion alias)
MOTION_EVENT_ACCELERATE|(model alias)|(motion alias)
MOTION_EVENT_CHANGE|(model alias)|(model alias)
MOTION_EVENT_DELETE|(model alias)|(motion alias)

Moves

Let the model move, turn or rotate. GLOBAL means that the position coordinates and rotations angles are treated as global coordinates, and LOCAL means they are local to each model.

MOVE_START|(model alias)|x,y,z
MOVE_START|(model alias)|x,y,z|(GLOBAL or LOCAL position)
MOVE_START|(model alias)|x,y,z|(GLOBAL or LOCAL position)|(move speed)
MOVE_STOP|(model alias)
MOVE_EVENT_START|(model alias)
MOVE_EVENT_STOP|(model alias)
TURN_START|(model alias)|x,y,z
TURN_START|(model alias)|x,y,z|(GLOBAL or LOCAL position)
TURN_START|(model alias)|x,y,z|(GLOBAL or LOCAL position)|(rotation speed)
TURN_STOP|(model alias)
TURN_EVENT_START|(model alias)
TURN_EVENT_STOP|(model alias)
ROTATE_START|(model alias)|rx,ry,rz
ROTATE_START|(model alias)|rx,ry,rz|(GLOBAL or LOCAL rotation)
ROTATE_START|(model alias)|rx,ry,rz|(GLOBAL or LOCAL rotation)|(rotation speed)
ROTATE_STOP|(model alias)
ROTATE_EVENT_START|(model alias)
ROTATE_EVENT_STOP|(model alias)

Stage

Make a stage.

STAGE|(stage file name, .xpmd or .pmd)
STAGE|(bitmap file name for floor),(bitmap file name for back)

Lighting

Light color and direction. Color value should be [0..1].

LIGHTCOLOR|(Red,Green,Blue)
LIGHTDIRECTION|x,y,z

Camera

Move the camera view. Each parameter values in the CAMERA message is exactly the the one displayed on the bottom of the screen when you enable a log view by d key. Or you can give a “camera motion” file as .vmd file.

Transition time period controls the speed of camera move. If given a positive value, the camera will move to the specified location linearly in the specified seconds. If set to 0, the camera parameter will immediately jump to the value with no camera motion. When set to -1, the camera move will be an accellerated smooth move (this is default).

You can “mount” the camera on the model by giving model alias and bone name.

CAMERA|x,y,z|rx,ry,rz|(distance)|(fovy)
CAMERA|x,y,z|rx,ry,rz|(distance)|(fovy)|(transition time period)
CAMERA|x,y,z|rx,ry,rz|(distance)|(fovy)|(transition time period)|(model alias)
CAMERA|x,y,z|rx,ry,rz|(distance)|(fovy)|(transition time period)|(model alias)|(bone name)
CAMERA|(camera motion file name)

User Input to message

Some user action will also generates message.

xxxxx_yyyyy_wwwww_hhhhh shows information about the position where user has long-pressed or released. (x,y) is the screen position, and (w,h) is the screen width and height, both in pixel.

DRAGANDDROP|(file name)
KEY|(key name)
SCREEN_EVENT_LONGPRESSED|xxxxx_yyyyy_wwwww_hhhhh
SCREEN_EVENT_LONGRELEASED|xxxxx_yyyyy_wwwww_hhhhh

You can add a menu via message. It gives users a choise to issue some predefined messages by menu tap. alias corresponds to a page of a menu you are going to create. You can add up to 20 menu pages in the app. id is the vertical position of the item in a page from 0 to 29. The type and argN are the message body to be generated when the menu item is selected.

MENU|ADD|(alias)
MENU|ADD|(alias)|backgroundImagePath
MENU|DELETE|(alias)
MENU|SETITEM|(alias)|(id)|(label)|(type)|(arg1)|(arg2)|...
MENU|DELETEITEM|(alias)|(id)

MENU_EVENT|ADD|(alias)
MENU_EVENT|DELETE|(alias)
MENU_EVENT|SETITEM|(alias)|(id)
MENU_EVENT|DELETEITEM|(alias)|(id)

You can define a content-specific menu by issuing menu related message. See the reference page of Menu messages for full details and options of each message.

Structure of menu

The structure of menu is as follows. Each page can be flipped by gesture.
Some menu pages are already defined in MMDAgent-EX.

page 1
  |- item 1
  |- item 2
  ...
  +- item 30
page 2
  |- item 1
  |- item 2
  ...
  +- item 30
...
page 20
  |- item 1
  |- item 2
  ...
  +- item 30

Add a new menu page

Issuing MENU|ADD message will add a new page to menu. The example message below will add a new blank menu page with alias name mymenu.

MENU|ADD|mymenu

Assign message to an item

MENU|SETITEM message can assign a message string to a menu item. The assigned message will be issued, when user opens the menu page and tap the item. The following message will assign to the top (index 0) item in the mymenu page, a label text “Test1”, and message string THIS|IS|TEST|MESSAGE. If already exist, the command will override the assignment to the new one.

MENU|SETITEM|mymenu|0|Test1|THIS|IS|TEST|MESSAGE

When the item is tapped, the specified message string will be issued, so you can write any menu interaction with your dialogue scenario.

Prompt

Display message dialog to user in front of the scene, and get response.

PROMPT_SHOW|(main text)|(select text 0)|(select text 1)|...
PROMPT_EVENT_SELECTED|(selected number)

KeyValue

You can set any KeyValue by message.

KEYVALUE_SET|(key name)|(value)

InfoText

Display document at full screen.

buttonLabels is a text label to display at the bottom of the text area. When user taps the button, the window will close. Giving multiple button labels like “Yes,No,Cancel” will show users multiple buttons for each, and the button user tapped is returned by INFOTEXT_EVENT_CLOSE. The colors are hex style like “ff00cc”. Alpha value can be set like “ff00ccaa”.

INFOTEXT_FILE|(filepath)|(titleLabel)|(buttonLabels)
INFOTEXT_FILE|(filepath)|(titleLabel)|(buttonLabels)|(scale)
INFOTEXT_FILE|(filepath)|(titleLabel)|(buttonLabels)|(scale)|(BACKGROUNDCOLOR)|(TEXTCOLOR)
INFOTEXT_STRING|textbody|(titleLabel)|(buttonLabels)
INFOTEXT_STRING|textbody|(titleLabel)|(buttonLabels)|(scale)
INFOTEXT_STRING|textbody|(titleLabel)|(buttonLabels)|(scale)|(BACKGROUNDCOLOR)|(TEXTCOLOR)

INFOTEXT_EVENT_SHOW
INFOTEXT_EVENT_CLOSE|(selecteDButtonLabel)

Logging

Start user input logging and send it to server. User inputs and system actions between LOG_START and LOG_FINISH will be stored in local, and all the logs are sent to server by LOG_UPLOAD.

These messages are just for controlling the logging timings inside interaction. The kind of data and server are managed in content package.

LOG_START
LOG_FINISH
LOG_UPLOAD

Plugin_Julius

Speech recognition engine.

  • Default system jconf file is jconf_ja_JP.txt for desktop, jconf_ja_JP_for_mobile.txt for mobile devices.
  • Also load “xxx.jconf” in the content folder, where xxx is the base name of the .mdf file.
  • User dictionary “xxx.dic” will be also loaded.
  • If a model has a morph named “volume”, its rate will be dynamically changed by the input volume.
  • If a model has a morph named “trigger”, its rate will be switched to 1.0 while performing recognition, and 0.0 while not.
  • This plugin can be enabled and disabled either by message or by menu. When disabled, RECOG_EVENT_AWAY|ON event will be issued and audio input will be muted input by setting audio scale to 0.0. When enabled, RECOG_EVENT_AWAY|OFF event will be issued and audio input will be restored.

Recognition result notification:

RECOG_EVENT_START
RECOG_EVENT_STOP|(word1,word2,...)
RECOG_EVENT_OVERFLOW
RECOG_EVENT_GMM

Away notification:

RECOG_EVENT_AWAY|ON
RECOG_EVENT_AWAY|OFF

Set Audio scaling (1.0 = read as is):

RECOG_MODIFY|GAIN|scaling_value
RECOG_EVENT_MODIFY|GAIN

Switch jconf and re-start engine. jconf_file path should be relative to AppData/Julius, not the content folder:

RECOG_MODIFY|CHANGE_CONF|jconf_file

Swap user dictionary:

RECOG_MODIFY|USERDICT_SET|(dictionary file path)
RECOG_MODIFY|USERDICT_UNSET
RECOG_EVENT_MODIFY|USERDICT_SET
RECOG_EVENT_MODIFY|USERDICT_UNSET

Word prediction (internal use):

RECOG_MODIFY|PREDICTWORD_FACTOR|float
RECOG_MODIFY|PREDICTWORD|word1,word2,word3,...

Enable / disable:

PLUGIN_ENABLE|Julius
PLUGIN_DISABLE|Julius
PLUGIN_EVENT_ENABLE|Julius
PLUGIN_EVENT_DISABLE|Julius

Plugin_Open_JTalk

Japanese speech synthesis engine.

SYNTH_START|(model alias)|(voice alias)|(synthesized text)
SYNTH_STOP|(model alias)
SYNTH_EVENT_START|(model alias)
SYNTH_EVENT_STOP|(model alias)
LIPSYNC_START|(model alias)|(phoneme and millisecond pair sequence)
LIPSYNC_STOP|(model alias)
LIPSYNC_EVENT_START|(model alias)
LIPSYNC_EVENT_STOP|(model alias)
PLUGIN_ENABLE|Open_JTalk
PLUGIN_DISABLE|Open_JTalk
PLUGIN_EVENT_ENABLE|Open_JTalk
PLUGIN_EVENT_DISABLE|Open_JTalk

Plugin_Audio

Playing audio file. Supported formats are mp3 and wav.

SOUND_START|(sound alias)|(sound file name)
SOUND_STOP|(sound alias)
SOUND_EVENT_START|(sound alias)
SOUND_EVENT_STOP|(sound alias)
PLUGIN_ENABLE|Audio
PLUGIN_DISABLE|Audio
PLUGIN_EVENT_ENABLE|Audio
PLUGIN_EVENT_DISABLE|Audio

Plugin_LookAt

Eye gazing to mouse (desktop).

PLUGIN_ENABLE|LookAt
PLUGIN_DISABLE|LookAt
PLUGIN_EVENT_ENABLE|LookAt
PLUGIN_EVENT_DISABLE|LookAt

Plugin_Variables

Common variable and count-down timer.

Be careful for “TIMER_EVENT_STOP“: when “TIMER_STARToverrides existing count down timer, MMDAgent-EX issues “TIMER_EVENT_CANCELLED”, whereas Normal MMDAgent issues “TIMER_EVENT_STOP”. This is non-compatible point, so take care when moving dialogue contents into MMDAgent-EX.

VALUE_SET|(variable alias)|(value)
VALUE_SET|(variable alias)|(minimum value for random)|(maximum value for random)
VALUE_UNSET|(variable alias)
VALUE_EVAL|(variable alias)|(EQ or NE or LE or LT or GE or GT for evaluation)|(value)
VALUE_EVENT_SET|(variable alias)
VALUE_EVENT_UNSET|(variable alias)
VALUE_EVENT_EVAL|(variable alias)|(EQ or NE or LE or LT or GE or GT for evaluation)|(value)|(TRUE or FALSE)
TIMER_START|(count down alias)|(value)
TIMER_STOP|(count down alias)
TIMER_EVENT_START|(count down alias)
TIMER_EVENT_STOP|(count down alias)
TIMER_EVENT_CANCELLED|(count down alias)
PLUGIN_ENABLE|Variables
PLUGIN_DISABLE|Variables
PLUGIN_EVENT_ENABLE|Variables
PLUGIN_EVENT_DISABLE|Variables

Plugin_VIManager

PLUGIN_ENABLE|VIManager
PLUGIN_DISABLE|VIManager
PLUGIN_EVENT_ENABLE|VIManager
PLUGIN_EVENT_DISABLE|VIManager
FST_LOAD|(.fst file path)
FST_LOAD|(.fst file path)|(initial state number)

Plugin_WindowController

EXECUTE|(file name)
KEY_POST|(window class name)|(key name)|(ON or OFF for shift-key)|(ON or OFF for ctrl-key)|(On or OFF for alt-key)
PLUGIN_ENABLE|WindowController
PLUGIN_DISABLE|WindowController
PLUGIN_EVENT_ENABLE|WindowController
PLUGIN_EVENT_DISABLE|WindowController

Plugin_TextArea

TEXTAREA_ADD|(textarea alias)|(width,height)|(size,margin,exlinespace)|r,g,b,a|r,g,b,a|x,y,z
TEXTAREA_ADD|(textarea alias)|(width,height)|(size,margin,exlinespace)|r,g,b,a|r,g,b,a|x,y,z|rx,ry,rz
TEXTAREA_ADD|(textarea alias)|(width,height)|(size,margin,exlinespace)|r,g,b,a|r,g,b,a|x,y,z|rx,ry,rz|(parent model alias)
TEXTAREA_ADD|(textarea alias)|(width,height)|(size,margin,exlinespace)|r,g,b,a|r,g,b,a|x,y,z|rx,ry,rz|(parent model alias)|(parent bone name)
TEXTAREA_SET|(textarea alias)|(text)
TEXTAREA_DELETE|(textarea alias)

TEXTAREA_EVENT_ADD|alias
TEXTAREA_EVENT_SET|alias
TEXTAREA_EVENT_DELETE|alias
PLUGIN_ENABLE|TextArea
PLUGIN_DISABLE|TextArea
PLUGIN_EVENT_ENABLE|TextArea
PLUGIN_EVENT_DISABLE|TextArea

Explanation of TEXTAREA_ADD:

TEXTAREA_ADD
  |(textarea alias)
      a new alias name
  |(width,height)
      text area box parameter
          >0 to use fixed size (longer text will shrink to fit within the box)
          <=0 to use variable size (varies according to the text)
      default: 0,0
  |(scale,margin,exlinespace)
      font scale, margin, additional linespace
      default: 1.0, 1.0, 0.0
  |r,g,b,a
      background color (0 ~ 1), set 0 to 'a' for disable background drawing
  |r,g,b,a
      text color (0 ~ 1)
  |x,y,z
      offset coordinate of the center of this area
  |rx,ry,rz
      (optional) rotation in angles
  |(parent model alias)
      (optional) model alias name to be put on
  |(parent bone name)
      (optional) bone of the model to be put on

Explanation of TEXTAREA_SET:

TEXTAREA_SET|(textarea alias)|(text)
  text: text to be displayed. should be braced by "" to handle white space. Use "\n" (backslash + 'n') for multiple line text

Plugin_Remote

PLUGIN_ENABLE|Remote
PLUGIN_DISABLE|Remote
PLUGIN_EVENT_ENABLE|Remote
PLUGIN_EVENT_DISABLE|Remote

Plugin_ErrorBlock

Plugin_Network

NETWORK_GET|(net alias)|(URI)|(save file name)
NETWORK_EVENT_GET|(net alias)
PLUGIN_ENABLE|Network
PLUGIN_DISABLE|Network
PLUGIN_EVENT_ENABLE|Network
PLUGIN_EVENT_DISABLE|Network

Plugin_Flite_plus_hts_engine

Plugin_Kafka



Last modified May 7, 2020: Re-organizing further (4b20b5d)