CraftAssist - LLMs on minecraft (?)

Craftassist is a cool project where the wuthors figured out a way to use natural language to communicate with a minecraft bot via chat. The even manage to give natural instructions to the robot to do certain tasks.

The first version of the bot had 3 main components:

1. Parsing incoming text into logical form (“action dictionary”)
2. This parsed data is then interpreted by a dialogue object (not sure what that is).
3. This dialogue object thingy queries a memory module.

I think the memory module is the special part. It is a symbolic representation of the both’s awareness of the world around it.

The bot has some low level actions like MOVE, PUT BLOCK X etc.

Given below is an example of a parsed human command:

## input: "go to the blue house"
## output:
{
    "dialogue_type": "HUMAN_GIVE_COMMAND",
    "action": {
        "action_type": "MOVE",
        "location": {
            "location_type": "REFERENCE_OBJECT",
            "reference_object": {
                "has_colour": [0, [3, 3]],
                "has_name": [0, [4, 4]]
}}}}

(WIP)