Skip to content

Commit 723cc28

Browse files
onepiecefreak3onepiecefreak3
authored
Merge pull request #11 from onepiecefreak3/refactoring
Refactoring
2 parents 49ea5e6 + 8f730ab commit 723cc28

File tree

424 files changed

+20900
-2206
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

424 files changed

+20900
-2206
lines changed

‎FormatSpecification.md

+179
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# XQ32/XSEQ Format Specification
2+
3+
## Introduction
4+
5+
XQ32 (and in extension XSEQ) is a script format used in 3DS Level5 games to script any sort of flow or high level behaviour.
6+
It is used to describe menu construction and behaviour, as well as environment management, and other similar tasks.
7+
The main motivation for this script format seems to be to elevate as much logic as possible out of the compiled executable of the game.
8+
It may have been helpful in debugging and hot switching behaviour in development.
9+
10+
## Datatypes
11+
12+
These are the data types of data units as they will appear in this documentation.
13+
14+
| Type | Size in bytes | Notes |
15+
| - | - | - |
16+
| short | 2 | |
17+
| ushort | 2 | Unsigned |
18+
| int | 4 | |
19+
| uint | 4 | Unsigned |
20+
| float | 4 | |
21+
| string[n] | n | Encoded as ASCII, if not stated otherwise |
22+
23+
## Container
24+
25+
This script format defines 4 tables and a string blob to store an arbitrary amount of methods of arbitrary length, callable through the games engine or the script itself.
26+
It is unknown what exactly the entry point into the script is or if it is even a set method to invoke the scripts logic.
27+
28+
## Compression
29+
30+
Each table is compressed by Level5's own container specification, and borrows pre-existing compression schemes by Nintendo from their SDK.
31+
Read the first 4 bytes of a table in little endian as an int. It assumes the name 'methodSize'.
32+
Decompress the table from the 4th byte onwards until the end, according to the method portion of 'methodSize'.
33+
34+
The value 'methodSize' is structured as follows:
35+
method = methodSize & 0x7
36+
decompressedSize = methodSize >> 3
37+
38+
The methods map to the following compressions:
39+
| Method | Compression |
40+
| - | - |
41+
| 0 | None |
42+
| 1 | LZ10 |
43+
| 2 | Huffman 4Bit |
44+
| 3 | Huffman 8Bit |
45+
| 4 | RLE |
46+
| 5 | ZLib |
47+
| 6 | - |
48+
| 7 | - |
49+
50+
## Structure
51+
52+
All values are read as little endian. All values are read consecutively in one structure, if not stated otherwise.
53+
A structure member can have a constant value, since they wouldn't make sense logically otherwise or are inferable by static analysis.
54+
55+
### Header
56+
57+
The header declares the start of each table and the string blob. It also defines the number of entries each of those tables have.
58+
The absolute offset to a table or the string blob can be calculated by left shifting the read value by 2.
59+
60+
| Datatype | Name | Default value |
61+
| - | - | - |
62+
| string[4] | magic | "XQ32" |
63+
| short | functionCount | |
64+
| short | functionOffset | 0x20 >> 2 |
65+
| short | jumpOffset | |
66+
| short | jumpCount | |
67+
| short | instructionOffset | |
68+
| short | instructionCount | |
69+
| short | argumentOffset | |
70+
| short | argumentCount | |
71+
| short | globalVariableCount | |
72+
| short | stringOffset | |
73+
74+
### Function
75+
76+
The function structure declares one function of the script.
77+
It declares the instructions and jumps used in this function, by giving the offset into the instruction and jump table and how many of them to read for this function.
78+
It declares the parameter count, the amount of values that are passed into the method from an external caller.
79+
It declares the amount of local and object values used in the function. Those values are used to allocate memory for the stack used in the function.
80+
It declares its own name, by giving the offset into the string blob and its CRC32/CRC16 checksum.
81+
82+
#### Xq32Function
83+
84+
| Datatype | Name |
85+
| - | - |
86+
| int | nameOffset |
87+
| uint | crc32 |
88+
| short | instructionIndex |
89+
| short | instructionEndIndex |
90+
| short | jumpIndex |
91+
| short | jumpCount |
92+
| short | localVariableCount |
93+
| short | objectVariableCount |
94+
| int | parameterCount |
95+
96+
#### XseqFunction
97+
98+
| Datatype | Name |
99+
| - | - |
100+
| int | nameOffset |
101+
| ushort | crc16 |
102+
| short | instructionIndex |
103+
| short | instructionEndIndex |
104+
| short | jumpIndex |
105+
| short | jumpCount |
106+
| short | localVariableCount |
107+
| short | objectVariableCount |
108+
| int | parameterCount |
109+
110+
### Jump
111+
112+
The jump structure declares one jump to another instruction inside the instruction table by index.
113+
It declares its own name, by giving the offset into the string blob and its CRC32/CRC16 checksum.
114+
A function invokes a jump by checksum or name. It's common to use the checksum for performance.
115+
A jump can only be performed in the current function. Jumps referenced outside the function may not be executed.
116+
117+
#### Xq32Jump
118+
119+
| Datatype | Name |
120+
| - | - |
121+
| int | nameOffset |
122+
| uint | crc32 |
123+
| int | instructionIndex |
124+
125+
#### XseqJump
126+
127+
| Datatype | Name |
128+
| - | - |
129+
| int | nameOffset |
130+
| ushort | crc16 |
131+
| int | instructionIndex |
132+
133+
### Instruction
134+
135+
The instruction structure delcares one instruction in a function of the script.
136+
It declares the arguments used in this instruction, by giving the offset into the argument table and how many of them to read for this instruction.
137+
It writes the result into the stack value indexed by 'targetVariable' (see "Variables").
138+
The logic to execute is defined by 'instructionType' (see "Instructions" in the script specification).
139+
140+
| Datatype | Name |
141+
| - | - |
142+
| short | argumentIndex |
143+
| short | argumentCount |
144+
| short | targetVariable |
145+
| short | instructionType |
146+
| int | zero |
147+
148+
### Argument
149+
150+
The argument structure declares one argument in an instruction.
151+
It declares its data type and the corresponding value.
152+
The base type of an argument can be calculated by taking its 4 least significant bits.
153+
154+
| Datatype | Name |
155+
| - | - |
156+
| int | type |
157+
| uint | value |
158+
159+
Base types of arguments typically found in scripts:
160+
| Basetype | Datatype | Notes |
161+
| - | - | - |
162+
| 1 | int | A signed integer value |
163+
| 2 | uint | An unsigned integer value |
164+
| 3 | float | A floating point value |
165+
| 4 | int | An index to a stack value (see "Variables") |
166+
| 8 | int | An absolute offset into the string blob. Normally SJIS-encoded. Null-terminated. |
167+
168+
## Variables
169+
170+
Variables represent values on the stack. There are multiple stack regions when a script is executed, each with their own implications.
171+
There are generally up to 1000 slots per stack region.
172+
173+
| Start | End | Description |
174+
| - | - | - |
175+
| 0 | 999 | Values, that persist through multiple scripts. Can contain any data, including primitive values and arrays. |
176+
| 1000 | 1999 | Values, that persist only in the function they were set in. 1000 is reserved as the return value for a function. Mostly used for primitive values. |
177+
| 2000 | 2999 | Values, that persist only in the function they were set in. Can contain any data, including primitive values and arrays. |
178+
| 3000 | 3999 | Values, that exclusively hold the input parameters to a function. Can contain any data, including primitive values and arrays. |
179+
| 4000 | 4999 | Values, that persist through multiple functions only in the script they were set in. Can contain any data, including primitive values and arrays. |

‎FunctionDictionary.json

-10
This file was deleted.

‎README.md

+39-8
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,42 @@
11
# XtractQuery
2-
A program to extract and recreate xq files from different 3DS games by Level5.<br>
3-
It currently support XSEQ and XQ32 files. Both use the extension .xq
42

5-
# Usage
6-
This is a command line tool.
3+
## Description
4+
A command line tool to de- and recompile .xq files from various 3DS games by Level5.<br>
5+
It supports all known format specifications.
76

8-
To see the help text explaining its options, type:
9-
```
10-
XtractQuery.exe -h
11-
```
7+
## Usage
8+
9+
Various options have to be set to properly use the command line tool.
10+
11+
| Option | Description |
12+
| - | - |
13+
| -h | Shows a help text explaining all the options listed here and examples on how to use use them. |
14+
| -o | The operation to execute. Has to be followed by either:<br>`d` to decompress a script<br>`e` to extract a script to human readable code<br>`c` to create a scripot from human readable code |
15+
| -t | The type of .xq file to process. Is only necessary for operation `c`. Has to be followed by either:<br>`xq32`<br>`xseq` |
16+
| -f | The file or directory to execute the operation on. |
17+
18+
### Method name mapping
19+
20+
In the file `methodMapping.json` instruction types, that are not known by the program (see "Instructions" in the script specification), can be mapped to a human readable name.<br>
21+
Since those unknown instructions are normally game specific logic, they have to be figured out by the user and added to the mapping for themselves.
22+
23+
If an unknown instruction type has no corresponding mapping, its name will be set to `subXXX`, where `XXX` is the instruction type.
24+
25+
### Reference scripts
26+
27+
Scripts can call methods from within themselves and other scripts currently loaded in the engine. Normally, those calls happen via the CRC32/CRC16 of the function name to invoke them.<br>
28+
To resolve those checksums back into human readable names, reference scripts can be placed in the folder `reference` next to the command line tool.
29+
30+
It is recommended to put every script of a game in the references to have the highest probability of properly resolving all checksums.<br>
31+
However, there is no guarantee that a checksum will be resolved, so user action has to be taken.
32+
33+
## Examples
34+
35+
To extract a script to human readable code:<br>
36+
```XtractQuery.exe -o e -f Path/To/File.xq```
37+
38+
To create a XQ32 script from human readable code:<br>
39+
```XtractQuery.exe -o c -t xq32 -f Path/To/File.txt```
40+
41+
To decompress the tables in a script (see "Compression" in format specification):<br>
42+
```XtractQuery.exe -o d -f Path/To/File.xq```

0 commit comments

Comments
 (0)