author: William 'psyFi' Hatcher <william@hatcher.work> 2017-08-13 13:32:12 +0200
committer: Tiger Wang <ziwei.tiger@outlook.com> 2017-08-13 13:32:12 +0200
commit: 0cd8482f07aadab6de27aa7395de5408737093ad (patch)
tree: f6809d1a12db5ed885a98f260bdf95a21e3c2343 /dev-docs/Cubeset file format.html
parent: Replaced includes with forward declarations (diff)
download: cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar
cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.gz
cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.bz2
cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.lz
cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.xz
cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.zst
cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.zip
1 files changed, 325 insertions, 0 deletions
diff --git a/dev-docs/Cubeset file format.html b/dev-docs/Cubeset file format.html
new file mode 100644
index 000000000..6ead2e700
--- /dev/null
+++ b/dev-docs/Cubeset file format.html
@@ -0,0 +1,325 @@
+<html>
+<head>
+<title>Cubeset file format</title>
+<link rel="stylesheet" type="text/css" href="style.css">
+</head>
+<body>
+<h2>Contents</h2>
+<ul>
+<li><a href="#abstract">Abstract - what and why</a></li>
+<li><a href="#details">Detailed description of the format</a>
+<ul>
+<li><a href="#csmeta">Cubeset metadata</a></li>
+<li><a href="#piece">Individual piece</a></li>
+<li><a href="#piecemeta">Piece metadata</a></li>
+</ul>
+</li>
+<li><a href="#example">Example</a></li>
+</ul>
+<hr>
+<a name="abstract"><h2>Abstract - what and why</h2></a>
+<p>We're seeing an increased need to store "prefabs" - little areas with predefined block contents, such as village houses or fortress rooms - in collections. We have one collection of village houses for the plains village, one collection for the desert village, one collection for the nether fortress... And there are plans in the future to use even more collections - trees, overworld fortresses, more village types and even custom structures. The point that they have in common is that they need to store not only the prefabs, but also metadata for those prefabs - how often they generate, how they connect together. There's even need for metadata for the entire collection, such as what the accepted biomes are, what block should village roads use, and various other generator parameters. So we need a file format that could store all this information together.</p>
+<p>There are some existing formats available to consider first:
+<ul>
+<li><a href="http://minecraft.gamepedia.com/Schematic_file_format"><b>schematic</b></a> - file format native to MCEdit / Bukkit / WorldEdit communities. Can store the prefab, the block entities and regular entities, and any metadata. Cannot store multiple prefabs. No effort to read or write, there's already code to do that (except for the entities) in the server.</li>
+<li><a href="https://dev.bukkit.org/projects/terrain-control"><b>bob / bo2 / bo3</b></a> - file format created for prefabs in the Terrain Control mod. Can store the prefab and any metadata. Support for block entities and regular entities is unknown. Cannot store multiple prefabs. Medium difficulty for reading and writing, would need new parser and serializer. Unknown (but assumed true) whether the format truly supports any metadata.</li>
+<li><a href="https://github.com/cuberite/cuberite/tree/master/src/Generating/Prefabs"><b>cpp</b></a> - export from our Gallery server directly into C++ source files. Can store the prefab and any metadata, block entities and regular entities currently not implemented but could be added. Very difficult for reading, writing already implemented. Only usable when compiling directly into the server. Can store multiple prefabs and metadata for the entire set.</li>
+</ul>
+Obviously none of these fully satisfy our needs, so we'll need to either extend one of them or create yet another one. Extending the .schematic file would mean that the exporter plugin would need to change most of the export code, which was deemed too unmaintainable. Because the bob format is not implemented at all, it wasn't even considered. The cpp format would have been a great candidate if it weren't so difficult to parse. However, it sparked an idea - something similar in form to the cpp format, but easily parsed. Since we already have the Lua interpreter, why not base the new format in Lua?</p>
+<p>
+With Lua, we could store any metadata for the prefabs, any additional information related to the entire set of prefabs. There's nothing stopping us from adding more items in a backward- and forward-compatible way. The prefabs can be stored very similar to the cpp format, an array of strings plus a charmap, or they can be stored externally in individual .schematic files and the Lua file would provide just the metadata. The server has already vast amounts of Lua-related support code that can be used for accessing the data. In the end this became the chosen solution. The format has been named "Cubeset" - a set of cube-based prefabs.</p>
+
+<hr/>
+
+<a name="details"><h2>Detailed description of the format</h2></a>
+<p>
+The Cubeset file has a .cubeset extension. Internally it is a Lua source file that provides a global value, Cubeset, which is a table containing the structured data. The loader checks the file's first 8 KiB to see if there is a "CubesetFormatVersion =" string in it (note the space between the word and the equals-sign), and if it is, the file is considered a Cubeset file and further loading is attempted. It is therefore crucial that tools producing this file format include this string as early as possible in the file.</p>
+<p>
+The top-level Cubeset table must contain at least two sub-tables: Metadata and Pieces. The Metadata table contains the metadata relevant to the entire set of prefabs in the file, the Pieces table contains the definitions and metadata for the individual prefabs. It is advised to make the Metadata table the first one, because it contains the signature used to identify the file ("CubesetFormatVersion ="). Apart from these two subtables the server ignores everything else.</p>
+
+<a name="csmeta"><h3>Cubeset metadata</h3></a>
+<p>
+The Cubeset.Metadata table is used to store metadata for the entire set of prefabs, and also for the format and version identification. It is a regular dictionary-style Lua table; the following elements are recognized:
+<table>
+<tr><th>Name</th><th>Type</th><th>Content</th></tr>
+<tr><td>CubesetFormatVersion</td><td>number</td><td>This is the format identification and at the same time it specifies the version of the file. Currently the file version is 1. Note that Cuberite checks the presence of the string "CubesetFormatVersion =", including the space between the word and the equals-sign, within the first 8 KiB of the file.</td></tr>
+<tr><td>ExportDate</td><td>string</td><td>The date when this file was exported, in the ISO format ("2015-06-16 13:34:03"). Inserted by <a href="https://github.com/madmaxoft/GalExport">GalExport</a> for versioning purposes. Ignored elsewhere.</td></tr>
+<tr><td>ExternalSchematic</td><td>boolean</td><td>Flag inserted by <a href="https://github.com/madmaxoft/GalExport">GalExport</a> to indicate that the individual prefabs are stored in separate .schematic files. Ignored elsewhere.</td></tr>
+<tr><td>IntendedUse</td><td>string</td><td>String identifying the generator part that the cubeset file is intended for. The server logs a warning when loading a cubeset file without an IntendedUse metadata; individual generators log a warning if a wrong IntendedUse is detected in a file they are asked to use.</td></tr>
+</table>
+</p>
+<p>Additional values are recognized by the specific generator (which is indicated by the IntendedUse value): </p>
+<table><tr><th>Generator (IntendedUse)</th><th>Name</th><th>Type</th><th>Content</th><th>Notes</th></tr>
+<tr><td>Village / PieceStructures / Trees</td><td>AllowedBiomes</td><td>string</td><td>Comma-separated list of biomes</td><td>The generator only generates the structure / village / tree in the specified biomes. If empty, all biomes are eligible.</td></tr>
+<tr><td rowspan=6>Village</td><td>MaxDensity</td><td>number</td><td>Maximum density (0 - 100) at which the connectors are populated.</td><td rowspan=2>The village generator picks a density between Min and Max, and then only that many percent of the free connectors are actually attempted. This eventually reduces the number of houses to make the village look more "roomy".</tr>
+<tr><td>MinDensity</td><td>number</td><td>Minimum density (0 - 100) at which the connectors are populated.</td></tr>
+<tr><td>VillageRoadBlockType</td><td>number</td><td>The block type used in the village for regular roads on the solid surface</td><td rowspan=4>The generator replaces the top terrain block with the specified block type and meta to form the roads. The generator can distinguish when it's replacing water and when it's replacing regular terrain, allowing the villages to include "bridges" as their roads.</td></tr>
+<tr><td>VillageRoadBlockMeta</td><td>number</td><td>The block meta used in the village for regular roads on the solid surface</td></tr>
+<tr><td>VillageWaterRoadBlockType</td><td>number</td><td>The block type used in the village for roads on the surface of water</td></tr>
+<tr><td>VillageWaterRoadBlockMeta</td><td>number</td><td>The block meta used in the village for roads on the surface of water</td></tr>
+<tr><td rowspan=8>PieceStructures</td><td>GridSizeX</td><td>number</td><td rowspan=2>Size, in blocks, of the seed grid</td><td rowspan=2>The generator starts each generated structure in a "seed", these two parameters control the (average) distances between two adjacent seeds.</td></tr>
+<tr><td>GridSizeZ</td><td>number</td></tr>
+<tr><td>MaxOffsetX</td><td>number</td><td rowspan=2>Maximum offset, in blocks, of the seed from the grid's center</td><td rowspan=2>The generator starts each generated structure in a "seed", these two parameters control the maximum distance of the seed from the regular grid (specified by GridSizeX and GridSizeZ). When zero, the structures are generated exactly on a rectangular grid. Recommended value is about half of the grid's size.</td></tr>
+<tr><td>MaxOffsetZ</td><td>number</td></tr>
+<tr><td>MaxStructureSizeX</td><td>number</td><td rowspan=2>Size, in blocks, of the bounding box for a single structure.</td><td rowspan=2>The generator will not connect any prefab to the rest of the structure if it crosses the bounding box.</td></tr>
+<tr><td>MaxStructureSizeZ</td><td>number</td></tr>
+<tr><td>MaxDepth</td><td>number</td><td>Maximum depth of the generated piece tree</td><td>The maximum number of connectors, away from the starting piece</td></tr>
+<tr><td>SeedOffset</td><td>number</td><td>Number that is added to world's seed for this generator</td><td>Each cubeset file should have a different seed offset, otherwise it runs the risk of its structures being generated directly over other cubeset file's that the server admin has enabled. Since the seed is used for placement, if two cubeset files share the same seed, they will share the structure placement as well. </td></tr>
+
+</table>
+
+<a name="piece"><h3>Individual piece</h3></a>
+<p>
+The Cubeset.Pieces table is an array containing individual prefabs. Each element describes a single prefab and its associated metadata. The following values are recognized:
+<table>
+<tr><th>Name</th><th>Type</th><th>Content</th></tr>
+<tr><td>OriginData</td><td>table</td><td>Inserted by <a href="https://github.com/madmaxoft/GalExport">GalExport</a> to identify the gallery area from which the prefab is exported. Ignored elsewhere.</td></tr>
+<tr><td>Hitbox</td><td>table</td><td>The relative coords of the prefab's hitbox (where the prefab is not allowed to overlap another prefab when generating). Members: MinX, MinY, MinZ, MaxX, MaxY, MaxZ, all numbers.</td></tr>
+<tr><td>Connectors</td><td>table</td><td>Array of <a href="#conndef">connector definitions</a>. The table must be present for each prefab, even if the prefab doesn't have any connectors (use an empty table, then).</td></tr>
+<tr><td>SchematicFileName</td><td>string</td><td>Name of the .schematic file that contains the block data for the prefab.</td></tr>
+<tr><td>Size</td><td>table</td><td>Table containing the dimensions of the prefab, if it is inlined into the BlockData element. Contains three members, x, y, z, each is a number.</td></tr>
+<tr><td>BlockData</td><td>table</td><td>Array of strings that are processed to produce the block data for the prefab. Each letter in the string corresponds to one block, the type of the block is translated through the BlockDefinitions table. The blocks are ordered YZX, that is, the X coord varies the most.</td></tr>
+<tr><td>BlockDefinitions</td><td>table</td><td>Array of strings that defines the mapping of each letter in BlockData to a specific blocktype. Each string should have the format "Letter: BlockType: BlockMeta".</td></tr>
+<tr><td>Metadata</td><td>table</td><td>Dictionary-style table of various per-prefab <a href="#piecemeta">metadata values</a>.</td></tr>
+</table>
+The prefab can either have the SchematicFileName element, in which case the specified schematic file is loaded as the block data, or it can have the Size, BlockData and BlockDefinitions elements, then the server parses the block data from those. If both data members are included, the SchematicFileName takes precedence and the server loads the data from the schematic file (note that this behavior may change, having both definitions is considered "undefined behavior").</p>
+<a name="conndef"><p>
+The connector definitions table is an array of tables, each element describing one connector. The following values are recognized:
+<table>
+<tr><th>Name</th><th>type</th><th>Content</th></tr>
+<tr><td>Type</td><td>number</td><td>The connector's type. The piece generator will only connect the connectors of inverse types ('1'-type connector will connect only to '-1'-type connector).</td></tr>
+<tr><td>RelX</td><td>number</td><td>X coord of the connector, relative to the prefab's zero point ({0, 0, 0} - the first block in the image).</td></tr>
+<tr><td>RelY</td><td>number</td><td>Y coord of the connector, relative to the prefab's zero point ({0, 0, 0} - the first block in the image).</td></tr>
+<tr><td>RelZ</td><td>number</td><td>Z coord of the connector, relative to the prefab's zero point ({0, 0, 0} - the first block in the image).</td></tr>
+<tr><td>Direction</td><td>number</td><td>The direction in which the connector is facing. Corresponds to the eBlockFace constants:
+	<table>
+	<tr><th>Value</th><th>Direction</th></tr>
+	<tr><td>0</td><td>Y-</td></tr>
+	<tr><td>1</td><td>Y+</td></tr>
+	<tr><td>2</td><td>Z-</td></tr>
+	<tr><td>3</td><td>Z+</td></tr>
+	<tr><td>4</td><td>X-</td></tr>
+	<tr><td>5</td><td>X+</td></tr>
+	</table>
+</td></tr>
+</table>
+If a connector definition is missing any of the fields, the server will not add the connector to the prefab upon loading. If a prefab doesn't have any connectors, it still needs to provide an empty Connectors table.</p></a>
+
+<a name="piecemeta"><h3>Piece metadata</h3></a>
+<p>
+Each piece contains additional metadata describing its properties. The server ignores metadata that it doesn't understand. The following values are recognized:
+<table>
+<tr><th>Name</th><th>Type</th><th>IsRequired</th><th>Contents</th></tr>
+<tr><td>IsStarting</td><td>number</td><td>Yes</td><td>Zero means that the piece is a regular piece, nonzero means that the piece is a starting piece (the "seed" of the structure). Required even for cubesets that don't represent a piece-generator data (such as trees). </td></tr>
+<tr><td>AllowedRotations</td><td>number</td><td>&nbsp;</td><td>Number representing a bitmask for which rotations the piece supports. Defaults to 0 (no rotations). Bit 0 (value 1) represents whether 1 counter-clockwise rotation is allowed, bit 1 (value 2) represents whether 2 rotations (180 degrees) are allowed, bit 2 (value 4) represents whether 1 clockwise rotation is allowed.</td></tr>
+<tr><td>AddWeightIfSame</td><td>number</td><td>&nbsp;</td><td>How much weight (chance to generate) should the piece generator add to this piece when the parent piece is the same. It is possible to have negative values, meaning that the piece doesn't like repeating itself. Defaults to 0.</td></tr>
+<tr><td>DefaultWeight</td><td>number</td><td>&nbsp;</td><td>How much weight (chance to generate) does the piece have by default, without any modifiers (AddWeightIfSame, DepthWeight). Defaults to 0.</td></tr>
+<tr><td>DepthWeight</td><td>string</td><td>&nbsp;</td><td>Override for DefaultWeight for specific depth (in the tree used by the piece generator). String in the format "Depth1:Weight1|Depth2:Weight2|...". Each unlisted depth gets the DefaultWeight. Defaults to empty string (no override).</td></tr>
+<tr><td>MergeStrategy</td><td>string</td><td>&nbsp;</td><td>Which merge strategy should be used when drawing the prefab into the world. String representation of one of the cBlockArea:eMergeStrategy constants: "msOverwrite", "msFillAir", "msImprint", "msLake", "msSpongePrint", "msDifference", "msSimpleCompare", "msMask". Defaults to "msSpongePrint".</td></tr>
+<tr><td>MoveToGround</td><td>number</td><td>&nbsp;</td><td>Zero means that the piece will stay where generated by the piece generator, nonzero means that the piece will be moved Y-wise so that its first connector will be on the top block of the existing terrain. Useful for village houses. Defaults to 0.</td></tr>
+<tr><td>ShouldExpandFloor</td><td>number</td><td>&nbsp;</td><td>Nonzero means that the prefab's lowest slice will be repeated downwards until it hits a solid block, effectively creating a foundation for the piece. Useful for nether fortresses and village houses. Defaults to 0.</td></tr>
+</table>
+Each value that should be a number also allows a string that represents a number. This makes it easier for automated exporters - they can export all values as strings.</p>
+
+<hr/>
+
+<a name="example"><h2>Example</h2></a>
+<p>
+The following example defines a cubeset with two pieces. The first piece is inlined into the cubeset file, the second piece uses an external schematic file.</p>
+<pre>
+Cubeset =
+{
+	Metadata =
+	{
+		CubesetFormatVersion = 1,
+		IntendedUse = "PieceStructures",
+		GridSizeX = 128,
+		GridSizeZ = 128,
+		MaxStructureSizeX = 64,
+		MaxStructureSizeZ = 64,
+		MaxOffsetX = 16,
+		MaxOffsetZ = 16,
+		MaxDepth = 4,
+		SeedOffset = 13,
+	},
+
+	Pieces =
+	{
+		-- The following piece was exported from the Gallery server by the GalExport plugin in the "cubeset" format:
+		{
+			OriginData =
+			{
+				ExportName   = "DarkCorridor",
+				Name         = "Nether 3",
+				GalleryName  = "Nether",
+				GalleryIndex = "3",
+				ID           = "30",
+				CreatorName  = "STR_Warrior",
+			},
+			Size =
+			{
+				x = 14,
+				y = 6,
+				z = 5,
+			},
+			Hitbox =
+			{
+				MinX = 0,
+				MinY = 0,
+				MinZ = 0,
+				MaxX = 13,
+				MaxY = 5,
+				MaxZ = 4,
+			},
+			Connectors =
+			{
+				{
+					Type = 1,
+					RelX = 0,
+					RelY = 1,
+					RelZ = 2,
+					Direction = 4,  -- X-
+				},
+				{
+					Type = 1,
+					RelX = 13,
+					RelY = 1,
+					RelZ = 2,
+					Direction = 5,  -- X+
+				},
+				{
+					Type = -1,
+					RelX = 0,
+					RelY = 1,
+					RelZ = 2,
+					Direction = 4,  -- X-
+				},
+				{
+					Type = -1,
+					RelX = 13,
+					RelY = 1,
+					RelZ = 2,
+					Direction = 5,  -- X+
+				},
+			},
+			Metadata =
+			{
+				["DefaultWeight"] = "100",
+				["IsStarting"] = "0",
+				["AllowedRotations"] = "7",
+				["MergeStrategy"] = "msSpongePrint",
+				["DepthWeight"] = "",
+				["ShouldExpandFloor"] = "1",
+				["MoveToGround"] = "0",
+				["AddWeightIfSame"] = "0",
+			},
+			BlockDefinitions =
+			{
+				".:  0: 0",  -- air
+				"a:112: 0",  -- netherbrick
+				"b:113: 0",  -- netherbrickfence
+				"c:114: 2",  -- netherbrickstairs
+				"d:114: 3",  -- netherbrickstairs
+				"m: 19: 0",  -- sponge
+			},
+			BlockData =
+			{
+				-- Level 0
+				"aaaaaaaaaaaaaa",  --  0
+				"aaaaaaaaaaaaaa",  --  1
+				"aaaaaaaaaaaaaa",  --  2
+				"aaaaaaaaaaaaaa",  --  3
+				"aaaaaaaaaaaaaa",  --  4
+
+				-- Level 1
+				"aaaaaaaaaaaaaa",  --  0
+				"..............",  --  1
+				"..............",  --  2
+				"..............",  --  3
+				"aaaaaaaaaaaaaa",  --  4
+
+				-- Level 2
+				"aabaaaaaaaabaa",  --  0
+				"..............",  --  1
+				"..............",  --  2
+				"..............",  --  3
+				"aabaaaaaaaabaa",  --  4
+
+				-- Level 3
+				"aabaaaaaaaabaa",  --  0
+				"..............",  --  1
+				"..............",  --  2
+				"..............",  --  3
+				"aabaaaaaaaabaa",  --  4
+
+				-- Level 4
+				"aabaaaaaaaabaa",  --  0
+				"..............",  --  1
+				"..............",  --  2
+				"..............",  --  3
+				"aabaaaaaaaabaa",  --  4
+
+				-- Level 5
+				"cccccccccccccc",  --  0
+				"aaaaaaaaaaaaaa",  --  1
+				"aaaaaaaaaaaaaa",  --  2
+				"aaaaaaaaaaaaaa",  --  3
+				"dddddddddddddd",  --  4
+			},
+		},  -- DarkCorridor
+
+		-- The following piece was exported from the Gallery server by the GalExport plugin in the "cubesetext" format:
+		{
+			OriginData =
+			{
+				ExportName   = "DoublePlantBed",
+				Name         = "Plains 5",
+				GalleryName  = "Plains",
+				GalleryIndex = "5",
+				ID           = "20",
+				CreatorName  = "tonibm1999",
+			},
+			Size =
+			{
+				x = 15,
+				y = 8,
+				z = 9,
+			},
+			Hitbox =
+			{
+				MinX = 0,
+				MinY = 0,
+				MinZ = 0,
+				MaxX = 14,
+				MaxY = 7,
+				MaxZ = 8,
+			},
+			Connectors =
+			{
+				{
+					Type = -1,
+					RelX = 7,
+					RelY = 2,
+					RelZ = 8,
+					Direction = 3,  -- Z+
+				},
+			},
+			Metadata =
+			{
+				["DefaultWeight"] = "100",
+				["IsStarting"] = "0",
+				["AllowedRotations"] = "7",
+				["MergeStrategy"] = "msSpongePrint",
+				["DepthWeight"] = "",
+				["ShouldExpandFloor"] = "1",
+				["MoveToGround"] = "1",
+				["AddWeightIfSame"] = "0",
+			},
+			SchematicFile = "PlainsVillage/20.schematic",
+		},  -- DoublePlantBed
+	}  -- Pieces
+}
+</pre>
+
+</body>
+</html>
author	William 'psyFi' Hatcher <william@hatcher.work>	2017-08-13 13:32:12 +0200
committer	Tiger Wang <ziwei.tiger@outlook.com>	2017-08-13 13:32:12 +0200
commit	0cd8482f07aadab6de27aa7395de5408737093ad (patch)
tree	f6809d1a12db5ed885a98f260bdf95a21e3c2343 /dev-docs/Cubeset file format.html
parent	Replaced includes with forward declarations (diff)
download	cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.gz cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.bz2 cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.lz cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.xz cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.tar.zst cuberite-0cd8482f07aadab6de27aa7395de5408737093ad.zip