GSoC project: Tiled compatibility support Enigma #2302

KartikShrivastava · 2022-06-05T09:15:55Z

In A Nutshell - This project implements two importers: .tmx/.tsx file importers in RadialGM and .egm file importer in Tiled

My GSoC project spans into three PRs in three different projects of two separate organizations. The two organizations involved are Enigma-dev and Tiled. fundies and Josh are my mentors from Enigma-dev and bjorn is my mentor from Tiled.

My first PR (this one which is main PR) #2302 goes in enigma-dev/enigma-dev repository
Second PR #228 GSoC project: Tiled compatibility support RGM goes in enigma-dev/RadialGM repo
Third PR #3435 GSoC project: Enigma importer plugin goes in mapeditor/tiled repo

Requirement: You have to use Arch linux OS specifically, as Enigma-dev's RadialGM only compiles in Arch linux at the time of writing these steps. Compiling RadialGM can be tough, so feel free to ping in enigma-dev discord server.

Steps to test the project:

Compile Enigma-dev's RadialGM project and Tiled projects. (Note that Enigma-dev's enigma-dev project is a submodule of RadialGM so it gets included with RadialGM)
Open terminal and change directory to RadialGM, checkout to PR #228
Change directory to RadialGM/Submodules/enigma-dev, checkout to PR #2302
Setup a system environment variable named ENIGMA_PATH and set its value to RadialGM/Submodules/enigma-dev. This path will be used by Tiled editor's "enigma" plugin.
Change directory to Tiled, checkout to PR #3435
Compile RadialGM and Tiled projects separately
Run RadialGM project to open RadialGM editor, test the TMX importer by importing some .tmx file
Run Tiled project to open Tiled editor, enable "enigma" plugin by checking it in Edit-Preferences-Plugins tab and finally test the EGM importer by importing some .egm file

Short demo video of the final outcome: https://www.youtube.com/watch?v=ZUJd5VhqQo8
Weekly project work logs: https://kartikshrivastava.github.io/

Completed list of tasks in GSoC coding period:

Tiled TMX and TSX importer support status in RadialGM and enigma-dev:

Enigma's EGM importer support status in Tiled:

Add enigma plugin in Tiled
Update enigma plugin to depend on libEGM using qbs module
Add functionality to parse .egm file into protocol buffer
Add functionality to create a Tiled map using .egm based protocol buffer

Follow-up list of subtasks after GSoC:

Add proper separation of GameMaker resources between EGMRoom.proto and GMRoom.proto
Add translation routine to convert GMRoom to EGMRoom, after loading GameMaker based formats in GMRoom
Replicate Tiled animated tiles in RadialGM editor renderer and engine
Add parallax effect
Add more gtests
Add support to wangsets in TMX importer
Add support to rest of missing TMX features in EGMRoom (needs updates in RadialGM renderer and enigma enigne)
Fix Tiled ci which is breaking due to missing libEGM headers
Add support to missing features of internal tileset in EGMRoom

codecov · 2022-06-05T10:30:08Z

Codecov Report

Merging #2302 (82cc729) into master (2ddad07) will increase coverage by 0.26%.
The diff coverage is n/a.

❗ Current head 82cc729 differs from pull request most recent head 411f672. Consider uploading reports for the commit 411f672 to get more accurate results

@@            Coverage Diff             @@
##           master    #2302      +/-   ##
==========================================
+ Coverage   35.27%   35.54%   +0.26%     
==========================================
  Files         213      215       +2     
  Lines       20474    20498      +24     
==========================================
+ Hits         7223     7286      +63     
+ Misses      13251    13212      -39

see 10 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ddad07...411f672. Read the comment docs.

CommandLine/libEGM/tsx.cpp

CommandLine/libEGM/file-format.h

Co-authored-by: Thorbjørn Lindeijer <bjorn@lindeijer.nl>

fundies · 2022-06-15T18:45:44Z

CommandLine/libEGM/tsx.cpp

+          // PackRes(resMap, dir, ids, child, msg, depth + 1);
+          break;
+        }
+        case CppType::CPPTYPE_INT32: {


This should be the same between the xml-based readers. Feel free to abstract this and use an include rather than copy-pasting.

Alright will try

JoshDreamland

Still have to look over the actual TSX logic in good detail, but here's some initial comments. Looking good so far; thank you!

JoshDreamland · 2022-07-13T02:38:01Z

shared/protos/Room.proto

@@ -86,17 +86,18 @@ message Room {
    optional string name = 1;
    optional int32 id = 2 [(id_start) = 10000001];
    optional string background_name = 3 [(resource_ref)="background", (gmx) = "bgName"];
-    optional double x = 4;
-    optional double y = 5;
+    optional double x = 4 [(tmx) = "x"];


Do we need this property when the tmx field has the same name? You'll notice we omit the gmx attributes when the name is he same.

I'm only loading the values of those fields which has tmx extension, rest are ignored. I also realized that number of fields not directly corresponding to tmx attributes >> number of fields which have direct link. So I decided to keep tmx for same name fields as well.

JoshDreamland · 2022-07-13T02:48:04Z

shared/protos/Background.proto

-  optional int32 vertical_spacing   = 10 [(gmx) = "tilevsep"];
+  optional int32 tile_width         = 5 [(gmx) = "tilewidth", (tmx) = "tilewidth"];
+  optional int32 tile_height        = 6 [(gmx) = "tileheight", (tmx) = "tileheight"];
+  optional int32 horizontal_offset  = 7 [(gmx) = "tilexoff", (tmx) = "margin"]; // tiled use single integer for both...


Haven't seen the rest of the PR, yet, but are these annotations useful? Are we able to sort by tag number (in this case, 7 and 8) and just comma-separate, in all cases?

I didn't get that. Like sorting and comma-separate thing? Are you talking about proto fields extension, then it loads automatically. Here is loading logic https://github.com/KartikShrivastava/enigma-dev/blob/5614cfbd6d67e668d1170ce7a52f6ffb9f89d2f6/CommandLine/libEGM/General/tiled_util.cpp#L36

shared/libbase64_util/libbase64_util.cpp

JoshDreamland · 2022-07-13T02:56:14Z

ENIGMAsystem/SHELL/Universal_System/estring.cpp

 namespace enigma_user {

 bool is_base64(unsigned char c) {
-  return (isalnum(c) || (c == '+') || (c == '/'));
+  return ::is_base64(c);
 }

 string base64_encode(string const& str) {


In the header, you can just put using ::base64_encode; (etc) instead of declaring them as wrapper impls, here.

Did that but is not it less readable and more confusing?

JoshDreamland · 2022-07-13T02:59:13Z

CommandLine/libEGM/General/zlib_util.cpp

@@ -0,0 +1,389 @@
+#include "zlib_util.h"


Did Robert write this? Or is this Zlib-licensed Zlib code? We should include a license header either way.

I'm not sure if its zlib-licensed code, I copied it from gmk so copied its license.

JoshDreamland · 2022-07-13T03:01:00Z

CommandLine/libEGM/gmk.cpp

-  libpng_encode32_file(rgba.data(), w, h, write_fname.u8string().c_str());
-}
-
-void writeTempBGRAFile(const std::filesystem::path& write_fname, std::unique_ptr<char[]> bytes, size_t width, size_t height) {


Where did this logic go? Was it unused...? It does look a little hacky.

I copied it to zlib_util.cpp where its used, just didn't put it in Decoder class, as it wasn't there previously as well. Its static right now.

…mpression

…le[Width[Height]]

bjorn

Just a few comments based on browsing the latest changes.

CommandLine/libEGM/tmx.h

bjorn · 2022-08-19T09:58:30Z

CommandLine/libEGM/tmx.cpp

      tile->set_x(currX * (mapTileWidth - (hexMapUtil->hexSideLength / 2)));

-      if((currX%2 == 0 && hexMapUtil->staggerIndex == "odd") || (currX%2 != 0 && hexMapUtil->staggerIndex == "even"))
+      if((currX%2 == 0 && hexMapUtil->staggerUtil.staggerIndex == "odd") ||
+         (currX%2 != 0 && hexMapUtil->staggerUtil.staggerIndex == "even"))


If we're going to do these comparisons a lot (like, for each tile), it could be a good idea to avoid doing string comparisons.

Replaced string staggerIndex and staggerAxis with enum class members

CommandLine/libEGM/tmx.h

bjorn · 2022-08-19T10:05:19Z

CommandLine/libEGM/tmx.h

+  /**
+   * @brief Hex map is a staggered map
+   */
+  StaggerUtil staggerUtil;


There may be reasons to prefer aggregation, but in this case I would be inclined to derive HexMapUtil from StaggerUtil instead.

Regarding the naming, I'm a bit confused about the choice of "Util", since at least personally I expect a utility to do something rather than just store some data. Maybe "Info", "Parameters" or "Properties" would be more suitable (or shorter versions, if you prefer).

Admittedly I'm pretty bad at naming, updated those structs with Info suffix and inherited StaggerInfo from HexMapInfo

Also use GMRoom in Gamemaker specific formats

Add tsx src files, 1st tiled compat commit

a723e88

bjorn reviewed Jun 6, 2022

View reviewed changes

CommandLine/libEGM/tsx.cpp Outdated Show resolved Hide resolved

CommandLine/libEGM/tsx.cpp Outdated Show resolved Hide resolved

CommandLine/libEGM/file-format.h Show resolved Hide resolved

KartikShrivastava and others added 3 commits June 6, 2022 23:45

background_walker typo fix

6a8b85b

Co-authored-by: Thorbjørn Lindeijer <bjorn@lindeijer.nl>

Add tmx as a new proto option

cc8e869

Updates to load tsx into bg using reflection

3e1e699

fundies reviewed Jun 15, 2022

View reviewed changes

KartikShrivastava added 12 commits June 20, 2022 19:23

Add tmx source files

aec2829

Added tile load functionality to tmx

2d81313

Trashing pugi xml traversal impl

0ea925f

First iteration of TMX importer

477adc1

Fixed scaleC; Added rotation, alpha, hori/vert flip support

82cc729

Add method for layer data support, and some refactoring

49ab543

Moved base64 to shared from estring

829d4ef

Fix naming of base64 encoder/decoder in shared

a85d177

Moved zlib decoder from gmk to libEGM/General dir

e91e5f3

Add working code to load compressed zlib data tiles

21c2c6b

Fixed null pointer and localid bug

b3e0817

Refactored duplicate code, added shared/libbase64_util to make

5614cfb

JoshDreamland reviewed Jul 13, 2022

View reviewed changes

Refactoring, license copy paste

55673b2

KartikShrivastava requested a review from JoshDreamland July 14, 2022 04:54

KartikShrivastava added 8 commits July 16, 2022 17:44

Fix gid signed char bug and refactored tsx

593fb3d

Add support to csv encoded data layer tiles

0e04e22

Add zstd library for Tiled compression support

5270098

Fix zstd library name in Makefile pkg-config

ac9223b

Add zstd decompression support, update zlib_util to support gzip deco…

2cf6829

…mpression

Add missing code for uncompressed tile layer data load

3d5899f

Fixed zstd makefile bug and added background color fetch from TMX map

7273269

Added hexagonal map import support

c8415a8

KartikShrivastava added 7 commits July 25, 2022 12:28

Fixed a bug related mix-match of mapTile[Width[Height]] and tilesetTi…

96ac8a8

…le[Width[Height]]

Fixed tileset load error because of empty background name

a0ea4ea

Added docs to tsx header file

ebf6f07

Added tmx documentation and updates tsx docs

4f1b870

Added isometric map support

c50fff0

Added staggered map support and infinite Tiled map support

2c6040e

Add separation btw EGMRoom and GMRoom

2f555df

bjorn reviewed Aug 19, 2022

View reviewed changes

KartikShrivastava added 7 commits August 22, 2022 00:46

Replace Room with EGMRoom, use objects instead of tiles

1769b4e

Some typo and type fixes

2615040

Updates in tmx importer structure

a2866a1

New EGMRoom support iteration 2

068660c

Add chunk support in new EGMRoom itr 3

661784d

Fix a small bug pugi node iterator bug

aabdda1

Add gtest for tiled maps with objects

d7f0d50

KartikShrivastava changed the title ~~WIP: Tiled compatibility support Enigma~~ GSoC project: Tiled compatibility support Enigma Sep 8, 2022

This was referenced Sep 8, 2022

GSoC project: Tiled compatibility support RGM enigma-dev/RadialGM#228

Open

GSoC project: Enigma importer plugin mapeditor/tiled#3435

Open

fundies added the GSOC-Done label Feb 7, 2023

KartikShrivastava added 5 commits June 4, 2023 00:20

fix typo in tiled_util.h and a bug in yyp.cpp

76b2848

split-up combined Room into respective GM and EGM Rooms

c024ff7

Also use GMRoom in Gamemaker specific formats

rearrange EGMRoom.proto to make difference from GMRoom clear

50f1cf2

feat: add gm room to egm room translator

db73bd5

add gmtoegm translator to gmx, sog, yyp

411f672

KartikShrivastava force-pushed the kash/tiled-comp-enigma branch 4 times, most recently from e8e7edd to 411f672 Compare July 8, 2023 06:38

KartikShrivastava requested a review from fundies July 8, 2023 06:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GSoC project: Tiled compatibility support Enigma #2302

GSoC project: Tiled compatibility support Enigma #2302

KartikShrivastava commented Jun 5, 2022 •

edited

codecov bot commented Jun 5, 2022 •

edited

fundies Jun 15, 2022

KartikShrivastava Jun 16, 2022

JoshDreamland left a comment

JoshDreamland Jul 13, 2022

KartikShrivastava Jul 13, 2022

JoshDreamland Jul 13, 2022

KartikShrivastava Jul 13, 2022

JoshDreamland Jul 13, 2022

KartikShrivastava Jul 13, 2022

JoshDreamland Jul 13, 2022

KartikShrivastava Jul 13, 2022

JoshDreamland Jul 13, 2022

KartikShrivastava Jul 13, 2022

bjorn left a comment

bjorn Aug 19, 2022

KartikShrivastava Aug 22, 2022 •

edited

bjorn Aug 19, 2022

KartikShrivastava Aug 22, 2022

GSoC project: Tiled compatibility support Enigma #2302

Are you sure you want to change the base?

GSoC project: Tiled compatibility support Enigma #2302

Conversation

KartikShrivastava commented Jun 5, 2022 • edited

In A Nutshell - This project implements two importers: .tmx/.tsx file importers in RadialGM and .egm file importer in Tiled

Completed list of tasks in GSoC coding period:

Follow-up list of subtasks after GSoC:

codecov bot commented Jun 5, 2022 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoshDreamland left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bjorn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KartikShrivastava Aug 22, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KartikShrivastava commented Jun 5, 2022 •

edited

codecov bot commented Jun 5, 2022 •

edited

KartikShrivastava Aug 22, 2022 •

edited