Auto Maple

Auto Maple is an intelligent Python program that plays MapleStory, a 2D side-scrolling MMORPG, using simulated key presses, TensorFlow machine learning, OpenCV template matching, and other computer vision techniques.

Minimap

Auto Maple uses OpenCV template matching to determine the bounds of the minimap as well as the various elements within it, allowing it to accurately track the player's in-game position. If record_layout is set to True in the routine file, Auto Maple will record the player's previous positions in a quadtree-based Layout object, which is periodically saved to a file in the "layouts" directory. Every time a new routine is loaded, its corresponding layout file, if it exists, will also be loaded. This Layout object uses the A* search algorithm on its stored points to calculate the shortest path from the player to any target location, which can dramatically improve the accuracy and speed at which routines are executed. _{Click here to view the above routine.}

Command Books

_{The above video shows Auto Maple consistently performing a mechanically advanced ability combination.}

Designed with modularity in mind, Auto Maple can operate any character in the game as long as it is provided with a list of in-game actions, or a "command book". A command book is a Python file that contains multiple classes, one for each in-game ability, that tells the program what keys it should press and when to press them. Once a command book is imported, its classes are automatically compiled into a dictionary that Auto Maple can then use to interpret commands within routines. Commands have access to all of Auto Maple's global variables, which can allow them to actively change their behavior based on the player's position and the state of the game.

Routines

_{Click here to view the entire routine.}

A routine is a user-created CSV file that tells Auto Maple where to move and what commands to use at each location. A custom-made interpreter within Auto Maple parses through the selected routine and converts it into a list of objects that can then be executed by the program. An error message is printed for every line that contains invalid parameters, and those lines are ignored during the conversion.

Points are created using *, <x position>, <y position>
Each point stores the commands below it and will execute them in that order once the character reaches that point. There are also a couple optional keyword arguments:
_{- "adjust": Fine-tunes character position to be within adjust_tolerance (a routine setting).}
_{- "frequency": How often to execute this point.}
_{- "counter": Set's the initial value of this point's counter. The counter increments every cycle (wrapped back to 0 at frequency) and the point will only execute if its counter is 0.}

Labels are created using @, <label name>
They can be jumped to using the "goto" command, which allows users to create loops and organize routines into sections.

Commands are created using <command name>, <p1>, <p2>, ...
The <command name> corresponds with the class names inside command book files, and <p1>, <p2>, ... correspond with the class's __init__ parameters. Keyword arguments are also supported.

Settings are updated using s, <setting name>, <value>
All the editable settings can be found at the bottom of config.py.

Runes

Auto Maple has the ability to automatically solve "runes", or in-game arrow key puzzles. It first uses OpenCV's color filtration and Canny edge detection algorithms to isolate the arrow keys and reduce as much background noise as possible. Then, it runs multiple inferences on the preprocessed frames using a custom-trained TensorFlow model until two inferences agree. Because of this preprocessing, Auto Maple is extremely accurate at solving runes in all kinds of (often colorful and chaotic) environments.

Video Demonstration

Click below to see Auto Maple in action:

Reflection

From working on this project, I truly learned a lot. Auto Maple not only allowed me to apply the knowledge I learned in lectures to a challenging problem, but it also introduced me to many exciting and complex concepts such as machine learning and Canny edge detection. However, perhaps more importantly, working on Auto Maple has given me a deeper appreciation of human problem solving and a clearer understanding of just how hard it is for a computer to emulate that.

Early in this project, when I was still trying to get the character to move to locations on the minimap, I resorted to letting the program blindly move horizontally and vertically until it reached its target. This was very inefficient and often resulted in the character getting stuck. Later, I realized that memory was the main aspect Auto Maple was missing: human gamers remember where they've been, which places are safe to walk on, and which places aren't. This gave me the idea to create a Layout class to help the program chart a path in advance based on where it has already been, much like a human.

In the end, Auto Maple is still far from perfect. It can't predict and prevent missteps like we can. It can't reproduce the fluid actions of a practiced gamer. But acknowledging these shortcomings inspires me to continue learning and experimenting with new ways to make programs faster and smarter.

yyzhou94 / auto-maple