Project Inspiration
Me and I friends recently started playing Minecraft again, so naturally, I began thinking about how I can build a project that improves my overall enjoyment while playing.
THEN IT HIT ME
Imagine having a wingman with you in your Minecraft world, AND having that wingman be able to give you intelligent callouts based on the actual game state.
So that's exactly what I'm building. I'm building a fork of my existing BT project that gives BT access to Minecraft game state and, eventually, an actual mech suit in the game.
BT's model in Titanfall 2, for reference
Planning and constraints
While I would love to just give BT access to everything in the game, this would likely result in BT getting too much context and giving bad responses. Therefore, we can wrap this information in Tools, then write Skills to tell BT how and when to use these Tools to get information about the game!
This approach does require that we manually write information about how this data is provided, when to call upon the Tools, how to use them, etc. But this approach will make it easier to maintain in the future.
So now that BT can fetch context, what context can we actually give?
To answer this question, I had to dive deep into how modding APIs interact with Minecraft, the difference between client and server mods, and how quickly this data is updated and provided.
Overall, this table compares the differences:
| Approach | Runs on | What BT can see | How often it updates | The catch |
|---|---|---|---|---|
| Fabric Client | My game | My health, hunger, inventory, chat, and mobs or blocks inside my render distance | Every client tick (20/sec), or every frame if needed | Only knows what the player knows |
| Fabric Server / NeoForge | The server | Every player, entity, and loaded chunk | Every server tick (20/sec) | I have to run the server |
| Paper | The server | About the same as a server mod, through Paper's event API | On events, plus scheduled tasks each tick | Needs a Paper server, which doesn't mix with Fabric mods |
| Bot client (Mineflayer) | Node.js client that joins as a player | Same as client, but with its own player | At game speed | Needs its own account and player slot, and it sees from the bot's position, not mine |
| RCON | The server, over the network | Only what commands print, like /list or /data get | Only when BT asks | Slow, text-only output that BT has to parse |
A few of these approaches can easily be ruled out:
- RCON, which only gives us access to commands
- Mineflayer, which requires building BT to literally play the game as a player
- Paper, since my server runs Fabric
That leaves us with choosing between running this on the server or on the clients.
The biggest difference between these is who needs to install it. Server-side mods need to be installed by everyone that wants to play on the server, but client-side mods only need to be installed by whoever wants them. This means that if only I want to use BT, I can install it for me, and only me. However, this does limit BT to information known to the player AND doesn't allow us to implement custom items, entities, etc.
Therefore, I'm going to take a two-stage approache:
- Implement a client-side mod to test BT's responses, access to information, skills, etc.
- Implement a server-side mod to:
- Add BT's mech body to the game
- Add BT's other items/crafting recipes to the game
- Restrict BT functionality behind item gates in survival (i.e., "Craft AI core before you can chat with BT")
Based on this, there are MANY features that come to mind.
To name a few:
- AI Core, which enables chatting with BT, without any way to access game data (no sensors)
- Helmet that establishes a "link" with BT, allowing him to see what you see (full client-side data access)
- Data Sensors, which give BT remote access to view information close to those sensors (partial remote data)
- I'm thinking cheap to make but less view than a player/mech (maybe 2x2 chunks or smaller?)
- Mech Suit with sensors and combat abillities
- Plus ammo for reloading
- Arm mods for different guns/sensors?
- Mining?
- Orbital Launcher that gives the ability for Titanfall (dropping the mech to your location)
ALL of these features would require a server-side mod, so that should be the final implementation goal
In case my writing doesn't reflect it, I'm so excited to be working on this!