Very Secure

TheFleet Overview

My previous article detaling TheFleet's pseudocode was poorly worded and contained extraneous implementation details. This revision aims to give a concise summary of how TheFleet works.

In order to log the irc space TheFleet uses fleetbot, a class similar to logbot. Each instance of fleetbot connects to 1 network and joins the max channels allowed per nick on that network.i Once connected to its network, a fleetbot logs messages in a local postgres db.ii Certain messages - join, part, kick, and privmsgiii - are logged to a table named irclog. Events that are specific to a fleetbot (disconnection from the server, being kicked from a channel, joining a channel, and failing to reconnect) are logged to a separate table named fleetlog.iv

Fleetbots have an aggressive reconnection strategy. The code that receives and sends messages to the irc network is surrounded in a handler-case.v If a runtime error is signaled a fleetbot attempts to reconnect. A fleetbot also attempts to reconnect if it hasn't received a pong from the server in *max-lag* (60) seconds.vi

I still need a way to orchestrate the large number of fleetbots connected to various networks. Previously I ran all the fleetbots for 1 network together in their own sbcl process. This design needs to be discarded since each process takes over 40MB of RAM and my VMs only have 1GB available. Other constraints I have to deal with are: A VM costs $5 / month, most networks allow only 3 connections per ip, and I can only run 1,000 concurrent threads on each VM.vii

  1. If a network has 130 channels and a max 50chans/nick, TheFleet creates 3 fleetbots - 2 join 50 channels and 1 joins 30 channels. []
  2. This postgres db is shared amongst all fleetbots running on the same VM. []
  3. Privmsg is the name for a standard irc message you see in a direct message or in a channel. A privmsg is sent to a target. A target is either a nick or a channel. []
  4. I also currently print some debugging statements and unhandled messages from networks to standard-output, which I redirect to a log file when I run the program. I plan to remove this extra logging to save disk space once TheFleet is running in full gear. []
  5. I.e. a try/catch block. The robustness of this design is dubious since it allows for unknown errors to persist. []
  6. This behavior is inherited from fleetbot's superclass, ircbot. []
  7. The thread per VM limit is possibly adjustable. My understanding is that the limit comes from each thread allocating ~ 1MB for its own stack. []

2 Responses to “TheFleet Overview”

  1. > Other constraints I have to deal with are: A VM costs $5 / month

    What's this mean ?!

  2. whaack says:

    Assuming I run TheFleet for 2 months, I gotta pay 10 orcbux per ip. 100 VMs should be more than enough for my needs, and in fact I would probably be fine with 20 or so if I forgo logging channels with <3 people in them.

    Spending $200-$1,000 is not a problem, but I am not rich enough to spend it w/ 0 contemplation.

Leave a Reply