whaack

So far TheFleet logged 52 out of 100 attempted networks.¹I began logging another batch of 45 networks on Fleet2. Fleet3 is currently inactive and waiting for assignment.

Before giving Fleet3 its next assignment, I need to pay some technical debt. The programs I use to query the irc networks for their metadata² are poorly written and undocumented.

Channel-snagger, the program used for grabbing the list of all channels, needs to be rewritten. It currently keeps a cache of the list of channels for individual networks and never updates this cache. This made sense when I first used the program, but now the networks' channel lists are stale. I could simply clear the cache, but the better solution is to make the program more efficient and reliable so caching is not required at all.

Another quirk of the channel-snagger is the makeshift way it handles the ping/pong dance with the servers. Networks require a ping in response to their pongs every N seconds. As a half-hearted attempt to keep the connection alive, I hard coded channel-snagger to send one ping after 30 seconds. Otherwise channel-snagger has no ping/pong logic. So if it takes too much time to receive the channel list from the network, the connection may be dropped for failure to perform the ping/pong routine. This can be fixed by refactoring channel-snagger so that it extends ircbot, which implements a proper ping/pong thread.

The code that queries the network for its maximum number of channels allowed per nick also needs a rewrite. It currently exists as a disorganized clump of scripts. I plan to consolidate them into the channel-snagger.

The above components are ultimately needed for keeping an up to date pool of the next channels to assign for logging. The only other missing element I can think of is a process for removing networks/channels that have already been logged (or have been deemed unloggable) from this pool. Once I have a system I am content with for assigning the next channels to my VMs running TheFleet, I will plan how to analyze the collected data.

1. For a network to be considered logged, at least 1 channel in the network was logged for 2 continuous weeks without any 1 interruption lasting more than 30 minutes. [↩]
2. Their list of channels and their max number of channels allowed per nick. [↩]

Over the past few weeks I attempted to log about 2,500 channels across 100 IRC networks. TheFleet ran on 2 VMs named Fleet2 and Fleet3.¹ Fleet2 was originally logging ~80 networks, but after about 9 days the program crashed from running out of memory. I restarted Fleet2, and to reduce the chance of the OOM problem repeating,² I set Fleet2 to log only 35 networks.

The summary of the data collected:³

VM 1 - "Fleet2"

Began logging on March 4th, 2020 at 23:42 UTC and ended logging on March 27th, 2020 at 21:42 UTC.

Logged 2,490,376 IRC messages.

Attempted to log 35 networks.⁴

Managed to produce logs for 17 of the 35 attempted networks.

Joined 547 channels and succesfully logged 463.⁵

VM 2 - "Fleet3"

Began logging on February 21st, 2020 at 03:41 UTC and ended logging on March 28th, 2020 at 21:46 UTC.

Logged 5,160,930 IRC messages.

Attempted to log 65 networks.⁶

Managed to produce logs for 35 of the 65 attempted networks.

Joined 1929 channels and successfully logged 578.⁷

1. You may be curious as to what happened to Fleet1. I had a problem with digitalocean's service. I lost the ability to SSH into my machine and could not recover access. I never started logging with TheFleet on this VM, so I gave up trying to regain access and instead created a new VM. [↩]
2. I still do not have an understanding of what causes/caused my program to spike in memory usage. I plan to continue with my band-aid solution of logging fewer networks per VM while I work to increase my understanding of how my program and computers in general manage memory. [↩]
3. To count a channel as successfully logged I allowed for an unlimited number of disconnections as long as each of them individually did not last more than 30 minutes. [↩]
4. MaddShark, LunarIRC, LUGS, Langochat.net, KewlFun, Italian-Amici, IRDSI, IrcWorld, IRCstorm, IRCnet.gr, ircHax0r, IRCFreakz, IRC2, IRC-Ban.de, IndoGaul, Immortal-Anime, IdleChat, Hub4Ever.Org, homo.net, Griefplay, GreekIRC, GreekChat, Gotham, Global-Irc.Org, GIMPnet, GigaIRC, German-Elite, Genscripts, GeekShed, GeekNode, FyreChat, Futuragora, FreeUniBG, ForumCerdas, ForeverChat [↩]
5. Networks where TheFleet joined at least 1 channel but in the end was not able to successfully log any channel: IRDSI, IRCnet.gr, GreekIRC

   Successfully logged channels:

   GeekNode: #bitcoin, #troll-gloo, #vdmbot, #pr0nfactory, #RMLL2014-mpl, #aigris, #ffdn-mastodon, #testtest, #aprilchat, #marsnet, #edmsio, #quadrature-ops, #gresille, #34c3, #cfai-lr, #fleurix, #ffdn-suivi, #ffdn-dn42, #alternatives87, #gixopdop, #louvilug, #iloth-ca, #neocade, #fdn-projets, #style-python-dev, #3wa-staff, #lae, #monodon, #minecraft_chiantos, #rezine, #lau, #TERRORIMSE, #Midways-Network, #planete-ASM, #AuvernIX, #adipolab, #vlsk, #ilerouge, #homomachinus, #esigetel, #Midway, #ph7net, #geekfault, #dadazedazeaz, #ldn, #tilima, #hivane, #partipirate, #Troll, #franciadmin, #alternp, #joron, #sudix, #manip, #fromagiens.net, #Magellan, #ceriz-admin, #pouetpouet, #cuisine, #larevolutionrousse, #thinkcode, #axinite-staff, #globenet, #chadoc, #freebsd, #zebulon.fr, #sporthi, #elament, #SCANI, #fosdem, #style-potins, #nniiiaaaaaa, #funlab, #hadoly, #ffdn-formation, #octogen, #evilirc, #hackerspacerouen, #style-python, #vic, #duchesse, #hmm-la-bd, #ppmp, #fdn2, #illyse-admin, #rmllbe, #42admin, #picnat, #arn, #united, #Aqualitad, #RPG, #Chikoumi, #ewo, #ffdn-bistro, #yolorandom, #pp-ektek, #april, #wireless-fr, #RadioAutiste, #Latex, #axinite, #RMLL_2018, #exe-dev, #miricorps, #LaboLyon, #sameswifi, #CnedSIO, #geeknode, #capitole, #betaseries, #Jeanne_D'Hack, #lockpicking-fr, #cryptoloutre, #rmll-noc, #igwan, #tdn, #annarx, #labriqueinter.net, #pe, #pan, #gitoyen-tech, #virtualisation, #42, #staffwm, #wdntw, #testwm, #pclight, #extrifusstory, #ndh, #geeknodecamp, #grenode, #naivart, #Tealer, #pp_ploire, #franctest, #mycelium, #francinet, #AuverNET

   German-Elite: #secnews, #aspies, #meinchan, #drehtür, #Nasu_gaming, #reallife, #talk, #shizetag, #german-elite, #magick, #quizfueralle, #QQQQQQQQQQQQQQ, #eule, #serienjunkies, #radio-tekula.de, #Mathematik, #schwanz, #leechomatics, #lecters-cafe, #joschi, #kontrolle, #maniacommunity, #platinum, #talk-signal, #informatik, #Dexit.Party, #subcentral, #hive, #ru2, #music, #soso, #german_alliance, #brains-sub-chan, #froschinger, #beastarium, #hoerbuch, #, #bsd, #froschi, #wbc-exil, #FrinkC, #rcbintern, #kde-kirigami, #nerds, #shadowlamb, #mo, #Nachhilfe, #planetopia, #Unique-Board, #windows, #diamonds, #alietsche, #radio4us.de, #debian, #fotografie, #konsoleros, #hackers, #debianforum.de, #coding, #klagemauer, #linux, #helper

   FreeUniBG: #belene, #manutd, #test, #balchik, #Sexy, #Bulchat, #irc.Gorna.net, #juventus, #plovdiv, #Game, #kex, #viva, #cska, #bourgas, #FreeZNC, #Bulgaria, #svalki, #Jambol, #irc.telnet.bg, #starazagora, #OnlineDJRadio, #5on5, #Dimitrovgrad, #youtube, #metal, #extreme, #shell, #breakdance, #debian, #welcome, #lamerzone, #gotham, #teteven, #Linux, #pleven, #real, #sofia, #IRC, #chatpat, #Rakovski, #alcoholic, #30-40, #Sladur, #London, #svishtov, #lovech, #ignore, #NorthSide, #realchat, #tyrnovo, #Sliven, #sex, #Varna, #burgaz, #burgas, #linuxhelp, #IRCHelp, #ruse, #irc.sabo.bg, #refresh, #novazagora, #empire, #radio, #shivachevo, #velingrad, #20-30, #doom, #Sweet, #overwatch, #BestShells, #VIP

   Italian-Amici: #TriviaTime, #IRCHelp, #chatitalia, #Divx-Italia, #InFiniTY, #eggdrop, #amici-radio, #vhost, #services, #amici-radio-staff

   Genscripts: #WRnet, #Mapantz, #Genscripts, #speedy

   Futuragora: #ajuda, #cosmos, #zeitgeist, #tecnoponics, #lobby, #IRCBrasil, #opers, #fadev, #feed, #radio, #futuragora, #tvp

   Langochat.net: #langochat, #SpamFilter, #aide

   FyreChat: #kosekroken, #kosebakken, #help, #vana-commits, #!stats@fyrechat.net, #fyrechat, #sandbox

   ForeverChat: #rto, #attractmode, #emuchina, #tosecdev, #MFM, #redump, #MFM-dev, #tosec2.0, #roms4italy, #tosec, #wii-rush, #castroom, #rawdump, #PRCows, #oldemuisos, #foreverchat, #aumame, #RomShepherd, #emulab, #gipsforfun, #collectors, #romvault, #retrofe, #retrogamers, #ut99redirect, #retrodatorer, #therealconacrew, #rivalroms, #mame, #RocketLauncher, #gaming.tombstones, #NGC-PAL, #CAX_hackathon, #vgm, #ssotk, #discord, #darkwater, #uconn

   IRC2: #bit-tech, #insane, #gettoduma, #muhely, #zirc, #V1X, #linux, #autocad, #depresszo, #civil3d, #pc-muhely, #tarskereso, #carp-kviz, #asd, #blender, #30_feletti

   Griefplay: #empireoftanks, #wh40k, #worstheap, #thedancefloor, #irchelp, #ff14, #hearthstone, #starbound, #eve-online, #failheap, #RasberryPi, #griefplay

   IRCFreakz: #ninja, #idlerpg, #australia, #eggdrops, #SaD, #sydney, #Trains, #jedstars.net, #Melbourne, #cncirc, #pc-talk, #ircfreakz, #quiltchat, #axon

   ircHax0r: #servers, #irpg, #sec

   KewlFun: #War, #kewlfun, #Yahtzee, #Profiles, #seen, #truelies

   ForumCerdas: #black_sky, #makassar, #satu, #jakartashells, #Yogyakarta, #ForumCerdas, #dsnnet, #abocy, #games, #makmur, #dago25, #depok, #alowaini, #malang, #bitcoin, #medan, #ukhuwah, #game, #batu, #smart, #rusuh, #mp3, #irene, #waroeng, #bot, #our, #jogjakarta, #kediri, #radio

   Global-Irc.Org: #egghelp, #formula1, #GlobaL-CreW, #syslog, #eggies, #idlerpg, #mp3passion-chat, #services, #worldofwarcraft, #hhchat, #friends, #twitter, #opers, #vhost, #speelhal, #walhalla, #movieplanet, #digitalcity, #cservice, #monitoring

   Hub4Ever.Org: #Rock_in_Roll, #brasil, #Linux, #DreaMule-German, #ahummm, #denis, #, #minasgerais, #DreaMule-English, #hub4ever, #Offtopic-Spanish, #Rodrigo, #DreaMule-French, #Games, #Ajuda, #idle, #Bots [↩]

6. UniBG, ShadowWorld.Net, QuickFox, DarkerNet, DALnet, Cognet, freenode, IRCsource, Chatters, Blafasel, Black-Cell.net, ZwergenIRC, Zoite, zeolia, Xerologic, WNet, UplinkIRC, UKChatBox, TwistedNet, Tweakers, TripSit, TransAdvice, tm-irc, Teepi, tchattons.com, synIRC, swissIRC.net, SwiftIRC, SurrealChat.net, Subluminal, Striked, SoundsNWaves.net, Snyde, SlashNET, SKYROCK, SiberChat.Org, ShadowFire, SceneP2P, SA-IRC, RootWorlD, RisposteInformatiche, RelicNet, RelaxedIRC, Recycled-IRC, PurpleSurge, PSIGenix, Powers.cl, PowaNet, PIRC.pl, Pilgerer, paranode.net, P2PChat, P2P-NET, oz.org, out-dated.net, Otaku-IRC, Omninet, NFNet, Netrusk, mircphantom.net [↩]
7. Networks where TheFleet joined at least 1 channel but in the end was not able to successfully log any channel:
   NFNet, TwistedNet, WNet, MagicStar, DALnet, ZwergenIRC, zeolia, UniBG, Mibbit, mircphantom.net, PurpleSurge, Pilgerer, P2P-NET, SA-IRC, Tweakers, Subluminal, SwiftIRC, tchattons.com, Zoite, Snyde, PowaNet, out-dated.net, PIRC.pl, synIRC, IRCsource, oz.org, Otaku-IRC

   Successfully logged channels:

   ShadowFire: #slow, #deathrow, #greytower, #ham.ee, #otc, #triumvirate, #chat, #chase, #hack, #selfharm, #shadowfire, #chucknorris, #americas_army, #warble, #finsprite, #avatarshq, #eve, #bandofbrothers, #monkeybong, #programming, #GTClassroom, #ut, #warez, #tatsi, #insolence, #diablo2, #pimpsanonymous, #non-bus, #wow, #afrikaans, #myadsl.co.za, #jeffftest, #guildwars, #weezel, #tempus-test, #elsewhere, #compsci, #tempus, #avatars, #obliq2, #Dent-O-Soft, #zagamers, #tce, #home, #jefff

   DarkerNet: #net.goth

   tm-irc: #de.wikipedia.huggle, #en.wikipedia.huggle, #irc, #simple.wikipedia.huggle, #serviceslog, #es.wikipedia.huggle, #no.wikipedia.huggle, #GrumpyChat, #konnectirc, #support, #meta.wikimedia.huggle, #area51, #opers, #petan, #staff, #pt.wikipedia.huggle

   Recycled-IRC: #KAISHII-FANSUB, #ANIMES-XDCC, #kigen, #linux, #Mangas-Arigatou, #Help-Fansub, #R0bust, #azk-vip, #honeypot04, #TEAM_OHZORA, #GENSOTEAM, #slam_dunk_fansub, #waazaa, #ZEROFANSUB, #Chuushin-Equipe, #help, #g-corp, #ASGARD-PROJECT, #zero-absolu, #frenchzfourtout_chat, #gabou, #ARCADIAN-PROJECT, #honeypot06, #APPLEFIELD_PROJECT, #MOONLIGHT_PROJECT, #shareyuken, #frenchzfourtout, #loups-garous, #roswell, #SFTD.requests, #SFTD.invite, #disku, #news, #oxygene, #Nyawko, #SOT, #ebooks, #pornparadise, #share-movies, #nt3, #foret, #ah-work, #YYH_Fansubs, #enjoy-hosting, #issekigan-animes, #AONOSCANTRAD, #staff-mirage, #p-claymore, #mibu-fansubs, #tenebreuse, #mangaart, #PREDISTRO-TF, #S&T, #honeypot20, #honeypot23, #ouifm, #JOHNNY-SUBS, #sokaoa, #YU-GI-OH!_FANSUBS, #MIRAGE-STAFF-BAKUMAN, #AH-SECRET, #honeypot21, #honeypot22, #honeypot25, #honeypot24, #honeypot27, #honeypot26, #honeypot29, #honeypot28, #TG, #Soumission-Dressage-Esclave, #numero_du_slot, #inulogic, #TOUCH_PRODUCTIONS, #soai-no-naka, #epoke, #honeypot14, #honeypot15, #honeypot16, #honeypot10, #honeypot11, #honeypot12, #honeypot13, #honeypot18, #honeypot19, #irf, #Eden, #Team_25, #ogs-team, #SASALAND, #INFECTIOUS, #RNK-FANSUBS, #honeypot09, #honeypot08, #honeypot03, #honeypot02, #honeypot01, #honeypot07, #honeypot05, #moches, #extAsia, #SUB-NO-NAKAMA, #honeypot17, #NEMESIS, #Briggs, #silex, #scripts, #neoryu-team, #tglobe, #mft, #oldies-fansub, #potos&co, #korben, #azk-check, #pcm, #shinsekai_STAFF, #SPACE, #Asgard-private, #Nemesix, #mofo, #azk-team, #pangya-fr, #EIEN-NO-KIMOCHI, #honeypot30, #The_Eden, #ANIME-HEART, #iNFEXiOUS-ViP, #nt3-staff, #sexe, #wtf, #workforce, #geekeries, #irf-test, #azk-recrute, #issekigan-staff, #SHINSEKAI, #hayakutrad, #darkside

   Chatters: #gatherstyle, #ppm, #jokes-gr, #rock-cafe, #linux, #cs.gather, #test123, #chatters, #enforcers, #uno, #Zougla-ops, #sex

   RootWorlD: #RocK-HeLP, #ATOLLO-FRIENDS, #irchelp

   TransAdvice: #tss, #ops, #stuff, #politics, #opers, #social, #lobby, #geeks, #bopm, #wiki, #support, #420, #mtf, #lobby-nsfw

   P2PChat: #German, #shareaza-french, #Shareaza-Portuguese, #Shareaza-Dutch, #shareaza-Italian, #Trivia, #shareaza-dev, #thebighack, #shareaza-chat, #diapergals, #shareaza-Arabic, #sharelin, #xdcc, #botpark, #Shareaza-Russian, #Shareaza-Polish, #help, #stux-notes, #p2pchat, #shareaza, #crawler, #Morpheus, #Shareaza-Spanish, #alternative, #razaops, #operhelp

   Netrusk: #agelidProd, #linux62, #wafonly, #cul, #heroeschronicles, #terrasse, ##linux, #services, #dehors, #blabla, #Saloon, #jdr, #clan_drows, #poissonrouge, #backroom, #templars, #aide, #sp4nk-test, #secret, #agelidTest, #PUC, #fsmsb, #marchéspublics, #yevon, #privanst, #maps, #linux, #blatest, #yop, #secrets, #anst, #celibataire-fr

   RisposteInformatiche: #Risposte_Informatiche, #raccontieros, #capre, #CAH, #caiusmaximus, #IRCHelp, #ADeltaX, #libera_chat, #olga, #vHost, #Enrico, #Minecraft, #largodellosvago, #mangaring, #Dragone2, #risoluto, #PA, #PokeMMO

   RelaxedIRC: #doghouse, #movies4earth-chat, #Freedom-zone-chat, #tv-caps, #Freedom-zone, #LimitLessChat, #movies4earth, #siosios

   Black-Cell.net: #zavislaci-admins, #zavislaci, #bcserv4, #bcserv5, #the-lounge

   QuickFox: #4paws, #ash-fox, #Reakt0r, #SIN, #psystra, #gimpgirl, #insilico, #insilico.bots, #Quickfox, #void, #cumboys, #insilico.north, #IceRealm, #yiffle, #subnova, #help, #looney, #IS.East.1, #IS.East.2, #exodus

   Powers.cl: #Metal, #linux

   TripSit: #tripsit.me, #tripsit1, #tripsitters, #welcome

   SoundsNWaves.net: #dj-ops, #justicequest, #frozzbozz, #awesomeradio, #One_Day_At_A_Time

   UKChatBox: #help, #UKChatBox

   Striked: #starburst, #d&d, #xenophase, #boobies, #rawr, #ashtear, #unreal, #idle

   SlashNET: #newtotse, #skulserve, #perlmonks, #lp301, #CriticalFailures, #lp201, #xkcd-music, #pugglewump, #mineraft, #xkcd-sciencefiction, #offtopic, #m-ei, #newrepublic, #innercircle, #newswire, ################################, #kotkamafia, #SETI@home, #zotsluts, #tagmods, #sex, #死, #☭, #paforums_tf2, #boringman, #bucketlog, #pax_gw2, #nricciar, #geohashing, #xkcd, #REDinfo, #idiots-club, #shakespeare, #jupiter, #retrocomputing, #gmg.minecraft, #thegentlemanloser, #codelove, #help, #xkcd-gaming, #xkcd-coding, #eccentric, #LTI_OTR, #℻, #omgcheckplease, #xkcd-signal, #secret-channel, #compsci, #pax_ffl, #twowords, #death, #urination_fans

   ShadowWorld.Net: #ZZZzzzz, #sanctuary, #AAaasstt, #_zzZZZzzz, #cybex.bots

   Teepi: #tribu, #musique, #pcinpact-linux, #asw, #aide, #cab, #wic, #teepi

   UplinkIRC: #minecraft, #UplinkCorp, #jgkamat_, #defcon, #multiwinia, #darwinia, #usurper, #xplice, #wow

   swissIRC.net: #just4fun, #newnigma2, #newnigma2-np, #oscam, #linux, #!/bin/sh, #schaildisoft, #2k5, #newnigma2-team, #Hilfe, #ump, #Treffpunkt

   Xerologic: #ux-chat, #COSMIC, #apps, #MP3JAROCHOS, #ultra-xero, #MASTERWAREZ, #TH3D3N, #zuul-central-xdcc, #pwnd, #scorpio, #xboxland, #1warez, #warez, #moviegods, #toonland, #tdz, #beast-xdcc, #UX-BOTS, #console-world, #search, #psp-moviez, ##COSMIC-BOTS, #MP3GROSEROS, #xboxland-mp3, #MP3GHOSTZ, #xboxland~boobies~bits, #hidd3n-zauberbude, #attrition, #the-club, #cracks, #xvids, #psp-world, #all-new-movies, #movies-r-us, #sb-movies, #mp3, #warez.net

   MauriceRadioLibre: #hopm

   RelicNet: #lobby

   Blafasel: #pybottest, #support, #, #w, #flipdot, #perl, #EstrBny, #test123, #apollo, #cccamp, #go, #freifunk, #fubbi, #unix, #iridium-messages, #33c3, #Badewanne, #ffmuc, #WakeUpFpletz, #game-on

   Omninet: #caleb, #omnomirc, #sorunome, #nspire-lua, #knexflux, #private, #SWLing, #games, #radio, #thebot, #programming, #prizm, #~#withgznc, #br330t, #digital, #lgbt

   MeFalcon: #Das_gebluemelte_Sofa, #Northsman, #Smugglers-BaR, #mefalcon, #radio-hilfe, #quizz, #blubb, #susi, #help, #News, #miroos-privaty, #radio, #DeichMupfel, #neurobeat, #party

   MindForge: #entre_nubes_

   Cognet: #acknak, #dc, #void, #food, #cats, #ccdc, #nomail, #uiuc.notlosers, #SHADOWRUN, #matrixtest, #matrix, #dorks, #tamulug-officers, #minecraft

   SKYROCK: #irc

   PSIGenix: #maplestory, #fitegaem, #aethera, #nwifi, #sushi, #pfalzerwald, #frotzbotdev, #test, #oscuro-destiny, #support, #bottestlab, #coperia3, #deep13, #, #spellbound, #ddb, #rtfo, #gamemusic, #s3pmemag, #werwerwer, #j-poop

   paranode.net: #thelounge, #test, #medusa, #geek, #jungletrain

   SurrealChat.net: #Diagnostics

   [↩]