Very Secure

Who Said That? Or, The Importance of V

February 17th, 2020

Growing up I acquired a distaste for appeals to authority. Nothing was more infuriating than requesting an explanation and getting the response "Because I said so". This lead me to my mistaken view that an evaluation of a text should be based only on the text itself and not its source. Afterall, one doesn't need to know the author to determine whether statements in a text are true.1 And if the text is in the form of code, the code will not execute differently depending on who wrote it. But experience has shown that, coming from the right mouth, "Because I said so" holds weight.

People are not equal. Knowledge and cognitive ability varies greatly from person to person. A good strategy for learning is to pick the smartest person in the room and listen to what they say. Maybe others will occasionally offer something intelligent, but it is expensive to evaluate useless babble. Ain't nobody got time for digging through piles and piles of pseudoscience to find little nuggets of information.

Indeed, two identical texts can have different values depending on their respective sources. If a text comes from someone you trust, it is worth considering. If a text comes from anon391, there is a risk it is guaranteed time spent evaluating the text will be wasted. Should the text turn out to be nonsense, there is no one to punish. Anon391 has not wagered his reputation by producing the text and thus has no skin in the game.

It would be nice if you could find a person whose brain worked better than yours and treat them as a fountain of truth. Unfortunately, although your brain may not work the best, it is the only one that works for *you*. So you must still evaluate for yourself the text that comes from a trusted source.

The above ideas are embodied in the proper use of V, The Most Serene Repbulic's version control system. Code is considered only if someone in the V operator's wot has put their reputation on the line by singing a vpatch. Making sure the code comes from a valid source is done *in addition* to reading and understanding the code. This is opposite to the method of the common layman, who neither reads the source nor knows the author of the code he runs on "his" machine.

  1. Apart from statements such as, "I, the author, have visited every country in Latin America." of course. [↩]

Posted in Bitcoin, tmsr | 2 Comments »

The RAV4

February 16th, 2020

Because of my ~0 experience buying cars, I hired my taxi driver to help me with the process. He found prospective deals, made appointments with owners, drove me to them, and then helped me evaluate the vehicles. The 2 most common models for sale that met my criteria1 were the Toyota RAV4 and the Daihatsu Terios Bego.2 Per BingoBoingo's advice I called local car rental companies to rent the models I was interested in for 1-2 days. Unfortunately they did not have them in stock. So I skipped the trial step and went straight to seeing the cars.

The first RAV4 looked promising from its advertisement. It was a 2002 model w/ 158,351 miles being sold for 4 million colones.3 It was owned by an older lady who lived in Santa Cruz.4 When we arrived to inspect the car I realized my first mistake, having worn a nice shirt. The next time I go used car shopping I'll be sure to wear ~rags; inspecting a car requires getting under it and getting dirty. Anyways, we found a problem that immediately rules out any car - an oil leak. As taxi amigo said, "la moto estaba llorando". We left immediately.

The next car we saw was a 2007 RAV4 with 144,310 miles. The car appeared to be in great condition. But its price was 6.8 million colones, a little on the high end of my budget, so we didn't inspect the vehicle thoroughly. Instead we told the owner we may come back after viewing a few more cars.

Then we went to a dealership. I had a sense of uneasiness there because of the unlevel playing field. The dealers are professionals at *selling* cars. Everything felt like a trap. The interiors of the cars, including the organs under their hoods, were squeaky clean and polished. And the dealership had all the lubricant ready to facilitate a quick purchase - a magic plastic acceptor and on premise lawyers. Regardless, I checked out a 2007 Honda CRV. This was the first vehicle I gave a test spin. As I was driving it, it had the tendancy to slightly veer off to the right. I had to keep a constant counter clockwise pressure on the steering wheel. I expressed my concern and I was told that the car just needs a realignment, a cheap procedure. The Honda was otherwise a nice car, but there were two key problems: its price and its mileage. With 213,000 miles at 6.5 million colones I was sure to realize a hefty loss when I tried to resell it.

I bought the fourth car I saw, a 2004 Toyota RAV4 w/ 148,636 miles for 4.5 million colones ($7,951):

rav4-4

rav4-2

rav4-3
Replacement front tires

rav4-1

Upon arrival we gave the car a visual inspection. The front tires were a bit worn but otherwise the vehicle looked good. I took it for a test drive. I had read reviews about the RAV4 that contained complaints about the engine's noise permeating through the car while driving. That seemed like the kind of thing that could annoy the hell out of me fast. But the engine sounded good / could hardly be heard from inside, and I found driving the car pleasant. After the test run I wanted to purchase the car.

I asked taxi amigo for his opinion. He said it was a good deal and if I liked the car I should negotiate and buy it. The car was being sold for 4.5 million colones. The result of my mediocore haggling was to get that price to include the lawyer fee and a set of new front tires.

I informed the owner that I wanted to have a mechanic inspect the car before I bought it.5 Since the car needed to get its Riteve6 in March I offerred to buy the car same day should the car pass the required inspection.7 So I drove for an hour with the owner to the Riteve office. Indeed, the inspection was thorough.8 The car passed so we drove to the lawyer's office.

The lawyer we chose was someone I had met a couple times already, my taxi driver's aunt. We ran into an issue because I was not able to do a same day wire into the owner's bank account. He agreed to sign over the car in exchange for a down payment of 900,000 colones cash, and my promise that I'd pay him as soon as the fiat systems acknowledged him as a wire recipient. Although the owner physically kept the car until he received all of the funds, my taxi amigo thought it was bold of him to sign over the ownership after only receiving a down payment and a promise.9 I think the enjoyable conversation I had with the owner on the car ride to the Riteve office built some trust. Two days later the money was in his account and the car was in my backyard.

  1. A $6-12k vehicle common to CR with the ability to traverse the unkept "roads" that lead to surf spots. [↩]
  2. On the day of the appointments the Terios Bego's were sold out, so we wound up seeing mostly RAV4's. [↩]
  3. At the time of writing the USD/Colones exchange rate is 1:566, so the car was roughly $7,067. [↩]
  4. Being owned by an older lady in the city decreases the chance that the car was used frequently for trips along the coast via Costa Rica's "roads". [↩]
  5. At this point I was worried about my tendancy to throw pebbles at glass to check to see if the glass is bulletproof. [↩]
  6. A Riteve is a sticker a car in CR needs to show it passed an inspection. [↩]
  7. I thought the Riteve was a sort of beaucratic dance but my taxi driver informed me that they do a thorough inspection and passing that inspection was as good as any thumbs up you could get from a mechanic. [↩]
  8. The inspection occurs in a building that looks like a car wash. You wait in line behind other cars to go to various stations where different parts of the vehicle are reviewed. First an agent comes and requires the driver to demonstrate the ability to turn on all of their lights: blinkers, headlights, hazard lights, etc. Then he comes in the vehicle and tests all the seatbelts, all the windows, and other functionalities such as the windshield wipers. The next station contains a device that shakes the vehicle to evaluate the suspension. After the suspension tests you drive over a narrow hole in the ground where another agent views the vehicle from below, checking for issues such as oil leaks. The last station tests the breaks. [↩]
  9. Especially since we were working with a lawyer I knew and he didn't. [↩]

Posted in Travel | 1 Comment »

Stuyvesant Admissions

February 14th, 2020

Admittance to my alma mater, Stuyvesant High School, is based solely on an entrance exam.1 Last year the NY Times created a clickbait post, Only 7 Black Students Got Into Stuyvesant, N.Y.'s Most Selective High School, Out of 895 Spots. The racism of multiple choice math and reading comprehension questions was exposed, and in response mayor Bill De Blasio pretended to take action.2 De Blasio's efforts to "diversify" Stuyvesant by altering the admissions criteria were met with a counter campaign from various students/parents. But all the noise from this debate is a distraction from Stuyvesant's real problem, teacher admissions.

The implication of the NY Times's post is that Stuyvesant is a good school in its own right. Swap out a few Xiaoping's for a few Tyrone's, some Mikhail's for some Pedro's, and the only difference will be who receives the great education provided by the Department of Education's crown jewel. This is nonsense, the only special aspect of Stuyvesant is that it has an objective standard for student admissions.3 The students' success does not come from the Stuyvesant staff, who are mostly welfare recipients pretending to do a job.

I speak from personal experience when I say Stuyvesant is infected with terrible teachers. The most memorable was Ms. Garber, a morbidly obese 300 pound chain smoker that taught... health. I had an English teacher who fell asleep during his own class. One history teacher never read his students' essays; he would just skim them and circle "key facts and dates". Many of these "teachers" were protected by tenure or some teacher's union measure.

Sol Stern wrote about Stuyvesant's staffing problem in an excerpt from the book, Breaking Free: Public School Lessons and the Imperative of School Choice.

Yet all these accolades tend to mask the school’s dirty little secret. While Stuyvesant admits only the city’s finest students, through a process that is free of subjectivity and nepotism, its hiring decisions are far less merit-driven. Like every other public school in the city, Stuyvesant is plagued by bureaucratic regulations and corrosive work rules that favor seniority and paper credentials over a teacher’s knowledge and skill in the classroom. For instance, according to the teachers’ contract, half of Stuyvesant’s teaching vacancies each year must be posted and set aside for teachers seeking transfers from other city schools. These set-asides are required to be filled solely on the basis of seniority. The fact that many of those selected lack the academic qualifications to teach to the level of Stuyvesant’s students is irrelevant to the union and the school system. The underlying premise of the contract is that any teacher with a state license is fit to teach at Stuyvesant.

This was written in 2003. Things haven't changed much in the last 20 years.4

The issue of transfers became Stuyvesant principal J. (Jinx) Cozzi Perullo’s bête noire—and ultimately led to her resignation. She had begun ruffling feathers from the moment she took the school’s helm in 1994. She didn’t like the seniority system and the work rules, believing they undermined excellence. And she said so in public. For this she made many enemies.

Perullo once described to me a conversation she had with the outgoing principal, Abraham Baumel, in which he told her that she now had the best principalship in the city. At first Perullo was flattered, until she realized what he meant: that Stuyvesant was a great place to be a principal because there was virtually no way to fail. The students guaranteed the school’s academic success. No matter what the principal did, the students would still achieve average SAT scores of 1400 points, and 99.5 percent would go to college.

Perullo conceded that Baumel had a point. “There’s never been a discipline issue here,” she told me. “The kids always do their homework. In some ways Stuyvesant isn’t a ‘real’ school. A teacher could fake it here for 35 years, because even when the teaching is inadequate the kids will find a way to do well on their tests.”

And if a teacher can fake it for 35 years, they must.

I support the move to remove Stuyvesant's test based admissions. Let the old folk receiving social security checks while pretending to teach deal with kids failing the tests that serve as their performance reviews. A few generations of bright incoming high school students will get shafted by this change, having NYC's intellectual safe haven swept away just before they got there. But at least those students won't graduate with the false impression that they received a good education. Hopefully when the smart kids are off the sinking ship they'll be able to find a school where teachers are held to a higher standard than the students.

  1. The exam is called the SHSAT - Specialized High School Admissions Test. It is open to all 8th graders in New York City. Roughly 30,000 students take the exam, competing for about 6,000 seats spread out across 9 schools. On the exam each student creates a preference list for the 9 schools. The top scoring student gets to go to their first choice school, then the student with the 2nd best score also gets to go to their first choice school, then the nth highest scoring student gets to go to their highest ranked school that still has a seat available. The bottom 24,000 students who do not get into any specialized school are doomed to attend a NYC district high school, a fate worse than being sent to the breeding farms. [↩]
  2. He proposed a plan to guarantee admittance to a specialized science high school to any student in the top 7% of their class in their middle school. [↩]
  3. And that comes at the hefty price of selecting for exam takers. [↩]
  4. To my great amusement Stern's excerpt contains a part about the difficulties faced by his son's favorite math teacher, a "Romanian refugee". Indeed, the story of a talented Romanian math teacher infiltrating Stuyvesant's hiring bureaucracy only to leave a few years later repeats itself! [↩]

Posted in Reflection | No Comments »

TheFleet Overview

February 7th, 2020

My previous article detaling TheFleet's pseudocode was poorly worded and contained extraneous implementation details. This revision aims to give a concise summary of how TheFleet works.

In order to log the irc space TheFleet uses fleetbot, a class similar to logbot. Each instance of fleetbot connects to 1 network and joins the max channels allowed per nick on that network.1 Once connected to its network, a fleetbot logs messages in a local postgres db.2 Certain messages - join, part, kick, and privmsg3 - are logged to a table named irclog. Events that are specific to a fleetbot (disconnection from the server, being kicked from a channel, joining a channel, and failing to reconnect) are logged to a separate table named fleetlog.4

Fleetbots have an aggressive reconnection strategy. The code that receives and sends messages to the irc network is surrounded in a handler-case.5 If a runtime error is signaled a fleetbot attempts to reconnect. A fleetbot also attempts to reconnect if it hasn't received a pong from the server in *max-lag* (60) seconds.6

I still need a way to orchestrate the large number of fleetbots connected to various networks. Previously I ran all the fleetbots for 1 network together in their own sbcl process. This design needs to be discarded since each process takes over 40MB of RAM and my VMs only have 1GB available. Other constraints I have to deal with are: A VM costs $5 / month, most networks allow only 3 connections per ip, and I can only run 1,000 concurrent threads on each VM.7

  1. If a network has 130 channels and a max 50chans/nick, TheFleet creates 3 fleetbots - 2 join 50 channels and 1 joins 30 channels. [↩]
  2. This postgres db is shared amongst all fleetbots running on the same VM. [↩]
  3. Privmsg is the name for a standard irc message you see in a direct message or in a channel. A privmsg is sent to a target. A target is either a nick or a channel. [↩]
  4. I also currently print some debugging statements and unhandled messages from networks to standard-output, which I redirect to a log file when I run the program. I plan to remove this extra logging to save disk space once TheFleet is running in full gear. [↩]
  5. I.e. a try/catch block. The robustness of this design is dubious since it allows for unknown errors to persist. [↩]
  6. This behavior is inherited from fleetbot's superclass, ircbot. [↩]
  7. The thread per VM limit is possibly adjustable. My understanding is that the limit comes from each thread allocating ~ 1MB for its own stack. [↩]

Posted in tmsr | 2 Comments »

The Story Told to Alcinoüs - the Cyclops

February 6th, 2020

In Book IX of The Odyssey, Odysseus recounts the part of his journey where his arrogance gets his entire crew killed. Odysseus doesn't see it that way and certainly doesn't make a point to acknowledge his culpability to his audience in Alcinoüs's hall. But unfortunately for Odysseus, I have the luxury of being able to reread and re-frame his tale.

Before getting some of his crew killed and the rest doomed to be killed later, Odysseus docks his fleet at an island abundant with food. Odysseus describes how fertile the land was, how the island had a stream of fresh water, how it had a bay safe for ships... Yet despite having found a goddamn oasis after toiling away at sea, Odysseus decides to leave for a nearby island. And once there, he thinks it smart to walk into a cave covered in massive piles of shit.1 Somehow the smell doesn't phase Odysseus and his men as they help themselves to some cheese.

The cyclops appears and isn't happy to find a bunch of men making themselves at home in his kitchen. He reclaims his food by eating the men who ate it. Odysseus and company can't escape because the cyclops moved a large boulder to block the door of the cave.2 So Odysseus comes up with a plan to get the cyclops drunk and stab him in the eye.

After Odysseus offers the cyclops wine, the cyclops becomes grateful and asks for his name. Wiley Odysseus tells the Cyclops that his name is "Noman." Odysseus pats himself on the back for this clever lie because later when the cyclops screams for help shouting, "I'm being attacked by Noman!" his other monster friends respond, "oh well if you're being attacked by 'no man' then I guess no one is attacking you hur der."

Odysseus and what remains of his crew manage to escape the cave, but Odysseus is not done being stupid. As he's sailing away he starts taunting the cyclops. The cyclops rips off part of a mountain and throws it at Odysseus's ship, missing by a hair. Having barely survived, Odysseus begins to taunt the cyclops again while his remaining crew is begging him to please stfu.

Unfortunately for the crew Odysseus does not listen and undoes his earlier act of cleverness by revealing his name. The cyclops then prays to his dad Poseidon to "please kill Odysseus, but if you can't do that then at least make his life hard and kill all his friends." The later part of the prayer comes true.

Odysseus's tale of being an idiot is well received in the hall of Alcinoüs. Stupidity mixed with "bravery" makes for fun stories.

  1. Odysseus doesn't mention the shit at first. He describes the pile of dung only later when he tells how he employed it as a hiding spot for the spear he used to blind the cyclops. [↩]
  2. Odysseus conveniently doesn't mention the massive boulder until after it locked them inside the cave. But this boulder must have been rolled to the side of the entrance when they entered, right? Odysseus does say that when entering the cave in his "stout heart" he "suspected" he would meet a powerful savage. A real Sherlock Holmes, this Odysseus. [↩]

Posted in Reflection | 2 Comments »

The Right Kind of Positivity

February 4th, 2020

Jfw's refreshing and amusing "From the forum log" series includes a summary of a recent monologue from mp. The summary caught my attention and I daresay I have something to add to it after reading an article on human behavior linked in a comment mp left here.

First, the summary:

Trinque not having replied yet, MP expanded his own point on the suppression machine: going through life, one is constantly uncovering new personal deficiencies. Each time, one faces a difficult choice: to face and overcome it, painful because it requires killing a part of one's youth; to accept insufficiency; or to pretend it doesn't exist or doesn't matter and eventually end up an overgrown teenage loser (possibly hanging out in #asciilifeform to commiserate). He went on to note that the existence of evil(i) is essential for life to be interesting, but that this doesn't make it any less evil.

i. Or problems1

And the original:

mircea_popescu: every time you discover something, you get the same exact choice, again and again each time. you get the same choice that also happens to be the ~only~ meaningful choice you ever get, or ever could possibly get : are you going to fix it ?
mircea_popescu: there's strictly two possible answers you can ever produce, and they're yes or no. yes sucks because, well... it is suicide. [http://ossasepia.com/2019/10/31/working-with-ideals-and-perfections/#comment-6848][]quite literally], the little girl gotta die.
mircea_popescu: no sucks because... it really doesn't. once you decide you actually "like" your dysfunction, or in any case you're not equal to its remedial, you're stuck hanging out in #asciilifeform with all the other cripples.
mircea_popescu: your life becomes a carousel of a) identifying partitions upon the world that'll neatly divide it across the line of "this part exposes my dysfunction" / "this part doesn't expose my dysfunction" (definitionally an impossible task) and b) making up stories about how it all works out, or will, or whatever.
mircea_popescu: in the end, what you do is hang out with the other cripples. that's it, the loser table at the highschool cafeteria, where you tell each other stories of "how you aren't really geeks", and how one day you'll get the cheerleader, the usual fare. making pacts about losing your virginity by end of school...
mircea_popescu: it's not exactly novel, but on the contrary well documented to date. lotta cripples have lived since the practical implementation of rochester's all men be cowards if they durst.
mircea_popescu: it gets you stuck saying patently dumb shit like "alliance of the smart against the stupid" when what you quite transparently ~really~ mean is very much "i thought this guy was gonna vouch for my imaginary partitions of a) above!" ;
mircea_popescu: and everyone else is stuck somehow reconciling your supposed "great technical acumen" with your apparent incapacity to intellectually function enough to match a age-adequately developed nine year old slut -- which is incidentally how the elaborate narrative of the naysayers even evoloved over time : the others dealing with the anal child is how anal childhood even gets implanted in heads in the first place.
mircea_popescu: from whence it can further spread -- much like camels who never saw other camels spit don't know how to spit nor ever spit, while camels who did see other camels spit do know how and actually do it, just so people don't come up with evil naturally, they just replicate it from having seen it before.
mircea_popescu: this isn't a complaint -- i don't personally mind evil exists, nor do i believe it shouldn't exist or that its absence would signal any kind of improvement. without evil the world's boring as fuck, which is why idiots asking dumb questions like "how come an infinitely good god has small children raped to death" are fucking stupid. evil is a better addition to the world than fucking cinema, it'd be way too borning to try
mircea_popescu: and live without.
mircea_popescu: nevertheless -- this introduces no relativism. the ethical choice stays the ethical choice, and the moral state remains the moral state. yes or no, the perpetual question, and well... https://www.youtube.com/watch?v=3CTZty62iqo

The perpetual struggle to make the right decision at every juncture is made extra difficult by the brain's tendency to turn to fatlogic. It's not enough to make one tough choice to fix a deficiency. To mature one has to repeatedly go against their conditioning while fighting against extinction bursts.

This requires forming the habit of running towards the scary. And in order to build a habit of running towards the scary, one needs a positive outlook. This positive outlook is not the rhasta "don't worry, be happy" mentality, but instead a healthy focus on positive experience that lets one bear the cost required to improve.2

The rhasta mentality is one I have lived with for oh too long. Dealing with issues via imagination doesn't work; even with heroin you can't fully escape reality. I like my cheery disposition, but I need to ensure that my happiness is grounded in reality. Growing up I saw my father3, the live warning, reach the point where he was unemployed, playing solitaire all day, and taking molly-lite.4

It is difficult to be positive while experiencing the pain from exposing flaws for dissection. Temporarily suffering to improve in the long term is required. But being alive means *always* moving in the direction of improvement, never planning to stop at some higher comfort level. So one needs to learn to bear the pain and focus on enjoying the reality that manifests from making tough choices.

  1. Is this how one quotes a footnote - by manually writing it in the bottom of the blockquote? [↩]
  2. As Naggum explains while discussing depression:

    depression is all about focusing on the negative things you observe, about expanding the impression of being "unlucky" to some kind of cosmic condition and from then on doing the opposite of what normal people do, which is to ignore the negative and the hardships because they _expect_ every ounce of positive experience to have a pound of cost, and they _know_ that errare humanum est, and that errors have costs, too. on top of this, the depressed tend not even to _see_ anything positive.

    [↩]

  3. Who always appeared to be happy [↩]
  4. Some antidepressant. I believe it was one of the popular SSRIs (Selective Serotonin Reuptake Inhibitors) [↩]

Posted in Reflection | No Comments »

ircbot no suicide on reconnect

February 2nd, 2020

ircbot-no-suicide-on-reconnect.vpatch
ircbot-no-suicide-on-reconnect.vpatch.whaack.sig

While writing fleetbot, whose functionality requires maintaining a whole lot of connections, I ran into a bug with ircbot's reconnection logic. The bug likely has stayed hidden because it manifests only when a specific thread calls ircbot-reconnect.1

The explanation of the bug is as follows:

Ircbot uses two threads. One thread is the run-thread that reads messages from the irc network and dispatches them to functions. The other thread, the ping-thread, sends a ping to the server every *ping-freq* seconds. The ping-thread also calls ircbot-reconnect if ircbot has not received a pong in *max-lag* seconds. ircbot-reconnect first calls ircbot-disconnect, which closes the connection and terminates the ping-thread. But since the active thread is the ping-thread when ircbot-disconnect is called, the ping-thread terminates itself. Thus the call to ircbot-reconnect never reaches the "connect" step of reconnection.

This vpatch changes ircbot so that ircbot-disconnect does not terminate the ping-thread if the ping-thread is nil or the active thread is the ping-thread. So now when the ping-thread successfully reconnects, it becomes the run-thread and spawns a new ping-thread to do its old job.2

In this patch I included two other changes:

1. I removed channel reconnection on kick. It's not explicitly stated in the #trilema bot spec, but I get the impression that bots are to do as told. If someone with the authority to do so kicks a bot, the bot should not fight back.

2. I fixed a misnamed parameter in make-ircbot.

1 diff -uNr a/botworks/ircbot/ircbot.lisp b/botworks/ircbot/ircbot.lisp
2 --- a/botworks/ircbot/ircbot.lisp c74e827a6c6c5cc4baf2bb72564e02a10057fd608cb1b11e504f81730ac30e2692bf54f5735faa6e4bd69be9f1ec9837cf12608551c0936653cf2880a54e2a6a
3 +++ b/botworks/ircbot/ircbot.lisp 9741409ae7fa93d709c729335c5396899d1b8f8f376d8fdd5284d9f1ddd073d1db49419fa859efe6b540c979990f16070137e849ee11e1ff57405ecdbc48a6cd
4 @@ -40,11 +40,6 @@
5 (add-hook conn 'irc-err_nicknameinuse-message (lambda (message)
6 (declare (ignore message))
7 (ircbot-randomize-nick bot)))
8 - (add-hook conn 'irc-kick-message (lambda (message)
9 - (declare (ignore message))
10 - (map nil
11 - (lambda (c) (join (ircbot-connection bot) c))
12 - (ircbot-channels bot))))
13 (add-hook conn 'irc-notice-message (lambda (message)
14 (ircbot-handle-nickserv bot message)))
15 (add-hook conn 'irc-pong-message (lambda (message)
16 @@ -65,7 +60,8 @@
17 (setf (ircbot-connection bot) nil)
18 (if (not (null (ircbot-run-thread bot)))
19 (sb-thread:terminate-thread (ircbot-run-thread bot)))
20 - (sb-thread:terminate-thread (ircbot-ping-thread bot))))
21 + (if (not (or (null (ircbot-ping-thread bot)) (equal sb-thread:*current-thread* (ircbot-ping-thread bot))))
22 + (sb-thread:terminate-thread (ircbot-ping-thread bot)))))
23
24 (defmethod ircbot-reconnect3
25 (let4)))
26 @@ -133,10 +129,10 @@
27 do (return t)))
28
29
30 -(defun make-ircbot (server port nick password channel)
31 +(defun make-ircbot (server port nick password channels)
32 (make-instance 'ircbot
33 :server server
34 :port port
35 :nick nick
36 :password password
37 - :channel channel))
38 + :channels channels))
39 diff -uNr a/botworks/manifest b/botworks/manifest
40 --- a/botworks/manifest b4d189ecf4e93ea7539d9512dc7554c44ca759712e0b035f04a850f508c84a244dae778387edd6e6a463204d35bf3ed8246ded018581cd4b8347189948cef643
41 +++ b/botworks/manifest 22d9a0246e251298ccfa31918fc9193b306aa202bf103ac2ad92b9920c40eb5cca7b10fa4e400e92b2b39a8a2483fe7d2a0295add1ce548d66b9373c12fa4928
42 @@ -1,2 +1,3 @@
43 424545 ircbot-genesis trinque ircbot genesis, http://trinque.org/2016/08/10/ircbot-genesis/
44 447104 ircbot-multiple-channels-corrected ben_bulpes multiple channel support for ircbot, http://cascadianhacker.com/correction-multiple-channel-patches-for-irclogbot
45 +615488 ircbot-no-suicide-on-reconnect whaack fixes reconnect bug + removes rejoin on chan kick, http://ztkfg.com/2020/02/ircbot-no-suicide-on-reconnect/
46

Although not included in this vpatch, ircbot would benefit from a better reconnection strategy. Currently it attempts to reconnect only once. And since the bot detected lag prior to the moment it attempts to reconnect, it is likely that that the one reconnect attempt will not succeed. So although this vpatch fixes a logical error, I am doubtful that it will increase ircbot's uptime by much. I will submit another vpatch with a more aggressive reconnection strategy once I have fleshed out all the details for its use case with fleetbot.

  1. This bug will be invisible to someone calling ircbot-reconnect in the repl. [↩]
  2. This is not the case if one started ircbot by running ircbot-connect-thread. In that situation, the ping-thread calls ircbot-connect-thread and terminates. ircbot-connect-thread creates a new run-thread as well as a new ping-thread. [↩]
  3. bot ircbot) &optional (quit-msg "..." [↩]
  4. threaded-p (not (null (ircbot-run-thread bot [↩]

Posted in tmsr | 2 Comments »

Network Socket Notes

January 31st, 2020

While working on TheFleet I came across questions difficult to answer with my minimal understanding of the system level functions used to send messages over a network. So I read a guide on the subject1 and took notes. The aim of this article is to present those notes in a manner that gives clarity to how certain c functions allow one to communicate over a network via the abstraction of writing to and reading from a unix file.

File descriptors

Each unix process has its own file descriptor table.2 Each row in that table contains the memory address of a file. When a process calls a system level function that interacts with a file, the process must pass the file's file descriptor3 as a parameter to the function.

Sockets

int socket(int domain, int type, int protocol);4

A socket is a special type of unix file used for either local inter process communication or communication over a network. The function socket creates a socket and returns its socket descriptor.5

Making a socket available to the world

int bind(int sockfd, struct sockaddr *my_addr, int addrlen);6

Calling bind tells the kernel to tie a socket to an (ip address, port) pair.7 The pair serves as the address of the socket on the internet.

Establishing connections

int connect(int sockfd, struct sockaddr *serv_addr, int addrlen);8

connect establishes a connection to a remote socket. Just like bind, connect tells the kernel to tie the given socket to an (ip address, port) pair. This pair is used as the return address for messages sent to the remote socket. The kernel picks a random available port for this purpose. Thus when a process sends a packet over the internet, it almost always uses a port number different than the one used in the foreign address.9

A process can implement typical server behavior by passing a socket descriptor to the functions bind, listen, and accept in order to establish many connections on the same local port.

int listen(int sockfd, int backlog);10

listen tells the kernel to create a limited size queue for incoming connections. Connection attempts are rejected whenever the queue is full.

int accept(int sockfd, struct sockaddr *addr, socklen_t *addrlen);

accept pops connections off of the listen queue. It returns a new socket descriptor to be used for the connection.

Established connections are uniquely determined by (source ip, source port, foreign ip, foreign port) tuples. So a machine with one ip address has a theoretical limit of 65536 * 65536 established connections per foreign ip address. Resource limitations make the maximum number of possible connections much smaller in practice.

One can see established connections with the command:

netstat -natu | grep 'ESTABLISHED'

Sending and receiving messages

Once a socket has an established connection, the functions send and recv are used to write and read from the socket.

int send(int sockfd, const void *msg, int len, int flags);
int recv(int sockfd, void *buf, int len, int flags);

send and recv do not guarantee to write or read all len bytes.

send returns the number of bytes actually sent or -1 on error.

recv returns the number of bytes actually read, 0 if the remote side closed the connection, or -1 on error.

Closing connections

int close(int sockfd);

close, the function used for closing files in general, is used to close a connection.

  1. Perhaps I should have spent more time looking for a better source of information. The author of this guide one seems to have some right ideas and occasionally is funny,

    They say absence makes the heart grow fonder, and in this case, I believe it to be true. (Or maybe it’s age.) But what I can say is that after a decade-plus of not using Microsoft OSes for my personal work, I’m much happier! As such, I can sit back and safely say, “Sure, feel free to use Windows!” …Ok yes, it does make me grit my teeth to say that.

    but despite knowing better he bows to the crowd

    At this point in the guide, historically, I’ve done a bit of bagging on Windows, simply due to the fact that I don’t like it very much. But I should really be fair and tell you that Windows has a huge install base and is obviously a perfectly fine operating system.

    The guide also includes sections on writing IPV6 compatible code. I skimmed over them. IPV6 is the network equivilant of Gavin monster blocks. [↩]

  2. If a process forks the child process is given a copy of the parent process's table. Changes in the two tables are not shared between the child and the parent. If both parent and child try to read from the same file, the one that calls read first gets the data. [↩]
  3. i.e. the index of the file in the process's file descriptor table [↩]
  4. The parameters given to socket determine whether the socket will use the tcp or udp as well as whether it will be used for network or local communication. [↩]
  5. A socket descriptor is a file descriptor that refers to a socket. Socket descriptors and file descriptors share the same table. [↩]
  6. sockaddr is a struct of address information that includes the (ip address, port) pair. [↩]
  7. The ip address can be a loopback address such as 127.0.0.1 if the socket is being used for local communication. [↩]
  8. for connect, sockaddr refers to the remote address. [↩]
  9. Previously, I had the misconception that the same port was used for each side of the client and server pair. [↩]
  10. The parameter backlog determines the length of the queue of incoming connections. It should be <= to the system limit, which is usually around 20 [↩]

Posted in Reflection, tmsr | 1 Comment »

A confession of how I waste time 2

January 29th, 2020

A confession of how I waste time 1

Last week my productivity slacked. I constantly paused work to do mindless activities. Lack of focus is an ongoing problem,1 but this past week my symptoms were more severe than usual. I figure one way to address the issue is to write an article revealing some of the embarrassing ways I waste my time.

The first is I listen to music during work and in between tasks. Before I do the dishes or some mundane chore, I put on a song. This makes all the small jobs I need to do around the house take longer. I recently knocked over a pot of hot coffee while changing a song, splattering shattered glass and liquid all over my kitchen.

I've mentioned this problem before. What I didn't mention was the exact "music" I listen to. I always put on something by Russ, a rapper who raps about being hard working and self made. So yeah, just about everyday for months I've been being lazy and listening to the same chord progressions accompanied with lyrics about how someone else's hard work led them to success.

The lyrics of Russ's song, Me You, are ohso fitting. Russ raps about the difference in work ethic between him and the listener.

...
Yeah, mm yeah, there's a difference
Oh yeah, there's a difference
Oh, yeah, there's a difference
There's a difference, yeah
Yeah, yeah, yeah

Me, got it out the mud, they respect that
Me, always spread the love when I get back
You, got your hand out, that's a bitch move
You, always complaining like a bitch do
Me, still right here with who I came with
Me, self-made, my circles on the same shit
You, got a different crew every weekend
You, don't want it that bad, keep sleeping
Yeah, you wished for it, me, I worked for it
...

2

Another way I burn the precious minutes of life is by watching surf videos on youtube.3 I've tried to prevent myself from doing this by null routing www.youtube.com by inserting "0.0.0.0 www.youtube.com" into my hosts file. But the fake resistance doesn't help, I always find some justification for commenting out the line.

These two specific avoidance behaviors are likely symptoms and not the source of my problem. But maybe analyzing the symptoms can help discover the root. The connection I see between the two aforementioned activities are they are low effort ways I use to hallucinate success. While listening to Russ, I am imagining myself as having already reached success through hard work. While watching the surf videos, I am imagining myself on large perfect waves. I might as well spend the time watching videos on other sites. I'd be doing the same thing except I'd have something to show for it when I finish.

  1. As I'm sure it is in the lives of most. [↩]
  2. Quite the poetry when you see the lyrics written out, yeah, yeah, yeah. [↩]
  3. I also go out and surf, that is after all a major reason I live where I do now. But I've reduced the time spent surfing to a healthy 5-7 hours per week. It has been easy to not avoid work through surfing, because there has been very little swell the past few months. Am I going to be able to forgo getting in the water when I see glassy barrels out my window? [↩]

Posted in Uncategorized | 6 Comments »

Fleetbot Pseudocode, Bugs and Fixes, and the New Next Steps

January 26th, 2020

Below is an overview of how TheFleet works on a per network basis. The article also touches on a bug I found in cl-irc, enumerates the hurdles I have to overcome related to orchestrating the fleet of fleetbots, and lists my next steps forward.

Table of Contents:

I. Fleetbot class template
II. Connecting basics
III. How fleetbot handles various irc messages
IV. Reconnecting
V. Notes on logging
VI. A bug in cl-irc
VII. Problems with orchestration
VIII. Next steps

I. Fleetbot class template

Below is the fleetbot class template. Some of the fields are inherited from its superclass, ircbot.

Field Name Datatype Description
db list a list of four strings (db_name, db_user, db_user_password, db_ip_addr) used for connecting to the postgres db.
sunk-p boolean set to true if the bot has disconnected, false otherwise
resurface-attempts integer the number of consecutive failed attempts to connect to a network. gets reset to 0 upon a successful connection. a connection is deemed successful when we receive our join message from the server1 for any channel in our lists of channels.
connection cl-irc class an object representing the connection to the network. ~all cl-irc commands take this object as the first parameter.2
channels list a list of strings that are the names of channels the bot will attempt to connect to.
active-channels list a list of strings that are the names of channels we are currently connected to. upon being kicked from a channel, we remove the channel from active-channels. upon being disconnected, we remove all channels by setting active-channels to nil. we add the corresponding channel to active-channels when we receive back our join message from the network.
networkname string the name of the irc network the bot connects to
server string the hostname of the network
port int the port number of the network (usually 6667)
nick string the nick we use for the network
current-nick string the nick that we are using for our current connection. this is the same as nick, unless the server told us we could not use nick (most likely because nick is already being used.) in this case we append "-???" to nick where each ? is a random digit 0-9.
password string the password for the nick. since we do not use registered nicks for fleetbot, this field is always nil.
connection-security :ssl or :none the connection security for the network. currently we do not connect to networks via ssl, so this is always set to :none
run-thread sb-thread the thread that is used for communicating with the server
ping-thread sb-thread the thread that is used for sending pings to the server
lag int the difference between the last time we sent the server a ping and the time we received the server's corresponding pong.
lag-track list everytime we send the server a ping we store a reference to the ping in the lag-track. when we receive the corresponding pong from the server we delete the ping from the lag-track. if we have not received a pong in response to a ping for > *max-lag* (default = 60) seconds we determine we have disconnected, and the ping-thread will attempt to reconnect our bot to the network.

II. Connecting basics

The fleetbot constructor takes as parameters: the server information (networkname, server, and port), a nick, a list of channels, and the db used for logging.

We then connect fleetbot via the overwritten method ircbot-connect-thread. ircbot-connect-thread creates a new thread, assigns that thread to the bot's run-thread field, and within the new thread calls ircbot-connect.

ircbot-connect is surrounded in a handler-case.3

First, ircbot-connect calls cl-irc's function connect, which takes a nickname, server, port, and connection-security and returns a connection object. The connection object provides an interface to send and receive messages from the irc server via cl-irc's api. Next, we use cl-irc's api to add a list of hooks to the connection object. A hook is a mapping of an irc message type to a function. When we receive a message via the socket stored in the connection object, cl-irc searches for hooks that are assigned to the message's message type and calls the hook's function with the message passed as the sole parameter.

If at any point an error is thrown, then we mark the bot as "sunk" and attempt to reconnect the bot. If we cannot reconnect after +MAX-RESURFACE-ATTEMPTS+ (default = 5) attempts then we determine the bot has sunk for good and stop trying to reconnect.

III. How fleetbot handles various irc messages

We have the following hooks setup:

IRC Message Type Function Description
irc-err_nicknameinuse-message We change our nick by appending a random suffix. (i.e. jenny -> jenny-482)
irc-kick-message The intended behavior is to log that we are kicked and then remove the channel from our active-channels.

Currently upon being kicked from a channel, we have a bug where we immediately try to rejoin all channels in fleetbot's channels field.4

irc-notice-message A notice message is an arbitrary message from the server. Different networks send different notice messages. For our purposes, we parse the message to see if the server is telling us that the nickname we're trying to use is registered. If so, then we pick a new randomize nick, just like we do when we receive irc-err_nicknameinuse-message. Afaik this is only useful for freenode.
irc-pong-message When we receive a pong message we calculate our lag with the server and mark that the server has received our ping.
irc-rpl_welcome-message Upon receiving the welcome message from the server, we start the ping-thread and connect our bot to its channels via cl-irc's join method.
irc-privmsg-message A privmsg is any normal user message sent to a target.5 We log privmsg's to the irclog table in our postgres db.
irc-part-message A part message gets sent when a user leaves a channel. We log these to our irclog along with privmsg's.
irc-join-message A join message gets sent when a user joins a channel. We log joins to our irclog along with part and privmsg's. If the join message is saying that ~we~ joined the channel, then: we add the channel to our list of active channels, we log to fleetlog that we joined a channel, and we set our consecutive-resurface-attempts back to 0.

After we set all the above hooks, we enter an infinite loop reading messages from the server. If we have a hook for the message type of a message we received, we dispatch to the corresponding function. If we receive a message type that we don't have a hook for, the default-hook from cl-irc gets called, which usually just prints the message to STDOUT. If at anypoint during our infiniteloop the server sends us an EOF or we throw any error, we attempt to reconnect.

IV. Reconnecting

Reconnecting is a bit tricky because there are two different threads that can call reconnect: the bot's run-thread and the bot's ping-thread. The run-thread would be reconnecting because the run-thread received an EOF from the server or hit an error during its operation. The ping-thread would be reconnecting because it hasn't received a pong to one of our pings from the server in *max-lag* seconds.

The reconnect flow is as follows:

First, we call ircbot-disconnect. ircbot-disconnect sets sunk-p to true6 and then logs an internal DISCONNECTED message to fleetlog for all the active-channels the bot was connected to. Then we set active-channels to nil. If we have an active connection to the network, we close that connection. Then we set the ircbot's lag-track and ircbot-connection to nil. Then, if we are calling disconnect from the run-thread, we kill the ping-thread. If we are calling disconnect from the ping-thread, we kill the run-thread.7 We then set the ping-thread to nil.

Once we've disconnected cleanly, we attempt to connect. If we are the run-thread we call ircbot-connect. If we are the ping-thread, we make a new thread with ircbot-connect-thread, and then call sb-thread:abort-thread to end the current thread.8 Then we set sunk-p to false.

V. Notes on logging

Our schema has two tables - irclog and fleetlog. We insert into irclog the types of messages one would normally see in their irc client: privmsg, join, part, and kick messages. Fleetlog, on the other hand, keeps a record of our bot's events that we choose to store. For each event we log: a custom message describing the event, the channel (if applicable), the nick (if applicable), the networkname, and the time the event occurred. The custom messages currently logged are:

Message Description
JOINED logged for a channel+nick when we receive a join message from the server where we were the ones who joined
KICKED logged for a channel+nick when we get kicked from a channel
DISCONNECTED logged for every active-channel when we call ircbot-disconnect
COULD-NOT-RECONNECT logged for every channel we failed to connect to when we have failed a consecutive series of attempts to reconnect
ARMADA-ALL-DEAD logged for a network when all bots (within a process) connected to a network have crashed
PROCESS-TERMINATED logged for a network when we end the process running fleetbot (either because all ships have sunk or because the process received a kill signal)

The schema for the postgres db is pasted below. Noticably missing is the indexing for fleetlog.


set search_path = public;

create table irclog (
    id serial primary key,
    target text not null,
    message text not null,
    host text,
    source text not null,
    "user" text,
    networkname text not null,
    irc_message_type text not null,
    received_at timestamp without time zone not null default (now() at time zone 'utc')
);

create index irclog_received_at on irclog (received_at);
create index irclog_target on irclog (target);
create index irclog_source on irclog (source);
create index irclog_networkname on irclog (networkname);

create table fleetlog (
    id serial primary key,
    channel text,
    message text not null,
    nick text,
    networkname text not null,
    received_at timestamp without time zone not null default (now() at time zone 'utc')
);

Also, when I run fleetbot, I redirect standard out to a log file. The log file contains prints of unhandled irc messages and debugging print statements I put inside fleetbot.

VI. A bug in cl-irc

I gave the cl-irc source a pass and noticed a bug that affects fleetbot. cl-irc has a global variable *unknown-reply-hook* that can be assigned a function. That function is supposed to be called anytime an irc network sends a message of unknown type. However, the code that throws the no-such-reply error in cl-irc is malformatted.

This9

(error "Ignore unknown reply." 'no-such-reply :reply-number reply-number)

should be:

(error 'no-such-reply :reply-number reply-number)

The bug in cl-irc made the no-such-reply error bypass cl-irc's own handler-case10, throwing the error upstream to fleetbot. Fleetbot handles all errors by reconnecting. So upon receiving an unknown reply, fleetbot would reconnect and then usually receive the same unknown reply, causing fleetbot to reconnect again ad infinitum. I will patch cl-irc with the above fix and then set *unknown-reply-hook* to a function that logs unknown replies to fleetlog.

VII. Problems with orchestration

I need to rewrite how I orchestrate the fleet of fleetbots. Here are the constraints I am dealing with:

1. An irc network typically only allows 3 connections per IP.

2. A VM with 1GB of RAM costs me $5 / month.

3. A unix process running sbcl and asdf consumes 30MB of RAM at minimum. Currently a running fleetbot consumes closer to 100 MB.

4. A unix sbcl process has a maximum number of threads that it can have running concurrently.11

5. A unix sbcl process has a maximum number of sockets it can have open.

6. A unix process has a maximum number of filedescriptors it can use. (adjustable with ulimit)

VIII. Next steps12

1. Publish vpatch addressing ircbot's reconnect bug

2. Fix reconnect on kick bug

3. Increase delay before reconnecting to a network

4. Patch cl-irc + create way to distribute fix to VMs (after I patch I can no longer load cl-irc via quicklisp)

5. Set *unknown-reply-hook* to a function that logs unknown replies to fleetlog

6. Fix fleetbot's db schema

7. Write article planning how to address orchestration of bots

  1. To join a channel, we send the server a join message. If the join is successful, we receive the same join mesage we sent back from the server. [↩]
  2. Under the hood, connection uses usocket:socket-connect to create a socket connected to the irc network. Then the socket is passed to usocket:socket-stream to get the network-stream. cl-irc creates an output-stream (used for sending messages to the network) by passing the network-stream to flexi-streams:make-flexi-stream. I haven't explored the usocket nor flexi-streams library at this time. [↩]
  3. A handler-case is Common Lisp's version of a try/catch block. [↩]
  4. I discovered this only now and it is the cause of problems I've run into.. The weird part of the bug is that we rejoin all channels when we are kicked from only one channel. This may explain why I saw "ERR_TOOMANYCHANNELS" messages despite limiting the length of channel-list to the max number of channels alllowed per nick on the network. [↩]
  5. A target is either a nick or a channel. [↩]
  6. I realized that ircbot-reconnect was setting sunk-p to false, ~before~ we call ircbot-disconnect. So all of our bots that reconnected once were being incorrectly marked as sunk. This has been fixed. [↩]
  7. This is handled incorrectly in the current version of ircbot. In ircbot, the ping-thread kills the ping-thread (itself) when trying to reconnect, thus crashing the bot. Since there is a few pieces of republican infrastructure sitting ontop of ircbot, it is a top priority to create a vpatch that fixes this. [↩]
  8. But I realize there is no reason for this "create new thread and then self destruct". At this point the ping-thread can stop doing its pinging job and just become the run thread. [↩]
  9. I do not know why the author of cl-irc put the string "Ignore unknown reply." as the first parameter. The first parameter instead should be the condition type. [↩]
  10. try/catch block [↩]
  11. 2048 iirc. There is likely a way to increase this number. [↩]
  12. Updated from my last plan [↩]

Posted in tmsr | 1 Comment »

« Older Entries
Newer Entries »