Archive for the ‘tmsr’ Category

Vpatch Study Part 2 - Topological Sort Example

Friday, October 18th, 2019

I have written a program that topologically sorts a list of python tuples representing vpatches1 as an exercise to understand V. A few tests are included. From this exercise I learned that I don't know exactly what it means for a vpatch to be a parent of another vpatch. I hope to resolve this point of confusion shortly.

I do not plan to continue working on this program unless otherwise directed. My next step is to publish my understanding of the what and why of V, followed by my annotations of asciilifeform's v.py.

  1. The datatype has the minimum information I believe is necessary to be able to perform a topological sort. []

V Study Part 1 - Vpatches and Vdiff

Tuesday, October 15th, 2019

Creating source using V is done by sequentially applying a set of vpatches through a process known as pressing. To press, V is given the most recent vpatch and an output directory. V then finds a path from the given vpatch to the genesis vpatch. Starting with the genesis vpatch, V applies each vpatch along the found path and dumps the result into the given output directory. In this post I go over how the vpatches used in this process are created.

To make a vpatch, a developer starts with a copy of the source already pressed to the previous most recent vpatch. We'll say for example this source is in a directory named oldversion. The developer then copies the source in oldverison to another directory that we'll call newversion. In the directory newversion he makes the source modifications that will constitute the vpatch. When finished, the developer runs

vdiff oldversion newversion

An example of the code for the vdiff program, taken from the bitcoin foundation, is reproduced in one line below:


diff -uNr $1 $2 | awk 'm = /^(---|\+\+\+)/{s="sha512sum \"" $2 "\" 2>/dev/null  " | getline x; if (s) { split(x, a, " "); o = a[1]; } else {o = "false";} print $1 " " $2 " " o} !m { print $0 }'

Running vdiff on the two directories creates the vpatch file, which is similar to a diff file obtained from running

diff -uNr oldversion newversion

The difference is vdiff replaces vanilla diff's file modification timestamps with hashes1 of the file's content. spyked articulates the importance of this in a recent thread he had with me in #o.

spyked: whaack, problem is that classical diff/patch leave room for ambiguity, i.e. in principle it's possible to (cleanly) apply a single hashless patch to different files, which results in different presses. so hashes are needed in order to identify the file (not only path/name, which is only metadata required for retrieval) as it is before/after applying the patch.

I still need to fully digest the awk command that is replacing the file timestamps with the file content hashes. But one quirk I noticed was that certain crafted files would cause the awk command to incorrectly match on certain lines. For example if you have


$tree
.
├── newversion
│   └── fool.txt
├── oldversion

$cat newversion/fool.text
++ trick.txt this_should_be_in_the_vpatch2

then

$vdiff oldversion newversion

will produce


diff -uNr oldversion/fool.txt newversion/fool.txt
--- oldversion/fool.txt false
+++ newversion/fool.txt 27991f54fb2534c59b6c0667f9a91d8bd9173b5cc3184aeea251c2435b7808457a5492add5646793738a1f3e9c32892a2261e18eb0e3a2d0d7a0486735bf43a8
@@ -0,0 +1 @@
+++ trick.txt false

the last line should be

+++ trick.txt this_should_be_in_the_vpatch

but it was mistakenly altered by the awk command. This incorrect modification to the vpatch makes the resulting fool.txt file have the wrong contents after pressing.3 However if, while pressing, V checks that the hashes of the resulting files match the intended files hashes found in the vpatch, V will correctly spot this error and fail to press. This gives an example of how diana_coman was right when responding to my point of confusion here

whaack: got it, i understand that the hashes are needed to identify the files. but regarding hashing the files yourself after every patch, the vpatches already let you know what the output hash will be. so if you trust the vpatch to the point where you're going to run the code outputted by it, then you should trust its claim of what the output of the hash would be. hashing the output files yourself after every patch then becomes more of a
whaack: data-integrity check.
diana_coman: the vpatches let you know what the output hash *should be*
diana_coman: nobody can let you know upfront what it *will be*; in general

  1. Originally, the hash function used in vdiff was sha512, as I have in the vdiff program I posted. Now the hash function used is keccak. The benefits of using keccak over sha512 are beyond the scope of both me and the post. []
  2. Note that I put two +'s at the start of the one line in this file. To show this line was added, the diff command's output will contain a "+" followed by the line's contents. This will cause there to be a line in the diff output with three sequential +'s that refers to a file's content. The awk command will incorrectly match to this line and attempt to replace this_should_be_in_the_vpatch with the hash of the non existent file trick.txt. []
  3. "this_should_be_in_the_vpatch" was replaced with the word "false" because the hash of the file "trick.txt" does not exist. []

V Study Reference Links

Tuesday, October 15th, 2019

My master diana_coman has assigned me the task of creating a report on my understanding of V that includes a v.py with my own annotations. I have collected related links for my reference as well as for the reference for anyone else on a similar path. I will update this post as I locate more material.

  1. ben_vulpes's V-tronics 101: A gentle introduction to The Most Serene Republic of Bitcoin's cryptographically-backed version control system 1
    1. mp's ode to V / ode to Genesis patch2
    2. mp's v manual genesis.
    3. mod6's perl v
  2. esthlos's a v-tron, a cl V he made himself. He has many updates to this vtron that are found on his homepage.
  3. asciilifeform's v.py3
  4. A note from spyked's blog about the shortcoming of v.py.
  5. trinque's v-manifest spec draft4
  1. This post has links to other useful material some of which I include on this post as well. []
  2. link dead at time of writing mod6 has fixed the link. []
  3. This was written by asciilifeform but I only have found phf's signature. There is a patch by phf to have asciilifeform's v.py use vtools. []
  4. Linking archived version until trinque recovers his blog. []

Recovery

Thursday, October 10th, 2019

The little hut known as ztkfg.com was engulfed by the flames of the recent fire within tmsr. This blog now temporarily lives in enemy territory while I wait for republic lands to return to their inhabitable state.

Previous blog posts should be restored shortly.

UPDATE: Previous posts have been salvaged from the fire.1

  1. Thanks to BingoBoingo saving my ass with the help of trinque. []

Past TMSR Work, Potential Future TMSR Work

Sunday, September 22nd, 2019

My contributions to the republic, while having spent years twiddling my thumbs reading the logs, are as follow:

1. 5 Qntra posts, found here: 1

My first post was an inside perspective of MIT's "blockchain" curriculum. It confirmed what the republic already knew, namely that there was no interesting work going on at MIT re bitcoin, and any "work" being done there was hostile towards republican interests. Two other posts were tabloidal, making fun of pantsuitism. And the last two posts were reports on Coinbase shenanagins during the bitcoin cash hard fork.

2. Setting up my own trb bitcoin node. (failed)

There was naivety in my attempts to setup a running bitcoin node. When I first attempted to setup a node, I tried to get it goig on an old unused laptop. One mistake was believing that 2gb of ram is enough to get a timely block sync. I had thought at the time that the only bottleneck to getting a node up to speed was downloading the blocks, and I did not intuit the time it takes to locally verify all the blocks along the way2 I later attempted to sync a node on a dedicated machine hosted by dreamhost.com,3 paying a little over $100 usd per month. I can't quite recall what happened, but I think around block 350,000 it got stuck. Later, without trying to reboot bitcoind, I decided to cut my expense with dreamhost and gave up on running a full node.

3. Researching how many bitcoins are tied up in P2SH4 (failed)

The goal was getting an upper bound of how many coins are in anyone-can-spend scripts in order to answer the question: how many coins are in addresses related to segwit?

To do this, I first used ben_vulpes's block explorer5 to grab sexprs containing the data for every block. This was obtained by looping from 0...max_block_height and running
wget -0 http://mimisbrunnr.cascadianhacker.com/blocks/blk{n}
where {n} was the block number. While I was running a loop performing this task I noticed that occasionally ben's block explorer would give me some malformed file - and I had to simply re-wget the same url until i got back a properly formatted sexpr. It took a while to download all the blocks from Ben (even though I was not verifying them) and so I paid for a digitalocean droplet to run my scraper script on.6

Once I had blocks 0..n, I ran a script7 that would go through a chunk of blocks and keep an ongoing hashmap mapping "(txn hash, output number) -> num_satoshis_sent_to_output" for all the outputs in the block that were sent to non-trb conforming addresses. For each new block, the script would first iterate through the txns in the block to see if any of them spent the coins in the ongoing hashmap obtained from all the previous blocks. If a txn in the new block consumed one of the P2SH UTXOs that was being stored, that UTXO would be deleted from the ongoing hashmap. Once the purge of transaction outputs that had just been spent was completed, the script reiterated through the new block's txns to add any txn outputs that were directed to non-trb P2SH's to the ongoing hashmap. After iterating through all the blocks, one could calculate how many satoshis were in non-trb addresses by summig up all the values in the obtained hashmap.

I don't recall at what point/why I just faded away and stopped working on this tool. It may have been because I hit a problem with running out of memory for storing all the segwit UTXOs. It was an interesting investigation and perhaps the republic would still find a counter of coins that are contained in non-trb addresses useful. Which brings us to part two of this post:

Potential Future TMSR Work

diana_coman: whaack_pura_vida: that8 is obsolete so not a lot of help in itself; nobody is going to make the list ready for you to pick and choose, wtf.
diana_coman: whaack_pura_vida: publish what you figure out by Sunday together with *how you went about* the figuring out

My initial internal response to diana_coman was " (1) why is that a ridiculous expectation since there previously was a list of entry points? and (2) how is that list obsolete if a young hand such as shrysr is digesting V, which is more or less a task on that list?"

The best answer I can come up with to my own questions are "Yes, a list was once generously made, but doesn't mean that lords have time to keep an up to date task list for noobs. That post was THREE YEARS AGO and now there are new tasks to do - which you must find yourself. The current task list may or may not coincide with the three year old post, you have to have read the logs to find out."

With that being said, and keeping with the "how you went about figuring out", here is a list of potential tasks, with an annotation denoting how/why I came to choose that task.

1. Creating my own V9
2. Related to 1, taking up the task of maintaining a vpatch viewer10
3. Creating a new trb block explorer11
4. Continuing fighting the war on Segwit, first by completing the task of sizing up the coins held in P2SH. 12
5. Learning ADA and completing Stan's FFA series.13

Tasks (3) and (4) seems the most interesting to me, but I believe the v-related tasks (1) and (2) should be my starting point.

  1. These were edited by BingoBoingo, and one was improperly formatted wthen sent to him. So these contributions may have even been net negative depending on how much time BingoBoingo had to spend to correcret my mistakes. []
  2. My intuition was likely skewed because my first experience with running a bitcoin node was using power ranger software which used SPV, effectively making my computer search for the longest chain instead of the longest valid chain. []
  3. Originally I had thought, what help is it to run a node on someone else's iron? I still believe it is not that useful, you are only temporarily increasing the redundancy of the bitcoin network, but at any moment the enemy can flip a switch and you go offline. Adding a node to pizarro also has dubious utility, because from my understanding the republic already has a few nodes there at 161.0.121.248 and 161.0.121.250. []
  4. pay to script hash []
  5. dead at time of writing []
  6. This was also discontinued when I did some cleaning out of expenses. And I wiped everything off the droplet without first taking a local copy. []
  7. A lot of CL weird and sloppy code. Some of it is copy and pasted from code Ben was using to analyze his own block explorer. []
  8. http://trilema.com/2016/how-to-participate-in-the-affairs-of-the-most-serene-republic/ []
  9. The initial idea was planted by the trilema post of entry to affairs. That being said, V seems a natural starting point for working with republican code. It demonstrates understanding of the tool required to publish and use any code in the republic []
  10. per the suggestion of trinque. []
  11. Found this may be useful by going through old tasks and noting that Ben's old block explorer mimisbrunnr had died. []
  12. I figure that the lords best spend their time fortifying their castle walls rather than going out to fight against nonsense like Segwit. But perhaps a noob could prove his worth by taking on this neglected task. []
  13. This task seems a useful start for the same reason the V tasks seem useful: to prepare a young hand by learning the tools used to contribute to the republic. In addition, from my understanding only a few have gone through any of Stan's series. But apart from the additional proofread, this is a personal development goal rather than a contribution. []