Very Secure

Notes on how to speed up the trb resync process after data corruption

July 14th, 2020

I was syncing my local trb node and I saw this error in my debug.log file:

NSt8ios_base7failureE

My most recent .dat file had been corrupted, from a cause unknown but most likely from my computer powering off during a write operation. I took some notes on how to avoid having to redownload all my hard earned blocks. The commands I've pasted below are specific to my case, but you can use them as an example.

I) Get asciilifeform's blkcut tool.

II) Remove your most recent blk000n.dat file, since it should be the file that is corrupted. Then cut up the remaining blk000*.dat files and organize the individual block files into a directory. How you do that second step is up to you. I moved all the blocks to ~/.bitcoin/cutblocks.

rm ~/.bitcoin/blk0003.dat # corrupted file.
cutblock ~/.bitcoin/blk0001.dat
cutblock ~/.bitcoin/blk0002.dat

mkdir ~/.bitcoin/cutblocks

for i in {0..9}
do
mv blk0001.dat.$i* ~/.bitcoin/cutblocks; mv blk0002.dat.$i* ~/.bitcoin/cutblocks;
done

III) Remove your old .dat files. I put them in a backup folder.

mkdir ~/bitcoin-data-backup
mv ~/.bitcoin/*.dat ~/bitcoin-data-backup

IV) Start bitcoind with the -caneat flag.

LC_ALL=C nohup ./bitcoind -caneat -myip=234.3.12.222 -addnode=108.31.170.3 -addnode=205.134.172.27 -verifyallll 2>&1 &

V) Now we need to run a script to eatblock on all our blocks. First find out how many blocks each old blkdata file had.

cutblock -c ~/bitcoin-data-backup/blk0001.dat

Then we load those blocks with a script to iterate through the eatblock commands. Run the following command replacing 188528 with the number you received from cutblock -c minus one1. If you have more than 1-3 blk000*.dat files, you'll want to create a more sophisticated script.

for i in {0..188528}; do echo "eating block $i"; ~/v/trb054/bitcoin/build/bitcoind eatblock ~/.bitcoin/cutblocks/blk0001.dat.$i.blk; done
  1. The first blk is blk0001.dat.0.blk []

Representing Data - Notes on Big And Little Endian Encodings

July 13th, 2020

While working on my block explorer I realized that my grasp on what it means for a system to be big or little endian was weak at best. I took the time to think about what exactly was the difference between the two. In the process I cleared up confusions in my overall understanding of encodings.

***

At a lower level, data structures in computer programs are all kept as series of bits. The state of these bits are stored on a piece of hardware such as a flash drive, an ssd drive, a mechanical hard drive, a stick of RAM, a CD, a cache, etc. Each bit has some physical location within its storage device. We can abstract this observation and give each location of a bit state a unique numerical address. Ostensibly numerically adjacent addresses map to physically adjacent bits, but this need not be the case.

We are often curious to know the state of the bits on our physical devices. Our goal becomes to bring an invisible state of the world into our field of consciousness. To do this we need our machine to read the bits on its storage device and construct light on our monitor that form characters we can perceive with our eyes. The characters we choose to represent the raw data are usually the same ones we use when we write numbers in hexadecimal: 0123456789ABCDEF. Since there are 16 characters in our symbol system, each character can represent an ordered list of 4 bits. To get a better understanding, we can work through an example.

Here is a mapping of addresses to bit states, with the address written in base 10 and bit state written as a 1 for on and a 0 for off.

0  -> 1
1  -> 0
2  -> 1
3  -> 0
4  -> 1
5  -> 1
6  -> 1
7  -> 0
8  -> 1
9  -> 1
10 -> 0
11 -> 1
12 -> 1
13 -> 1
14 -> 0
15 -> 1

Writing out the 1's and 0's with the lowest address on the left, the above becomes:

1010111011011101

If we want to represent the data above with the hexadecimal characters we do the following. We write out a hexadecimal string such that if we were to convert each hexadecimal character into its 4 digit binary equivilant we would have the bit state of each address, with the leftmost bit being the lowest address (0) and the rightmost bit being the highest address (15)


1010 | 1110 | 1101 | 1101 |
  A  |   E  |   D  |   D  |
AEDD

It's important to understand that so far we are not using the hexadecimal characters nor their 4-digit binary equivilant to represent numbers. These characters are only being used to represent the state of the data as it is stored in some storage device.

However, we do often use the state of chunks of memory to represent numbers. Coming up for a standard for how bit states map to numbers allows one to do arithmetic operations. We could, in theory, come up with any arbitrary mapping. For example, one could make the 5th position in memory represent the most significant bit of a number, then the 0th the 2nd most significant bit, the 9th the 3rd, etc. This would admittedly be a stupid and confusing scheme for representing numbers. In practice there are two prevailing conventions - big endian and little endian.

In a big endian1 system, the bit state at the lowest address represents the most sigificant bit in a binary number and the state in the highest address represents the least significant bit. The physical state of the machine above in a big endian context represents the following number: (I've written the same number 3 times in different bases)

1010111011011101 (base 2)
AEDD (base 16)
44,765 (base 10)

In little endian, the least significant byte is placed at the lowest memory address. However, within that byte, the bit with the lowest address is the most significant.

0  -> 1 (9th  most significant bit)
1  -> 0 (10th most significant bit)
2  -> 1 (11th most significant bit)
3  -> 0 (12th most significant bit)
4  -> 1 (13th most significant bit)
5  -> 1 (14th most significant bit)
6  -> 1 (15th most significant bit)
7  -> 0 (16th most significant bit)
8  -> 1 (1st  most significant bit)
9  -> 1 (2nd  most significant bit)
10 -> 0 (3rd  most significant bit)
11 -> 1 (4th  most significant bit)
12 -> 1 (5th  most significant bit)
13 -> 1 (6th  most significant bit)
14 -> 0 (7th  most significant bit)
15 -> 1 (8th  most significant bit) 

When in the context of little endian, the same above state now represents a different number, again written 3 times in different bases:

1101110110101110 (base 2)
DDAE (base 16)
56,750 (base 10)

If you tell a computer "store 56,750 starting at address 0 for me", the state of the bits at the addresses in memory on a big/little endian machine will differ. But if you give the more specfic instruction "write byte 9F at address 0", then the resulting state will be the same on the two machines.

***

As jfw points out, some fields in trb have the endianness of their internal and external representations swapped. So when you extract the hex characters that represent the "state of the machine" that is storing the block data, you need to (sometimes) reverse the order of the bytes so that the resulting hexadecimal string represents the correct numerical value for the field you're observing.

Creating/finding an exhausitive list of which fields in trb are represented in little endian and which are represented in big endian is on the todo list.

  1. Also known as Network Byte Order, since it is the standard way to organize bits to send accross networks. []

A Bitcoin Block Explorer - The Why & The How

July 9th, 2020

I've embarked on a mission to create a bitcoin block explorer. My motivation is the network is sorely missing a public block explorer that has a public vpatch'd source. Afaik the only public block explorer made by someone in the web of trust was ben_vulpes's mimisbrunnr, which has been dead for years.

Current heathen block explorers1 have the wrong ethos. They do not publish their source prominently, they have no guarantee that they won't break their api, they support all sorts of various scam coins, they support non-standard bitcoin transactions, they likely run off prb, etc. etc. They are operated by amorphous identities, rather than by an individual in the web of trust.

So a better block explorer is a low-hanging fruit. But it's worth questioning whether making a block explorer that serves data over http is worth creating in the first place. Bitcoin operators should treat their locally verified blockchain as the source of truth instead of the output of some site.2 The thing is... trb operators cannot view what's inside their blockchain because the design for its storage is atrocious.3 If trb stored its data in a sane manner to begin with, a public block explorer wouldn't be necessary. Trb operators would just query their local blockchain.

Since we are stuck with the trb abomination, what's needed is a second program that monitors the growing tumor blockchain and packages the information into a second, sane database. This program, which provides a reasonable way to actually understand wtf's inside all those blk****.dat files, is likely desired by ~all trb node operators.4 So while I do plan to provide an http service, the goal is first and foremost to make a block explorer that one can run locally on top of trb.

Much of the hard work for this job has already been completed by jfw. He released gbw-node, a piece of software for the online portion of his wallet that provides a way to track the balances of provided addresses. His program scans the trb blockchain, inserts relevant information in a sanely structured sqlite3 database, and then provides a way to query that db.

A few tweaks and additions are needed to make his gbw-node provide a way to access all the information that a block explorer is generally expected to provide. For example, the db needs a new table for blocks. Also, instead of tracking specific addresses, the gbw-node needs to track all addresses.5

The next task on my plate is figuring out the full list of necessary changes and creating an estimation of how much storage space trb + the mirror'd db will require. Meanwhile I need find a home to host my block explorer and begin syncing a trb node in that home asap.

  1. Here are some:

    https://www.blockchain.com/explorer
    https://blockchair.com/
    https://tokenview.com/en
    https://explorer.bitcoin.com/btc
    https://insight.bitpay.com/
    https://tradeblock.com/bitcoin/explorer
    https://bitcoinchain.com/block_explorer
    https://btc.com/
    https://live.blockcypher.com/
    https://blockchain.coinmarketcap.com/ []

  2. Perhaps a proper http block explorer signs its responses to queries when requested. []
  3. And trb is something that the world is stuck with. Fork the source and eventually you'll fork the chain. And who has the authority to say that their new chain, one that is not the accepted longest chain in og trb, is to be called "Bitcoin" proper? Bitcoin defends itself economically, you know. []
  4. So long as that node operator is using trb to manage their own bitcoins, rather than just to place a node in the world to increase the robustness of the network / number of copies of the blockchain. []
  5. Arguably this is a feature a wallet user with extra storage space / cpu cycles to spare would want anyways. []

Spanish Study Log 13, July 8th 2020

July 9th, 2020

I. Leer El Entenado Por José Saer (30 minutos)

Revista

I. Vocabulario

disimular - ocultar con astucia o habilidad lo que se siente o se padece.
la intemperie - ambiente atmosférico
el alcance - distancia que alcanza la acción o la influencia de una cosa
negrura - cualidad de negro
chisporroteantes - sizzling
acribillada de un volcán - ??
entrever - ver una cosa de manera confusa o imprecisa
la incandescencia - estado de un cuerpo incandescente
incandescente - [cuerpo] que adquiere un color rojo o blanco por haber sido sometido a altas temperaturas, especialmente el carbón y los metales.
la orfandad - condición de huérfano (alguien sin padres)
el cáñamo humedecido - the moistened hemp
amontonamiento de mercaderías - stacking of merchandise
capitanes - captains
me acunó - it cradled me
mandadero - persona que hace mandados o recados
changador - porter (some ship job in South America)
un acoplamiento gratuito - a free hookup
me bastaban - They were enough for me
torpeza - cualidad de torpe
torpe - [persona, animal] que se mueve con lentitud y dificultad sin dominar del todo los movimientos.
la incomodidad - the discomfort
grumete - muchacho que en un barco ayuda a la tripulación en sus tareas para aprender el oficio de marinero.
el poniente - the West

Horario:
8:40PM Empieza
9:08PM Fin

How To Test gbw-node By Viewing The First Ever Transaction Containing a p2pkh Address

July 8th, 2020

After setting up gbw-node and getting a local trb running, I wanted to do a basic test to make sure that my instance of jfw's wallet software was able to show a balance for an address that had received some bitcoins. Dorion had done a similar exercise, except he scanned up to block 634,000 which took about 24 hours.

Being impatient, I attempted to track the address found in the first non-coinbase txn in history, found in block 170. But I discovered that gbw-node cannot track the receiving address in that transaction, since gbw-node only supports addresses that receive bitcoins via the p2pkh script. The first transaction in bitcoin's history was paid directly to the public key, not the public key's hash.

I believe the first address in the form of a p2pkh script can be found in block 728 in a transaction that sent the address 100 bitcoins.

The corresponding bitcoin address and txn-id are 12higDjoCCNXSA95xZMWUdPvXNmkAduhWv and 6f7cf9580f1c2dfb3c4d5d043cdbb128c640e3f20161245aa7372e9666168516, respectively.

So to quickly test basic functionality of gbw-node, run the following commands:

gbw-node watch firstp2pkh
12higDjoCCNXSA95xZMWUdPvXNmkAduhWv

gbw-node reset
gbw-node scan # Wait for block 728 to pass, then 
gbw-node unspent-outs

You should see:

12higDjoCCNXSA95xZMWUdPvXNmkAduhWv 100.00000000 6f7cf9580f1c2dfb3c4d5d043cdbb128c640e3f20161245aa7372e9666168516 0 #blk 728 tx 1

Building TRB on CentOS 6.9, Notes on a Few Gotchas

July 8th, 2020

Yesterday I built trb on my local computer running CentOS 6.9. I imagine I saved a bit of time from already having keccak V setup. The only dependency from the list on the bitcoin foundation's website I was missing was "bc."1

After grabbing the vpatches from the foundation's website I also pulled in two other vpatches written by jfw, one that provides an improved way to get/send rawtx's and another that fixes up the bitcoind build process in various ways. The commands to grab the patches and seals:

wget http://fixpoint.welshcomputing.com/v/bitcoin/bitcoin_rawtx_get_send.vpatch
wget http://fixpoint.welshcomputing.com/v/bitcoin/bitcoin_system_compiler.vpatch
wget http://fixpoint.welshcomputing.com/v/bitcoin/bitcoin_rawtx_get_send.vpatch.jfw.sig
wget http://fixpoint.welshcomputing.com/v/bitcoin/bitcoin_system_compiler.vpatch.jfw.sig

After a successful press, I ran into a couple of errors when trying to build the beast that is trb. The first one was some error related to my LD_LIBRARY_PATH variable. (UPDATE: As jfw pointed out in the comments, this error was received when I tried to do the build for the first time, without his system_compiler patch.)

You seem to have the current working directory in your
LD_LIBRARY_PATH environment variable. This doesn't work.
make[2]: *** [core-dependencies] Error 1
make[2]: Leaving directory `/home/whaack/v/trb054/bitcoin/build/buildroot-2015.05'
make[1]: *** [buildroot-2015.05] Error 2
make[1]: Leaving directory `/home/whaack/v/trb054/bitcoin/build'
make: *** [build] Error 2

I updated my environment by running:

LD_LIBRARY_PATH="";

in my terminal.

The next problem I ran into was building Boost. I caught the issue because I actually stared at the screen and watched all that compiler spew. Python errors flowed in front of my eyes, and I figured that the problem must be related to the fact that the native python on CentOS 6.9 is python 2.6.6 rather than python 2.7.x.

Getting my environment to use python 2.7.x instead of python 2.6.6 was a little bit tricky. Apparently you cannot update the native python without putting your OS in danger. So you need to install this program scl as well as python27 through yum and then use scl to start a temporary environment that has python pointed at python27. In this environment you build bitcoind.

scl enable python27 bash
cd ~/v/trb054/bitcoin/
make ONLINE=1
  1. I also had to downgrade my gpg from 2.X to 1.4.23, as well as tweak my path so that my gcc was 4.4.7. (Previously my gcc was pointed to some gcc 8.x used by a 2019 version of GNAT.) []

Spanish Study Log 11, July 6th 2020

July 8th, 2020

I. Read La Buena Suerte (30 mins)

II. Read Que Bien Suena (30 mins)

== Review

I. Vocab

búho - an owl
sutilmente - subtly
la revancha - the rematch
despistar - hacer perder la pista o el camino a alguien (mislead/sidetrack)
enfurecer - hacer que una persona o un animal se pongan furiosos
encajar - (1) to fit in (2) put up with
malévola - que tiene tendencia a hacer el mal
aullidos - voz quejumbrosa y prolongada que emiten el lobo, el perro, y otros cánidos
asió con firmeza - grasped firmly
la empuñadura - the grip
refunfuñando - emitir [una persona] sonidos no articulados o palabras murmuradas entre dientes en señal de enojo o desagrado
brotar - nacer o salir [una planta] de la tierra.
pringosa - sticky
la losa - the slab [of rock]

II. Que Bien Suena

When one word ends in a consonant and the next begins with a vowel, the words are linked and not separated with glottal stops [?]. The linked words ALSO respect the rule of preferred open vowels. SO:

un alma = [u.nal.ma]

los olmos = [lo.sol.mos]

remember to combine the consonants if a word ends with a consonant and the following word begins with the same one

el lago [e.la.go]
es Sara [e.sa.ra]

same thing happends when you end/start on the same vowel

este estudiante [es.tes.tu.djan.te]
río Orinoco [ri.o.ri.no.co]

you make the vowel above a little longer than normal vowels, but not 2x as long. You can note the increased duration with a :

[es.te:s.tu.djan.te]
[ri.o:.ri.no.co]

va a hacer bien [ba.ser.bjen] / [ba:.ser.bjen]

double consonants are also reduced to one within words, example:

innecesario = [i.ne.ce.sa.rjo]

dipthongs can be formed between words

casi olvidado = [ca.sjol.vi.da.do]
esta idea = [es.tai.de.a]
siete u ocho [sje.te.wo.cho]
la unidad [lau.ni.dad]

(Finished on pg 63)

Timesheet:

9:14AM: Start reading.
9:36AM: Lookup vocab words
9:39AM: Begin Reading Que Bien Suena
10:12AM: Finish

Spanish Study Log 12, July 7th 2020

July 8th, 2020

I. Read La Buena Suerte (30mins)

II. Read Que Bien Suena (30mins)

==

I. Vocab

vaya paradoja - What a paradox
un cosquilleo en el tobillo - a tickle in the ankle
una brizna - a blade (of a tree, I believe)
rozar - to touch
los osados - the daring
cotidiano - que ocurre, se hace o se repite todos los dias.

II. Notes

La "g" dura en espanol puede escribir con solo "g" o "gu"

guedeja [ge.dé.xa]
guillermo [gi.jér.mo]
juegue [xwé.ge]

If you are going to pronounce the u after a g you need an umlaut:

güero [gwé.ro]
güira [gwí.ra]
averigüé [a.be.ri.gwé] (I found out)

Or if the gu is followed by something that is not e or i:

guantanamera [gwan-ta-na-mé-ra]
antiguo [an-tí-gwo]

g
gemir [xe.mír]
gitano [xi.ta.no]
Argentina [ar.xen.tí.na]

j
jornada [xor.ná.da]
jujuy [xu.xwí]

Page 69 has a list of notes on fonetic symbols for Spanish.

(Finished at page 70.)

I have a lot of exercise debt.

Review

TimeSheet:
6:05PM Begin Reading
6:23PM Begin translating (finished book)
6:25PM Begin reading Que Bien Suena
6:54PM Finished.

Spanish Study Log 10, July 5th 2020

July 8th, 2020

I. Read La Buena Suerte (20 mins)

II. Read Que Bien Suena (20 mins)

== Review

I. Vocab

aconsejable - que es conveniente o recomendable
se sobresaló - it excelled ?
imprescindible - que es o se considera tan necesario que no puede prescindir de el
desplegar - poner en práctica una actividad o manifestar una cualidad
musgo - moss
indumentaria - nombre genérico de la ropa que cubre y resguarda el cuerpo humano
manchar - ensuciar una cosa dejando una señal o una marca.

(finished on pg. 26)

II. Que Bien Suena

triptongo

semiconsonante-vocal-semivocal

buey = [bwei] (ox)

miau = [mjau] (meow)

* i and u are semivocals.

vocales fuertes tienen sílabas distintas - se llaman "en hiato"

leo (le-o)
fealdad (fe-al-dad)
bacalao (ba-ca-la-o)
creer (cre-er)

(I skipped exercises B & C + Poems and went to chapter 3)

Note that hacia preposition is different than imperfecto de hacer

hacia = ha-cja
hacía = ha-ci-a

continuo (adjetivo) con-ti-nwo
continúo (1st persona, presente de continuar) con-ti-nu-o

Skipped exercises here as well and went to chapter 4

El enlace entre vocales

Don't glottal stop inbetween words

una estudiante = u.na_es.tu.djan.te and NOT u.na?es.tu.djan.te

Que va a hacer? = ke.ba_a_ser

If a word ends with a consonant and starts with the same consonant, you can use just one consonant

i.e.

el lago = e.lá.go

## = Divisón entre palabras
V = vocal
C = consonante
C1/C1 = la misma consonante

Hay enlace en los siguientes ambientes fonéticos:

V##V (mi hermana) [mi_er.má.na]
C##V (las hermanas) [las_er.má.nas]
C1##C1 (las serenas) [las_se.ré.nas]

No hay enlace en los siguientes:

V##C (mi cuñado) [mi.ku-ñá.do] (sin enlace)
C1##C1 (los cuñados) [los.ku.ñá.dos] (sin enlace)

Spanish Study Log 9, July 4th 2020

July 4th, 2020

I. Read La Buena Suerte (30 mins)

II. Review Ordinales (10 mins)

III. Qué Bien Suena (30 mins)

== Review

I. Vocab:

desolado - que está arruinado, yermo, vacío o sin vida
deprimido - que padece una depresión
abatido - que ha perdido el ánimo, las fuerzas
abatir - tirar o hacer caer al suelo a una persona o cosa que está de pie o en un lugar alto, generalmente con fuerza o impetu
un corcel - caballo, especialmente el que es ligero, de gran alzada y bella figura, como los que servían para los torneos y batallas.
venerar - sentir y demostrar gran amor y respeto a una cosa o a una persona por su virtud, dignidad, méritos o santidad.
un súbdito - que está sujeto a la autoridad de otro al que tiene la obligación de obedecer.
despejar - dejar un espacio libre de personas o cosas que lo ocupan.
raudo - que van a gran velocidad o que invierte poco tiempo o mentos tiempo del que se considera normal en moverse o desarrollarse.
"actúa y no postergues" - act and don't delay
una alforja - a saddlebag
una espada - a sword
imprescindible - que es o se considera tan necessario que no se puede prescindir de el o no se puede dejar de tener en consideración
oscurecer - 1. volverse oscuro el cielo al ponerse el sol. 2. volverse oscurco el cielo al nublarse. 3. pasar [una cosa] o estar oscura o más oscura.
apesadumbrado - que muestra tristerza, aflicción y disgusto. "me siento apesadumbradada por el error que he cometido"
un rasgo - a trait ??
una risita burlona - a mocking giggle
se desternillaba de risa - ??
se alegró con amargura - he was glad with bitterness??
aconsejable -que es conveniente o recomendable.

(Leftoff on page 22)

II. Ordinales Revista

1 - primero (el primer)
2 - segundo
3 - tercero (el tercer)
4 - cuarto
5 - quinto
6 - sexto
7 - séptimo
8 - octavo
9 - novano noveno
10 - décimo
11 - décimo primero (o undécimo)
12 - décimo segundo (o duodécimo)
13 - décimotercero
14 - décimocuarto
15 - décimoquinto
16 - décimosexto
17 - décimoséptimo
18 - décimoctavo
19 - décimonoveno

Divisible By 10

20th - vigésimo
30th - trigésimo
40th - cuadragésimo
50th - quincuagésimo
60th - sexagésimo
70th - septuagésimo
80th - octogésimo
90th - nonagésimo

III. Que Bien Suena (Diptongo)

bien; buen // vocal debil es el primer
seis; aun // vocal debil es el ultimo

Si la vocal débil es el primer miembro hay una menor abertura en la cavidad oral (o bucal.)
Son semiconsonantes - se representan con [j] y [w].

Si la vocal débil es el segundo, hay mayor abertura en la cavidad oral.
Son semivocales1 - se representan con [i] y [u] (NOTE: These [i] and [u] symbols have a little ^ under them, i don't know the escape sequence to make the right characters.)

Si hay dos vocales débiles la primera funciona como la semiconsonante y la segunda será la vocal nuclear.

cuidado (cwi-da-do)
ciudad (cju-dad)

Ex A pg. 32 Done on paper.

TimeSheet

11:20AM: Start reading La Buena Suerte
11:50AM: Go over time reading la buena suerta to finish vocab
11:58AM: Begin Ordinales Revista
12:03PM: Start Reading Que Bien Suena
12:34PM: Bonus time finishing an exercise in Que Bien Suena
12:39PM: Finish

  1. From Catford's A Practical Introduction To Phonetics - A semivowel is a momentary approximant. A [w] is an ultra short [u] and a [j] is an ultra short [i]. []