Block Explorer Progress - What's Done, What's Next
I) What's Done
I've taken jfw's gbw-node and repurposed it to serve as a block explorer. The functions I'm left with after my changes are view-block, view-txn-by-hash/view-txn-by-pos, view-ancestors/view-descendents, view-address, balance, utxos, and push.
Per jfw's suggestion, I merged the input table into the output table in the sql schema. I renamed the output table as output_input. Thinking of the output of one txn and the corresponding input in the txn that spends that output as being a single structure has helped me form a clearer picture of how chains of bitcoin transactions are connected.
Here is a snapshot of the current sql schema with the new "output_input" table.
--- Gales Bitcoin Wallet: node (online component) schema
--- J. Welsh, December 2019
--- Dialect: SQLite (3.7.0 for WAL)
PRAGMA journal_mode=WAL;
BEGIN;
CREATE TABLE block (
block_id INTEGER PRIMARY KEY,
height Integer NOT NULL,
size INTEGER NOT NULL,
version INTEGER NOT NULL,
prev_hash BLOB NOT NULL,
hash BLOB NOT NULL,
root BLOB NOT NULL,
timestamp INTEGER NOT NULL,
target INTEGER NOT NULL, -- Should we include this?
nonce INTEGER NOT NULL
);
CREATE UNIQUE INDEX i_block_hash on block(hash);
CREATE UNIQUE INDEX i_block_height on block(height);
CREATE TABLE tx (
tx_id INTEGER PRIMARY KEY,
hash BLOB NOT NULL,
block_id INTEGER NOT NULL REFERENCES block,
pos INTEGER NOT NULL, --pos in block
comment TEXT,
size INTEGER NOT NULL,
fee INTEGER
);
CREATE INDEX i_tx_hash ON tx(hash);
CREATE UNIQUE INDEX i_tx_block_id_pos ON tx(block_id, pos);
-- Every input begins its life as an output.
CREATE TABLE output_input (
output_input_id INTEGER PRIMARY KEY,
creating_tx_id INTEGER NOT NULL REFERENCES tx,
out_pos INTEGER NOT NULL,
address_id INTEGER NOT NULL REFERENCES address, -- aka script pub key
value INTEGER NOT NULL,
spending_tx_id INTEGER REFERENCES tx, -- If null, it hasn't been spent.
in_pos INTEGER, -- position in input vector in spending txn
scriptsig BLOB,
flags TEXT
);
CREATE UNIQUE INDEX i_output_txid_out_pos ON output_input(creating_tx_id, out_pos);
CREATE INDEX i_output_addrid ON output_input(address_id);
CREATE INDEX i_input_txid_n ON output_input(spending_tx_id);
CREATE TABLE address (
address_id INTEGER PRIMARY KEY,
address BLOB NOT NULL
);
CREATE UNIQUE INDEX i_address_address ON address(address);
CREATE TABLE state (
scan_height INTEGER NOT NULL DEFAULT(-1)
);
INSERT INTO state DEFAULT VALUES;
COMMIT;
II. What's Next
A) Refactor commands so that they can be used by both the command line and web interface.
I've decided to use flask to run the web server portion of the block explorer. From reading the logs, this python package appears to be a handy utility whose use is a mortal sin. So I don't want to make flask's installation a requirement for running the block explorer locally.
I plan to do the following. I'm going to design the block explorer so that I can run a public web interface using flask. The source of everything will be public, so anyone will be able to install the block explorer along with flask and use the explorer locally via the web interface. Alternatively, they will be able to install the explorer without flask, but in this case they will only be able to use the block explorer via a command line interface similar to the one gbw-node currently employs.
In order to allow for the two uses of the explorer, I need to split all the command functions into two parts - one that returns structured data and the other that prints the structure data. The command line interface and web interface will stringify the data appropriately.
B) Write a view-raw-hex of block command.
As an exercise in understanding and in order to check the integrity of the explorer's stored data, I want to make sure that I can take the tables in the gbw-node sql database and reconstruct a bit-perfect block. In order to provide this feature I need to store some data considered extraneous by the original gbw-node wallet, such as the input field for a coinbase as well as a transaction's sequence number, version, and locktime.
C) Get domain names and configure servers.
I now have one box currently syncing trb on asciilifeform's rack. But setting up at least one other mirror in a different geographical location seems prudent.
D) Continous trb scanning.
Currently gbw-node has no way to handle reorgs. It pulls data from the bitcoin rpc up until the 'block height - CONFIRMATION'th block. This is done via the command "scan", which halts when it reaches the most recent block. To keep the explorer's data up to date, the block explorer must always be scanning. I can either modify the scan command to run on an infinite loop, sleeping for ~10 mins when it hits the max block height, or I can just continually rescan via a crontask.
E) Provide a way to show information about transactions in the mempool / recent blocks.
The main use cases I have for a block explorer are obtaining utxo data for spending bitcoins, pushing raw bitcoin transactions to the network, and confirming that recently pushed transactions were received by the network. The block explorer in its current state has no way to store transactions in the mempool. The schema requires a transaction to have an associated block id and block position. So currently the block explorer is not useful for showing recent blocks, nor for showing recent unconfirmed transactions.
I plan to create a separate table, mempool_transaction, that displays information about transactions in the mempool. The scan function will delete mempool transactions whenever it finds the transaction successfully placed in a deep block. I also will want to figure out how to store recent blocks that may be reorg'd. I think that I'll handle this in a similar manner to mempool_transaction, with some volatile table named recent_blocks. The corresponding row from this table will be deleted when the block has confirmed its place in the explorer's "main chain."