With Web 3.0 and blockchain becoming more mainstream every day, do you know what blockchain is? Do you know its technical advantages and use-cases?
The goal of this tutorial is to introduce blockchain technology from a technical perspective by building one from scratch.
Forget everything you've heard about blockchain from social media. Now, you will build a blockchain system from ground zero to really understand the ins and outs of this peer-to-peer, distributed technology.
Afterwards, make your own mind up about its future and advantages. Spoiler alert: you will fall in love with programming blockchain software.
You will follow the story of a software developer who is looking to revolutionize his local bar by implementing blockchain technology for its payment system.
Although blockchain has several undeniable use-cases, at the moment, the number one application is payments. This is because banks are still running on an inefficient, 40 year old infrastructure powered by CSV files and FTP.
The story comes with a lot of fun and intriguing facts about the overall blockchain ecosystem and different protocols such as Bitcoin, Ethereum and XRP.
So let's get started and jump into our story.
Andrej is a bar owner by night and a software developer by day in a small Slovakian town called Bardejov.
Andrej is tired of:
Andrej would love to:
His goal is to write a simple program and keep all the balances of his clients in virtual form.
Andrej shares his thoughts here:
"Every new customer will give me cash, and I will credit them an equivalent amount of my digital tokens (coins/cryptocurrency). The tokens will represent a monetary unit within and outside the bar.
The users will use the tokens for all bar functionalities from paying for drinks, borrowing and lending them to their friends, and playing table tennis, poker and kicker.
Having a bar powered by blockchain tokens will generate tons of value for my customers. Contrary to my competition and other bars on this street, where the customers only spend money and get a hangover in exchange, my bar customers holding bar's tokens will have shareholders rights.
Similar to owning a large portion of stocks in a company like Apple or Microsoft, the customers holding these bar tokens will be able to decide how the bar will operate by voting and deciding on:
Oh, this will be a programming dream!
I will call the tokens: The Blockchain Bar tokens, TBB!"
Now that Andrej has shared his dream, we'll get started.
Let's dive into our tutorial. I recommend 2+ years of programming experience in Java/PHP/Javascript, or another language similar to Go.
If you want to get a good quick intro to go, here's a free course that'll get you started.
You can also complete the official 17 lectures of A Tour Of Go to get familiar with the language syntax and basic concepts (~20 mins).
Because like blockchain, it's a fantastic technology for your overall programming career. Go is a trendy language and Go devs are better paid than the average Java/PHP/Javascript positions.
Go is optimized for multi-core CPU architecture. You can spawn thousands of light-weight threads (Go-routines) without problems. It's extremely practical for highly parallel and concurrent software such as blockchain networks.
By writing your software in Go, you achieve nearly C++ level of performance out of the box without killing yourself for that one time you forgot to free up memory.
Go also compiles to binary which makes it very portable.
This article has a dedicated open-sourced Github repository with full source code so you can compile the code and run the program on your own local machine.
If you get stuck at any chapter or a particular line of code, create a Github Issue in this repository describing your problem and I will help you out ASAP!
↓ Visit the Github repository and follow the installation instructions ↓
✍ git checkout c1_genesis_json
Andrej mastered relational SQL databases in the 90s. He knows how to make advanced data models and how to optimize the SQL queries.
It's time for Andrej to catch up with innovation and start building Web 3.0 software.
Luckily, after reading "The Lean Startup" book last week, Andrej feels like he shouldn't over-engineer the solution just yet. Hence, he chooses a simple but effective JSON file for the bar's MVP database.
In the beginning, there was a primitive centralized database.
Blockchain is a database.
Monday, March 18.
Andrej generates 1M utility tokens.
In the blockchain world, tokens are units inside the blockchain database. Their real value in dollars or euro fluctuates based on their demand and popularity.
Every blockchain has a "Genesis" file. The Genesis file is used to distribute the first tokens to early blockchain participants.
It all starts with a simple, dummy genesis.json.
Andrej creates the file ./database/genesis.json where he defines that The Blockchain Bar's database will have 1M tokens and all of them will belong to Andrej:
The tokens need to have a real "utility", that is a use case. Users should be able to pay with them from day 1!
Andrej must comply with law regulators (the SEC). It is illegal to issue unregistered security. On the other hand, utility tokens are fine, so right away he prints and sticks a new pricing white p̶a̶p̶e̶r̶ poster on the bar's door.
Andrej assigns a starting monetary value to his tokens so he can exchange them for euro, dollars, or other fiat currency.
Andrej also decides he should be getting 100 tokens per day for maintaining the database and having such a brilliant disruptive idea.
Blockchain is a database.
The token supply, initial user balances, and global blockchain settings you define in a Genesis file.
✍ git checkout c2_db_changes_txt
Monday, March 25.
After a week of work, the bar facilities are ready to accept tokens. Unfortunately, no one shows up, so Andrej orders three shots of vodka for himself and writes the database changes on a piece of paper:
To avoid recalculating the latest state of each customer's balance, Andrej creates a ./database/state.json file storing the balances in an aggregated format.
Tuesday, March 26.
To bring traffic to his bar, Andrej announces an exclusive 100% bonus for everyone who purchases the TBB tokens in the next 24 hours.
Bing! He gets his first customer called BabaYaga. BabaYaga pre-purchases 1000€ worth of tokens, and to celebrate, she immediately spends 1 TBB for a vodka shot. She has a drinking problem.
DB transactions written on a piece of paper:
Blockchain is a database.
The token supply, initial user balances, and global blockchain settings you define in a Genesis file.
The Genesis balances indicate what was the original blockchain state and are never updated afterwards.
The database state changes are called Transactions (TX).
✍ git checkout c3_state_blockchain_component
Developers used to event-sourcing architecture must have immediately recognized the familiar principles behind transactions. They are correct.
Blockchain transactions represent a series of events, and the database is a final aggregated, calculated state after replaying all the transactions in a specific sequence.
Tuesday evening, March 26.
It's a relaxing Tuesday evening for Andrej. Celebrating his first client, he decides to play some Starcraft and clean up his local development machine by removing some old pictures.
Unfortunately, he prematurely pressed enter when typing a removal command path in terminal sudo rm -rf /. Oops.
All his files, including the bar's genesis.json and state.json are gone.
Andrej, being a senior developer, repeatedly shouted some f* words very loudly for a few seconds, but he didn't panic!
While he didn't have a backup, he had something better — a piece of paper with all the database transactions. The only thing he needs to do is replay all the transactions one by one, and his database state will get recovered.
Impressed by the advantages of event-based architecture, he decides to extend his MVP database solution. Every bar's activity, such as individual drink purchases, MUST be recorded inside the blockchain database.
Each customer will be represented in DB using an Account Struct:
Each Transaction (TX - a database change) will have the following four attributes: from, to, value and data.
The data attribute with one possible value (reward) captures Andrej's bonus for inventing the blockchain and increases the initial TBB tokens total supply artificially (inflation).
The Genesis DB will remain a JSON file:
All the transactions, previously written on a piece of paper, will be stored in a local text-file database called tx.db, serialized in JSON format and separated by line-break character:
The most crucial database component encapsulating all the business logic will be State:
The State struct will know about all user balances and who transferred TBB tokens to whom, and how many were transferred.
It's constructed by reading the initial user balances from genesis.json file:
Afterwards, the genesis State balances are updated by sequentially replaying all the database events from tx.db:
The State component is responsible for:
Adding new transactions to Mempool:
Persisting the transactions to disk:
Changing, Validating the state:
Andrej wants to have a convenient way to add new transactions to his DB and list the latest balances of his customers. Because Go programs compile to binary, he builds a CLI for his program.
The easiest way to develop CLI based programs in Go is by using the third party github.com/spf13/cobra library.
Andrej initializes Go's built-in dependency manager for his project, called go modules:
✍ cd ogramming-newsletter-edition
✍ go mod init wsletter-edition
The Go modules command will automatically fetch any library you reference within your Go files.
Andrej creates a new directory called: cmd with a subdirectory tbb:
✍mkdir -p ./cmd/tbb
Inside he creates a main.go file, serving as the program's CLI entry point:
The Go programs are compiled using the install cmd: ✍ go install ./cmd/tbb/...
Go will detect missing libraries and automatically fetch them before compiling the program. Depending on your $GOPATH the resulting program will be saved in the $GOPATH/bin folder.
✍echo $GOPATH
✍which tbb
You can run tbb from your terminal now, but it will not do anything because the Run function inside the main.go file is empty.
The first thing Andrej needs is versioning support for his tbb CLI program.
Next to the main.go file, he creates a version.go command:
Compiles and runs it: ✍ go install ./cmd/tbb/... ✍ tbb version
Version: 0.1.0-beta TX Add && Balances List
Identically to the version.go file, he creates a balances.go file:
The balances command will be responsible for loading the latest DB State and printing it to the standard output:
Andrej verifies if the cmd works as expected. It should print the exact balances defined in the Genesis file because the tx.db file is still empty.
✍ go install ./cmd/tbb/...
✍ tbb balances list
Works well! Now he only needs a cmd for recording the bar's activity.
Andrej creates ./cmd/tbb/tx.go cmd:
The tbb tx add cmd uses State.Add(tx) function for persisting the bar's events into the file system:
The tbb tx add cmd has 3 mandatory flags: --from, --to and --value.
The CLI is done!
Andrej migrates all transactions from paper to his new DB:
✍ tbb tx add --from=andrej --to=andrej --value=3
✍tbb tx add --from=andrej --to=andrej --value=700
✍tbb tx add --from=babayaga --to=andrej --value=2000
✍tbb tx add --from=andrej --to=andrej --value=100 --data=reward
✍tbb tx add --from=babayaga --to=andrej --value=1
Read all TXs from disk and calculate the latest state:
Bar data successfully restored! Phew, what a night!
The good thing about the Cobra lib for CLI programming is the additional features it comes with. For example, you can now run: tbb help cmd and it will print out all TBB registered sub-commands with instructions on how to use them.
The token supply, initial user balances, and global blockchain settings are defined in a Genesis file.
The Genesis balances indicate what the original blockchain state was and are never updated afterwards.
The database state changes are called Transactions (TX).
Transactions are old fashion Events representing actions within the system.
Commit: 5d4b0b
Let's talk about greed.
✍ git checkout c4_caesar_transfer
Wednesday, March 27.
BabaYaga invested a bit too much. She forgot her flat rent payment was around the corner, and she doesn't have the money. BabaYaga calls her flat owner, Caesar.
BabaYaga: Hey Caesar, I am sorry, but I don't have the cash to pay you the rent this month…
Caesar: Why not?
BabaYaga: The Blockchain Bar ICO offered a massive bonus, and I purchased 2000€ worth of tokens for just 1000€. It was a great deal!
Caesar: What the heck are you talking about? What is an ICO? What on earth are tokens? Can you pay me in some other way?
BabaYaga: Oh, not again. I can give you 1000 TBB tokens worth 1000€, and you can use them in the bar to pay for your drinks! Let me call the bar owner, Andrej, and make the transfer!
Caesar: All right... I will take it.
Andrej performs the transfer, but decides to charge an extra 50 TBB tokens for his troubles. He doesn't want to, BUT the bar shareholders who invested in him a few years ago are forcing him to generate profit as soon as possible.
BabaYaga won't notice this relatively small fee most likely anyway, Andrej tells himself. In the end, only he has the DB access.
// Rent payment
✍tbb tx add --from=babayaga --to=caesar --value=1000
// hidden fee charge
✍ tbb tx add --from=babayaga --to=andrej --value=50
// new reward for another day of maintaining the DB
✍ tbb tx add --from=andrej --to=andrej --value=100 --data=reward
Commit: 00d6ed
✍ git checkout c5_broken_trust
Thursday, March 28.
BabaYaga enters the bar for her birthday.
BabaYaga: Hey, Andrej! Today is my birthday! Get me your most expensive bottle!
Andrej: Happy birthday! Here you go: Crystal Head Vodka. But you need to purchase one additional TBB token. The bottle costs 950 tokens, and your balance is 949.
BabaYaga: What?! My balance is supposed to be 999 TBB!
Andrej: The funds transfer to Caesar you requested last week cost you 50 tokens.
BabaYaga: This is unacceptable! I would never agree to such a high fee. You can't do this, Andrej. I trusted your system, but you are as unreliable as every other business owner. Things must change!
Andrej: All right, look. You are my most loyal customer, and I didn't want to charge you, but my shareholders forced me.
Let me re-program my system and make it completely transparent and decentralized. After all, if everyone were able to interact with the bar without going through me, it would significantly improve the bar's efficiency and balance the level of trust!
BabaYaga: Well, it certainly sounds good, but is this even possible?
Andrej: Yes, I think so. With a bit of hashing, linked lists, immutable data structure, distributed replication, and asymmetric cryptography!
BabaYaga: I have no idea what you have just said but go and do your geeky thing, Andrej!
Closed software with centralized access to private data allows for just a handful of people to have a lot of power. Users don’t have a choice, and shareholders are in business to make money.
Blockchain developers aim to develop protocols where applications' entrepreneurs and users synergize in a transparent, auditable relationship. Specifications of the blockchain system should be well-defined from the beginning and only change if its users support it.
Blockchain is a database. The token supply, initial user balances, and global blockchain settings are defined in a Genesis file. The Genesis balances indicate what was the original blockchain state and are never updated afterwards.
The database state changes are called Transactions (TX). Transactions are old fashion Events representing actions within the system.
Commit: 642045
✍ git checkout c6_immutable_hash
The technical difficulty starts with this section! The concepts will only get more challenging but at the same time, very exciting. Buckle up :)
Friday, March 29.
If Andrej wants to figure out how to program an immutable DB, he has to realize why other database systems are mutable by design.
He decides to analyze an all-mighty MySQL DB Table:
In MySQL DB, anyone with access and a good enough reason can perform a table update such as:
Updating values across different rows is possible because the table rows are independent, mutable, and the latest state is not apparent.
What’s the latest DB change? Last column changed? Last row inserted? If so, how can Andrej know what row was deleted recently? If the rows and table state were tightly coupled, dependent, a.k.a, updating row 1 would generate a completely new, different table, Andrej would achieve his immutability.
Hashing is process of taking a string input of arbitrary length and producing a hash string of fixed length. Any change in input, will result in a new, different hash.
Try it: https://play.golang.org/p/FTPUa7IhOCE
Andrej also requires some level of security for his database, so he decides for a Cryptographic Hash Function with the following properties:
Saturday Evening, March 30.
Andrej modifies the Persist() function to return a new content hash, Snapshot, every time a new transaction is persisted.
The Snapshot is produced by this new sha256 secure hashing function:
The doSnapshot() is called by the modified Persist() function. When a new transaction is written into the tx.db file, the Persist() hashes the entire file content and returns its 32 bytes "fingerprint" hash.
From this moment, everyone can 100% confidently and securely refer to any particular database state (set of data) with a specific snapshot hash.
1/4 Run the tbb balances list cmd and check the balances are matching.
2/4 Remove the last 2 rows from ./database/tx.db and check the balances again.
3/4 Reward Andrej for the last 2 days (from 28th to 30th of March):
Reward Transaction 1:
Reward Transaction 2:
4/4 Run the tbb balances list cmd and ensure the balances and the snapshot hash is the same as at the beginning.
Done!
Because the cryptographic hash function sha256 produces the same output (given the same inputs (current tx.db and 2x tbb tx add)), if you follow the exact steps on your own computer, you will generate the exact same database state and hashes!
Closed software with centralized access to private data puts only a few people to the position of power. Users don’t have a choice, and shareholders are in business to make money.
Blockchain developers aim to develop protocols where applications' entrepreneurs and users synergize in a transparent, auditable relation. Specifications of the blockchain system should be well defined from the beginning and only change if its users support it.
Blockchain is an immutable database. The token supply, initial user balances, and global blockchain settings you define in a Genesis file. The Genesis balances indicate what was the original blockchain state and are never updated afterwards.
The database content is hashed by a secure cryptographic hash function. The blockchain participants use the resulted hash to reference a specific database state.
Commit: b99e51
You finished the first few chapters! Congratulations!
█▒▒▒▒▒▒▒▒▒ 10%
But this was just a quick warm-up. Blockchain is a very challenging and extensive technology, and you would need an entire book explaining how to build the full system and all of its components from scratch - so I wrote one.
You can continue reading in the next free chapter in my newsletter version of "The Blockchain Way of Programming" eBook.
07 | The Blockchain Programming Model
Learning: You redesign and refactor your MVP database into a blockchain architecture.
Thanks for reading!