Skip to content

Commit d89db6a

Browse files
jrakibijrakibi
authored
Merge pull request #60 from jrakibi/comapctsize-feerate
Add 3 new topics: compactSize, Fee rate
2 parents d9d5a48 + fae7690 commit d89db6a

File tree

3 files changed

+372
-3
lines changed

3 files changed

+372
-3
lines changed

‎decoding/compact-size.mdx

+159-1
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,162 @@ images:
1414
parent: "technical-foundation"
1515
---
1616

17-
(coming soon)
17+
Compact Size (also known as CompactSize or var_int) is a variable-length integer encoding used in Bitcoin to efficiently represent numbers while minimizing space usage. It's primarily used in transaction data and network messages to indicate:
18+
19+
- Number of inputs/outputs in a transaction
20+
- Script sizes
21+
- Number of witness elements
22+
- Length of upcoming data fields
23+
24+
## Structure
25+
26+
The encoding uses a prefix byte to determine how to read the subsequent bytes:
27+
28+
<div className="overflow-x-auto">
29+
<table className="min-w-full bg-white dark:bg-gray-900 rounded-lg overflow-hidden">
30+
<thead className="bg-orange-100 dark:bg-orange-900">
31+
<tr>
32+
<th className="px-6 py-3 text-left text-sm font-semibold">Prefix</th>
33+
<th className="px-6 py-3 text-left text-sm font-semibold">Format</th>
34+
<th className="px-6 py-3 text-left text-sm font-semibold">Range</th>
35+
<th className="px-6 py-3 text-left text-sm font-semibold">Total Size</th>
36+
</tr>
37+
</thead>
38+
<tbody className="divide-y divide-gray-200 dark:divide-gray-800">
39+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
40+
<td className="px-6 py-4 whitespace-nowrap"><code>≤ 0xFC</code></td>
41+
<td className="px-6 py-4">Direct value</td>
42+
<td className="px-6 py-4">0-252</td>
43+
<td className="px-6 py-4">1 byte</td>
44+
</tr>
45+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
46+
<td className="px-6 py-4 whitespace-nowrap"><code>0xFD</code></td>
47+
<td className="px-6 py-4">Next 2 bytes (LE)</td>
48+
<td className="px-6 py-4">253-65,535</td>
49+
<td className="px-6 py-4">3 bytes</td>
50+
</tr>
51+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
52+
<td className="px-6 py-4 whitespace-nowrap"><code>0xFE</code></td>
53+
<td className="px-6 py-4">Next 4 bytes (LE)</td>
54+
<td className="px-6 py-4">65,536-4,294,967,295</td>
55+
<td className="px-6 py-4">5 bytes</td>
56+
</tr>
57+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
58+
<td className="px-6 py-4 whitespace-nowrap"><code>0xFF</code></td>
59+
<td className="px-6 py-4">Next 8 bytes (LE)</td>
60+
<td className="px-6 py-4">4,294,967,296-2^64-1</td>
61+
<td className="px-6 py-4">9 bytes</td>
62+
</tr>
63+
</tbody>
64+
</table>
65+
</div>
66+
67+
### How to Read It
68+
69+
Let's analyze real Bitcoin transactions to understand compact size encoding:
70+
71+
1. **Transaction with 1 Input**
72+
-> <a href="https://mempool.space/tx/5e6e1a9b4ce3f9f39467d7129e3ecfbe6c81c08dd377aac666fedc9a758fe614" target="_blank">5e6e1a9b4ce3f9f...fedc9a758fe614</a>
73+
- Looking at input count: `01`
74+
- Since 1 < 252, it's a direct value
75+
- Meaning: Transaction has 1 input
76+
- Total bytes used: 1
77+
78+
2. **ScriptSig of 107 bytes**
79+
-> <a href="https://mempool.space/tx/c27c4d2236fce2a7542e024408d7f89b95e50e42a2c3d10be499c3102ccb45ef" target="_blank">c27c4d2236fce2a...99c3102ccb45ef</a>
80+
- Looking at scriptsig size: `6b`
81+
- Since 107 (0x6b) < 252, it's a direct value
82+
- Meaning: ScriptSig is 107 bytes long
83+
- Total bytes used: 1
84+
85+
3. **ScriptPubKey of 4,026 bytes**
86+
-> <a href="https://mempool.space/tx/e411dbebd2f7d64dafeef9b14b5c59ec60c36779d43f850e5e347abee1e1a455" target="_blank">e411dbebd2f7d64...347abee1e1a455</a>
87+
- Looking at scriptpubkey size: `fd ba 0f`
88+
- First byte is 0xFD, so read next 2 bytes
89+
- Next bytes: `ba 0f` (in little-endian)
90+
- Convert from little-endian: `0fba` = 4,026 in decimal
91+
- Meaning: ScriptPubKey is 4,026 bytes long
92+
- Total bytes used: 3
93+
94+
<ExpandableAlert title="Understanding Little-Endian" type="info">
95+
In the third example, we see `ba 0f` instead of `0f ba` because of little-endian encoding.
96+
Think of it as reading the bytes from right to left: `0fba` is the actual number in hexadecimal.
97+
</ExpandableAlert>
98+
99+
<ExpandableAlert title="Tip" type="warning">
100+
You can verify these values yourself by looking at the raw transaction data on the block explorer.
101+
Click the transaction IDs above and look for the "Raw Transaction" section.
102+
</ExpandableAlert>
103+
104+
## Implementation
105+
106+
Here's a Python implementation for reading and writing compact size integers:
107+
108+
<CodeSnippet
109+
code={`def read_varint(s):
110+
'''Reads a variable integer from a stream
111+
112+
The first byte determines the format:
113+
- If < 0xfd: directly contains the number
114+
- If 0xfd: next 2 bytes contain number
115+
- If 0xfe: next 4 bytes contain number
116+
- If 0xff: next 8 bytes contain number
117+
'''
118+
i = s.read(1)[0]
119+
if i == 0xfd:
120+
return int.from_bytes(s.read(2), 'little')
121+
elif i == 0xfe:
122+
return int.from_bytes(s.read(4), 'little')
123+
elif i == 0xff:
124+
return int.from_bytes(s.read(8), 'little')
125+
else:
126+
return i
127+
128+
def encode_varint(i):
129+
'''Encodes an integer as a compact size
130+
131+
Returns bytes object with the encoded number
132+
'''
133+
if i < 0xfd:
134+
return bytes([i])
135+
elif i < 0x10000:
136+
return b'\\xfd' + i.to_bytes(2, 'little')
137+
elif i < 0x100000000:
138+
return b'\\xfe' + i.to_bytes(4, 'little')
139+
else:
140+
return b'\\xff' + i.to_bytes(8, 'little')`}
141+
language="python"
142+
/>
143+
144+
## Common Uses in Bitcoin
145+
146+
1. **Transaction Structure**
147+
- Input count
148+
- Output count
149+
- ScriptSig size
150+
- ScriptPubKey size
151+
- Witness element count
152+
153+
<TransactionCreation enabledFields={["input_count", "output_count", "scriptsig_size", "scriptpubkey_size"]} />
154+
155+
2. **Block Data**
156+
- Number of transactions in a block
157+
158+
3. **Network Messages**
159+
- Length of upcoming message data
160+
- Number of inventory items
161+
162+
163+
<ExpandableAlert title="Warning" type="warning">
164+
The compact size format is part of Bitcoin's consensus rules. Incorrect encoding
165+
or decoding can lead to invalid transactions or network messages.
166+
</ExpandableAlert>
167+
168+
<ExpandableAlert title="Important Notes" type="info">
169+
- Most transactions use single-byte encoding (≤ 252)
170+
- The FF prefix (8-byte numbers) is rarely used in practice
171+
- All multi-byte numbers must be encoded in little-endian format
172+
- This encoding has been part of Bitcoin since its first release
173+
</ExpandableAlert>
174+
175+
This variable-length integer encoding plays a crucial role in keeping transaction and block data compact while maintaining flexibility for larger values when needed.

‎decoding/endianness.mdx

+58-1
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ Big-endian is considered more "human-readable" because the data is stored in the
7575

7676
Little-endian stores the **least significant byte** first. This might feel counterintuitive to humans but is more efficient for modern processors.
7777

78-
Using the same number **12345678** (`0x00BC614E`), here’s how it looks in **little-endian**:
78+
Using the same number **12345678** (`0x00BC614E`), here's how it looks in **little-endian**:
7979

8080
<CodeSnippet
8181
language="Hex"
@@ -146,3 +146,60 @@ Most modern CPUs are little-endian and the network protocols typically use big-e
146146
This duality requires developers to frequently convert between the two formats when working with Bitcoin data.
147147

148148
---
149+
150+
## 4. Working with Endianness in Code
151+
152+
When working with Bitcoin data, you'll frequently need to convert between little-endian and big-endian formats. Here are some common Python functions for handling endianness:
153+
154+
### Little-Endian to Integer
155+
156+
<CodeSnippet
157+
language="python"
158+
code={`def little_endian_to_int(b):
159+
'''Convert little-endian bytes to integer'''
160+
return int.from_bytes(b, 'little')
161+
162+
# Example usage
163+
bytes_data = bytes.fromhex('4E61BC00') # 12345678 in little-endian
164+
number = little_endian_to_int(bytes_data)
165+
print(number) # Output: 12345678`}
166+
/>
167+
168+
### Integer to Little-Endian
169+
170+
<CodeSnippet
171+
language="python"
172+
code={`def int_to_little_endian(n, length):
173+
'''Convert integer to little-endian bytes of specified length'''
174+
return n.to_bytes(length, 'little')
175+
176+
# Example usage
177+
number = 12345678
178+
bytes_data = int_to_little_endian(number, 4)
179+
print(bytes_data.hex()) # Output: 4e61bc00`}
180+
/>
181+
182+
### Common Gotchas
183+
184+
1. **Byte Order Confusion**: When reading transaction IDs or hashes:
185+
186+
<CodeSnippet
187+
language="python"
188+
code={`# INCORRECT - reading txid directly from hex
189+
txid = "4e61bc0000000000000000000000000000000000000000000000000000000000"
190+
191+
# CORRECT - reverse the bytes for proper txid display
192+
txid_bytes = bytes.fromhex(txid)
193+
txid_display = txid_bytes[::-1].hex()`}
194+
/>
195+
196+
2. **Length Specification**: When converting to little-endian, always specify the correct byte length:
197+
198+
<CodeSnippet
199+
language="python"
200+
code={`# For Bitcoin amounts (8 bytes)
201+
amount = int_to_little_endian(12345678, 8) # Correct
202+
amount = int_to_little_endian(12345678, 4) # Wrong - too short!`}
203+
/>
204+
205+
---

‎decoding/fee-rate.mdx

+155-1
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,158 @@ images: ["/bitcoin-topics/static/images/topics/thumbnails/musig-thumbnail.webp"]
1111
parent: "fee-calculation"
1212
---
1313

14-
(coming soon)
14+
When miners select transactions for inclusion in a block, they don't just look at the raw fee amount, they evaluate transactions based on their _fee rate_.
15+
16+
This is how we calculate the fee rate:
17+
18+
<div className="text-center my-4 text-orange-500">
19+
$$\mathbf{Fee\ Rate = \frac{Transaction\ Fee}{Transaction\ Size}}$$
20+
</div>
21+
22+
Miners use the fee rate to maximize their profit by comparing the profitability of including different transactions in their blocks.
23+
24+
For example, consider these two transactions:
25+
26+
1. Transaction A: 50,000 sats fee, 250 vbytes size
27+
2. Transaction B: 30,000 sats fee, 100 vbytes size
28+
29+
**Fee Rates:**
30+
- Transaction A: 200 sats/vbyte (50,000 ÷ 250)
31+
- Transaction B: 300 sats/vbyte (30,000 ÷ 100)
32+
33+
<ExpandableAlert title="Fee Rate Priority" type="info">
34+
Even though Transaction A pays a higher absolute fee, Transaction B is more profitable for miners because it provides more fees per unit of block space.
35+
</ExpandableAlert>
36+
37+
## Fee Rate Units
38+
39+
There are three common ways to measure fee rates:
40+
41+
1. **sats/vbyte (Most Common)**
42+
2. **sats/wu (Internal)**
43+
3. **sats/byte (Legacy)**
44+
45+
46+
<div className="overflow-x-auto">
47+
<table className="min-w-full bg-white dark:bg-gray-900 rounded-lg overflow-hidden">
48+
<thead className="bg-orange-100 dark:bg-orange-900">
49+
<tr>
50+
<th className="px-6 py-3 text-left text-sm font-semibold">Unit</th>
51+
<th className="px-6 py-3 text-left text-sm font-semibold">Description</th>
52+
<th className="px-6 py-3 text-left text-sm font-semibold">Usage</th>
53+
<th className="px-6 py-3 text-left text-sm font-semibold">Example</th>
54+
</tr>
55+
</thead>
56+
<tbody className="divide-y divide-gray-200 dark:divide-gray-800">
57+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
58+
<td className="px-6 py-4 whitespace-nowrap font-semibold"><code>sats/vbyte</code></td>
59+
<td className="px-6 py-4">Virtual bytes account for witness discount</td>
60+
<td className="px-6 py-4">Most common unit used by wallets and block explorers</td>
61+
<td className="px-6 py-4">50 sats/vbyte</td>
62+
</tr>
63+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
64+
<td className="px-6 py-4 whitespace-nowrap font-semibold"><code>sats/wu</code></td>
65+
<td className="px-6 py-4">Weight units used internally by Bitcoin Core</td>
66+
<td className="px-6 py-4">Internal Bitcoin Core (4M WU block limit)</td>
67+
<td className="px-6 py-4">12.5 sats/wu</td>
68+
</tr>
69+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
70+
<td className="px-6 py-4 whitespace-nowrap font-semibold"><code>sats/byte</code></td>
71+
<td className="px-6 py-4">Deprecated since SegWit activation</td>
72+
<td className="px-6 py-4">Legacy software only</td>
73+
<td className="px-6 py-4">50 sats/byte</td>
74+
</tr>
75+
</tbody>
76+
</table>
77+
</div>
78+
79+
## Understanding Virtual Bytes (vBytes)
80+
81+
Virtual bytes (vBytes) are a way to express transaction size that accounts for the SegWit discount. The formula is simple:
82+
83+
<div className="text-center my-4 text-orange-500">
84+
$$\mathbf{Virtual\ Bytes = Transaction\ Weight ÷ 4}$$
85+
</div>
86+
87+
<div className="overflow-x-auto">
88+
<table className="min-w-full bg-white dark:bg-gray-900 rounded-lg overflow-hidden">
89+
<thead className="bg-orange-100 dark:bg-orange-900">
90+
<tr>
91+
<th className="px-6 py-3 text-left text-sm font-semibold">Transaction Type</th>
92+
<th className="px-6 py-3 text-left text-sm font-semibold">Weight Units</th>
93+
<th className="px-6 py-3 text-left text-sm font-semibold">Virtual Bytes</th>
94+
</tr>
95+
</thead>
96+
<tbody className="divide-y divide-gray-200 dark:divide-gray-800">
97+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
98+
<td className="px-6 py-4">Legacy (non-SegWit)</td>
99+
<td className="px-6 py-4">size × 4</td>
100+
<td className="px-6 py-4">Same as raw size</td>
101+
</tr>
102+
<tr className="hover:bg-gray-100 dark:hover:bg-gray-800">
103+
<td className="px-6 py-4">Native SegWit</td>
104+
<td className="px-6 py-4">(base × 4) + witness</td>
105+
<td className="px-6 py-4">~25% smaller than raw size</td>
106+
</tr>
107+
</tbody>
108+
</table>
109+
</div>
110+
111+
<ExpandableAlert title="Example Calculation" type="info" expandable={true} initialLines={2}>
112+
Consider a SegWit transaction with:
113+
- Base size: 200 bytes
114+
- Witness size: 100 bytes
115+
116+
1. Weight = (200 × 4) + 100 = 900 weight units
117+
2. Virtual Size = 900 ÷ 4 = 225 vBytes
118+
119+
Even though the raw size is 300 bytes (200 + 100), the virtual size is only 225 vBytes thanks to the SegWit discount!
120+
</ExpandableAlert>
121+
122+
## Minimum Relay Fee Rate
123+
124+
Nodes enforce a minimum fee rate to prevent spam:
125+
126+
- Default minimum: 1 sat/vbyte
127+
- Transactions below this rate are typically rejected
128+
- You can check your node's setting:
129+
130+
<CodeSnippet
131+
language="bash"
132+
code={`$ bitcoin-cli getmempoolinfo
133+
{
134+
"loaded": true,
135+
"size": 47293,
136+
"bytes": 32883821,
137+
"usage": 148075552,
138+
"maxmempool": 300000000,
139+
"mempoolminfee": 0.00001000, # 1 sat/vbyte
140+
...
141+
}`}
142+
/>
143+
144+
<ExpandableAlert title="NOTE" type="info">
145+
💡 The minimum relay fee is a policy rule, not a consensus rule. While
146+
theoretically possible, getting a below-minimum fee transaction mined is
147+
extremely unlikely unless you have a direct connection to a miner.
148+
</ExpandableAlert>
149+
150+
## RPC: Current Fee Rates
151+
152+
You can estimate current fee rates using Bitcoin Core:
153+
154+
<CodeSnippet
155+
language="bash"
156+
code={`$ bitcoin-cli estimatesmartfee 6
157+
{
158+
"feerate": 0.00008124, # BTC/kB
159+
"blocks": 6
160+
}`}
161+
/>
162+
163+
Or query popular block explorers:
164+
- mempool.space
165+
- blockstream.info
166+
- bitcoinfees.earn.com
167+
168+
Remember that fee rates are dynamic and can change rapidly based on network demand.

0 commit comments

Comments
 (0)