BitTorrent Protocol: Overview

BitTorrent is a protocol for distributing files over the internet.

Unlike traditional file client/server systems, BitTorrent is peer-to-peer. It doesn't rely on a single central server to host the file. Instead, it relies on a distributed network of "peers".

In this concept, we'll learn more about how the BitTorrent protocol works. You'll learn about trackers, peers, pieces and more.

Pieces

A file is broken down into multiple pieces. This allows a file to be downloaded from multiple peers in parallel.

Each piece is typically 256KB in size.

When all pieces are downloaded, they can be combined to assemble the original file.

Piece Hashes

Each piece is identified by a 20-byte SHA1 hash. This allows peers to verify the integrity of the pieces they download.

This is important because peers can't trust each other. A malicious peer could send a corrupt piece to another peer. As long as the receiving peer verifies the hash of each piece they download, they can be sure that the piece is valid.

Trackers

A peer needs to know which other peers it can download a file from. To do this, they contact a "tracker".

Trackers are servers that keep track of which peers have which files. Peers can ask a tracker for a list of peers that have a file. They can then contact those peers to download the file.

You might've noticed that we said that BitTorrent doesn't rely on a central server, but a tracker sounds a lot like a "central" server. What gives?

A tracker is only used to find peers, not to download files. Once a peer has found other peers, it no longer relies on the tracker to download the file.

That said, relying on a tracker even if it's just to find peers isn't ideal.

DHT (distributed hash table) is an extension to the BitTorrent protocol that allows peers to find each other without a tracker. We won't cover this in detail, but you can read more about it here.

Torrent Files

A .torrent file contains all the information a peer needs to download a file.

Most importantly, it contains:

• List of all pieces and their hashes
• The URL(s) for one or more trackers

The data in a .torrent file is encoded using the Bencode encoding format. We have a separate concept that covers Bencode in detail.

Here's an example of .torrent file for Debian (a Linux distribution):

d
  8:announce
    41:http://bttracker.debian.org:6969/announce
  7:comment
    35:"Debian CD from cdimage.debian.org"
  13:creation date
    i1573903810e
  4:info
    d
      6:length
        i351272960e
      4:name
        31:debian-10.2.0-amd64-netinst.iso
      12:piece length
        i262144e
      6:pieces
        (binary blob of the hashes of each piece)
    e
e

(Note: the above example is formatted for readability, a real .torrent file would be a single long line)

We won't cover Bencode in detail here. View the Bencode concept to learn more.

Peer Messages

We now know that given a .torrent file a peer can do the following:

• Connect to a tracker to find other peers
• Know which pieces it needs to download

But how do peers actually communicate with each other?

This is where peer messages come in. Peers send each other messages to request pieces, send pieces, and more.

We have a separate concept that covers peer messages in detail.

For now, we'll look at a common sequence of messages that peers send to each other.

We'll use the following terminology:

• "downloader": a peer that is downloading a file
• "uploader": a peer that is uploading a file

When the downloader connects to the uploader, here's roughly what the sequence of messages looks like:

1. downloader connects to uploader and performs a "handshake"
2. downloader waits to receive a bitfield message from the uploader
3. uploader sends a bitfield message to the downloader, indicating which pieces it has
4. downloader sends an interested message to the uploader, indicating that it wants to download pieces from the uploader
5. uploader sends an unchoke message to the downloader, indicating that the downloader can start requesting pieces
6. downloader sends multiple request messages to the uploader, indicating which pieces it wants
7. uploader sends multiple piece messages to the downloader, containing the pieces the downloader requested

We aren't going to cover these messages in detail here. View the Peer Messages concept to learn more.

Summary

In this concept, we learned about the following:

• Pieces: A file is broken down into pieces so that it can be downloaded in parallel from multiple peers.
• Piece Hashes: Each piece is identified by a 20-byte SHA1 hash, which allows peers to verify the integrity of the pieces they download.
• Trackers: Trackers are servers that keep track of which peers have which files
• Torrent Files: A .torrent file lists the pieces that make up a torrent, along with their hashes. It also includes the URL(s) for one or more trackers
• Peer Messages: Peers send each other messages to request pieces, send pieces, and more