What the scanner and the generator do

Running the scanner

Overview

The scanner measures the bandwidth of each relay in the Tor network (except the directory authorities) by creating a two hops circuit with the relay. It then measures the bandwidth by downloading data from a destination Web Server and stores the measurements.

The generator read the measurements, aggregates, filters and scales them using torflow’s scaling method.

Then it generates a bandwidth list file that is read by a directory authority to report relays’ bandwidth in its vote.

_images/scanner.svg

Intialization

  1. Parse the command line arguments and configuration files.
  2. Launch a Tor thread with an specific configuration or connect to a running Tor daemon that is running with a suitable configuration.
  3. Obtain the list of relays in the Tor network from the Tor consensus and descriptor documents.
  4. Read and parse the old bandwidth measurements stored in the file system.
  5. Select a subset of the relays to be measured next, ordered by:
    1. relays not measured.
    2. measurements age.
data sources

Classes used in the initialization:

classes initializing data

Source code: sbws.core.scanner.run_speedtest()

Measuring relays

  1. For every relay:
  2. Select a second relay to build a Tor circuit.
  3. Build the circuit.
  4. Make HTTPS GET requests to the Web server over the circuit.
  5. Store the time the request took and the amount of bytes requested.
activity measuring relays

Source code: sbws.core.scanner.measure_relay()

Selecting a second relay

  1. If the relay to measure is an exit, use it as an exit and obtain the non-exits.
  2. If the relay to measure is not an exit, use it as first hop and obtain the exits.
  3. From non-exits or exits, select one randomly from the ones that have double consensus bandwidth than the relay to measure.
  4. If there are no relays that satisfy this, lower the required bandwidth.
activity select second relay

Source code: sbws.core.scanner.measure_relay()

Selecting the data to download

  1. While the downloaded data is smaller than 1GB or the number of download is minor than 5:
  2. Randomly, select a 16MiB range.
  3. If it takes less than 5 seconds, select a bigger range and don’t keep any information.
  4. If it takes more than 10 seconds, select an smaller range and don’t keep any information.
  5. Store the number of bytes downloaded and the time it took.

Source code: sbws.core.scanner._should_keep_result()

Writing the measurements to the filesystem

For every measured relay, the measurement result is put in a queue. There’s an independent thread getting measurements from the queue every second. Every new measurement is appended to a file as a json line (but the file itself is not json!). The file is named with the current date. Every day a new file is created.

Source code: sbws.lib.resultdump.ResultDump.enter()

Running the generator

Every hour, the generator: #. Aggregate all the measurements (not older than 6 six days) for every relay. #. Filter the measurements #. Scale the measurements #. Write the bandwidth file

Source code: sbws.lib.v3bwfile.V3BWFile.from_results()

Filtering the bandwidth measurements

Each relay bandwidth measurements are selected in the following way:

  1. At least two bandwidth measurements (Result s) MUST have been obtained within an arbitrary number of seconds (currently one day). If they are not, the relay MUST NOT be included in the Bandwith File.
  2. The measurements than are are older than an arbitrary number of senconds in the past MUST be discarded. Currently this number is the same as data_period (5 days).

If the number of relays to include in the Bandwidth File are less than a percententage (currently 60%) than the number of relays in the consensus, additional Header Lines MUST be added (see XXX) to the Bandwith File and the relays SHOULD NOT be included.

Scaling the bandwidth measurements

Consensus bandwidth obtained by new implementations MUST be comparable to the consensus bandwidth, therefore they MUST implement torflow_scaling. The bandwidth_file_spec appendix B describes torflow scaling and a linear scaling method.

Writing the bandwidth file

The bandwidth file format is defined in the bandwidth_file_spec.