Rust Reqwest via Proxy and SSL Certificate captured by Burp

Make browser or API call requests via Rust script can be done with Reqwest module. It’s easier than using Hyper, which recently has a lot of changes. There nothing many articles, guidelines or working examples with the latest version of Hyper and Reqwest for advanced usage.

Several things that I want to achieve here where there lack of coverage on:

1. How to enable reqwest working with proxy
2. How to use Authorization Beaver in reqwest 
3. How to applied self signed certificate 
4. How to capture the traffic via Burp for monitoring

I tried with Hyper and Hyper Proxy, but it seems it’s more complicated and harder to do than Reqwest.

Here is an step by step to enable proxy, certification and captured by Burp.

cargo.toml

[dependencies]
hyper = "0.13.5"
hyper-native-tls = "0.3.0"
reqwest = { version = "0.10.4", features = ["blocking", "json", "gzip"] }
tokio = { version = "0.2", features = ["full"] }
diesel = { version = "1.4.4", features = ["postgres", "chrono"] }
dotenv = "0.15.0"
chrono = { version = "0.4.11", features = ["serde"] }
serde = { version = "1.0.110", features = ["derive"] }

1. Enable Certification in Reqwest

let mut buf = Vec::new();
File::open("cacert.der").unwrap().read_to_end(&mut buf).unwrap();
let cert = reqwest::Certificate::from_der(&buf).unwrap();

2. Add custom headers

// Set required header to mimic browser / application
let mut headers = header::HeaderMap::new();

headers.insert(header::AUTHORIZATION, header::HeaderValue::from_static("Bearer XXXX"));
headers.insert(header::USER_AGENT, header::HeaderValue::from_static("YOUR USER AGENT"));
headers.insert(header::ACCEPT_ENCODING, header::HeaderValue::from_static("gzip, deflate"));
headers.insert(header::HeaderName::from_static("x-custom-header"), header::HeaderValue::from_static("x-value"));

For more details about Reqwest headers

3. Add proxy

// Proxy for burp
let raw_proxy = format!("http://burp-ip-proxy:8080");
let mut proxy = reqwest::Proxy::all(&raw_proxy).unwrap();

4. Final configuration for Client

let client = reqwest::blocking::Client::builder()
            .default_headers(headers)
            .gzip(true)
            .timeout(Duration::from_secs(10))
            .add_root_certificate(cert)
            .proxy(proxy)
            .build()
            .unwrap();

Wrap-up all script into single main.rs

//! Hello, and welcome to the main code for the scrapper!
//!
//! This API documentation is highly technical and is purely a reference
extern crate chrono;
extern crate diesel;
extern crate stock;
extern crate job_scheduler;
extern crate select;
extern crate reqwest;

use reqwest::header;

use std::fs::File;
use std::io::Read;
use std::time::Duration;

/// Main function to be executed either from command-line or cargo run
fn main() {
    fetch("https://yodiw.com");
}

/// Fetch URL
fn fetch(url: &str) {
    let mut buf = Vec::new();
    File::open("cacert.der").unwrap().read_to_end(&mut buf).unwrap();
    let cert = reqwest::Certificate::from_der(&buf).unwrap();

    // Set required header to mimic browser / application
    let mut headers = header::HeaderMap::new();

    headers.insert(header::AUTHORIZATION, header::HeaderValue::from_static("Bearer XXXXXXXXX"));
    headers.insert(header::USER_AGENT, header::HeaderValue::from_static("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36"));
    headers.insert(header::ACCEPT_ENCODING, header::HeaderValue::from_static("gzip, deflate"));
    headers.insert(header::HeaderName::from_static("x-custom-header"), header::HeaderValue::from_static("custom value"));
 
    // Proxy for burp
    let raw_proxy = format!("http://ip-burp-proxy:8080");
    let mut proxy = reqwest::Proxy::all(&raw_proxy).unwrap();

    let client = reqwest::blocking::Client::builder()
                .default_headers(headers)
                .gzip(true)
                .timeout(Duration::from_secs(10))
                .add_root_certificate(cert)
                .proxy(proxy)
                .build()
                .unwrap();

    // Calling the API
    let  mut response = client.get(url).send().unwrap();
    println!("{:#?}", response);
    println!("{}", response.status());

    // Print response
    let mut buf = String::new();
    response.read_to_string(&mut buf).expect("Failed to read response");
    println!("{}", buf);
}

Hope this will help you in using Reqwest for more advance usage. I have a plan to open source my rust crawler engine at https://github.com/yodiaditya

Please stay tune