Introduction #
For the past three years as a security consultant testing web applications, I've encountered several projects with unique edge cases—whether it's a client using an unusual serialization format or an outdated technology stack that Burp Suite doesn’t support.
After searching for solutions and finding none, I was left with two options: perform the tests manually or patch together different tools to achieve the desired outcome. In this writeup, I’ll share how I overcame these challenges.
Imagine you're assigned to a security testing engagement where the client uses a SOAP API that requires digitally signing XML, similar to AWS's Sigv4. Unfortunately, Burp Suite doesn’t support this feature and has no plans to add it due to the outdated nature of the technology.
So, how can we script this process?
One limitation with Burp Suite is its Jython integration, which only supports Python 2.7, meaning you can't import external libraries like you could with a regular Python environment. However, one method that worked for me was to route traffic through Burp Suite to Mitmproxy, where a Python function signs the XML before forwarding it to the server.
Here's a method I used:
- curl sends the request to Burp Suite.
- Burp forwards the request to Mitmproxy.
- MITMProxy runs a Python function that signs the XML and sends it to the web server.
Flow
graph LR;
curl -->|HTTP Traffic | Burp[Burp Suite 8080]
Burp -->|Upstream Proxy| mitmproxy[mitmproxy 8088]
mitmproxy -->|Signs XML| WebServer[WebServer]
⚠️ A word of warning: While this approach works, it's far from painless. SOAP XML signing is extremely brittle — a single whitespace difference, namespace mismatch, or canonicalization issue will silently break your signature with little to no useful error output. The script provided is a starting point, but expect to spend significant time adapting it to your specific SOAP schema. This is a last resort when other options aren't available, not a plug-and-play solution.
Setup #
First, we need to install mitmproxy and the required libraries. I recommend installing mitmproxy using pip in a virtual environment to keep your environment clean.
1# Set up a virtual environment and activate it
2python3 -m venv venv
3source venv/bin/activate
4
5# Install required dependencies
6pip install mitmproxy xmlsec lxml
7
8# Ensure the certificate, key, and XML template are in the same directory
9mitmweb -s mitm-xml-signer.py --listen-port 8088 # Web version
10mitmproxy -s mitm-xml-signer.py --listen-port 8088 # CLI version
11
Configure Burp Suite to Use mitmproxy as an Upstream Proxy #
To forward traffic from Burp Suite to mitmproxy, configure Burp Suite to use mitmproxy as an upstream proxy.
Steps in Burp Suite:
- Open Burp Suite and navigate to Settings.
- In the Connections section under Network, select Upstream Proxy Servers.
- Click Add to configure the upstream proxy settings.
- Enter the following details:
- Destination Host:
*(forward all traffic tomitmproxy) - Destination Port:
* - Proxy Host:
127.0.0.1(or the IP where mitmproxy is running) - Proxy Port:
8088(the portmitmproxyis listening on)
- Destination Host:
- Save the configuration by clicking OK.
{: .light }
{: .dark }
Burp Suite "Connections" with upstream proxy set to 127.0.0.1:8088 for all traffic.
XML Template #
For my mitmproxy script to sign the XML body correctly, I tailored the script to parse and look for the Body element with an Id attribute—that is, <Body Id="">——as this was the requirement based on the SOAP API I was testing. Depending on how your client's SOAP API is structured, you will need to tailor the script accordingly.
1<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
2 xmlns:ns="http://example.com/namespace">
3 <soapenv:Header/>
4 <soapenv:Body Id="1">
5 <ns:MyRequest>
6 <ns:Data>Sample data</ns:Data>
7 </ns:MyRequest>
8 </soapenv:Body>
9</soapenv:Envelope>
Python Script to Sign XML #
Below is the Python script that will be executed by mitmproxy to sign the XML. Make sure you place the certificate, private key, and this script in the same directory.
1import mitmproxy.http
2from mitmproxy import http
3import xmlsec
4from lxml import etree
5from OpenSSL import crypto
6import os
7
8'''
9Instructions:
101. Update the following variables as needed:
11 - private_key_password = "secret" (or set to None if not required)
12 - private_key_path = "privatekey.pem"
13 - cert_path = "cert.pem"
14
152. To set the PRIVATE_KEY_PASSWORD environment variable, use the following command:
16 - Linux/macOS: export PRIVATE_KEY_PASSWORD="your_password_here"
17 - Windows: set PRIVATE_KEY_PASSWORD=your_password_here
18
193. To run the script with mitmproxy:
20 mitmweb -s mitm-xml-signer.py --listen-port 8888
21'''
22
23def load_private_key_with_password(private_key_path, password):
24 try:
25 with open(private_key_path, "rb") as key_file:
26 private_key_data = key_file.read()
27
28 private_key = crypto.load_privatekey(crypto.FILETYPE_PEM, private_key_data, passphrase=password.encode())
29 return private_key
30 except Exception as e:
31 print(f"Failed to load or decrypt private key: {e}")
32 return None
33
34def load_certificate_details(cert_path):
35 try:
36 with open(cert_path, "rb") as cert_file:
37 cert_data = cert_file.read()
38 certificate = crypto.load_certificate(crypto.FILETYPE_PEM, cert_data)
39
40 issuer = certificate.get_issuer()
41 serial_number = certificate.get_serial_number()
42 return certificate, issuer, serial_number
43 except Exception as e:
44 print(f"Failed to load certificate: {e}")
45 return None, None, None
46
47# Updated add_key_info to only include O and OU
48def add_key_info(signature_node, issuer, serial_number):
49 key_info = xmlsec.template.ensure_key_info(signature_node)
50 x509_data = xmlsec.template.add_x509_data(key_info)
51
52 issuer_serial_node = etree.SubElement(x509_data, "{http://www.w3.org/2000/09/xmldsig#}X509IssuerSerial")
53 issuer_name_node = etree.SubElement(issuer_serial_node, "{http://www.w3.org/2000/09/xmldsig#}X509IssuerName")
54
55 # Filter issuer components to include only 'O' and 'OU'
56 desired_components = [b'O', b'OU'] # Components you want to include
57 issuer_components = [
58 (name.decode(), value.decode())
59 for name, value in issuer.get_components()
60 if name in desired_components
61 ]
62
63 # Build the issuer name string
64 issuer_name_node.text = ", ".join([f"{name}={value}" for name, value in issuer_components])
65
66 serial_number_node = etree.SubElement(issuer_serial_node, "{http://www.w3.org/2000/09/xmldsig#}X509SerialNumber")
67 serial_number_node.text = str(serial_number)
68
69def sign_xml(xml_str, private_key_password):
70 xml = etree.fromstring(xml_str)
71
72 private_key_path = "privatekey.pem"
73 cert_path = "cert.pem"
74
75 private_key = load_private_key_with_password(private_key_path, private_key_password)
76 if private_key is None:
77 return None
78
79 certificate, issuer, serial_number = load_certificate_details(cert_path)
80 if certificate is None:
81 return None
82
83 sign_ctx = xmlsec.SignatureContext()
84 key = xmlsec.Key.from_file(private_key_path, xmlsec.KeyFormat.PEM, password=private_key_password)
85 key.load_cert_from_file(cert_path, xmlsec.KeyFormat.PEM)
86 sign_ctx.key = key
87
88 # Extract namespace mappings
89 nsmap = xml.nsmap
90 nsmap_rev = {v: k for k, v in nsmap.items()}
91
92 # Get the SOAP envelope namespace and prefix
93 soapenv_ns = 'http://schemas.xmlsoap.org/soap/envelope/'
94 soapenv_prefix = nsmap_rev.get(soapenv_ns)
95 if soapenv_prefix is None:
96 raise ValueError("SOAP envelope namespace is not defined in the XML.")
97
98 # Find the Body element using the namespace prefix
99 body = xml.find(f'.//{{{soapenv_ns}}}Body')
100 if body is None:
101 raise ValueError("SOAP Body not found!")
102
103 # Register the 'id' or 'Id' attribute
104 id_attr_name = None
105 for attr_name in ['Id', 'id']:
106 if body.get(attr_name) is not None:
107 id_attr_name = attr_name
108 break
109 if id_attr_name is None:
110 raise ValueError("Body element does not have 'Id' or 'id' attribute.")
111
112 # Register the ID attribute
113 xmlsec.tree.add_ids(body, [id_attr_name])
114
115 # Find the Header element; if it doesn't exist, create it
116 header = xml.find(f'.//{{{soapenv_ns}}}Header')
117 if header is None:
118 # Find the Envelope element
119 envelope = xml.find(f'.//{{{soapenv_ns}}}Envelope')
120 if envelope is None:
121 raise ValueError("SOAP Envelope not found!")
122
123 # Create the Header element with the existing namespace prefix
124 header_tag = f'{{{soapenv_ns}}}Header'
125 header = etree.SubElement(envelope, header_tag, nsmap={soapenv_prefix: soapenv_ns})
126
127 # Create the Signature node
128 signature_node = xmlsec.template.create(xml, xmlsec.Transform.EXCL_C14N, xmlsec.Transform.RSA_SHA1)
129
130 header.append(signature_node)
131
132 # Specify the URI to point to the Body's Id
133 ref = xmlsec.template.add_reference(signature_node, xmlsec.Transform.SHA1, uri=f"#{body.get(id_attr_name)}")
134 xmlsec.template.add_transform(ref, xmlsec.Transform.ENVELOPED)
135 xmlsec.template.add_transform(ref, xmlsec.Transform.EXCL_C14N)
136
137 add_key_info(signature_node, issuer, serial_number)
138
139 # Sign the signature node
140 try:
141 sign_ctx.sign(signature_node)
142 except Exception as e:
143 print(f"Error during signing: {e}")
144 return None
145
146 return etree.tostring(xml, pretty_print=True).decode()
147
148# Mitmproxy addon to intercept and modify requests
149class XMLSigner:
150 def __init__(self):
151 self.private_key_password = os.getenv('PRIVATE_KEY_PASSWORD')
152
153 def request(self, flow: mitmproxy.http.HTTPFlow):
154 # Check if the request is an XML (you can refine this check)
155 if "xml" in flow.request.headers.get("Content-Type", "").lower():
156 xml_content = flow.request.content
157
158 try:
159 signed_xml = sign_xml(xml_content, self.private_key_password)
160 if signed_xml:
161 flow.request.content = signed_xml.encode('utf-8')
162 print("XML request successfully signed.")
163 else:
164 print("Failed to sign XML.")
165 except Exception as e:
166 print(f"Error during XML signing: {e}")
167
168
169addons = [
170 XMLSigner()
171]
172
173
Running the Script #
To execute the script, ensure the necessary files (Certificate, Key, and XML template) are in the same directory as mitmproxy and run mitmweb with the provided script:
1mitmweb -s mitm-xml-signer.py --listen-port 8088
Once mitmweb is running, you can visit the web interface to inspect the flows: http://127.0.0.1:8081/#/flows
Finally, we send a curl request to Burp Suite and verify mitm is signing the xml
Verifying with Curl #
Finally, send a request using curl to Burp Suite, which will forward it to mitmproxy to sign the XML:
1curl --location \
2 --request POST "https://www.example.com/" \
3 --header "Content-Type: text/xml; charset=utf-8" \
4 --data @template.xml \
5 --proxy http://127.0.0.1:8080 \
6 --insecure \
7 -v
{: .light }
{: .dark }
mitmproxy script in action signing xml requests.
Limitations and Considerations #
While this solution offers a practical method for automating the signing of SOAP requests during testing, it's important to be aware of its limitations and potential impact on your workflow.
Firstly, the provided Python script is primarily intended for demonstration purposes. It's tailored to work with a specific structure of the SOAP XML envelope. If your SOAP messages use a different structure, namespaces, or include additional elements, you'll need to adjust the XML parsing logic within the script. This may involve modifying how the script locates the <Envelope>, <Header>, and <Body> elements, as well as how it handles namespaces. Ensuring that the script aligns with your specific XML schema is crucial for it to function correctly.
Another consideration is the effect of using mitmproxy as an upstream proxy on Burp Suite's functionality. Specifically, configuring Burp Suite to forward traffic to mitmproxy can interfere with the Burp Collaborator feature. Burp Collaborator relies on making HTTPS requests to detect out-of-band interactions, and when these requests are routed through mitmproxy, they may fail due to SSL issues. This happens because mitmproxy, acting as an upstream proxy, might not handle these HTTPS requests properly, leading to unsuccessful interactions.
To mitigate this issue, you have a couple of options. One approach is to toggle the upstream proxy settings in Burp Suite as needed. When you require Burp Collaborator for testing, you can disable the upstream proxy settings to allow Burp to handle HTTPS requests directly. Once you're done, you can re-enable the upstream proxy to continue using mitmproxy for signing SOAP requests. While this method works, it can be somewhat cumbersome to repeatedly change settings during your testing sessions.
Alternatively, you can configure Burp Collaborator to use unencrypted HTTP instead of HTTPS. This adjustment allows the Collaborator traffic to pass through mitmproxy without encountering SSL-related problems. However, this approach has security implications. Using HTTP instead of HTTPS means the data is transmitted in plain text, which could be a concern in environments where security is paramount. Before opting for this solution, ensure it complies with your organization's security policies and the requirements of your testing engagement.
Conclusion #
I hope this guide helps you overcome similar challenges in your next project or security engagement. By using mitmproxy alongside Burp Suite, you can handle complex scenarios like digitally signing SOAP requests or working with custom serializers—even when your usual tools fall short. This approach demonstrates how integrating different tools can extend your capabilities and adapt to unique testing requirements. I encourage you to experiment with this method and tailor it to your specific needs, pushing the boundaries of what's possible in security testing.