I often receive emails asking what is the difference between mitmproxy helper and Wireshark Helper and ordinary packet capture software.
The reason for the birth of these two softwares is that a friend who is engaged in network work felt that the current mobile phone packet capture software was not 'complete' enough, resulting in many packets not being captured, so we spent time together and made two softwares based on the scenes that network workers often work.
So here is a unified explanation.
The four-layer model of TCP/IP
Application layer | such as HTTP protocol
Transport layer | such as TCP/UDP protocol
Network layer | e.g. IPv4/IPv6 packets
link layer | e.g. Ethernet
Let's figure out two concepts first, one is system proxy, and the other is virtual network card.
Let's take the desktop system as an example.
This concept must be familiar to everyone, such as macOS, you can modify the system proxy in System Preferences->Network->Advanced->Proxy,
But after setting the proxy here, [Whether the application software will use this proxy depends entirely on the application software itself], for example, the Chrome browser will use the proxy set here, and many applications such as Terminal choose to ignore this proxy.
In fact, it works at the application layer. If it is intercepted here, the packet capture tool only needs to process the protocol packets of the application layer.
This is also the way most of the current packet capture tools are implemented.
The principle of virtual network card is the same as that of ordinary network card, here it works at the bottom link layer.
If we route all packets to this virtual network card, this also means that the application software will pass through it, no longer depends on whether the application software wants it or not.
Intercepting here is more troublesome to implement. It is not enough only understand the HTTP protocol. You need to care about the entire TCP/IP stack. First, you must process the IP protocol packets of the network layer, and then convert the IP packets to the TCP/UDP protocol of the network layer. , and then the TCP/UDP protocol corresponds to the application layer protocol processing.
mitmproxy helper and Wireshark Helper currently use virtual network cards to intercept.
We mentioned the difference between system proxy and virtual network card. iOS and Android also have the same concept, that is, the packet capture tool may be intercepted by system proxy, or it may be in virtual network card interception.
Note: [App can easily skip system proxy], for example: .connectionProxyDictionary = [AnyHashable : Any]()
mitmproxy helper and Wireshark Helper are currently using virtual network cards to intercept.
Packet capture on the iOS side requires the use of Network Extension technology, meaning that the VPN logo will appear on the top of the phone.
Answer: No.
Because the iOS application settings system proxy also need to use Network Extension technology,
So the VPN logo appears, and the packet capture tool may still be intercepted at the [system proxy] layer,
The author of some packet capture tools will explain in the introduction which layer is intercepted. If you don't see it, you need to ask the author.
At present, the Android system has built-in Private DNS (DoT), and the desktop and mobile versions of Chrome provide built-in Secure DNS (DoH)
When DoT or DoH is turned on, the query content cannot be intercepted
So we're going to close it:
First intercept the packet and then decrypt it. The principle of TLS decryption (the encryption of many protocols uses TLS encryption technology, such as HTTPS, etc.) is the middleman, that is, a root certificate needs to be installed in the mobile phone system.
If you haven't installed the root of trust certificate before, here are two videos Wireshark Helper Video and mitmproxy helper video.
Be sure to watch the video carefully, iOS needs to operate in two places before it takes effect, the video is showing iOS 14, and different versions of the system may operate slightly differently.
Android system also created the concept of user certificate and system certificate, Root is required to install into the system certificate.
If you understand the principle of encryption, you can get uncertain conclusions. For example, suppose the client and the server agree on a password and an encryption algorithm,
Then the client writes this password into the client code, and when it needs to be encrypted, it is encrypted with this password and sent to the server,
The server uses the same password to decrypt the received client data. At this time, only those who know the password and encryption algorithm can decrypt it.
That is to say, by installing a trusted root certificate, the middleman can decrypt the mostly TLS encryption, which is powerless for customized or magic-modified encryption.
Although their UI is not the best looking, but as a View layer there is no stronger than them, Wireshark is a tool that developed almost with the early Internet.
There are many packet capture tools with gorgeous UIs that are not usable for us, because compared with these two View layers, both the protocol viewing granularity and the operation are too simple, and they are not realistic.