What's the easiest way to set up a simulated network with ISP-style NAT? Preferably without buying dedicated hardware for the purpose, and preferably reproducible for other developers.
Usecase: I am developing a P2P system, and need a reasonably representative environment to test my software in, and how well it deals with shitty residential networking configurations.
I'm a developer, not a network engineer, so my knowledge of networks is limited to a developer perspective and I don't have the spoons to learn it in-depth.
Boosts appreciated 
@joepie91 set up a machine behind a consumer router and disable all port mapping protocols
I'd love to have an "internet" VM that simulates a (small scale) global internet, with packet losses, variable latency, etc., in a configurable way, along with a way to hook up individual client VMs to the fake-internet (with simulated NAT etc. like you mention). A turn-key solution really ought to be possible, but I'm not sure one exists.
https://www.nsnam.org/about/
@retr0id @joepie91 A generic linux VM with https://github.com/tylertreat/Comcast would get you the configurable flakiness. (and NAT is available in the basic system).
@joepie91 You could do this fairly easily (I think) in AWS using multiple VPCs (for your home networks) and then using EC2 instances with 2 ENIs to route traffic between the 'home networks' and your 'internet VPC'. You'd have to configure the EC2 instance to do the routing, firewall and NAT though (but there are images for that).
I don't know how scalable or reproducible it needs to be.
@joepie91 I work at an ISP, and I am not aware of anything that comes close to what you need. Here are my thoughts on P2P systems for residential internet connections:
If you look at IPv6, this is quite easy: All hosts know their own globally unique and reachable addresses, and can send packets to each other, which are not network address translated. 1/4
@joepie91 You only have to deal with firewalls. I would expect most CPEs to drop incoming packets without corresponding firewall state (they should), so you just need an out-of-band way to negotiate addresses and ports and start sending packets from both ends of the connection so that firewall state is established on both sides, before packets will actually reach the target hosts. (This will only work with UDP.) 2/4
@joepie91 Looking at IPv4: Due to constantly increasing demand for IPv4 addresses, in a year or two, almost all residential internet connections will be behind carrier-grade NAT (CGN). (Source: personal gut feeling) Protocols do exist to request incoming connections from CGN boxes (i.e. PCP), but I am not aware of any ISP actually implementing them. So I would expect most P2P nodes to not be able to support incoming connections over IPv4. 3/5
@joepie91 Current solutions for VOIP and WebRTC make use of centralized infrastructure that relays data between hosts behind NAT. (TURN) 4/5
@joepie91 So, to summarize: A P2P network has to be built primarily on IPv6. It should be possible to use the few P2P nodes with public IPv4 address or cooperating local NAT and no CGN to allow IPv4-only hosts to participate in the P2P network in a limited manner. I know this is extremely annoying and I am sorry. We fucked this up royally. 5/5
@joepie91 should be possible using Linux network namespaces, they basically give you network-isolated environments on the same machine, without having to run complete VMs.
@joepie91 I am in the same situation as you and I hate to say it but I have come to the conclusion that there's no substitute for the real thing.
In terms of simulating just the NAT part, the default networking configurations of Libvirt VMs and Docker containers should get you there. So for example, you could test two containers which are on different docker "bridged" networks, and see if you can get them to talk P2P in the usual way: the two will rendezvous somewhere like a STUN server, exchange each-others source ports, and then attempt to connect to each-other via UDP.
So for example:
* container A decides "I will create a socket with source port 12345"
* container B decides "I will create a socket with source port 43211"
* they exchange that info with eachother
* container A opens a UDP socket to the docker bridge IP address with source port 12345 and destination port 43211
* container B opens a UDP socket to the docker bridge IP address with source port 43211 and destination port 12345
To be honest I've never tried this, I have no idea if it will work but based on my somewhat limited understanding, it _should_ work for initial prototyping.
However the problem I see is that when it comes time to polish the application, you will run into situations where every home router is slightly different. I'm not aware of any tools which can simulate that. So far I have simply tried to deploy apps in as many different home networks as possible, those of my friends and family.
@joepie91 gns3 with any choice of vendor that works for the CGNAT part
@joepie91 at previous job we turned up ESXi in a VM, and ran quite a lot of DNS servers and other stuff to simulate a network
Today I would run the same in Kubernetes or dockers
There is nothing /that/ special about an ISP network
@joepie91 "ISP-style NAT" is a very wide range of weird varieties (varying as well by country/company), so it's fairly hard to be sure exactly what you're looking for.
That said, you might be looking for something along the lines of having multiple routers' WAN ports connected up to a single parent router, subnetted to be completely independent of each other but ultimately able to connect over the Internet.
I'm curious exactly what your P2P system is and why you'd need to test this specifically