VPN Troubleshooting Q&A
This document is still a work in progress and will continue to have informational and formatting changes.
General Questions:
What, if any, error messages are presented?
Is this a new connection being tested for the first time, or an existing one that encountered issues?
When did this issue start? When was the last successful connection?
Is the client aware of any changes since this was last working?
Do all commpoints/servers using this connection encounter the same issues?
How is this supposed to work? i.e. VPN, Transit Gateway/VPC peering connection, public Internet with IP and port Whitelisting, public Internet with DNS and the proxy server?
What is the IP address and port number the traffic is coming from and going to?
Has the client contacted their networking team to troubleshoot? Ask the client networking team the following quetions:
What is the behavior of the traffic? Do they see the traffic leaving interface server, reaching their firewall, and being sent out? Do they see any replies to the traffic being sent? Do the logs indicate any other issues, for instance VPN flapping, incorrect traffic selectors, etc?
General Troubleshooting:
Check the AWS VPN logs to see if the connection is stable. Search the logs for the word "traffic" and that will reveal every time a phase 2 rekey happened successfully. A stable connection with the standard settings should have one phase 2 rekey every hour and phase 1 rekey every eight hours. If rekeys are happening more frequently then assume both sides may be attemtping to initiate the connection and try setting one side to responder only (usually easier to set AWS to be responder).
If rekeys are happening with standard frequency, check the traffic selectors displayed in the phase 2 rekey message. What is displayed there is what the customer has configured on their gateway. Ensure the customer gateway is not sending multiple rekeys with a /32 IP on their side. If it is, explain AWS can only use one security association at a time and that the /32 needs to be replaced with a subnet that includes all hosts to be routed on the tunnel.
If the traffic selectors presented from the customer gateway are specific subnets but AWS is configured with the default 0.0.0.0/0 <> 0.0.0.0/0 then update the AWS traffic selectors to match the customer's.
Specific Troubleshooting:
Always first start by verifying how this connection should be working. Is it going over a VPN, and if so are they using the correct VPN NAT IPs? Is it going over a transit gateway connection, if so are they using the correct static IPs and is the EC2 instance configured to use that static IP for outbound traffic? Is it going over the Internet directly from the EC2 instance to a specific IP, if so ensure the security groups are configured to allow this traffic? Is it going to a URL
If a commpoint isn't working:
Is just one not working or all? If just one is not working but others are, assume it is specific to that port number and troubleshoot security groups/ACLs. If all are not working then assume the issue exists with the entire host and focus on traffic selectors or routing issues. Is the commpoint configured to talk to an IP or a URL? If IP check security groups. If URL check the proxy server to ensure that URL is allowed and ensure the commpoint is configured to use the proxy server. If both are configured correctly check the proxy server logs to see what is happening to the outbound traffic.
If an entire host is not working, are others working? For instance prod works but dev does not. This would likely indicate an issue with the traffic selectors.
How to check proxy server logs:
Connect to the proxy server and grep part of the URL in the file /var/log/squid/access.log. if you see TCP_DENIED/403 then the squid proxy is blocking the traffic and needs to be configured correctly. If it does not say TCP_DENIED/403 then the traffic is most likely leaving the proxy server successfully and the issue lies outside of our AWS environment. If you do not see the URL in the access.log at all, then the traffic never made it to the proxy server and the commpoint likely isn't configured correctly.
The following section is still a work in progress:
- if connectivity is intermittent
-- check vpn logs for tunnel establishment message. search for "traffic". phase 1 should rekey every hour and phase 2 every 8 hours (if configured with default lifetimes). If this is happening more frequently determine why
-- if frequent rekeys, confirm settings on gateways. likely both are set to start/main mode. Set one side to be responder only, often easier to set AWS to responder only. Startup action should be "add" and DPD timeout action should be "clear"
if VPN is up but no communication
- to analyze packet flow:
-- ensure VPC flow logs are enabled on both the site to site VPN and the TGW
-- find the TGW attachment ID for the VPN in question, easiest is from the route table checking a known route
-- examine the flow logs for that TGW attachment. this is the first point in the AWS network where the packet can be seen. if packets not seen here, they are not making it to the TGW
-- examine the flow logs for the ENI attached to the instance in question. if packets are not seen here, they are not making it to the instance
-- run a tcpdump from the OS with: timeout 1h tcpdump -vnni any
--- replace
--- look for the flag section of the packet capture. in a normal connection the TCP session is established first, you should see a SYN from the client [S], a SYN/ACK from the server [S.], and an ACK from the client [.]. If these three are seen the intial TCP handshake was successful. You should then see data pushed and acknowledged [P] and [P.] and [.].
---- if a TCP RST [R] is seen shortly after the initial TCP handshake, this suggests whichever side is sending it
-- if rhapsody receiving traffic but not replying (receive SYN but no SYN/ACK)
How to use netstat to ensure rhapsody is listening or has established connections:
If the customer states they cannot use a subnet on their traffic selector then they must either rebuild the connection as route-based, or build a separate VPN tunnel for each IP.
To set AWS as the responder, change the Startup Action to Add and the DPD Timeout Action to None.
what information needed from cloudops after resolving a ticket:
-- any changes made, what environment looked like before and after, why changes were made