Troubleshooting Private Access Connections
Prerequisites for deploying Secure App Connector
-
Provisioning Key - Copy the text string generated from the connector group from the Skyhigh Security UI (User Interface) and supply it as a parameter to the deployment script (infra.sh).
-
Proxy information - If you use a proxy server, then make sure to bypass the IP address of the proxy server along with certain domains. For the list of domains, see Deploy Connectors.
-
Gateway IP - Choose the nearest Gateway IP so that Connector establishes a secure tunnel with the private applications through Client Proxy. This is supplied as a parameter to the deployment script. Please refer to the product guide for list of Gateway URLs.
-
No Proxy - Make sure to add the CIDR range
10.0.0.0/8
inno_proxy
to assign IPv4 address to microk8s pods when you are using<no_proxy>
For information about deploying connectors, see Deploy Secure App Connectors.
Troubleshooting Secure App Connector
Log on to the connector host using SSH (Secure Shell) and execute the required commands to troubleshoot a connector.
The following table provides the list of commands you can execute:
Options | Description |
---|---|
1 (About) |
Displays the Secure App Connector image version |
2 (Status) |
Displays the utilization and status of all the services running on the connector |
3 (Troubleshooting) |
Help you troubleshoot the Secure App Connector related issues |
4 (Manage) |
Manage (start, restart, or stop) Secure App Connector |
5 (Run Diagnostics) |
Performs the diagnostics test and displays its results |
6 (Download Logs) |
Download all logs and configuration details to a temp (/tmp/) folder on the host |
0 (Exit) | Exit from the command console. |
Download Logs
You can download all logs and configuration details to/tmp/connector_state.tar.gz
- Log on to connector host using SSH.
- Execute pa_connector.sh > enter 6.
Downloads.tar.gz
file to the temp (/tmp/
) folder once the command is executed completely. -
You can upload the latest file
connector_state_currentdate_time.tar.gz
to the support portal for review.
Resolving Host Names
For a Connector to function, the system Domain Name Service (DNS) should be able to resolve both Skyhigh Security URLs and Private Application URLs. You can do the following to resolve host names using the system DNS:
Perform the following to check if the system DNS is resolving both Skyhigh Security and Private Application URLs:
-
Log on to connector host using SSH (Secure Shell).
-
Execute pa_connector.sh > enter 3 > enter a > enter the domain name.
Test the HTTP proxy set
Perform the following to test if a private application is reachable through a connector:
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 3 > enter b > enter the http proxy address.
Displays the connection established message if the private application is reachable.
Test the system proxy
Perform the following to test if a private application connection is established with the connector via system proxy:
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 3 > enter c > enter the system proxy address.
Displays if the private application connection is established with the connector via system proxy.
Test Ping
Perform the following to test ping:
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 3 > enter d.
Run TCP dump for Private Application
Perform the following to check the private application connectivity:
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 3 > enter e.
Downloads.pcap
file to the temp (/tmp/
) folder once the command is executed completely. -
You can upload the latest file
pa_capture-currentdate.pcap
to the support portal for review.
Manage connectors
Perform the following to manage connectors:
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 4 > enter <g or h or i>.
Start connector services
Perform the following to start all connector services:
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 4 > enter h.
Restart connector services
Perform the following to restart all services that run in a connector:
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 4 > enter i.
Stop a connector
-
Log on to connector host using SSH.
-
Execute pa_connector.sh > enter 4 > enter g.
Tips
- Get the Private Access connector pod name using
sudo kubectl get pods -n cwpp
- To log into the pod, run
sudo kubectl exec -ti <pod_name> -n cwpp -- bash
- To install a connector again on the same VMs, then run the following commands:
sudo rm -rf /opt/McAfee/
snap remove microk8s
-
Check the VPN (Virtual Private Network) tunnel: Tunnels are created between the connector and Private Access Gateway.
-
Run
ifconfig
tun0
andtun1
are created with an IP address 10.8.0.x. -
Check
/.../mount/logs/postRegistration.log
and/.../mount/logs/openvpn_client.log
-
-
Monitor Health Updates: Check
/.../mount/logs/healthUpdate.log
to ensure that the VPN tunnel is up. The health update requires the list of private applications monitored by the connector. So, check if the private application list sync is successful and/.../private_apps.json
is present. -
Proxy to private applications: Check
/.../mount/logs/debug.log
to ensure that an application is reachable and not blocked by the policy. Ensure that the DNS configured on the connector can access private applications and resolve their URLs.
CIS Hardening Scan on Linux IPV4 RHCE 8 Server
Fixing vulnerabilities experienced when performing the CIS Hardening scan on the Linux IPv4 RHCE 8 server:
- Ensure IP forwarding is disabled (Refer Page 227 on the CIS Guide)
- To enable IP forwarding, run the following commands
sysctl
-w net.ipv4.ip_forward=1
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002 | xargs chmod o-w
-
Ensure no world writable files exist (Refer page 618 on the CIS Guide)
- To view writable file list, run the command
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002
- Run the
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002 | xargs chmod o-w
command to remove write permission on all files
- Ensure no ungrouped files or directories exist (Refer page 621 on the CIS Guide)
- To view ungrouped files or directories, run
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -nogroup
- To change group ownership to root on all of them, run the following command:
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -nogroup | xargs chgrp root
- Run the view command to ensure there are no ungrouped files or directories.
- Ensure sticky bit is set on all world-writable directories (Refer Page 607 on the CIS Guide)
- To view sticky bit is set on all world-writable directories, run
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type d \( -perm -0002 -a ! -perm -1000 \) 2>/dev/null
- To add sticky bit to all of the directories
df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type d \( -perm -0002 -a ! -perm -1000 \) 2>/dev/null | xargs -I '{}' chmod a+t '{}' - Run the view command to ensure there are no sticky bit is set on all world-writable directories.
Common Errors and Workaround
Task | When | Error | Workaround |
---|---|---|---|
Unable to access EPEL Package | Not installed the packages correctly |
Package installation is failing |
Local package repository should be disabled and enable public repository EPEL yum repo Methods to check if repository is reachable:
Run the following commands when package installation fails:
Run the following commands to manually download and install packages:
Install subscription packages:
Ensure successful installation of packages by executing the below commands:
|
Connector Registration |
Entered incorrect provision key | Incorrect provision key Error decoding the provisioning key |
Check the |
Expired/Invalid client credentials | Exception while refreshing access token | Download the new connector package from Skyhigh Security UI and ensure that right provisioning key is used. | |
Connection to registration endpoint | Connection to registration endpoint - failed |
|
|
Firewall is used | Registration Failed |
Add The proxy port can be 8080, 8081, 443 |
|
Disable the firewall
|
|||
PA Connector is not coming up |
|
|
|
Unable to login to Connector Pod | Unable to login to connector pod | Server error | Update /etc/hosts of the VM.
|
DNS resolution failed for PA Gateway domain |
Error while resolving PA Gateway domain | ||
PoP Manager pod is not reaching completed state |
Disable the firewall
|
||
PoP is not showing healthy on the PoP Management page |
Create pop-manager pod again |
||
Bringing up the VPN tunnel | GW or HTTP_PROXY are not reachable |
|
Connector maintains an outbound VPN tunnel with PA Gateway.
|
Private Application list sync |
|
Check the /.../mount/logs/paListsync.log if Skyhigh cloud API is not reachable. If this is not reachable, then ensure it is reachable to synchronize the private application list with the connector. |
|
Private Application is not reachable |
Run the command on the connector pod:
If curl also fails, then ensure that the IP of the connector pod and host IP are not on the same subnet (10.254.254.x) |
||
Accessing Private Application | Able to access Private Application using Firefox browser, but failed to access the same application on the Chrome browser | Google Chrome - Taking Too Long to Load |
Disable the Use secure DNS option on Google Chrome. The Use secure DNS option should be disabled to access the private applications. Go to Chrome browser > Settings > Privacy and Security > Disable the Use Secure DNS option. |
Installing packages on closed environment |
sample of a snap output: [ec2-user@ip-172-22-21-231 ~]$ snap version
(you can install microk8s using snap sudo snap install --channel=1.24/stable microk8s --classic)[ec2-user@ip-172-22-21-231 ~]$ sudo microk8s statusmicrok8s is running addons: enabled: disabled: community # (core) The community addons repository
|