Skip to main content

Welcome to our updated site!

Skyhigh Security

Troubleshooting Private Access Connections

  1. Last updated
  2. Save as PDF

Prerequisites for deploying Secure App Connector

  • Provisioning Key - Copy the text string generated from the connector group from the Skyhigh Security UI (User Interface) and supply it as a parameter to the deployment script (infra.sh).

  • Proxy information - If you use a proxy server, then make sure to bypass the IP address of the proxy server along with certain domains. For the list of domains, see Deploy Connectors. 

  • Gateway IP - Choose the nearest Gateway IP so that Connector establishes a secure tunnel with the private applications through Client Proxy. This is supplied as a parameter to the deployment script. Please refer to the product guide for list of Gateway URLs.

  • No Proxy - Make sure to add the CIDR range 10.0.0.0/8 in no_proxy to assign IPv4 address to microk8s pods when you are using <no_proxy>

    For information about deploying connectors, see Deploy Secure App Connectors.

Troubleshooting Secure App Connector

Log on to the connector host using SSH (Secure Shell) and execute the required commands to troubleshoot a connector.

Connector.png

The following table provides the list of commands you can execute:

Options Description
1 (About) 

Displays the Secure App Connector image version

About.png
2 (Status)

Displays the utilization and status of all the services running on the connector

Status.png
3 (Troubleshooting)

Help you troubleshoot the Secure App Connector related issues

Troubleshoot.png
4 (Manage)

Manage (start, restart, or stop) Secure App Connector

manage.png
5 (Run Diagnostics)

Performs the diagnostics test and displays its results

diagnostics.png
6 (Download Logs)

Download all logs and configuration details to a temp (/tmp/) folder on the host

log.png
0 (Exit)  Exit from the command console.

Download Logs

You can download all logs and configuration details to/tmp/connector_state.tar.gz 

  1. Log on to connector host using SSH. 
  2. Execute pa_connector.sh > enter 6.
    Downloads .tar.gz file to the temp (/tmp/) folder once the command is executed completely.

  3. You can upload the latest file connector_state_currentdate_time.tar.gz to the support portal for review. 

Resolving Host Names  

For a Connector to function, the system Domain Name Service (DNS) should be able to resolve both Skyhigh Security URLs and Private Application URLs. You can do the following to resolve host names using the system DNS:

Perform the following to check if the system DNS is resolving both Skyhigh Security and Private Application URLs:

  1. Log on to connector host using SSH (Secure Shell). 

  1. Execute pa_connector.sh >  enter 3 >  enter a > enter the domain name.

domain.png

Test the HTTP proxy set

Perform the following to test if a private application is reachable through a connector:

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 3 > enter b > enter the http proxy address.
    Displays the connection established message if the private application is reachable.

    http.png

Test the system proxy

Perform the following to test if a private application connection is established with the connector via system proxy:

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 3 > enter c > enter the system proxy address.
    Displays if the private application connection is established with the connector via system proxy.

    systemp.png

Test Ping

Perform the following to test ping:

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 3 > enter d.

ping.png

Run TCP dump for Private Application

Perform the following to check the private application connectivity:

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 3 > enter e.
    Downloads .pcap file to the temp (/tmp/) folder once the command is executed completely.

  2. You can upload the latest file pa_capture-currentdate.pcap to the support portal for review. 

tcpdump.PNG

Manage connectors

Perform the following to manage connectors:

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 4 > enter <g or h or i>.

Connectorcommands.png

Start connector services

Perform the following to start all connector services:

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 4 > enter h.

Restart connector services

Perform the following to restart all services that run in a connector:

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 4 > enter i.

Stop a connector

  1. Log on to connector host using SSH. 

  1. Execute pa_connector.sh > enter 4 > enter g.

Tips

  • Get the Private Access connector pod name using sudo kubectl get pods -n cwpp
  • To log into the pod, run sudo kubectl exec -ti <pod_name> -n cwpp -- bash
  • To install a connector again on the same VMs, then run the following commands:
    • sudo rm -rf /opt/McAfee/
    • snap remove microk8s 

  • Check the VPN (Virtual Private Network) tunnel: Tunnels are created between the connector and Private Access Gateway.

    • Run ifconfig to check if the tunnels tun0 and tun1 are created with an IP address 10.8.0.x. 

    • Check /.../mount/logs/postRegistration.log and /.../mount/logs/openvpn_client.log

  • Monitor Health Updates:  Check/.../mount/logs/healthUpdate.log to ensure that the VPN tunnel is up. The health update requires the list of private applications monitored by the connector. So, check if the private application list sync is successful and /.../private_apps.json is present.

  • Proxy to private applications: Check/.../mount/logs/debug.log to ensure that an application is reachable and not blocked by the policy. Ensure that the DNS configured on the connector can access private applications and resolve their URLs.

CIS Hardening Scan on Linux IPV4 RHCE 8 Server

Fixing vulnerabilities experienced when performing the CIS Hardening scan on the Linux IPv4 RHCE 8 server:

  1. Ensure IP forwarding is disabled (Refer Page 227 on the CIS Guide)
  • ​​​​​​To enable IP forwarding, run the following commands
    ​​​sysctl -w net.ipv4.ip_forward=1df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002 | xargs chmod o-w

  1. Ensure no world writable files exist (Refer page 618 on the CIS Guide)

  • To view writable file list, run the commanddf --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002
  • Run the df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type f -perm -0002 | xargs chmod o-w command to remove write permission on all files
  1. Ensure no ungrouped files or directories exist (Refer page 621 on the CIS Guide)
  • To view ungrouped files or directories, run
    df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -nogroup
  • To change group ownership to root on all of them, run the following command: df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -nogroup | xargs chgrp root
  • Run the view command to ensure there are no ungrouped files or directories.
  1.  Ensure sticky bit is set on all world-writable directories (Refer Page 607 on the CIS Guide)
  • To view sticky bit is set on all world-writable directories, run
    df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type d \( -perm -0002 -a ! -perm -1000 \) 2>/dev/null
  • To add sticky bit to all of the directories
    df --local -P | awk '{if (NR!=1) print $6}' | xargs -I '{}' find '{}' -xdev -type d \( -perm -0002 -a ! -perm -1000 \) 2>/dev/null | xargs -I '{}' chmod a+t '{}'
  • Run the view command to ensure there are no sticky bit is set on all world-writable directories.

Common Errors and Workaround

Task When  Error Workaround
Unable to access EPEL Package  Not installed the packages correctly

Package installation is failing

Local package repository should be disabled and enable public repository EPEL yum repo

Methods to check if repository is reachable:

  • Use the dnf or yum check-update command to connect with configured RHEL repositories and check for updates. Displays an error message if the repository is not reachable.
  • Use the dnf or yum repolist allcommand to check if corresponding repository is enabled (in status column).

Run the following commands when package installation fails:

  • Check error message provided by package manager yum or dnf.
  • Useyum deplist <package-name> command to check any missing package dependency.
  • Use ping or curl to test connectivity to repo server.
  • Verify repo configuration file in the following directory path: /etc/yum.repo.d/<repo-configuration-file> to ensure repository URLs are correct and active.
  • Use theyum or dnf clean all command to clear package cache. 
  • Use the df -h command to check disk space.
  • Use therpm --checksig command to check package integrity by verifiying digital signature of package.

Run the following commands to manually download and install packages:

  1. wget <package-url> or curl -O <package-url>
  2. yum or dnf install <path-to-package-rpm file>
  3. rpm -i <rpm-file>, dpkg -i<rpm-file>
  4. verify installation rpm -q <package-name> or dpkg -s <package-name>

Install subscription packages:

subscription-manager repos --disable=<local repo name>
subscription-manager repos --enable=epel
sudo yum repolist all
sudo subscription-manager repos --enable=rhel-8-server-optional-rpms
subscription-manager repos --enable=rhel-8-baseos-rhui-rpms
subscription-manager repos --enable=rhui-client-config-server-8

Ensure successful installation of packages by executing the below commands:

  • Check the directory for package listing: /var/cache/yum/package-name or /var/cache/dnf/package-name. To check and download logs from this directory: /var/log/yum.log ro dnf.log
  • Use theyum info package-name command to check package size.
  • Use theyum list installed package-name command to check the list of installed packages.
  • Use the yum list package-name command to know the package download status.
Connector Registration

 		 Entered incorrect provision key Incorrect provision key

Error decoding the provisioning key

Check the/.../mount/logs/postRegistration.log for incorrect provision key and copy the correct provision key from Skyhigh Security UI and provide it toinfra.sh while deploying a connector. 
Expired/Invalid client credentials Exception while refreshing access token  Download the new connector package from Skyhigh Security UI and ensure that right provisioning key is used. 
Connection to registration endpoint  Connection to registration endpoint - failed 
  • Check whether Skyhigh Cloud API reachable
  • Check the existence of tun0 and tun1 
Firewall is used Registration Failed

Add <Proxy IP>: <proxy port> and <PA Gateway IP>: 443 to Azure network rules.

The proxy port can be 8080, 8081, 443
   

Disable the firewall

systemctl status firewalld.service

sudo systemctl stop firewalld

PA Connector is not coming up
  • Incorrect provisioning key. When you copy and paste the provisioning key, additional special characters gets added to the provisional key.
  • Insufficient memory in the Host – free up memory of the host
  
  • Check if Host (Ubuntu/RHEL) is configured with more than 3 DNS servers
  • Check the provisioning key.
Unable to login to Connector Pod Unable to login to connector pod Server error Update /etc/hosts of the VM. 

 

DNS resolution failed for PA Gateway domain 

 Error while resolving PA Gateway domain
PoP Manager pod is not reaching completed state    

Disable the firewall

systemctl status firewalld.service

sudo systemctl stop firewalld
PoP is not showing healthy on the PoP Management page     

Create pop-manager pod again
sudo kubectl apply -f Pop-manager.yaml
Bringing up the VPN tunnel  GW or HTTP_PROXY are not reachable
  • Error while resolving hostname
  • Exception while checking Proxy
  • Exception while checking PA Gateway
  • Unable to connect to Proxy
  • Unable to connect to PA Gateway
 Connector maintains an outbound VPN tunnel with PA Gateway.
  • Run ifconfig to check tun adapter and the IP address assigned by the Private Access Gateway.
  • Check whether the gateway port 443 provided to infra.sh is reachable from the connector. 
Private Application list sync   Check the /.../mount/logs/paListsync.log if Skyhigh cloud API is not reachable. If this is not reachable, then ensure it is reachable to synchronize the private application list with the connector. 
Private Application is not reachable    

Run the command on the connector pod:

curl -v -k https://app-server

If curl also fails, then ensure that the IP of the connector pod and host IP are not on the same subnet (10.254.254.x)
Accessing Private Application Able to access Private Application using Firefox browser, but failed to access the same application on the Chrome browser Google Chrome - Taking Too Long to Load

Disable the Use secure DNS option on Google Chrome. 

The Use secure DNS option should be disabled to access the private applications. Go to Chrome browser > Settings > Privacy and Security > Disable the Use Secure DNS option.
Installing packages on closed environment    
  1. curl
  2. python3
  3. snapd

sample of a snap output:

[ec2-user@ip-172-22-21-231 ~]$ snap version
snap    2.57.6-2.el8
snapd   2.57.6-2.el8
series  16
rhel    8.7
kernel  4.18.0-425.13.1.el8_7.x86_64

  1. microk8s 1.24 version

(you can install microk8s using snap

sudo snap install --channel=1.24/stable microk8s --classic)[ec2-user@ip-172-22-21-231 ~]$ sudo microk8s statusmicrok8s is running
high-availability: no
datastore master nodes: 127.0.0.1:19001
datastore standby nodes: none

addons:

enabled:
dns                  # (core) CoreDNS
ha-cluster        # (core) Configure high availability on the                                   current node

disabled:

community          # (core) The community addons repository
dashboard          # (core) The Kubernetes dashboard
gpu                     # (core) Automatic enablement of Nvidia                                   CUDA
helm                    # (core) Helm 2 - the package manager for                              Kubernetes
helm3                  # (core) Helm 3 - Kubernetes package                                      manager
host-access          # (core) Allow Pods connecting to Host                                     services smoothly
hostpath-storage  # (core) Storage class; allocates storage                                   from host directory
ingress                 # (core) Ingress controller for external                                       access
mayastor              # (core) OpenEBS MayaStor
metallb                 # (core) Loadbalancer for your Kubernetes                               cluster
metrics-server      # (core) K8s Metrics Server for API access                               to service metrics
prometheus          # (core) Prometheus operator for                                                monitoring and logging
rbac                      # (core) Role-Based Access Control for                                      authorisation
registry                 # (core) Private image registry exposed on                                localhost:32000
storage                # (core) Alias to hostpath-storage add-on,                                deprecated

  1. nfs-utils
  2. epel-release (based on RHEL 8/9)

 

  1. Back to top
  • Was this article helpful?

Recommended articles

  1. Article type
    Topic
  2. Tags
    1. ZTNA