Troubleshooting CWPP Issues

Cross Account use of single PoP 

• Agent and PoP VPC are in different accounts and the same region. The AWS administrator must allow listing the account before creating an endpoint by using helperscript. 
• Allow listing the account is a one-time process that the admin can perform. 

Cross-Region use of single PoP 

• Agent and PoP VPC are in different regions and have the same or different accounts.  
• The creation of Endpoints for CWPP services is supported within the same Region only. It is not supported to create an endpoint between a VPC and a service in a different Region. 

Locating Error Logs 

Error details are available in the install log located at /opt/.../cwpp/pop/install.log, which gets created as soon as installation initiates. 

• Check /var/log/cloud-init-output.log for PoP installation logs, which are initialized by the cloud. 
• Check CWPP service logs in the EFS folder (/opt/.../cwpp/pop/PoPDeployment/PoPCreation/aws/EFS/)  

Possible errors during prerequisite CFT Deployment

For prerequisite CFT: 

• The maximum number of VPCs has been reached. (Service: AmazonEC2; Status Code: 400; Error Code: VpcLimitExceeded) 
• When the VPC limit exceeded in the account 

Possible errors during PoP CFT Deployment

• Failed to configure auto-scaling group. Refer to the install log for more details. 
• Check if the auto-scaling group name is empty or already taken. For example, popname. 
• Failed to get registration token. Refer to the install log for more details. 
• Check if Client Credentials for IAM Registration Token are null. 
• Failed to configure EFS. Refer to the install log for more details. 
• Check if EFS ID is in an available state or validate if EFS Mounting is done properly. 
• Failed to configure microk8s installation. Refer to the install log for more details. 
• Failed to start the installation. Refer to the install log for more details. 
• Check if all the parameters are sent properly in the CFT. 
• Error message PoP CFT installation failed due to an unknown reason.
  • This error can happen when one uploads the "full" package named "PoPPackage.tar" as downloaded from UI to the file location (e.g. S3 bucket), instead of uploading only the tar file "PoPDeployment.tar" which needs to be extracted from the full "PoPPackage.tar" before. Make sure you extract the file "PoPDeployment.tar" from the downloaded PoP Installation package first.
  • For more details, refer to the instance system / cloud-init log.

PoP failed to send health data to Skyhigh CASB

The following are possible reasons when PoP failed to send health data: 

• Security Group rules are not updated.
• Verify the Secure Gateway rules associated with the PoP against these and change accordingly. 
• Security Group is not tagged with the Kubernetes tag.
• Add kubernetes.io/cluster/{PoPName} tag to security group. 
• Exceeded the rules configured in the SG as it has already exhausted the allowed rules limit. 
• Verify with the Secure Gateway rules against these and change accordingly. (Need to attach finalized Secure Gateway rules list.) 
• Provision of Load balancer failed.
• Check the quota limits of available load balancers. 
• Provisioning of the VPC endpoint service failed.  
• Verify if load balancers are created properly. 
• Subnets are not properly tagged with the Kubernetes tag 
• Add kubernetes.io/cluster/{PoPName} tag to subnet. 

PoP reporting as unhealthy

Heartbeat payload of PoP infrastructure and services health details are sent to display the latest status for PoP in Skyhigh CASB. PoP sends unhealthy status due to the following reasons: 

• CWPP PoP services are not up and running. 
• The primary instance is not running or not up. 
• Troubleshooting steps to make PoP healthy are provided. 
  

• Check PoP manager payload 

PoP Secondary node not getting added

• If all the 10 registration tokens are used and on the 11th time when we ask to create a new secondary node, it will not be added. 
• Manually run the following commands to generate a fresh token from the primary and add it to the secondary instance. 
  1. Log in to the PoP Master Node.
  2. Run the command:

sudo microk8s add-node

For example, the sample output is shown below:

Join node with: microk8s join 10.0.2.62:25000/iRyjXABuauMPtLEOrZfDuROeXRrYCTqB
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 10.0.2.62:25000/iRyjXABuauMPtLEOrZfDuROeXRrYCTqB
microk8s join 10.1.62.0:25000/iRyjXABuauMPtLEOrZfDuROeXRrYCTqB

3. Log in to the newly created slave node. To add this node successfully, run the above join command. For example, microk8s join 10.0.2.62:25000/iRyjXABuauMPtLEOrZfDuROeXRrYCTqB
4. Wait for 5 minutes and go to the Master node and run the command to check the slave status. The result shows the list of Slave Ips and their status as Ready or Not Ready.

sudo kubectl get nodes -n cwpp

For example, the sample result is shown below:

NAME          STATUS   ROLES    AGE   VERSION
10.0.2.63     Ready    <none>   5d    v1.18.20
gcpdemopop    Ready    <none>   5d    v1.18.20