Three months in Palo Alto - Part 3 (GPCS)

OK, I know I have missed part two, but I will be coming back to that sooner or later. But I thought I would put some stuff down on the blog while it is fresh in my memory. In part one, I started looking at Palo Alto NG firewalls in AWS as a new VPN solution. The test went well and was favorably received by all concerned. However, a single instance firewall offers no resiliency, and there are also regional issues. Now, this can be addressed by adding more gateways (VM instances in different AWS regions), but then this is more to manage. So, when discussing this with our VAR, they mentioned the Palo Alto GlobalProtect Cloud Service (GPCS), which sounded like it would fit the bill, so we decided to give it a go.

Palo Alto GlobalProtect Cloud Service (GPCS) Setup.

There are four components to GPCS, the logging service, infrastructure (service) subnet, mobile users, and remote networks. You also need Panorama to manage your devices and push out policies.

Panorama

To get started with GPCS you need to install Panorama, which does all the management. When you have installed Panorama, you need to edit the server to set the serial number to be the same one you get from Palo Alto, upload and install the Cloud service plugin and then generate and apply a one-time password (OTP). You need to be running a fairly recent version of Panorama. I started with 8.0.1 and hit some issues.

Before I set the serial number, I got this error when trying to apply the OTP: “Failed to verify account. Generic communication error occured. Please enable debugging for detailed info [get-panorama-cert.py:164] (“global name ‘fetch_cert’ is not defined”,).”

You can edit the serial number by going to Panorama > Setup > General Settings

Once I had set the serial number, I got this error: “Failed to verify account. Generic communication error occured. Please enable debugging for detailed info [get-panorama-cert.py:183] (2, ‘No such file or directory’)”.

After I upgraded Panorama to 8.0.6 all went well and you can start to pull in your licenses to activate the services.

Logging Service

The logging service does not take much configuration, just pick a location and that’s that. Let’s move on.

Service Setup

The service network controls access from Panorama to our cloud devices, and this cannot share the same network as a remote network. This is very important to note. So, what does this mean? Well, services such as Panorama live in this service subnet, which is connected to the GPCS via a site to site VPN. If you need anything like RADIUS or AD/LDAP, then these also need to live in this subnet. Sounds OK, right?

Well, this is one issue I have with it – basically, it is no different (in principle) to how the services setup works in the single instance we ran in part one, but less flexible. We still have a services network and a data network, but with the single instance, we could send particular traffic down the data network if we wanted to. This is not possible with the GPCS. So where we had a pretty flat network, with a large subnet to encompass AD, varying servers, and users, we had to carve out a new subnet to host these services (Panorama, an AD box (which we had to build from scratch as having a multi-homed server did not work as the default gateway was in the wrong subnet) and the RADIUS server (for DUO.com)), and (thankfully) we have second internet pipe for the infrastructure service VPN.

If you don’t have a second ISP, then I guess you are not completely out of luck as the VPN endpoint for the Infrastructure service component of GPCS is different to the Remote Networks one, so there won’t be any confusion in trying to send data for two different networks to the same endpoint – you just need to make sure that the IPSec and IKE details are named properly in Panorama as to not make it confusing for yourself. Nevertheless, it is a bit annoying having to start carving up your network for it to fit with the GPCS.

Once this is set up, you should get some nice results on the status page:

I only got the green light with version 1.0.2 of the cloud plugin. With version 1.0.0 and 1.0.1 I got a message saying that the IPSec tunnel state could not be determined. The tunnel still worked though regardless. This meant that I could push policies from Panorama and use AD and Duo for authentication. I just need some user to send some traffic.

Mobile Users

For the mobile users, you need to configure a pretty big subnet – it does not matter if you have two users, two hundred users or two thousand, you must specify /16 subnet for this part of the service. Seems a bit wasteful, but there you are.

Once you have this setup, which is pretty easy in Panorama, the benefits of the service can really be seen.

If we say that the portal is on a device in Europe, but the person making the VPN connection is based in San Francisco, then the portal answers the VPN connection attempt, but bounces the user over to their nearest “hub” – in this example, this would be North California. On the status map you should get something a bit like this:

This is one area that really shines with the GPCS, you get the speeds through the Amazon network, and all the cool stuff that comes with geolocation (like getting Amazon.com instead of Amazon.co.uk for example). This is one of the major factors for us, giving our mobile users the speeds they need, whilst keeping control of their VPN traffic, no more hauling all their traffic to the UK, only for it to go back over to the US, or APAC for example. The policies that are pushed out are local to the devices, and just so long as AD queries are kept to a minimum, there should be little lag.

By default, the address given for the portal and VPN connections is a yourchoice.gpcloudservice.com address (you get to choose the “yourchoice” bit). You can customize it though and use your own domain, you’ll need to create a CNAME back to the original address and generate an SSL/TLS certificate for it – and import it, but then you can give your users a better experience. It does take a little bit of time for the changes to get pushed out fully – normally policy changes take a few minutes to get deployed to all the GPCS endpoints, but the cert and name changed seemed to take quite a bit longer. So a bit of patience is advised.

But what if your users need to connect back to the offices? What happens to the traffic then?

Remote Networks

In the same way that the Service subnet is set up in a particular region (the one closest to your services), the remote networks are as well. You create a new remote network and you specify a region so that the GPCS can provision the VPN on a device closest to that particular endpoint (therefore benefitting from Amazon’s network speed when going from one remote network to another if both are connected to the GPCS. Again, if you are working with and endpoint far away, then be patient before trying the VPN as it takes time for the changes to propagate.

With the remote networks, you buy an allocation of bandwidth, up to 300Mbps and can carve this up as you see fit, giving a remote network anything between 10Mbps and 300Mbps. However, once you exhaust this, you cannot connect any more remote networks:

GPCS General thoughts

I have grown to really like working with Palo Alto devices. It does take a little bit of time to get used to making the switch from Cisco ASAs to PAN-OS, but if you have worked with Checkpoint it is very similar. Changes to devices and policies must first be committed to Panorama:

Then pushed to the devices:

This is a lot different to working with ASAs, where the apply button has an immediate effect. You can preview the changes as well, which is good. But this is not a post on Panorama, this is about the GPCS.

The benefits of the GPCS is that you get the ease of cloud setup with the benefit of local geographic presence and the speed. Now there is no need to spend out on Co-lo and expensive leased lines to give users the benefit of local speed. For example, we do not need to purchase a dedicated VPN appliance for people in Sydney, or (and much worse) have them connect to a VPN appliance in the UK. This is all taken care of with the GPCS.

Did I mention the speed benefits? Remember, this is on Amazon Web Services, so you get their network speed (or a small proportion of at least).

The tests I have been running have been met very favorably with my users, all over the world, and I look forward to moving this to production in the near future.

Caveats

The service is still in its early days, and with that comes a few caveats.

I would recommend being careful with your policies and logging. As part of PCI compliance, it is recommended to make an explicit deny everything rule right at the end of your policies. This is useful as you can log the rule hits and see what gets stopped. No, with the GPCS, having this rule can stop your ability to connect using the GlobalProtect client, and if you log against this rule, then you get to see a lot of very weird traffic (lots of random torrent traffic when I did it).

Similarly, do not do a permit all / all from untrusted to untrusted with logging for exactly the same reason. It makes it really hard to see your own traffic. The Palo Alto take was that this was just bot traffic looking to use anything it can to hook onto to move traffic around. It’s in the GPCS set up guide, so take heed.

The other issue I found is that if the Palo Alto network’s website is down then you cannot log tickets (obviously) but also pulling logs into Panorama is difficult (because they come via the logging service in the cloud) and you cannot push changes to the devices. I found this out after they made a bit of a boo-boo on their website: