If you are getting this error message read this article first BEFORE you try to rebooting your firewall.
I ran into this problem recently with a Palo Alto PA-200 series firewall. I tried to login via the WebUI and would get the error “Session timed out”. I could SSH into the firewall and the internet was working. I’ve had this problem before and a quick reboot of my PA-200 solved the problem. Not so this time.
This time when I rebooted the firewall it did not come back up. Well not fully at least. I could PING and SSH it and load the WebUI (still getting the “Session timed out” error when trying to login) but all network traffic stopped flowing. This was bad.
I did some searching online and found that the issue can occur if you run low on disk space on your Palo Alto.
I logged into my PA-200 via SSH and ran the following:
myuser@PA200-MySite> show system disk-space Filesystem Size Used Avail Use% Mounted on /dev/sda3 1.9G 1.9G 0M 100% / /dev/sda5 6.6G 1.3G 5.1G 20% /opt/pancfg /dev/sda6 1.9G 752M 1.1G 42% /opt/panrepo tmpfs 1.3G 67M 1.2G 6% /dev/shm /dev/sda8 2.4G 1.4G 874M 62% /opt/panlogs
You can see the problem. The root (/) volume of the PA-200 was out of disk space.
I reviewed the following Palo Alto KB’s:
Neither of them helped. They didn’t clear up any disk space on the root volume.
To fix the problem I had to call Palo Alto support. They generated a support key which allows them and only them access to the root file system on a Palo Alto device. Form there they cleared out a few logs in /var/log that were eating up all of the disk space. The main problem was /var/log/secure. If you’re familiar with Linux systems you’ll know that log gains entries for all successful and failed login attempts. In the Palo Alto’s case it was all of the successful and failed logins via SSH.
Once Palo Alto cleared out those logs files we gave our PA-200 another reboot and it came back up as per normal.
Then came the part where we wanted to prevent this from happening again. I knew for a fact we had never created a rule that allowed access to the PA-200’s SSH service and there was no way someone internally was hammering the PA-200 trying to break into it. Fortunately I had a really good support rep on the phone that knew exactly where to look. Management Profiles.
If you login to your Palo Alto via the WebUI and go to ‘Network’ and ‘Interfaces’ you’ll see a column labelled ‘Management Profile’. In our case we had a management profile assigned to our public interface that allowed for SSH. This is how the internet in general was accessing our PA-200’s SSH service. That’s not the best part though. The best part is traffic that is allowed via a Management Profile isn’t logged so you can’t even tell this is happening by looking at your traffic logs. Awesome right?
We changed our management profile to only allow ICMP (pings) and called it a day.
I’ve been told Palo Alto is aware this is an issue but it only really affects the PA-200 since it has the smallest hard drive. Palo Alto isn’t making it a priority to fix it by implementing something as simple as logrotate or even truncating the log after 50mb is written to it.
If you have this exact problem I really hope you have you have an active Palo Alto support contact. If you don’t you’re screwed. Palo Alto is the only one who can access the root file system.
I’m hoping they will eventually fix this problem in future PanOS releases.
Thx for this blogpost, I had the same issue.
I tried enter webui this weekend, and got the same error as you. I thought I would wait until monday to call PA.
When I was about to call PA support on monday I found that some logs might have been deleted themselves as the root system was only 99% full now.
So if you dont have an active support with PA (you should btw) It might be a solution to wait a few days like I did.
Might just be luck though :)
I had the same issue but on a PA-3020. Exact the same solution. Thanks!!
Had the same issue on PA-3020 running PAN OS 6.1.10. There was a messages log that inflated drastically with login attempts on this partition – which is a management plane. Once this was cleared and the partition was not utilized 100%, access to the web UI was regained.
Same as the author, I limited protocol access on the management profile. PAN support did indicate that this bug is rectified in PAN OS 7.
Ha. Only took Palo Alto 2 major revisions to fix this problem.
Hi! Thanks for the article!
In my case restarting management service without rebooting the dataplane was the solution.
The command for PanOS v5 & v6:
Glad it was helpful! Thanks for the tip.
Hey,
many thanks for article – helps a lot.
I had this same issue – 100% volume used.
In my case I removed log files through CLI and it helped reduce volume to 90% and I was able login to WebUI
Source: https://live.paloaltonetworks.com/t5/Management-Articles/How-and-When-to-Clear-Disk-Space-on-the-Palo-Alto-Networks/ta-p/55736
Step #3:
“Delete rotated files and files with extention .old as follows. These files contain monitoring details and service related logs on the firewall. Hence they can be deleted safely if you don’t need them. If TAC investigates an ongoing issue, you may prefer to keep them until you upload the tech support file to the case manager. ”