IdP memory (heap size) issues

The size of the UK federation metadata aggregate has grown sufficiently large that scaling issues are appearing for some metadata consumers. IdP operators have found that Java maximum heap size settings which were suitable in the past are now insufficient to process the current size of aggregate. The number of federated entities continues to increase, both from new entities in the UK federation and also the increase of entities imported through eduGAIN, so this issue will not go away.

Furthermore, out of memory errors are hard to diagnose. They may manifest themselves as follows:

• the IdP cannot read the metadata file properly, and as a result fails to authenticate SPs. Restarting the IdP makes the problem go away
• the IdP crashes or hangs

Recommendations for Java heap size

If you are running a Shibboleth v3 IdP, please see the Installation pages for your particular deployment to determine how to set an appropriate Java heap size, and what that size should be.

There is a page on the Shibboleth wiki about Shibboleth IdP Heap Management which may also provide some insight for operations.

Other mitigation

Rather than alter the Java heap size of the IdP, you might consider using the MDQ (Metadata Query) service that is being rolled-out by the UK federation. In this mode of operation, the IdP does not download and process the whole of the UK federation metadata aggregate. Instead, it downloads metadata as-and-when it requires it. Please see our MDQ documentation page for information.