Contexts staying alive through navigation to content item where it is forbidden

I have a strange situation that I am not sure about if it is an Umbraco bug or something in my code. I have context X which is loaded on page A. This context observes the UMB_DOCUMENT_WORKSPACE_CONTEXT. I then navigate to page B which has a condition which prevents context X from loading, but the observe still triggers in that context resulting in my code loading some things in which it shouldn’t do.

Context in question is this one:

And how I register it: