Thanks for the input Aaron!
I didn’t implement it as an external login provider, so my IDs shouldn’t end up in the database at any point.
I ended up contacting Umbraco Support and they told me that PR #18320 most likely would solve my issue. After upgrading to Umbraco 13.8.0 my issue was resolved.