This reminded me of a page I made about six months ago, so I checked the links and the source templates. Sometimes I make comments in the source about unexpected kludges but this time I didn't unfortunately. However I do remember the link URLs weren't what I expected (or preferred really).
So if you look at the paths, they show the full path in /user including Grav folder prefixes. (Note that a new version of this page not yet launched moves these files out of modular folders, but they still show /user/... and prefixes.)
I don't think you specify whether you are coding your link via page markdown or in a template. I used a template pulling filenames from frontmatter. For what it's worth, here is a simpification of the template that serves these URLs (current modular page version). It may be useful:
{% if page.header.abstracts is defined %}
{% if page.header.abstracts.file is defined -%}
<a href="{{ page.media[page.header.abstracts.file].url }}" class="image clear" type="application/pdf">
{%- endif -%}
....
{% endif %}
So I'm letting Grav derive the URL here.
I also thought of the possibility that multilingual paths are screwing up your media URLs, but simply removing the language path did not help. (My limited experience with multilingual pages makes me think paths can be buggy.)
Hope some of that is helpful.
