A couple of things I want to mention:
1) Right now about 1000 pages is the upper limit of what Grav will handle. Of course faster hardware (processor/ssd/memory) and also software (webserver/caching/etc) can make a big difference too. Grav can be optimized to not check for page changes, and this becomes essential on large sites. Also there currently is a static-cache plugin that effectively turns grav into a staticly served site, and makes it fast. We do have plans to revisit the 'scalability' and performance of Grav for large sites in the coming year.
2) Every 'folder' underneath pages/ is considered a page in Grav. Even if it has no markdown file in it. This feature allows you to have a user/images/ folder for example, and put images in there, and still make use of Grav's media functions by accessing the images page, and then in turn the images.
3) You could create a folder underneath each page for each media type. This would mean you would have to find the page first, then access its media, as the media would not be available directly in the main pag e sitting above the media. However, this is going to increase your overall page-count considerably, as each page could potentially have 4+ media child pages.
4) Grav already groups media into distinct types (images, videos, files). And it automatically does that for all media found in the current folder for the page. For Grav to work optimally, you would be better off putting all the media in the page folder, and use a consistent naming structure to ensure they were manageable. eg image-foo.jpg, image-bar.jpg, video-foo.avi, video-bar.avi, that kind of thing...