Disallow /files/ paths in robots.txt (railsdoc#203)

* Initial plan

* Add Disallow rule for /files/ path in robots.txt

Co-authored-by: toshimaru <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: toshimaru <[email protected]>

Author: Copilot

src/robots.txt

@@ -2,4 +2,4 @@
 
 # Allow crawling of all content
 User-agent: *
-Disallow:
+Disallow: /files/