

do they just want everything to be crawled
Yes. Web crawling has been a normal and vital part of the web from day 1. We’d have no search engines without crawlers.
The web is user-centric by design. I’m sick of tech companies trying to flip the script and hoard information, most of which is not theirs to begin with (e.g. Google, Reddit, Twitter, Facebook, etc.).
Better yet, use borg to back up. Managing your own tars is a burden. Borg does duduplication, encryption, compression, and incrementals. It’s as easy to use as rsync but it’s a proper backup tool, rather than a syncing tool.
Not the only option, but it’s open source, and a lot of hosts support it directly. Also works great for local backups to external media. Check out Vorta if you want a GUI.