I have a similar experience with uploading via FTP to non Github Pages.
When you rebuild
_site directory after creating one post, then generally you only need to upload the post, the homepage (assuming that’s where posts are listed) and the site map.
But what if your site has a Blog Posts page other than the homepage or you have another page which you edit and add to link to your new post. In those cases you’d have more to update.
If speed is important to you, then use an upload method like I have in my FTP application which will only add and update files and won’t try and upload content which is already on the server and has no diff.
You can also use Github Actions to publish via FTP and secret values to your target site so you can let Github take care of the rebuild.