is there a way to speed up building the site, after it has generated the file this part takes long time
Running php -d memory_limit=-1 bin/console app:convert-local-secrets dev --no-interaction
Note that there is no need to enable Encrypt Secrets during development, you only need to enable it and do it once when you generate the production version before deploying to your production site.