It's been two years since I have been designing my posts with Elementor and Now I feel like I should have continued with Custom made HTML CSS pages or any other simple page builders. Sometimes it seems like Elementor is heavy plugin and whenever caching is set to low level, it seems to crash often.
No, Elementor does not slow down your website. But Elementor adds external CSS files, Javascript Libraries, and above all the Page Builder adds multiple Divs alongside which may slow down the rendering processes. All these are negligible only if you aren't using multiple Elementor Addon plugins which use excessive amounts of Javascript, making your Page more functional as well as heavy.
So make sure you use the Elementor Page builder carefully, try not to install more and more addon plugins, keep designing simple, Optimize images, Avoid animations and you'll be good to go.