You have a misconception of what a sitemap is.
The sitemap is used to audit the crawling of the site by the bot search engine. The site plan and the exploration of a site are two different and independent things. Google will continue to explore your site regardless of any site map. The sitemap will be used to check / see if Google is able to correctly analyze your site. For example, if pages are in your sitemap and Google has not seen the page, Google can add the page to the recovery queue to include.
The opposite is not true. If a page is not in the sitemap, Google will not remove it from its index. Why? Because Google found it while exploring the site.
What you seem to believe is that the sitemap is the only authority used by Google to know which pages exist on a given site. This is not the case. The crawl is. The site map only allows Google to know if they can properly analyze your site and, if not, the missing pages in Google that need to be added to the recovery queue.
Your Google expectations no longer try to access pages because these pages no longer appear in your sitemap are incorrect. Sitemaps are cached and checked only periodically. Why? Because it is an audit process.
You have a real problem to solve.
You return a 500 error for pages not found. It's bad. Your website should return a 404 Not Found error. The error 500 is a system error and Google will treat the condition as temporary. If your site returned a 404 error, Google will continue to try the page several times over a period of time until it decides that the page does not exist anymore. If possible, you want to issue a Deleted 410 error for the pages you have deleted. If this is too much work or it is not possible, the 404 will be the same over time.
You must correct your 500 error.