Submitting this as an issue for now, but I may have contributions for you... #6
Description
I hacked apart your code for hours to purpose-build a script for a project, & it actually ended up being possibly useful for other people for many other applications. I simply wanted to scrape an entire GitBook into markdown very quickly & easily to keep it up to date as training data. But it does a bang up job of cleaning up the text for many different things. I sicked it on developers.android.com & it does quite well. Another couple docs pages, & it is good... It's very much driven towards my own goals, but if anybody is interested, it is currently nested inside of another project here: https://github.com/symmetricalboy/myshell-assistant/tree/main/md-tool
I may clean it up & generalize it & fork you eventually, but not at the moment.
One bit of feedback after living in your code for so long: threading sucks. You should use async. But I didn't actually implement async myself yet, so...
Also, I am using turndown.js to clean the text now, which is much better.
Cheers!