What happens if a miner or pool decides to turn to this system, generating those shares and then selling to themselves, how will that affect the rest of block generation?
It is not that easy, because you cannot easily produce Proof of Work out of thin air, you have to spend some "work" on that. And if something is committed to the block, then if you want to change your commitment, you have to re-mine it.
Because a single entity like Foundry could almost clog the network with just a few days of work and lost fees (which are right now 1% of the whole reward)
It is not that easy to clog the network if you have a Proof of Work, because it can be used to filter spam. For example, each node can adjust required work for commitments, based on how many of them are broadcasted: if there are too strong, then one commitment can cover N transactions. If they are too weak, then N commitments can cover a single transaction. If something is gameable, then it can require re-mining, then people couldn't easily change their minds after spending work on mining some commitment.
The fee also acts as a way to prevent spam, by incentivizing miners to include zero fees tx you're just turning the blockchain more vulnerable to spam where lower fees would take priority space
It is not a scheme, where lower fees are always included. They are conditionally included, if you can provide some Proof of Work for that. So, the minimal fee stays the same, it is just measured not only in satoshis, but rather in "satoshis+work" combination. Another thing is that it depends on what nodes accept, so they can decide, how much discount they want to provide, and how much work they are willing to validate from smaller miners. And in case where something goes wrong, it does not affect the whole network, because it is only relay-rule, and block-inclusion-rule, so no forks are needed, and the old system can be restored at any time if needed (and different nodes can try different settings, to find the right balance between requiring N satoshis, and supporting small miners). Also, older commitments can be later removed, because if a free transaction is included, then those commitments are no longer needed, as long as those transactions are deep enough to not be reversed.
secondary I don't see how this would work with thousands of small miners getting their "shares" of each block each day, without further more affecting the limited space
It is only an option. And it is only a discount for a miner's transaction. If it gets abused, then nodes could require more work per satoshi, or turn off that feature entirely. Another thing is that with proper sighashes, it is possible to create a negative fee transaction, if getting a higher reward in one shot is needed.
How would you do this as this would need to store each miner's contribution from the previous block to the next one and that is not happening right now?
Not "each miner's contribution", but only for those willing to use that system. Also, it can be turned on and off at any time, because it depends on each node, exactly in the same way as minimal fees depends on nodes. So, if you have 700k blocks, you don't need the whole history of all mined shares in the whole system. You need only a short, moving, on-the-fly calculated window of shares. You don't have to store the rest, because each miner can provide a proof when willing to get a discount. So, if you have a commitment for the block 700,000, then you are the only one who needs to backup that commitment, and then you reveal it later to some node. But as the whole chain is moving forward, your discount is getting less and less significant in the whole chainwork, so you are incentivized to share it early, then it will get combined with other commitments.
There is LN, and anyhow, home mining for BTC will probably be completely dead in a few years, right now is barely breathing.
Yes, we have LN for such things, this is just one more step, to cover for example discounts in channel opening and closing transactions. Because if you are a really small miner, then you probably don't have enough power to use that on-chain, for small miners there are millisatoshis (or maybe smaller units in the future) inside LN channels.