I currently have Armory 0.96.4 and Bitcoin Core 0.18.0.
Everything is fine and smooth.
1. If I upgrade to Bitcoin Core 0.21.0, will Armory 0.96.4 continue working fine? (Both online and airgapped)
I'm kinda tempted to say "If it ain't broke, don't fix it"... If everything is currently "fine and smooth"... then you might be better off just leaving it "as is"... there haven't been any "breaking changes" to Bitcoin Core... so 0.18 will quite happily continue to work with the network (as things currently stand).
2. (Not Armory question) Are there any reasons to even upgrade to Bitcoin Core 0.21.0? What are the benefits? I'm just a HODLer and I make just a few transactions every year.
You'd need to read the release notes of the versions after 0.18.0 up to 0.21.0 and see what changes have been made and what vulnerabilities have been addressed and assess whether or not they offer any benefit to you.
There have been a number of changes and additions... but that doesn't necessarily mean "better" for your particular use case.