Or is this the intended behavior of QGIS upon export to GeoJSON and I'm thinking about this wrong?
I don't know what the intended behaviour of QGIS is, but if the intention is to write GeoJSON according its specification (https://www.rfc-editor.org/rfc/rfc7946) or just RFC7946 for short. Then yes.
Section 4 of RFC7946 tells us that
The coordinate reference system for all GeoJSON coordinates is a
geographic coordinate reference system, using the World Geodetic
System 1984 (WGS 84) [WGS84] datum, with longitude and latitude units
of decimal degrees. This is equivalent to the coordinate reference
system identified by the Open Geospatial Consortium (OGC) URN
Note the datum
World Geodetic System 1984 (alias WGS 84) is defined by EPSG::6326
urn:ogc:def:crs:OGC::CRS84 is NOT the same as
urn:ogc:def:crs:EPSG::4326 They share the same datum, but the axis order is different, OGC::CRS84 is long/lat and EPSG::4326 is lat/long.
There is no 'crs' member in GeoJSON (GeoJSON is RFC7946), because the CRS is ALWAYS urn:ogc:def:crs:OGC::CRS84 or at least (for pedants) the 2D CRS is always urn:ogc:def:crs:OGC::CRS84.
You can by 'arrangement' (say in an internal application) use a different CRS, but if you're exchanging GeoJSON or publishing GeoJSON, the expectation will be that it's urn:ogc:def:crs:OGC::CRS84
GeoJSONLint, Right Hand Rule GeoJSON Fixer, etc. I keep getting the "Line 1: old-style crs member is not recommended, this object is equivalent to the default and should be removed" error.
So the warning is correct here if you have a crs member and the crs is stated as being OGC::CRS84.
Prior to the specification, in the draft (aka GeoJSON 2008), there was a crs member and other CRS were allowed.
Contrary to the advice of @user30184 you don't want to use the option
RFC7946=YES if you want a crs member and EPSG:4326 as the CRS; you want
RFC7946=NO, though it should be the default for GDAL/OGR.
RFC7946=YES/NO. (OGR >= 2.2) Whether to use RFC 7946 standard. Otherwise GeoJSON 2008 initial version will be used. Default is NO (thus GeoJSON 2008).