It is a LONG time since I used ArcScene and Exported to VRML. However, I used to do something similar a lot. The VRML exported from ArcScene is verbose and not always well-formed so I didn't like it and therefore do not recommend that you try to export your whole model in one go. It also didn't used to properly instantiate repeated objects - making the VRML rather inefficient and difficult to edit later. However, it would export the terrain reasonably efficiently.
My approach was to export just the terrain and any extruded polygons (building footprints etc). All objects represented by points I would create as a VRML object in a modelling package and then inline the object to the VRML file and add instantiate it at the point locations.
You need to adjust your coordinates unless you opt to use real-world coordinates. I used to use Excel to quickly generate pseudo VRML as I could create rows with my coordinates and then auto-fill the VRML code around them. To get the Z value I used extract values to points.
As a general note, VRML is VERY old and was superseded many years ago by X3D. I don't know if ArcScene ever did support X3D in the end (see this thread - where you will note that the OP there also suffered a similar crashing issue to yours). Even so, I would recommend that you export just the basic terrain to VRML, convert to X3D and go from there. X3D gives you much better animation, physics, editors and viewers etc
ArcScene is really nice for very quick demos but limited for anything beyond that. A much better alternative option is to use a 'proper' 3D modelling package like Blender. There are plugins that will take georeferenced rasters, DEMs and shapefiles. You can import all your layers into Blender, use the point-data shapefiles as particle instances to instantiate your point objects (or use a raster as a particle generator to control size and density of the instantiated objects). The great advantage of this approach is you have vastly more control over animate, lighting, fly-throughs, texturing etc.