How can I switch to layout view in ArcScene and export a map in jpg format?
There is no Layout View in ArcScene.
You can export your scene to JPG by using
Export Scene >
2D... and selecting
JPEG in the Save As Type drop-down of the Export Map dialog.