If I was to develop a mapping application to be a standalone windows app, is there an open source GIS engine that is somewhat comparable to ArcGIS Engine?
There are many options available for standalone windows app development. However the usability really depends on your programming platform and functionality requirements.
GeoTools : GeoTools is an open source Java library that provides tools for geospatial data.
SharpMap : SharpMap is an easy-to-use mapping library for use in web and desktop applications. It provides access to many types of GIS data, enables spatial querying of that data, and renders beautiful maps. The engine is written in C# and based on the .Net 4.0 framework. SharpMap is released under GNU Lesser General Public License.
DotSpatial : DotSpatial is a geographic information system library written for .NET 4. It allows developers to incorporate spatial data, analysis and mapping functionality into their applications or to contribute GIS extensions to the community.
You can check this link for a better description.