make your LOD proxy, basically the silhouette of the building, and bake the high def model you describe to a single sheet, preferably a tiny portion of a large sheet shared for many LODs.
It requires a new material set and a new model to make LODs. If you were going to just use the same model, it would defeat the purpose as mipmaps are already in the main asset. The LOD needs optimized assets for the tiny area of space on screen it will occupy.
Here is an example of a good LOD image. It is from Mafia2, though I don't know who made it or who to credit:
It's all just baked down from the high to a new low proxy.