You are right, "Mittag" means "midday" to begin with. But it is also used as a term for lunch. For instance you would say "Mittag essen" (to have lunch; "Mittagessen" written in one word is official for "lunch"), but this is from "zu Mittag essen".
If you take a look at Grimms Wörterbuch, you will find: "Mittag machen" (literally: "to make midday"; "to have lunch") "daher 'mittag' auch geradezu für das mittagessen selbst" ("Hence 'mittag' also for lunch itself"). Same in a modern dictionary.
By the way, in the south of Germany and in Switzerland "Mittag" stretches well into the afternoon.