Basically...it was the Don River.
The original Town of York (laid out in 1793) was located east of Jarvis St. and south of Queen St., in what is known as "Old Town" today. However the westward bias of Toronto growth was soon underway. Frederick Armstrong, in his book A City in the Making (1988, p. 17), notes that by the time the City of Toronto was incorporated in 1834, this area had "long ceased to be the centre of the city." As Armstrong explains:
"The westward spread of York, which had left the original town so undeveloped, had come about because of several factors. Most obvious were the unhealthy swampy areas on the town's eastern margins around the outlets of the Don River, which was already known as Ashbridge's Bay. But there were also positive reasons for the spread of the town to the west. Old Fort York (then usually called the Garrison), situated on the western Military Reserve which extended west from Peter Street, exerted a pull which became stronger when the provincial legislative buildings were moved to the west end. Yonge Street, opened by Simcoe's Queen's Rangers in 1796, provided another reason for the shift westward. It became the main route north to Toronto's rich agricultural heartland, stretching up to Lake Simcoe and potentially beyond to Penetanguishene on Georgian Bay. Yet another factor in the town's westward expansion was the pleasant lakeshore running from the mouth of the Don River towards the Garrison. Paralleling this strand, Front Street, then edging the habour, rapidly became the preferred residential location of many of the most prominent citizens. Finally, as road conditions began to improve in the years immediately before incorporation, Queen (Lot) Street, which was then the road leading to Dundas Street and the west, began to exert a north-westerly pull away from the original town plot. Consequently, by 1834 the growth of the city had resulted in a fairly solidly built up area stretching along the east-west axis of King Street, from Simcoe's town to York Street. West of York Street, a series of semi-built up areas continued on to Peter Street and the Garrison."
The west end - running out to High Park - was already well established by 1914, with the exception of the far northwest fringe (i.e. the Junction) which continued to grow in the interwar years. In contrast, the east end was much less developed and grew mostly in the interwar period. The opening of the Prince Edward Viaduct (aka the Bloor Viaduct) in 1918 spurred development east of the Don.
The difference is evident in looking at photos from the City Archives.
Here is Bloor and Delaware in 1917 - this is 3.5 km or 2 miles from Yonge St. - already well established, has a streetcar line etc.:
Meanwhile, at Danforth and Pape in 1913 - the same distance east from Yonge St. - the roads aren't paved yet and it's barely developed:
This streetcar map from 1912 shows that the streetcar and road network is much less developed east of the Don:
Here are some population figures for areas annexed by the city in the late 1800s and early 1900s. Note that the city stretched out to High Park by the 1880s, but the Beaches area was not annexed until 1908 and 1909. Here's an annexation map from Wikipedia:
As census tract populations aren't available until 1941, I had to draw from a combination from wards and 1921 federal riding populations (the 1921 Census broke the ridings into wards and included population figures for 1901 and 1911) and they do not match up exactly with these boundaries. For instance, what I call "Riverdale" is located south of the Danforth as far as Pape (although the Riverdale annexation stretched to Greenwood), "East Toronto" is everything south of the Danforth and east of Pape, and Parkdale and Brockton uses Dovercourt rather than Dufferin and the CPR tracks rather than Bloor as boundaries. I also included the West Toronto Junction, which became Ward 7 when annexed by the city in 1909. In addition, to these figures from the census, I tallied up the 1941 populations for these areas using census tracts. So here are populations are for 1901, 1911, 1921 and 1941.
Riverdale 11,179; 24,387; 33,747; 44,297
East Toronto 4,633; 28,151; 66,204; 85,835
Parkdale and Brockton 27,549; 67,954; 91,186; 94,591
The Junction 6,091; 18,860; 34,938; 44,908
Parkdale and Brockton then saw its great growth spurt between 1901 and 1911 and had flatlined by 1921, while East Toronto added the most to its population between 1911 and 1921, and continued to grow into the 1920s.
West Toronto Junction, the city's northwest fringe, grew later than the area directly to the west of the core, and like the area east of the Don also added the most population between 1911 and 1921 and grew significantly up until 1941. But it too developed a bit earlier.
* With special thanks to Tiger Master on the urbantoronto.ca discussion board, who made me aware of the archive pictures and the 1912 streetcar map.