Hematopoiesis is often pictured as a hierarchy of branching decisions, giving rise to all mature blood cell types from stepwise differentiation of a single cell, the hematopoietic stem cell. Various aspects of this process have been modeled using various experimental and theoretical techniques on different scales. Here we integrate the more common population-based approach with a single-cell resolved molecular differentiation model to study the possibility of inferring mechanistic knowledge of the differentiation process. We focus on a sub-module of hematopoiesis: differentiation of granulocyte-monocyte progenitors GMPs) to granulocytes or monocytes. Within a branching process model, we infer the differentiation probability of GMPs from the experimentally quantified heterogeneity of colony assays under permissive conditions where both granulocytes and monocytes can emerge. We compare the predictions with the differentiation probability in genealogies determined from single-cell time-lapse microscopy. In contrast to the branching process model, we found that the differentiation probability as determined by differentiation marker onset increases with the generation of the cell within the genealogy. To study this feature from a molecular perspective, we established a stochastic toggle switch model, in which the intrinsic lineage decision is executed using two antagonistic transcription factors. We identified parameter regimes that allow for both time-dependent and time-independent differentiation probabilities. Finally, we infer parameters for which the model matches experimentally observed differentiation probabilities via approximate Bayesian computing. These parameters suggest different timescales in the dynamics of granulocyte and monocyte differentiation. Thus we provide a multi-scale picture of cell differentiation in murine GMPs, and illustrate the need for single-cell time-resolved observations of cellular decisions.