The perception of faces involves a large network of cortical areas of the human brain. While several studies tested this network recently, its relationship to the lateral occipital (LO) cortex known to be involved in visual object perception remains largely unknown. We used functional magnetic resonance imaging and dynamic causal modeling (DCM) to test the effective connectivity among the major areas of the face-processing core network and LO. Specifically, we tested how LO is connected to the fusiform face area (FFA) and occipital face area (OFA) and which area provides the major face/object input to the network. We found that LO is connected via significant bidirectional connections to both OFA and FFA, suggesting the existence of a triangular network. In addition, our results also suggest that face- and object-related stimulus inputs are not entirely segregated at these lower level stages of face-processing and enter the network via the LO. These results support the role of LO in face perception, at least at the level of face/non-face stimulus discrimination.