As a multi-criteria decision-making (MCDM) problem, supplier selection plays a key role in achieving the objectives of a supply chain system. Multiple strategic, operational, quantitative, and qualitative criteria influence the supplier selection process. A wide spectrum of criteria have been introduced, classified, and used by researchers and practitioners to evaluate the suppliers' performance; however, measuring and employing all of these criteria is impractical in real-world scenarios due to the budget, time, and information limitations. In this study, a decision support system (DSS) is developed, which helps managers to select a set of most effective criteria for the supplier selection process. This DSS is a threefold integration of MCDM and simulation and optimization. In this framework, the MCDM module incorporates a combination of criteria to select the suppliers. Then, a simulation model is used to evaluate the performance of the supply chain system considering the selected suppliers. Based on the simulation results, a multi-objective metaheuristic algorithm is utilized to find the ideal combinations of the criteria to maximize the supply chain system performance.