The worldwide prevalence of diabetes has spurred numerous studies on the development of new antidiabetic medicines. As a result, dipeptidyl peptidase IV (DPP4) has been recognized as a validated target. In our efforts to discover new DPP4 inhibitors, we analyzed the complexed structures of DPP4 available in Protein Data Bank and designed a series of triazole compounds. After enzyme activity assays and crystallographic verification of the binding interaction patterns, we found that the triazole compounds can inhibit DPP4 with micromolar IC50 values. Liver microsome stability and cytochrome P450 metabolic tests were performed on this series, revealing undesirable pharmacokinetic profiles for the triazole compounds. To overcome this liability, we substituted the triazole ring with an amide or urea group to produce a new series of DPP4 inhibitors. Based on its enzyme activity, metabolic stability, and selectivity over DPP8 and DPP9, we selected compound 21 r for further study of its in vivo effects in mice using an oral glucose tolerance test (OGTT). The results show that 21 r has efficacy similar to that of sitagliptin at a dose of 3 mg kg−1. The crystal structure of 21 r bound to DPP4 also reveals that the trifluoromethyl group is directed toward a subpocket different from the subsite bound by sitagliptin, providing clues for the design of new DPP4 inhibitors.