Abstract: The even-weight modular forms of level N can be arranged into the common irreducible representations of the inhomogeneous finite modular group Gamma(N) and the homogeneous finite modular group Gamma(N)' which is the double covering of Gamma(N) , and the odd-weight modular forms of level N transform in the new representations of Gamma(N)'. We find that the above structure of modular forms can naturally generate texture zeros of the fermion mass matrices if we properly assign the representations and weights of the matter fields under the modular group. We perform a comprehensive analysis for the Gamma(3)' congruent to T' modular symmetry. The three generations of left-handed quarks are assumed to transform as a doublet and a singlet of T', and we find six possible texture-zero structures of the quark mass matrix up to row and column permutations. We present five benchmark quark models which can produce very good fits to the experimental data. These quark models are further extended to include the lepton sector, and the resulting models can give a unified description of both quark and lepton masses and flavor mixing simultaneously, although they contain a smaller number of free parameters than the observables.