The fractional Laplacian (-Delta)a, a is an element of(0,1), and its generalizations to variable-coefficient 2a-order pseudodifferential operators P, are studied in Lq-Sobolev spaces of Bessel-potential type Hqs. For a bounded open set omega subset of Rn, consider the homogeneous Dirichlet problem: Pu=f in omega, u=0 in Rn set minus omega. We find the regularity of solutions and determine the exact Dirichlet domain Da,s,q (the space of solutions u with f is an element of Hqs(omega over bar )) in cases where omega has limited smoothness C1+tau, for 2a<tau<infinity, 0 <= s<tau-2a. Earlier, the regularity and Dirichlet domains were determined for smooth omega by the second author, and the regularity was found in low-order Holder spaces for tau=1 by Ros-Oton and Serra. The Hqs-results obtained now when tau<infinity are new, even for (-Delta)a. In detail, the spaces Da,s,q are identified as a-transmission spaces Hqa(s+2a)(omega over bar ), exhibiting estimates in terms of dist(x, partial differential omega)a near the boundary.The result has required a new development of methods to handle nonsmooth coordinate changes for pseudodifferential operators, which have not been available before; this constitutes another main contribution of the paper.