There's info on it on suns Java Trail site here
Implementing Document Filter
Looks like you get the Document model associated with the JTextArea ( PlainDocument) and cast to AbastractDocument and create/set your own DocumentFilter for the JTextArea
The code is there in the DocumentFilter link on that page. You could use the class example provided and implement it something like this for a JTextArea
int maxChars=10;
JTextArea tArea=new JTextArea();
AbstractDocument pDoc=(AbstractDocument)tArea.getDocument();
pDoc.setDocumentFilter(new DocumentSizeFilter(maxChars));