[SOLVED] Using libmenu and ncurses, listing stops if char ’ is encountered

The problem is that I'm populating a menu array with file names. If any of the filenames contains a " ’ ", the menu list selection will only show up to (but not including) the file that has the " ’ " in the filename.

The menu gets populated by items that were sorted and put into a binary search tree.

My program is working as intended when there are no filenames that contain the " ’ ". Using gdb, When I look at the function at the time when items are being copied to the menu array that post_menu() uses, everything looks fine. But when post_menu() and refresh() are executed, the menu doesn't show the full list.

For example, if there is a file called custer’s last stand, at the menu selection window, I would see

... carp copying culvert

and the listing would be ended. I can still scroll up and back down, and do everything as normal. I can quit the program and get a "0" for a return code.

The ’ character appears to be 146 on the extended ASCII chart. I can reproduce the problem every time.

I normally build the program with ncurses 5.9, but I tried with the latest, 6.1, and the result was the same.

When it's not using ncurses, my program operates on filenames with the ’ character fine, as far as I can tell.

lines 350-376 of restore_rmw.c

    ITEM **my_items;
    MENU *my_menu;
    /* Initialize items */
    my_items = (ITEM **) calloc (n_choices + 1, sizeof (ITEM *));
    populate_menu (root, my_items, true);

    my_items[n_choices] = (ITEM *)NULL;

    my_menu = new_menu ((ITEM **) my_items);
    menu_opts_off (my_menu, O_ONEVALUE);

    mvprintw (LINES - 7, 0, "== %s ==", waste_curr->files);
    mvprintw (LINES - 6, 0,
          ngettext ("== contains %d file ==", "== contains %d files ==", n_choices), n_choices);

    /* TRANSLATORS: I believe the words between the < and > can be translated
     */
    mvprintw (LINES - 4, 0, _("<CURSOR-RIGHT / CURSOR-LEFT> - switch waste folders"));
    mvprintw (LINES - 3, 0, _("\
<SPACE> - select or unselect an item. / <ENTER> - restore selected items"));

    /* TRANSLATORS: the 'q' can not be translated. rmw can only accept a 'q'
     * for input in this case.
    */
    mvprintw (LINES - 2, 0, _("'q' - quit"));
    post_menu (my_menu);
    refresh ();

Lines 81-98 of bstc.

void
populate_menu (node * node, ITEM ** my_items, bool level_one)
{
  static int i;
  if (level_one)
    i = 0;

  if (node == NULL)
    return;

  /* first recur on left child */
  populate_menu (node->left, my_items, false);

  my_items[i] = new_item (node->data, node->desc);
  i++;

  /* now recur on right child */
  populate_menu (node->right, my_items, false);
}

If anyone is interested in trying to reproduce this problem, just use the git code from the rmw repo (it only take a few seconds to build).

I got an email from Norm saying that the fix for this is to link against ncursesw and menuw (to enable "wide-character" support.

Did that, and it works on Debian 9 now, but the OpenBSD and OSX build is now broken. I should be able to fix that tomorrow.

146=0x92 is a non-printable control-character in ISO-8859-1/Unicode, it's called PU2

On the other hand, in windows-1252 it is indeed a printable character:

Cp1252. Unicode Description
0x92	0x2019	#RIGHT SINGLE QUOTATION MARK

Iconv might help but I suggest simply deleting every files that have non-ASCII characters in their names.

Originally Posted by NevemTeve Iconv might help but I suggest simply deleting every files that have non-ASCII characters in their names.

That would be impossible in this case. The program is a console "recycle bin". I found the bug completely by chance because I had a single file present in my trashcan that had this silly char.

Or convert any non 0x21-0x5b,0x5d-0x7e character to \xXX sequence.

That could get pretty tricky. What's displayed in the list are the names of the files in the "Waste" folder; from there, selecting one or multiple files will restore them. If a filename gets changed, then it can't get restored.

The only way I can think of off the top of my head to do that would be keep track of the file that has it's name changed, the character that it was previously. And if it's selected for restoration, change it back.

I remodelled by autoconf file and the build is working on OpenBSD and Mac OSX again. Though I'm not too sure if wide-char support is available on osx yet. On the Travis code integration logs, it's showing that ncursesw and menuw can't be found; and it's linking against the "non-w" libraries.

As for iconv, I've heard of it course but I'm not at all familiar with it. Are you suggesting that would be a better long-term solution than using ncurses with wide-character support enabled?

> If a filename gets changed, then it can't get restored.

Trust me: the change I've suggested _is_ reversible, eg: tűző -> t\xc5\xb1z\xc5\x91 -> tűző