~ruther/qmk_firmware

751316c34465ea77e066c3052729b207f3d62e0c — Zach White 5 years ago 5d3bf8a
[CLI] Add a subcommand for getting information about a keyboard (#8666)

You can now use `qmk info` to get information about keyboards and keymaps.

Co-authored-by: Erovia <Erovia@users.noreply.github.com>
M docs/cli_commands.md => docs/cli_commands.md +35 -2
@@ 6,6 6,8 @@

This command allows you to compile firmware from any directory. You can compile JSON exports from <https://config.qmk.fm>, compile keymaps in the repo, or compile the keyboard in the current working directory.

This command is directory aware. It will automatically fill in KEYBOARD and/or KEYMAP if you are in a keyboard or keymap directory.

**Usage for Configurator Exports**:

```


@@ 73,8 75,9 @@ $ qmk compile -kb dz60

## `qmk flash`

This command is similar to `qmk compile`, but can also target a bootloader. The bootloader is optional, and is set to `:flash` by default.
To specify a different bootloader, use `-bl <bootloader>`. Visit the [Flashing Firmware](flashing.md) guide for more details of the available bootloaders.
This command is similar to `qmk compile`, but can also target a bootloader. The bootloader is optional, and is set to `:flash` by default. To specify a different bootloader, use `-bl <bootloader>`. Visit the [Flashing Firmware](flashing.md) guide for more details of the available bootloaders.

This command is directory aware. It will automatically fill in KEYBOARD and/or KEYMAP if you are in a keyboard or keymap directory.

**Usage for Configurator Exports**:



@@ 128,6 131,32 @@ Check your environment and report problems only:

    qmk doctor -n

## `qmk info`

Displays information about keyboards and keymaps in QMK. You can use this to get information about a keyboard, show the layouts, display the underlying key matrix, or to pretty-print JSON keymaps.

**Usage**:

```
qmk info [-f FORMAT] [-m] [-l] [-km KEYMAP] [-kb KEYBOARD]
```

This command is directory aware. It will automatically fill in KEYBOARD and/or KEYMAP if you are in a keyboard or keymap directory.

**Examples**:

Show basic information for a keyboard:

    qmk info -kb planck/rev5

Show the matrix for a keyboard:

    qmk info -kb ergodox_ez -m

Show a JSON keymap for a keyboard:

    qmk info -kb clueboard/california -km default

## `qmk json2c`

Creates a keymap.c from a QMK Configurator export.


@@ 152,6 181,8 @@ qmk list-keyboards

This command lists all the keymaps for a specified keyboard (and revision).

This command is directory aware. It will automatically fill in KEYBOARD if you are in a keyboard directory.

**Usage**:

```


@@ 162,6 193,8 @@ qmk list-keymaps -kb planck/ez

This command creates a new keymap based on a keyboard's existing default keymap.

This command is directory aware. It will automatically fill in KEYBOARD and/or KEYMAP if you are in a keyboard or keymap directory.

**Usage**:

```

A lib/python/qmk/c_parse.py => lib/python/qmk/c_parse.py +161 -0
@@ 0,0 1,161 @@
"""Functions for working with config.h files.
"""
from pathlib import Path

from milc import cli

from qmk.comment_remover import comment_remover

default_key_entry = {'x': -1, 'y': 0, 'w': 1}


def c_source_files(dir_names):
    """Returns a list of all *.c, *.h, and *.cpp files for a given list of directories

    Args:

        dir_names
            List of directories relative to `qmk_firmware`.
    """
    files = []
    for dir in dir_names:
        files.extend(file for file in Path(dir).glob('**/*') if file.suffix in ['.c', '.h', '.cpp'])
    return files


def find_layouts(file):
    """Returns list of parsed LAYOUT preprocessor macros found in the supplied include file.
    """
    file = Path(file)
    aliases = {}  # Populated with all `#define`s that aren't functions
    parsed_layouts = {}

    # Search the file for LAYOUT macros and aliases
    file_contents = file.read_text()
    file_contents = comment_remover(file_contents)
    file_contents = file_contents.replace('\\\n', '')

    for line in file_contents.split('\n'):
        if line.startswith('#define') and '(' in line and 'LAYOUT' in line:
            # We've found a LAYOUT macro
            macro_name, layout, matrix = _parse_layout_macro(line.strip())

            # Reject bad macro names
            if macro_name.startswith('LAYOUT_kc') or not macro_name.startswith('LAYOUT'):
                continue

            # Parse the matrix data
            matrix_locations = _parse_matrix_locations(matrix, file, macro_name)

            # Parse the layout entries into a basic structure
            default_key_entry['x'] = -1  # Set to -1 so _default_key(key) will increment it to 0
            layout = layout.strip()
            parsed_layout = [_default_key(key) for key in layout.split(',')]

            for key in parsed_layout:
                key['matrix'] = matrix_locations.get(key['label'])

            parsed_layouts[macro_name] = {
                'key_count': len(parsed_layout),
                'layout': parsed_layout,
                'filename': str(file),
            }

        elif '#define' in line:
            # Attempt to extract a new layout alias
            try:
                _, pp_macro_name, pp_macro_text = line.strip().split(' ', 2)
                aliases[pp_macro_name] = pp_macro_text
            except ValueError:
                continue

    # Populate our aliases
    for alias, text in aliases.items():
        if text in parsed_layouts and 'KEYMAP' not in alias:
            parsed_layouts[alias] = parsed_layouts[text]

    return parsed_layouts


def parse_config_h_file(config_h_file, config_h=None):
    """Extract defines from a config.h file.
    """
    if not config_h:
        config_h = {}

    config_h_file = Path(config_h_file)

    if config_h_file.exists():
        config_h_text = config_h_file.read_text()
        config_h_text = config_h_text.replace('\\\n', '')

        for linenum, line in enumerate(config_h_text.split('\n')):
            line = line.strip()

            if '//' in line:
                line = line[:line.index('//')].strip()

            if not line:
                continue

            line = line.split()

            if line[0] == '#define':
                if len(line) == 1:
                    cli.log.error('%s: Incomplete #define! On or around line %s' % (config_h_file, linenum))
                elif len(line) == 2:
                    config_h[line[1]] = True
                else:
                    config_h[line[1]] = ' '.join(line[2:])

            elif line[0] == '#undef':
                if len(line) == 2:
                    if line[1] in config_h:
                        if config_h[line[1]] is True:
                            del config_h[line[1]]
                        else:
                            config_h[line[1]] = False
                else:
                    cli.log.error('%s: Incomplete #undef! On or around line %s' % (config_h_file, linenum))

    return config_h


def _default_key(label=None):
    """Increment x and return a copy of the default_key_entry.
    """
    default_key_entry['x'] += 1
    new_key = default_key_entry.copy()

    if label:
        new_key['label'] = label

    return new_key


def _parse_layout_macro(layout_macro):
    """Split the LAYOUT macro into its constituent parts
    """
    layout_macro = layout_macro.replace('\\', '').replace(' ', '').replace('\t', '').replace('#define', '')
    macro_name, layout = layout_macro.split('(', 1)
    layout, matrix = layout.split(')', 1)

    return macro_name, layout, matrix


def _parse_matrix_locations(matrix, file, macro_name):
    """Parse raw matrix data into a dictionary keyed by the LAYOUT identifier.
    """
    matrix_locations = {}

    for row_num, row in enumerate(matrix.split('},{')):
        if row.startswith('LAYOUT'):
            cli.log.error('%s: %s: Nested layout macro detected. Matrix data not available!', file, macro_name)
            break

        row = row.replace('{', '').replace('}', '')
        for col_num, identifier in enumerate(row.split(',')):
            if identifier != 'KC_NO':
                matrix_locations[identifier] = (row_num, col_num)

    return matrix_locations

M lib/python/qmk/cli/__init__.py => lib/python/qmk/cli/__init__.py +1 -0
@@ 13,6 13,7 @@ from . import docs
from . import doctor
from . import flash
from . import hello
from . import info
from . import json
from . import json2c
from . import list

M lib/python/qmk/cli/cformat.py => lib/python/qmk/cli/cformat.py +6 -4
@@ 4,7 4,9 @@ import subprocess
from shutil import which

from milc import cli
import qmk.path

from qmk.path import normpath
from qmk.c_parse import c_source_files


def cformat_run(files, all_files):


@@ 45,10 47,10 @@ def cformat(cli):
    ignores = ['tmk_core/protocol/usb_hid', 'quantum/template']
    # Find the list of files to format
    if cli.args.files:
        files.extend(qmk.path.normpath(file) for file in cli.args.files)
        files.extend(normpath(file) for file in cli.args.files)
    # If -a is specified
    elif cli.args.all_files:
        all_files = qmk.path.c_source_files(core_dirs)
        all_files = c_source_files(core_dirs)
        # The following statement checks each file to see if the file path is in the ignored directories.
        files.extend(file for file in all_files if not any(i in str(file) for i in ignores))
    # No files specified & no -a flag


@@ 56,7 58,7 @@ def cformat(cli):
        base_args = ['git', 'diff', '--name-only', cli.args.base_branch]
        out = subprocess.run(base_args + core_dirs, check=True, stdout=subprocess.PIPE)
        changed_files = filter(None, out.stdout.decode('UTF-8').split('\n'))
        filtered_files = [qmk.path.normpath(file) for file in changed_files if not any(i in file for i in ignores)]
        filtered_files = [normpath(file) for file in changed_files if not any(i in file for i in ignores)]
        files.extend(file for file in filtered_files if file.exists() and file.suffix in ['.c', '.h', '.cpp'])

    # Run clang-format on the files we've found

A lib/python/qmk/cli/info.py => lib/python/qmk/cli/info.py +141 -0
@@ 0,0 1,141 @@
"""Keyboard information script.

Compile an info.json for a particular keyboard and pretty-print it.
"""
import json

from milc import cli

from qmk.decorators import automagic_keyboard, automagic_keymap
from qmk.keyboard import render_layouts, render_layout
from qmk.keymap import locate_keymap
from qmk.info import info_json
from qmk.path import is_keyboard

ROW_LETTERS = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnop'
COL_LETTERS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijilmnopqrstuvwxyz'


def show_keymap(info_json, title_caps=True):
    """Render the keymap in ascii art.
    """
    keymap_path = locate_keymap(cli.config.info.keyboard, cli.config.info.keymap)

    if keymap_path and keymap_path.suffix == '.json':
        if title_caps:
            cli.echo('{fg_blue}Keymap "%s"{fg_reset}:', cli.config.info.keymap)
        else:
            cli.echo('{fg_blue}keymap_%s{fg_reset}:', cli.config.info.keymap)

        keymap_data = json.load(keymap_path.open())
        layout_name = keymap_data['layout']

        for layer_num, layer in enumerate(keymap_data['layers']):
            if title_caps:
                cli.echo('{fg_cyan}Layer %s{fg_reset}:', layer_num)
            else:
                cli.echo('{fg_cyan}layer_%s{fg_reset}:', layer_num)

            print(render_layout(info_json['layouts'][layout_name]['layout'], layer))


def show_layouts(kb_info_json, title_caps=True):
    """Render the layouts with info.json labels.
    """
    for layout_name, layout_art in render_layouts(kb_info_json).items():
        title = layout_name.title() if title_caps else layout_name
        cli.echo('{fg_cyan}%s{fg_reset}:', title)
        print(layout_art)  # Avoid passing dirty data to cli.echo()


def show_matrix(info_json, title_caps=True):
    """Render the layout with matrix labels in ascii art.
    """
    for layout_name, layout in info_json['layouts'].items():
        # Build our label list
        labels = []
        for key in layout['layout']:
            if key['matrix']:
                row = ROW_LETTERS[key['matrix'][0]]
                col = COL_LETTERS[key['matrix'][1]]

                labels.append(row + col)
            else:
                labels.append('')

        # Print the header
        if title_caps:
            cli.echo('{fg_blue}Matrix for "%s"{fg_reset}:', layout_name)
        else:
            cli.echo('{fg_blue}matrix_%s{fg_reset}:', layout_name)

        print(render_layout(info_json['layouts'][layout_name]['layout'], labels))


@cli.argument('-kb', '--keyboard', help='Keyboard to show info for.')
@cli.argument('-km', '--keymap', help='Show the layers for a JSON keymap too.')
@cli.argument('-l', '--layouts', action='store_true', help='Render the layouts.')
@cli.argument('-m', '--matrix', action='store_true', help='Render the layouts with matrix information.')
@cli.argument('-f', '--format', default='friendly', arg_only=True, help='Format to display the data in (friendly, text, json) (Default: friendly).')
@cli.subcommand('Keyboard information.')
@automagic_keyboard
@automagic_keymap
def info(cli):
    """Compile an info.json for a particular keyboard and pretty-print it.
    """
    # Determine our keyboard(s)
    if not is_keyboard(cli.config.info.keyboard):
        cli.log.error('Invalid keyboard: %s!', cli.config.info.keyboard)
        exit(1)

    # Build the info.json file
    kb_info_json = info_json(cli.config.info.keyboard)

    # Output in the requested format
    if cli.args.format == 'json':
        print(json.dumps(kb_info_json))
        exit()

    if cli.args.format == 'text':
        for key in sorted(kb_info_json):
            if key == 'layouts':
                cli.echo('{fg_blue}layouts{fg_reset}: %s', ', '.join(sorted(kb_info_json['layouts'].keys())))
            else:
                cli.echo('{fg_blue}%s{fg_reset}: %s', key, kb_info_json[key])

        if cli.config.info.layouts:
            show_layouts(kb_info_json, False)

        if cli.config.info.matrix:
            show_matrix(kb_info_json, False)

        if cli.config_source.info.keymap and cli.config_source.info.keymap != 'config_file':
            show_keymap(kb_info_json, False)

    elif cli.args.format == 'friendly':
        cli.echo('{fg_blue}Keyboard Name{fg_reset}: %s', kb_info_json.get('keyboard_name', 'Unknown'))
        cli.echo('{fg_blue}Manufacturer{fg_reset}: %s', kb_info_json.get('manufacturer', 'Unknown'))
        if 'url' in kb_info_json:
            cli.echo('{fg_blue}Website{fg_reset}: %s', kb_info_json['url'])
        if kb_info_json.get('maintainer') == 'qmk':
            cli.echo('{fg_blue}Maintainer{fg_reset}: QMK Community')
        else:
            cli.echo('{fg_blue}Maintainer{fg_reset}: %s', kb_info_json.get('maintainer', 'qmk'))
        cli.echo('{fg_blue}Keyboard Folder{fg_reset}: %s', kb_info_json.get('keyboard_folder', 'Unknown'))
        cli.echo('{fg_blue}Layouts{fg_reset}: %s', ', '.join(sorted(kb_info_json['layouts'].keys())))
        if 'width' in kb_info_json and 'height' in kb_info_json:
            cli.echo('{fg_blue}Size{fg_reset}: %s x %s' % (kb_info_json['width'], kb_info_json['height']))
        cli.echo('{fg_blue}Processor{fg_reset}: %s', kb_info_json.get('processor', 'Unknown'))
        cli.echo('{fg_blue}Bootloader{fg_reset}: %s', kb_info_json.get('bootloader', 'Unknown'))

        if cli.config.info.layouts:
            show_layouts(kb_info_json, True)

        if cli.config.info.matrix:
            show_matrix(kb_info_json, True)

        if cli.config_source.info.keymap and cli.config_source.info.keymap != 'config_file':
            show_keymap(kb_info_json, True)

    else:
        cli.log.error('Unknown format: %s', cli.args.format)

M lib/python/qmk/cli/list/keymaps.py => lib/python/qmk/cli/list/keymaps.py +7 -11
@@ 4,7 4,7 @@ from milc import cli

import qmk.keymap
from qmk.decorators import automagic_keyboard
from qmk.errors import NoSuchKeyboardError
from qmk.path import is_keyboard


@cli.argument("-kb", "--keyboard", help="Specify keyboard name. Example: 1upkeyboards/1up60hse")


@@ 13,13 13,9 @@ from qmk.errors import NoSuchKeyboardError
def list_keymaps(cli):
    """List the keymaps for a specific keyboard
    """
    try:
        for name in qmk.keymap.list_keymaps(cli.config.list_keymaps.keyboard):
            # We echo instead of cli.log.info to allow easier piping of this output
            cli.echo('%s', name)
    except NoSuchKeyboardError as e:
        cli.echo("{fg_red}%s: %s", cli.config.list_keymaps.keyboard, e.message)
    except (FileNotFoundError, PermissionError) as e:
        cli.echo("{fg_red}%s: %s", cli.config.list_keymaps.keyboard, e)
    except TypeError:
        cli.echo("{fg_red}Something went wrong. Did you specify a keyboard?")
    if not is_keyboard(cli.config.list_keymaps.keyboard):
        cli.log.error('Keyboard %s does not exist!', cli.config.list_keymaps.keyboard)
        exit(1)

    for name in qmk.keymap.list_keymaps(cli.config.list_keymaps.keyboard):
        print(name)

M lib/python/qmk/commands.py => lib/python/qmk/commands.py +1 -0
@@ 64,6 64,7 @@ def compile_configurator_json(user_keymap, bootloader=None):
def parse_configurator_json(configurator_file):
    """Open and parse a configurator json export
    """
    # FIXME(skullydazed/anyone): Add validation here
    user_keymap = json.load(configurator_file)

    return user_keymap

A lib/python/qmk/comment_remover.py => lib/python/qmk/comment_remover.py +20 -0
@@ 0,0 1,20 @@
"""Removes C/C++ style comments from text.

Gratefully adapted from https://stackoverflow.com/a/241506
"""
import re

comment_pattern = re.compile(r'//.*?$|/\*.*?\*/|\'(?:\\.|[^\\\'])*\'|"(?:\\.|[^\\"])*"', re.DOTALL | re.MULTILINE)


def _comment_stripper(match):
    """Removes C/C++ style comments from a regex match.
    """
    s = match.group(0)
    return ' ' if s.startswith('/') else s


def comment_remover(text):
    """Remove C/C++ style comments from text.
    """
    return re.sub(comment_pattern, _comment_stripper, text)

M lib/python/qmk/constants.py => lib/python/qmk/constants.py +6 -0
@@ 7,3 7,9 @@ QMK_FIRMWARE = Path.cwd()

# This is the number of directories under `qmk_firmware/keyboards` that will be traversed. This is currently a limitation of our make system.
MAX_KEYBOARD_SUBFOLDERS = 5

# Supported processor types
ARM_PROCESSORS = 'cortex-m0', 'cortex-m0plus', 'cortex-m3', 'cortex-m4', 'MKL26Z64', 'MK20DX128', 'MK20DX256', 'STM32F042', 'STM32F072', 'STM32F103', 'STM32F303'
AVR_PROCESSORS = 'at90usb1286', 'at90usb646', 'atmega16u2', 'atmega328p', 'atmega32a', 'atmega32u2', 'atmega32u4', None
ALL_PROCESSORS = ARM_PROCESSORS + AVR_PROCESSORS
VUSB_PROCESSORS = 'atmega328p', 'atmega32a'

M lib/python/qmk/decorators.py => lib/python/qmk/decorators.py +5 -4
@@ 5,7 5,8 @@ from pathlib import Path

from milc import cli

from qmk.path import is_keyboard, is_keymap_dir, under_qmk_firmware
from qmk.keymap import is_keymap_dir
from qmk.path import is_keyboard, under_qmk_firmware


def automagic_keyboard(func):


@@ 67,18 68,18 @@ def automagic_keymap(func):
                        while current_path.parent.name != 'keymaps':
                            current_path = current_path.parent
                        cli.config[cli._entrypoint.__name__]['keymap'] = current_path.name
                        cli.config_source[cli._entrypoint.__name__]['keyboard'] = 'keymap_directory'
                        cli.config_source[cli._entrypoint.__name__]['keymap'] = 'keymap_directory'

                # If we're in `qmk_firmware/layouts` guess the name from the community keymap they're in
                elif relative_cwd.parts[0] == 'layouts' and is_keymap_dir(relative_cwd):
                    cli.config[cli._entrypoint.__name__]['keymap'] = relative_cwd.name
                    cli.config_source[cli._entrypoint.__name__]['keyboard'] = 'layouts_directory'
                    cli.config_source[cli._entrypoint.__name__]['keymap'] = 'layouts_directory'

                # If we're in `qmk_firmware/users` guess the name from the userspace they're in
                elif relative_cwd.parts[0] == 'users':
                    # Guess the keymap name based on which userspace they're in
                    cli.config[cli._entrypoint.__name__]['keymap'] = relative_cwd.parts[1]
                    cli.config_source[cli._entrypoint.__name__]['keyboard'] = 'users_directory'
                    cli.config_source[cli._entrypoint.__name__]['keymap'] = 'users_directory'

        return func(*args, **kwargs)


A lib/python/qmk/info.py => lib/python/qmk/info.py +249 -0
@@ 0,0 1,249 @@
"""Functions that help us generate and use info.json files.
"""
import json
from glob import glob
from pathlib import Path

from milc import cli

from qmk.constants import ARM_PROCESSORS, AVR_PROCESSORS, VUSB_PROCESSORS
from qmk.c_parse import find_layouts
from qmk.keyboard import config_h, rules_mk
from qmk.math import compute


def info_json(keyboard):
    """Generate the info.json data for a specific keyboard.
    """
    info_data = {
        'keyboard_name': str(keyboard),
        'keyboard_folder': str(keyboard),
        'layouts': {},
        'maintainer': 'qmk',
    }

    for layout_name, layout_json in _find_all_layouts(keyboard).items():
        if not layout_name.startswith('LAYOUT_kc'):
            info_data['layouts'][layout_name] = layout_json

    info_data = merge_info_jsons(keyboard, info_data)
    info_data = _extract_config_h(info_data)
    info_data = _extract_rules_mk(info_data)

    return info_data


def _extract_config_h(info_data):
    """Pull some keyboard information from existing rules.mk files
    """
    config_c = config_h(info_data['keyboard_folder'])
    row_pins = config_c.get('MATRIX_ROW_PINS', '').replace('{', '').replace('}', '').strip()
    col_pins = config_c.get('MATRIX_COL_PINS', '').replace('{', '').replace('}', '').strip()
    direct_pins = config_c.get('DIRECT_PINS', '').replace(' ', '')[1:-1]

    info_data['diode_direction'] = config_c.get('DIODE_DIRECTION')
    info_data['matrix_size'] = {
        'rows': compute(config_c.get('MATRIX_ROWS', '0')),
        'cols': compute(config_c.get('MATRIX_COLS', '0')),
    }
    info_data['matrix_pins'] = {}

    if row_pins:
        info_data['matrix_pins']['rows'] = row_pins.split(',')
    if col_pins:
        info_data['matrix_pins']['cols'] = col_pins.split(',')

    if direct_pins:
        direct_pin_array = []
        for row in direct_pins.split('},{'):
            if row.startswith('{'):
                row = row[1:]
            if row.endswith('}'):
                row = row[:-1]

            direct_pin_array.append([])

            for pin in row.split(','):
                if pin == 'NO_PIN':
                    pin = None

                direct_pin_array[-1].append(pin)

        info_data['matrix_pins']['direct'] = direct_pin_array

    info_data['usb'] = {
        'vid': config_c.get('VENDOR_ID'),
        'pid': config_c.get('PRODUCT_ID'),
        'device_ver': config_c.get('DEVICE_VER'),
        'manufacturer': config_c.get('MANUFACTURER'),
        'product': config_c.get('PRODUCT'),
        'description': config_c.get('DESCRIPTION'),
    }

    return info_data


def _extract_rules_mk(info_data):
    """Pull some keyboard information from existing rules.mk files
    """
    rules = rules_mk(info_data['keyboard_folder'])
    mcu = rules.get('MCU')

    if mcu in ARM_PROCESSORS:
        arm_processor_rules(info_data, rules)
    elif mcu in AVR_PROCESSORS:
        avr_processor_rules(info_data, rules)
    else:
        cli.log.warning("%s: Unknown MCU: %s" % (info_data['keyboard_folder'], mcu))
        unknown_processor_rules(info_data, rules)

    return info_data


def _find_all_layouts(keyboard):
    """Looks for layout macros associated with this keyboard.
    """
    layouts = {}
    rules = rules_mk(keyboard)
    keyboard_path = Path(rules.get('DEFAULT_FOLDER', keyboard))

    # Pull in all layouts defined in the standard files
    current_path = Path('keyboards/')
    for directory in keyboard_path.parts:
        current_path = current_path / directory
        keyboard_h = '%s.h' % (directory,)
        keyboard_h_path = current_path / keyboard_h
        if keyboard_h_path.exists():
            layouts.update(find_layouts(keyboard_h_path))

    if not layouts:
        # If we didn't find any layouts above we widen our search. This is error
        # prone which is why we want to encourage people to follow the standard above.
        cli.log.warning('%s: Falling back to searching for KEYMAP/LAYOUT macros.' % (keyboard))
        for file in glob('keyboards/%s/*.h' % keyboard):
            if file.endswith('.h'):
                these_layouts = find_layouts(file)
                if these_layouts:
                    layouts.update(these_layouts)

    if 'LAYOUTS' in rules:
        # Match these up against the supplied layouts
        supported_layouts = rules['LAYOUTS'].strip().split()
        for layout_name in sorted(layouts):
            if not layout_name.startswith('LAYOUT_'):
                continue
            layout_name = layout_name[7:]
            if layout_name in supported_layouts:
                supported_layouts.remove(layout_name)

        if supported_layouts:
            cli.log.error('%s: Missing LAYOUT() macro for %s' % (keyboard, ', '.join(supported_layouts)))

    return layouts


def arm_processor_rules(info_data, rules):
    """Setup the default info for an ARM board.
    """
    info_data['processor_type'] = 'arm'
    info_data['bootloader'] = rules['BOOTLOADER'] if 'BOOTLOADER' in rules else 'unknown'
    info_data['processor'] = rules['MCU'] if 'MCU' in rules else 'unknown'
    info_data['protocol'] = 'ChibiOS'

    if info_data['bootloader'] == 'unknown':
        if 'STM32' in info_data['processor']:
            info_data['bootloader'] = 'stm32-dfu'
        elif info_data.get('manufacturer') == 'Input Club':
            info_data['bootloader'] = 'kiibohd-dfu'

    if 'STM32' in info_data['processor']:
        info_data['platform'] = 'STM32'
    elif 'MCU_SERIES' in rules:
        info_data['platform'] = rules['MCU_SERIES']
    elif 'ARM_ATSAM' in rules:
        info_data['platform'] = 'ARM_ATSAM'

    return info_data


def avr_processor_rules(info_data, rules):
    """Setup the default info for an AVR board.
    """
    info_data['processor_type'] = 'avr'
    info_data['bootloader'] = rules['BOOTLOADER'] if 'BOOTLOADER' in rules else 'atmel-dfu'
    info_data['platform'] = rules['ARCH'] if 'ARCH' in rules else 'unknown'
    info_data['processor'] = rules['MCU'] if 'MCU' in rules else 'unknown'
    info_data['protocol'] = 'V-USB' if rules.get('MCU') in VUSB_PROCESSORS else 'LUFA'

    # FIXME(fauxpark/anyone): Eventually we should detect the protocol by looking at PROTOCOL inherited from mcu_selection.mk:
    # info_data['protocol'] = 'V-USB' if rules.get('PROTOCOL') == 'VUSB' else 'LUFA'

    return info_data


def unknown_processor_rules(info_data, rules):
    """Setup the default keyboard info for unknown boards.
    """
    info_data['bootloader'] = 'unknown'
    info_data['platform'] = 'unknown'
    info_data['processor'] = 'unknown'
    info_data['processor_type'] = 'unknown'
    info_data['protocol'] = 'unknown'

    return info_data


def merge_info_jsons(keyboard, info_data):
    """Return a merged copy of all the info.json files for a keyboard.
    """
    for info_file in find_info_json(keyboard):
        # Load and validate the JSON data
        with info_file.open('r') as info_fd:
            new_info_data = json.load(info_fd)

        if not isinstance(new_info_data, dict):
            cli.log.error("Invalid file %s, root object should be a dictionary.", str(info_file))
            continue

        # Copy whitelisted keys into `info_data`
        for key in ('keyboard_name', 'manufacturer', 'identifier', 'url', 'maintainer', 'processor', 'bootloader', 'width', 'height'):
            if key in new_info_data:
                info_data[key] = new_info_data[key]

        # Merge the layouts in
        if 'layouts' in new_info_data:
            for layout_name, json_layout in new_info_data['layouts'].items():
                # Only pull in layouts we have a macro for
                if layout_name in info_data['layouts']:
                    if info_data['layouts'][layout_name]['key_count'] != len(json_layout['layout']):
                        cli.log.error('%s: %s: Number of elements in info.json does not match! info.json:%s != %s:%s', info_data['keyboard_folder'], layout_name, len(json_layout['layout']), layout_name, len(info_data['layouts'][layout_name]['layout']))
                    else:
                        for i, key in enumerate(info_data['layouts'][layout_name]['layout']):
                            key.update(json_layout['layout'][i])

    return info_data


def find_info_json(keyboard):
    """Finds all the info.json files associated with a keyboard.
    """
    # Find the most specific first
    base_path = Path('keyboards')
    keyboard_path = base_path / keyboard
    keyboard_parent = keyboard_path.parent
    info_jsons = [keyboard_path / 'info.json']

    # Add DEFAULT_FOLDER before parents, if present
    rules = rules_mk(keyboard)
    if 'DEFAULT_FOLDER' in rules:
        info_jsons.append(Path(rules['DEFAULT_FOLDER']) / 'info.json')

    # Add in parent folders for least specific
    for _ in range(5):
        info_jsons.append(keyboard_parent / 'info.json')
        if keyboard_parent.parent == base_path:
            break
        keyboard_parent = keyboard_parent.parent

    # Return a list of the info.json files that actually exist
    return [info_json for info_json in info_jsons if info_json.exists()]

A lib/python/qmk/keyboard.py => lib/python/qmk/keyboard.py +111 -0
@@ 0,0 1,111 @@
"""Functions that help us work with keyboards.
"""
from array import array
from math import ceil
from pathlib import Path

from qmk.c_parse import parse_config_h_file
from qmk.makefile import parse_rules_mk_file


def config_h(keyboard):
    """Parses all the config.h files for a keyboard.

    Args:
        keyboard: name of the keyboard

    Returns:
        a dictionary representing the content of the entire config.h tree for a keyboard
    """
    config = {}
    cur_dir = Path('keyboards')
    rules = rules_mk(keyboard)
    keyboard = Path(rules['DEFAULT_FOLDER'] if 'DEFAULT_FOLDER' in rules else keyboard)

    for dir in keyboard.parts:
        cur_dir = cur_dir / dir
        config = {**config, **parse_config_h_file(cur_dir / 'config.h')}

    return config


def rules_mk(keyboard):
    """Get a rules.mk for a keyboard

    Args:
        keyboard: name of the keyboard

    Returns:
        a dictionary representing the content of the entire rules.mk tree for a keyboard
    """
    keyboard = Path(keyboard)
    cur_dir = Path('keyboards')
    rules = parse_rules_mk_file(cur_dir / keyboard / 'rules.mk')

    if 'DEFAULT_FOLDER' in rules:
        keyboard = Path(rules['DEFAULT_FOLDER'])

    for i, dir in enumerate(keyboard.parts):
        cur_dir = cur_dir / dir
        rules = parse_rules_mk_file(cur_dir / 'rules.mk', rules)

    return rules


def render_layout(layout_data, key_labels=None):
    """Renders a single layout.
    """
    textpad = [array('u', ' ' * 200) for x in range(50)]

    for key in layout_data:
        x = ceil(key.get('x', 0) * 4)
        y = ceil(key.get('y', 0) * 3)
        w = ceil(key.get('w', 1) * 4)
        h = ceil(key.get('h', 1) * 3)

        if key_labels:
            label = key_labels.pop(0)
            if label.startswith('KC_'):
                label = label[3:]
        else:
            label = key.get('label', '')

        label_len = w - 2
        label_leftover = label_len - len(label)

        if len(label) > label_len:
            label = label[:label_len]

        label_blank = ' ' * label_len
        label_border = '─' * label_len
        label_middle = label + ' '*label_leftover  # noqa: yapf insists there be no whitespace around *

        top_line = array('u', '┌' + label_border + '┐')
        lab_line = array('u', '│' + label_middle + '│')
        mid_line = array('u', '│' + label_blank + '│')
        bot_line = array('u', '└' + label_border + "┘")

        textpad[y][x:x + w] = top_line
        textpad[y + 1][x:x + w] = lab_line
        for i in range(h - 3):
            textpad[y + i + 2][x:x + w] = mid_line
        textpad[y + h - 1][x:x + w] = bot_line

    lines = []
    for line in textpad:
        if line.tounicode().strip():
            lines.append(line.tounicode().rstrip())

    return '\n'.join(lines)


def render_layouts(info_json):
    """Renders all the layouts from an `info_json` structure.
    """
    layouts = {}

    for layout in info_json['layouts']:
        layout_data = info_json['layouts'][layout]['layout']
        layouts[layout] = render_layout(layout_data)

    return layouts

M lib/python/qmk/keymap.py => lib/python/qmk/keymap.py +61 -14
@@ 2,8 2,8 @@
"""
from pathlib import Path

import qmk.path
import qmk.makefile
from qmk.path import is_keyboard
from qmk.keyboard import rules_mk

# The `keymap.c` template to use when a keyboard doesn't have its own
DEFAULT_KEYMAP_C = """#include QMK_KEYBOARD_H


@@ 47,6 47,14 @@ def _strip_any(keycode):
    return keycode


def is_keymap_dir(keymap):
    """Return True if Path object `keymap` has a keymap file inside.
    """
    for file in ('keymap.c', 'keymap.json'):
        if (keymap / file).is_file():
            return True


def generate(keyboard, layout, layers):
    """Returns a keymap.c for the specified keyboard, layout, and layers.



@@ 95,7 103,7 @@ def write(keyboard, keymap, layout, layers):
            An array of arrays describing the keymap. Each item in the inner array should be a string that is a valid QMK keycode.
    """
    keymap_c = generate(keyboard, layout, layers)
    keymap_file = qmk.path.keymap(keyboard) / keymap / 'keymap.c'
    keymap_file = keymap(keyboard) / keymap / 'keymap.c'

    keymap_file.parent.mkdir(parents=True, exist_ok=True)
    keymap_file.write_text(keymap_c)


@@ 103,37 111,76 @@ def write(keyboard, keymap, layout, layers):
    return keymap_file


def list_keymaps(keyboard_name):
def locate_keymap(keyboard, keymap):
    """Returns the path to a keymap for a specific keyboard.
    """
    if not is_keyboard(keyboard):
        raise KeyError('Invalid keyboard: ' + repr(keyboard))

    # Check the keyboard folder first, last match wins
    checked_dirs = ''
    keymap_path = ''

    for dir in keyboard.split('/'):
        if checked_dirs:
            checked_dirs = '/'.join((checked_dirs, dir))
        else:
            checked_dirs = dir

        keymap_dir = Path('keyboards') / checked_dirs / 'keymaps'

        if (keymap_dir / keymap / 'keymap.c').exists():
            keymap_path = keymap_dir / keymap / 'keymap.c'
        if (keymap_dir / keymap / 'keymap.json').exists():
            keymap_path = keymap_dir / keymap / 'keymap.json'

    if keymap_path:
        return keymap_path

    # Check community layouts as a fallback
    rules = rules_mk(keyboard)

    if "LAYOUTS" in rules:
        for layout in rules["LAYOUTS"].split():
            community_layout = Path('layouts/community') / layout / keymap
            if community_layout.exists():
                if (community_layout / 'keymap.json').exists():
                    return community_layout / 'keymap.json'
                if (community_layout / 'keymap.c').exists():
                    return community_layout / 'keymap.c'


def list_keymaps(keyboard):
    """ List the available keymaps for a keyboard.

    Args:
        keyboard_name: the keyboards full name with vendor and revision if necessary, example: clueboard/66/rev3
        keyboard: the keyboards full name with vendor and revision if necessary, example: clueboard/66/rev3

    Returns:
        a set with the names of the available keymaps
    """
    # parse all the rules.mk files for the keyboard
    rules_mk = qmk.makefile.get_rules_mk(keyboard_name)
    rules = rules_mk(keyboard)
    names = set()

    if rules_mk:
    if rules:
        # qmk_firmware/keyboards
        keyboards_dir = Path.cwd() / "keyboards"
        keyboards_dir = Path('keyboards')
        # path to the keyboard's directory
        kb_path = keyboards_dir / keyboard_name
        kb_path = keyboards_dir / keyboard
        # walk up the directory tree until keyboards_dir
        # and collect all directories' name with keymap.c file in it
        while kb_path != keyboards_dir:
            keymaps_dir = kb_path / "keymaps"
            if keymaps_dir.exists():
                names = names.union([keymap for keymap in keymaps_dir.iterdir() if (keymaps_dir / keymap / "keymap.c").is_file()])
                names = names.union([keymap.name for keymap in keymaps_dir.iterdir() if is_keymap_dir(keymap)])
            kb_path = kb_path.parent

        # if community layouts are supported, get them
        if "LAYOUTS" in rules_mk:
            for layout in rules_mk["LAYOUTS"].split():
                cl_path = Path.cwd() / "layouts" / "community" / layout
        if "LAYOUTS" in rules:
            for layout in rules["LAYOUTS"].split():
                cl_path = Path('layouts/community') / layout
                if cl_path.exists():
                    names = names.union([keymap for keymap in cl_path.iterdir() if (cl_path / keymap / "keymap.c").is_file()])
                    names = names.union([keymap.name for keymap in cl_path.iterdir() if is_keymap_dir(keymap)])

    return sorted(names)

M lib/python/qmk/makefile.py => lib/python/qmk/makefile.py +0 -32
@@ 2,8 2,6 @@
"""
from pathlib import Path

from qmk.errors import NoSuchKeyboardError


def parse_rules_mk_file(file, rules_mk=None):
    """Turn a rules.mk file into a dictionary.


@@ 51,33 49,3 @@ def parse_rules_mk_file(file, rules_mk=None):
                    rules_mk[key.strip()] = value.strip()

    return rules_mk


def get_rules_mk(keyboard):
    """ Get a rules.mk for a keyboard

    Args:
        keyboard: name of the keyboard

    Raises:
        NoSuchKeyboardError: when the keyboard does not exists

    Returns:
        a dictionary with the content of the rules.mk file
    """
    # Start with qmk_firmware/keyboards
    kb_path = Path.cwd() / "keyboards"
    # walk down the directory tree
    # and collect all rules.mk files
    kb_dir = kb_path / keyboard
    if kb_dir.exists():
        rules_mk = dict()
        for directory in Path(keyboard).parts:
            kb_path = kb_path / directory
            rules_mk_path = kb_path / "rules.mk"
            if rules_mk_path.exists():
                rules_mk = parse_rules_mk_file(rules_mk_path, rules_mk)
    else:
        raise NoSuchKeyboardError("The requested keyboard and/or revision does not exist.")

    return rules_mk

A lib/python/qmk/math.py => lib/python/qmk/math.py +33 -0
@@ 0,0 1,33 @@
"""Parse arbitrary math equations in a safe way.

Gratefully copied from https://stackoverflow.com/a/9558001
"""
import ast
import operator as op

# supported operators
operators = {ast.Add: op.add, ast.Sub: op.sub, ast.Mult: op.mul, ast.Div: op.truediv, ast.Pow: op.pow, ast.BitXor: op.xor, ast.USub: op.neg}


def compute(expr):
    """Parse a mathematical expression and return the answer.

    >>> compute('2^6')
    4
    >>> compute('2**6')
    64
    >>> compute('1 + 2*3**(4^5) / (6 + -7)')
    -5.0
    """
    return _eval(ast.parse(expr, mode='eval').body)


def _eval(node):
    if isinstance(node, ast.Num):  # <number>
        return node.n
    elif isinstance(node, ast.BinOp):  # <left> <operator> <right>
        return operators[type(node.op)](_eval(node.left), _eval(node.right))
    elif isinstance(node, ast.UnaryOp):  # <operator> <operand> e.g., -1
        return operators[type(node.op)](_eval(node.operand))
    else:
        raise TypeError(node)

M lib/python/qmk/path.py => lib/python/qmk/path.py +1 -25
@@ 4,20 4,10 @@ import logging
import os
from pathlib import Path

from qmk.constants import QMK_FIRMWARE, MAX_KEYBOARD_SUBFOLDERS
from qmk.constants import MAX_KEYBOARD_SUBFOLDERS, QMK_FIRMWARE
from qmk.errors import NoSuchKeyboardError


def is_keymap_dir(keymap_path):
    """Returns True if `keymap_path` is a valid keymap directory.
    """
    keymap_path = Path(keymap_path)
    keymap_c = keymap_path / 'keymap.c'
    keymap_json = keymap_path / 'keymap.json'

    return any((keymap_c.exists(), keymap_json.exists()))


def is_keyboard(keyboard_name):
    """Returns True if `keyboard_name` is a keyboard we can compile.
    """


@@ 68,17 58,3 @@ def normpath(path):
        return path

    return Path(os.environ['ORIG_CWD']) / path


def c_source_files(dir_names):
    """Returns a list of all *.c, *.h, and *.cpp files for a given list of directories

    Args:

        dir_names
            List of directories, relative pathing starts at qmk's cwd
    """
    files = []
    for dir in dir_names:
        files.extend(file for file in Path(dir).glob('**/*') if file.suffix in ['.c', '.h', '.cpp'])
    return files

M lib/python/qmk/tests/test_cli_commands.py => lib/python/qmk/tests/test_cli_commands.py +83 -21
@@ 4,89 4,151 @@ from qmk.commands import run

def check_subcommand(command, *args):
    cmd = ['bin/qmk', command] + list(args)
    return run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)
    result = run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
    return result


def check_returncode(result, expected=0):
    """Print stdout if `result.returncode` does not match `expected`.
    """
    if result.returncode != expected:
        print('`%s` stdout:' % ' '.join(result.args))
        print(result.stdout)
        print('returncode:', result.returncode)
    assert result.returncode == expected


def test_cformat():
    result = check_subcommand('cformat', 'quantum/matrix.c')
    assert result.returncode == 0
    check_returncode(result)


def test_compile():
    assert check_subcommand('compile', '-kb', 'handwired/onekey/pytest', '-km', 'default').returncode == 0
    result = check_subcommand('compile', '-kb', 'handwired/onekey/pytest', '-km', 'default', '-n')
    check_returncode(result)


def test_flash():
    assert check_subcommand('flash', '-b').returncode == 1
    assert check_subcommand('flash').returncode == 1
    result = check_subcommand('flash', '-kb', 'handwired/onekey/pytest', '-km', 'default', '-n')
    check_returncode(result)


def test_flash_bootloaders():
    result = check_subcommand('flash', '-b')
    check_returncode(result, 1)


def test_config():
    result = check_subcommand('config')
    assert result.returncode == 0
    check_returncode(result)
    assert 'general.color' in result.stdout


def test_kle2json():
    assert check_subcommand('kle2json', 'kle.txt', '-f').returncode == 0
    result = check_subcommand('kle2json', 'kle.txt', '-f')
    check_returncode(result)


def test_doctor():
    result = check_subcommand('doctor', '-n')
    assert result.returncode == 0
    assert 'QMK Doctor is checking your environment.' in result.stderr
    assert 'QMK is ready to go' in result.stderr
    check_returncode(result)
    assert 'QMK Doctor is checking your environment.' in result.stdout
    assert 'QMK is ready to go' in result.stdout


def test_hello():
    result = check_subcommand('hello')
    assert result.returncode == 0
    assert 'Hello,' in result.stderr
    check_returncode(result)
    assert 'Hello,' in result.stdout


def test_pyformat():
    result = check_subcommand('pyformat')
    assert result.returncode == 0
    assert 'Successfully formatted the python code' in result.stderr
    check_returncode(result)
    assert 'Successfully formatted the python code' in result.stdout


def test_list_keyboards():
    result = check_subcommand('list-keyboards')
    check_returncode(result)
    # check to see if a known keyboard is returned
    # this will fail if handwired/onekey/pytest is removed
    assert 'handwired/onekey/pytest' in result.stdout


def test_list_keymaps():
    result = check_subcommand('list-keymaps', '-kb', 'handwired/onekey/pytest')
    assert result.returncode == 0
    check_returncode(result, 0)
    assert 'default' and 'test' in result.stdout


def test_list_keymaps_long():
    result = check_subcommand('list-keymaps', '--keyboard', 'handwired/onekey/pytest')
    assert result.returncode == 0
    check_returncode(result, 0)
    assert 'default' and 'test' in result.stdout


def test_list_keymaps_kb_only():
    result = check_subcommand('list-keymaps', '-kb', 'niu_mini')
    assert result.returncode == 0
    check_returncode(result, 0)
    assert 'default' and 'via' in result.stdout


def test_list_keymaps_vendor_kb():
    result = check_subcommand('list-keymaps', '-kb', 'ai03/lunar')
    assert result.returncode == 0
    check_returncode(result, 0)
    assert 'default' and 'via' in result.stdout


def test_list_keymaps_vendor_kb_rev():
    result = check_subcommand('list-keymaps', '-kb', 'kbdfans/kbd67/mkiirgb/v2')
    assert result.returncode == 0
    check_returncode(result, 0)
    assert 'default' and 'via' in result.stdout


def test_list_keymaps_no_keyboard_found():
    result = check_subcommand('list-keymaps', '-kb', 'asdfghjkl')
    assert result.returncode == 0
    check_returncode(result, 1)
    assert 'does not exist' in result.stdout


def test_json2c():
    result = check_subcommand('json2c', 'keyboards/handwired/onekey/keymaps/default_json/keymap.json')
    assert result.returncode == 0
    check_returncode(result, 0)
    assert result.stdout == '#include QMK_KEYBOARD_H\nconst uint16_t PROGMEM keymaps[][MATRIX_ROWS][MATRIX_COLS] = {\t[0] = LAYOUT(KC_A)};\n\n'


def test_info():
    result = check_subcommand('info', '-kb', 'handwired/onekey/pytest')
    check_returncode(result)
    assert 'Keyboard Name: handwired/onekey/pytest' in result.stdout
    assert 'Processor: STM32F303' in result.stdout
    assert 'Layout:' not in result.stdout
    assert 'k0' not in result.stdout


def test_info_keyboard_render():
    result = check_subcommand('info', '-kb', 'handwired/onekey/pytest', '-l')
    check_returncode(result)
    assert 'Keyboard Name: handwired/onekey/pytest' in result.stdout
    assert 'Processor: STM32F303' in result.stdout
    assert 'Layout:' in result.stdout
    assert 'k0' in result.stdout


def test_info_keymap_render():
    result = check_subcommand('info', '-kb', 'handwired/onekey/pytest', '-km', 'default_json')
    check_returncode(result)
    assert 'Keyboard Name: handwired/onekey/pytest' in result.stdout
    assert 'Processor: STM32F303' in result.stdout
    assert '│A │' in result.stdout


def test_info_matrix_render():
    result = check_subcommand('info', '-kb', 'handwired/onekey/pytest', '-m')
    check_returncode(result)
    assert 'Keyboard Name: handwired/onekey/pytest' in result.stdout
    assert 'Processor: STM32F303' in result.stdout
    assert 'LAYOUT' in result.stdout
    assert '│0A│' in result.stdout
    assert 'Matrix for "LAYOUT"' in result.stdout